11:54 p.m.
Hi,
I am getting the following error messages in Lustre Clients continuously,
I am unable to find any clu on this, Please advoice.
I am posting one client messages.
Mar 20 00:05:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 00:05:20 node22 kernel: LustreError: Skipped 373 previous similar
messages
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803650175/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395254164 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 374 previous
similar messages
Mar 20 00:15:24 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:15:24 node22 kernel: LustreError: Skipped 367 previous similar
messages
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069f61c00 x1462271803659320/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395254732 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 368 previous
similar messages
Mar 20 00:28:04 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:28:04 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db400 x1462271803668389/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395255492 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:38:34 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:38:34 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803676412/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395256121 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:48:37 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 00:48:37 node22 kernel: LustreError: Skipped 16 previous similar
messages
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060f94800 x1462271803685186/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395256724 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 16 previous
similar messages
Mar 20 00:58:45 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:58:45 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499c00 x1462271803693596/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395257371 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:09:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 01:09:09 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803701825/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395257993 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 01:19:39 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:19:39 node22 kernel: LustreError: Skipped 18 previous similar
messages
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060e57800 x1462271803711034/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395258587 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 18 previous
similar messages
Mar 20 01:29:40 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 01:29:40 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db000 x1462271803719047/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259225 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError:
3255:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b272400 x1462271803727400/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881062400800 x1462271803727404/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 82 previous
similar messages
Mar 20 01:50:10 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:50:10 node22 kernel: LustreError: Skipped 190 previous similar
messages
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803735098/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395260417 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 189 previous
similar messages
Mar 20 02:00:17 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:00:17 node22 kernel: LustreError: Skipped 199 previous similar
messages
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803743031/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395261061 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 198 previous
similar messages
Mar 20 02:10:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.44@o2ib. The ost_write operation failed with -5
Mar 20 02:10:20 node22 kernel: LustreError: Skipped 210 previous similar
messages
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b5c1800 x1462271803750283/t0(0)
o4->lustre-OST0005-osc-ffff880819776400@192.168.1.44@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395261665 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 213 previous
similar messages
Mar 20 02:20:23 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:20:23 node22 kernel: LustreError: Skipped 191 previous similar
messages
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803758875/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395262269 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 191 previous
similar messages
Mar 20 02:30:25 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:30:25 node22 kernel: LustreError: Skipped 149 previous similar
messages
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803766929/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395262833 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 148 previous
similar messages
Mar 20 02:40:26 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:40:26 node22 kernel: LustreError: Skipped 225 previous similar
messages
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb800 x1462271803774144/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395263434 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 225 previous
similar messages
[root@node22 ~]#
Thank you
Venkat Reddy
12:47 a.m.
Hi,
the first point to look at is: "ost_write operation failed with -5" -
these error codes are listed in the manual, section troubleshooting:
http://wiki.lustre.org/manual/LustreManual20_HTML/LustreTroubleshooting.html
-5 means "The operation failed with a read or write error."
are the OSTs all Ok? All OST's mounted and readable? check dmesg or
syslog on the OSSes for read-write errors on the devices, messages
saying "remounting readonly". If there are any troubles like that, check
the cause of the problem: Are the block devices Ok? Any issues with RAID
controllers or disks...
If this is all ok, sometimes simply rebooting the client can solve the
problem. However, this will throw away cached data in the client memory.
best regards,
Martin
On 03/25/2014 01:54 PM, Venkat Reddy wrote:
Hi,
I am getting the following error messages in Lustre Clients continuously,
I am unable to find any clu on this, Please advoice.
I am posting one client messages.
Mar 20 00:05:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 00:05:20 node22 kernel: LustreError: Skipped 373 previous similar
messages
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803650175/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395254164 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 374 previous
similar messages
Mar 20 00:15:24 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:15:24 node22 kernel: LustreError: Skipped 367 previous similar
messages
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069f61c00 x1462271803659320/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395254732 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 368 previous
similar messages
Mar 20 00:28:04 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:28:04 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db400 x1462271803668389/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395255492 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:38:34 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:38:34 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803676412/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395256121 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:48:37 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 00:48:37 node22 kernel: LustreError: Skipped 16 previous similar
messages
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060f94800 x1462271803685186/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395256724 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 16 previous
similar messages
Mar 20 00:58:45 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:58:45 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499c00 x1462271803693596/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395257371 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:09:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 01:09:09 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803701825/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395257993 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 01:19:39 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:19:39 node22 kernel: LustreError: Skipped 18 previous similar
messages
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060e57800 x1462271803711034/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395258587 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 18 previous
similar messages
Mar 20 01:29:40 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 01:29:40 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db000 x1462271803719047/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259225 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError:
3255:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b272400 x1462271803727400/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881062400800 x1462271803727404/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 82 previous
similar messages
Mar 20 01:50:10 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:50:10 node22 kernel: LustreError: Skipped 190 previous similar
messages
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803735098/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395260417 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 189 previous
similar messages
Mar 20 02:00:17 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:00:17 node22 kernel: LustreError: Skipped 199 previous similar
messages
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803743031/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395261061 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 198 previous
similar messages
Mar 20 02:10:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.44@o2ib. The ost_write operation failed with -5
Mar 20 02:10:20 node22 kernel: LustreError: Skipped 210 previous similar
messages
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b5c1800 x1462271803750283/t0(0)
o4->lustre-OST0005-osc-ffff880819776400@192.168.1.44@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395261665 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 213 previous
similar messages
Mar 20 02:20:23 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:20:23 node22 kernel: LustreError: Skipped 191 previous similar
messages
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803758875/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395262269 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 191 previous
similar messages
Mar 20 02:30:25 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:30:25 node22 kernel: LustreError: Skipped 149 previous similar
messages
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803766929/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395262833 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 148 previous
similar messages
Mar 20 02:40:26 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:40:26 node22 kernel: LustreError: Skipped 225 previous similar
messages
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb800 x1462271803774144/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395263434 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 225 previous
similar messages
[root@node22 ~]#
Thank you
Venkat Reddy
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
5:04 a.m.
Hi Martin,
Thank you for the response,
All OSTs are OK, All OSTs are mounted and readable.
Last updated: Thu Mar 27 23:20:08 2014
Last change: Mon Jan 27 20:04:21 2014 via cibadmin on mds1
Stack: openais
Current DC: mds1 - partition with quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
4 Nodes configured, 4 expected votes
8 Resources configured.
============
Online: [ mds1 oss1 oss2 mds2 ]
mdt (ocf::heartbeat:Filesystem): Started mds1
ost0 (ocf::heartbeat:Filesystem): Started oss1
ost1 (ocf::heartbeat:Filesystem): Started oss1
ost2 (ocf::heartbeat:Filesystem): Started mds2
ost3 (ocf::heartbeat:Filesystem): Started mds2
ost4 (ocf::heartbeat:Filesystem): Started oss2
ost5 (ocf::heartbeat:Filesystem): Started oss2
ost6 (ocf::heartbeat:Filesystem): Started mds1
-----------------------------------------------------------------------------------
Storage is OK
CLI> show raid-groups
RAID Group RAID Assigned Status
Total Free
No. Name Level CM
Capacity(MB) Capacity(MB)
0 RG1 RAID5 CM#0 Available
7552512 0
1 RG2 RAID5 CM#1 Available
7552512 0
2 RG3 RAID5 CM#0 Available
7552512 0
3 RG4 RAID5 CM#1 Available
7552512 0
4 RG5 RAID5 CM#0 Available
7552512 0
5 RG6 RAID5 CM#1 Available
7552512 0
6 RG7 RAID5 CM#0 Available
7552512 0
7 RG8 RAID1+0 CM#1 Available
1678336 0
CLI>
CLI> show volumes
Volume Status Type Encryption
Expansion RG or TPP or FTRP Size(MB) Reserved Copy
No. Name
(Concatenation) No. Name Deletion Protection
1 LVOL1 Available Standard
OFF - 0 RG1 7552512 Disable
2 LVOL2 Available Standard
OFF - 1 RG2 7552512 Disable
3 LVOL3 Available Standard
OFF - 2 RG3 7552512 Disable
4 LVOL4 Available Standard
OFF - 3 RG4 7552512 Disable
5 LVOL5 Available Standard
OFF - 4 RG5 7552512 Disable
6 LVOL6 Available Standard
OFF - 5 RG6 7552512 Disable
7 LVOL7 Available Standard
OFF - 6 RG7 7552512 Disable
8 MVOL1 Available Standard
OFF - 7 RG8 1678336 Disable
CLI>
All the clients are getting this type of errors.
MDS
Mar 25 22:24:35 mds1 kernel: LustreError:
0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 101s: evicting client at 192.168.1.13@o2ib ns:
filter-lustre-OST0006_UUID lock: ffff8808681796c0/0x1f83925c1aa896e8 lrc:
3/0,0 mode: PW/PW res: 6908586/0 rrc: 9 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.13@o2ib remote: 0xf5d818ed856c63c0
expref: 165 pid: 2430 timeout 9228304163
Mar 25 22:26:16 mds1 kernel: LustreError:
0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 202s: evicting client at 192.168.1.21@o2ib ns:
filter-lustre-OST0006_UUID lock: ffff8807479feb80/0x1f83925c1aa896ef lrc:
3/0,0 mode: PW/PW res: 6908586/0 rrc: 8 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.21@o2ib remote: 0x7c2792f137aefb3e
expref: 154 pid: 2430 timeout 9228405000
Mar 25 22:26:16 mds1 kernel: LustreError:
3162:0:(client.c:1110:ptlrpc_import_delay_req()) @@@ IMP_CLOSED
req@ffff880c5a49e800 x1458394099405002/t0(0)
o105->lustre-OST0006@192.168.1.21@o2ib:15/16 lens 344/224 e 0 to 0 dl 0 ref
1 fl Rpc:N/0/ffffffff rc 0/-1
Mar 25 22:26:16 mds1 kernel: LustreError:
3162:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid
192.168.1.21@o2ib) returned 0 from completion AST ns:
filter-lustre-OST0006_UUID lock: ffff8804a6a3e100/0x1f83925c1aa89704 lrc:
4/0,0 mode: PW/PW res: 6908586/0 rrc: 8 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.21@o2ib remote: 0x7c2792f137aefb4c
expref: 99 pid: 14380 timeout 9228506003
Mar 25 22:26:16 mds1 kernel: LustreError:
3162:0:(client.c:1110:ptlrpc_import_delay_req()) @@@ IMP_CLOSED
req@ffff880c5a49e800 x1458394099405003/t0(0)
o105->lustre-OST0006@192.168.1.21@o2ib:15/16 lens 344/224 e 0 to 0 dl 0 ref
1 fl Rpc:N/0/ffffffff rc 0/-1
Mar 25 22:27:44 mds1 cib[1811]: info: cib_stats: Processed 1 operations
(0.00us average, 0% utilization) in the last 10min
Mar 25 22:27:57 mds1 kernel: LustreError:
0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 303s: evicting client at 192.168.1.12@o2ib ns:
filter-lustre-OST0006_UUID lock: ffff880868179ac0/0x1f83925c1aa89720 lrc:
3/0,0 mode: PW/PW res: 6908586/0 rrc: 4 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.12@o2ib remote: 0x8d21c3afd5cada74
expref: 85 pid: 14380 timeout 9228506004
Mar 25 22:27:57 mds1 kernel: LustreError:
3162:0:(client.c:1110:ptlrpc_import_delay_req()) @@@ IMP_CLOSED
req@ffff88030f177c00 x1458394099405364/t0(0)
o105->lustre-OST0006@192.168.1.12@o2ib:15/16 lens 344/224 e 0 to 0 dl 0 ref
1 fl Rpc:N/0/ffffffff rc 0/-1
Mar 25 22:27:57 mds1 kernel: LustreError:
3162:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid
192.168.1.12@o2ib) returned 0 from completion AST ns:
filter-lustre-OST0006_UUID lock: ffff88091a8ee9c0/0x1f83925c1aa89727 lrc:
4/0,0 mode: PW/PW res: 6908586/0 rrc: 5 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.12@o2ib remote: 0x8d21c3afd5cada7b
expref: 43 pid: 14380 timeout 9228607264
Mar 25 22:27:57 mds1 kernel: LustreError:
3162:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous
similar message
OSS
Mar 24 13:33:31 mds2 kernel: LustreError:
0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 101s: evicting client at 192.168.1.28@o2ib ns:
filter-lustre-OST0002_UUID lock: ffff8805d6ed0100/0x96fed93b234af472 lrc:
3/0,0 mode: PW/PW res: 6901880/0 rrc: 6 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.28@o2ib remote: 0x9f7ce9b31ba2bd6
expref: 31 pid: 32593 timeout 9110052417
Mar 24 13:33:31 mds2 kernel: LustreError:
3210:0:(client.c:1110:ptlrpc_import_delay_req()) @@@ IMP_CLOSED
req@ffff8807814dd800 x1458394128705870/t0(0)
o105->lustre-OST0002@192.168.1.28@o2ib:15/16 lens 344/224 e 0 to 0 dl 0 ref
1 fl Rpc:N/0/ffffffff rc 0/-1
Mar 24 13:33:31 mds2 kernel: LustreError:
3210:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid
192.168.1.28@o2ib) returned 0 from completion AST ns:
filter-lustre-OST0002_UUID lock: ffff88052ed28340/0x96fed93b234af479 lrc:
4/0,0 mode: PW/PW res: 6901880/0 rrc: 7 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.28@o2ib remote: 0x9f7ce9b31ba2bdd
expref: 16 pid: 32593 timeout 9110153000
Mar 24 13:35:12 mds2 kernel: LustreError:
9230:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 202s: evicting client at 192.168.1.29@o2ib ns:
filter-lustre-OST0002_UUID lock: ffff8805d6ed0700/0x96fed93b234af480 lrc:
3/0,0 mode: PW/PW res: 6901880/0 rrc: 4 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.29@o2ib remote: 0xa620576242647a3c
expref: 136 pid: 32593 timeout 9110153000
Mar 24 13:36:53 mds2 kernel: LustreError:
0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 303s: evicting client at 192.168.1.27@o2ib ns:
filter-lustre-OST0002_UUID lock: ffff880b5c7ab380/0x96fed93b234af48e lrc:
3/0,0 mode: PW/PW res: 6901880/0 rrc: 2 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.27@o2ib remote: 0x345c14aa1c07ae5d
expref: 37 pid: 32593 timeout 9110254597
Mar 26 15:54:11 mds2 kernel: LustreError:
9239:0:(filter_io.c:501:filter_preprw_read()) io error -5
Mar 26 15:54:11 mds2 kernel: LustreError:
9239:0:(filter_io.c:501:filter_preprw_read()) Skipped 7 previous similar
messages
Will there any benifit of rebooting entire storage and OSS, MDS.
Thanks & Regards,
Venkat
On Wed, Mar 26, 2014 at 7:17 PM, Martin Hecht <hecht(a)hlrs.de> wrote:
Hi,
the first point to look at is: "ost_write operation failed with -5" -
these error codes are listed in the manual, section troubleshooting:
http://wiki.lustre.org/manual/LustreManual20_HTML/LustreTroubleshooting.html
-5 means "The operation failed with a read or write error."
are the OSTs all Ok? All OST's mounted and readable? check dmesg or syslog
on the OSSes for read-write errors on the devices, messages saying
"remounting readonly". If there are any troubles like that, check the cause
of the problem: Are the block devices Ok? Any issues with RAID controllers
or disks...
If this is all ok, sometimes simply rebooting the client can solve the
problem. However, this will throw away cached data in the client memory.
best regards,
Martin
On 03/25/2014 01:54 PM, Venkat Reddy wrote:
Hi,
I am getting the following error messages in Lustre Clients continuously,
I am unable to find any clu on this, Please advoice.
I am posting one client messages.
Mar 20 00:05:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 00:05:20 node22 kernel: LustreError: Skipped 373 previous similar
messages
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803650175/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395254164 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 374 previous
similar messages
Mar 20 00:15:24 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:15:24 node22 kernel: LustreError: Skipped 367 previous similar
messages
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069f61c00 x1462271803659320/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395254732 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 368 previous
similar messages
Mar 20 00:28:04 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:28:04 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db400 x1462271803668389/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395255492 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:38:34 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:38:34 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803676412/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395256121 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:48:37 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 00:48:37 node22 kernel: LustreError: Skipped 16 previous similar
messages
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060f94800 x1462271803685186/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395256724 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 16 previous
similar messages
Mar 20 00:58:45 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 00:58:45 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499c00 x1462271803693596/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395257371 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:09:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 01:09:09 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803701825/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395257993 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 01:19:39 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:19:39 node22 kernel: LustreError: Skipped 18 previous similar
messages
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060e57800 x1462271803711034/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395258587 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 18 previous
similar messages
Mar 20 01:29:40 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with -5
Mar 20 01:29:40 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db000 x1462271803719047/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259225 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError:
3255:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b272400 x1462271803727400/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881062400800 x1462271803727404/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 82 previous
similar messages
Mar 20 01:50:10 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 01:50:10 node22 kernel: LustreError: Skipped 190 previous similar
messages
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803735098/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395260417 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 189 previous
similar messages
Mar 20 02:00:17 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:00:17 node22 kernel: LustreError: Skipped 199 previous similar
messages
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803743031/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395261061 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 198 previous
similar messages
Mar 20 02:10:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.44@o2ib. The ost_write operation failed with -5
Mar 20 02:10:20 node22 kernel: LustreError: Skipped 210 previous similar
messages
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b5c1800 x1462271803750283/t0(0)
o4->lustre-OST0005-osc-ffff880819776400@192.168.1.44@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395261665 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 213 previous
similar messages
Mar 20 02:20:23 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:20:23 node22 kernel: LustreError: Skipped 191 previous similar
messages
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803758875/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395262269 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 191 previous
similar messages
Mar 20 02:30:25 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:30:25 node22 kernel: LustreError: Skipped 149 previous similar
messages
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803766929/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395262833 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 148 previous
similar messages
Mar 20 02:40:26 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with -5
Mar 20 02:40:26 node22 kernel: LustreError: Skipped 225 previous similar
messages
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb800 x1462271803774144/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens 488/416
e 0 to 0 dl 1395263434 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 225 previous
similar messages
[root@node22 ~]#
Thank you
Venkat Reddy
_______________________________________________
HPDD-discuss mailing
listHPDD-discuss@lists.01.orghttps://lists.01.org/mailman/listinfo/hpdd-discuss
5:32 p.m.
On Thu, Mar 27, 2014 at 11:34:17PM +0530, Venkat Reddy wrote:
All the clients are getting this type of errors.
MDS
Mar 25 22:24:35 mds1 kernel: LustreError:
0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer
expired after 101s: evicting client at 192.168.1.13@o2ib ns:
filter-lustre-OST0006_UUID lock: ffff8808681796c0/0x1f83925c1aa896e8 lrc:
3/0,0 mode: PW/PW res: 6908586/0 rrc: 9 type: EXT [0->18446744073709551615]
(req 0->4095) flags: 0x20 nid: 192.168.1.13@o2ib remote: 0xf5d818ed856c63c0
expref: 165 pid: 2430 timeout 9228304163
is your infiniband fabric stable?
a single bad cable that is only partially working - often called "link
flapping" - can upset opensm and hence the whole fabric enough that you
can see that type of lustre message.
the easiest way to check might be to tail your opensm.log. you could
also run ibdiagnet et al. generally there should be no sm activity at
all unless you are rebooting a node. if you see up/down and sm
recalculating activity almost continuously then it's pretty easy to
work out which link port(s) are bad and then disable one end to
temporarily stabilise the fabric until the cable can be replaced.
a node that has crashed in a peculiar way that renders its ib card
unresponsive can also cause opensm to loop indefinitely and screw up
the fabric. find and power off the node.
a flakey ib switch would also do it, but that is rare.
cheers,
robin
12:05 p.m.
Can anybody guide, what causes the lustre errors ?
Jobs are conning out with no such file or directory ?
Is stripe count causes lustreerror ?
On Mar 25, 2014 6:24 PM, "Venkat Reddy" <reddy281(a)gmail.com> wrote:
Hi,
I am getting the following error messages in Lustre Clients continuously,
I am unable to find any clu on this, Please advoice.
I am posting one client messages.
Mar 20 00:05:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 00:05:20 node22 kernel: LustreError: Skipped 373 previous similar
messages
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803650175/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395254164 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:05:20 node22 kernel: LustreError:
3248:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 374 previous
similar messages
Mar 20 00:15:24 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with
-5
Mar 20 00:15:24 node22 kernel: LustreError: Skipped 367 previous similar
messages
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069f61c00 x1462271803659320/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395254732 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:15:24 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 368 previous
similar messages
Mar 20 00:28:04 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with
-5
Mar 20 00:28:04 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db400 x1462271803668389/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395255492 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:28:04 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:38:34 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with
-5
Mar 20 00:38:34 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803676412/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395256121 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:38:34 node22 kernel: LustreError:
3250:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 00:48:37 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 00:48:37 node22 kernel: LustreError: Skipped 16 previous similar
messages
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060f94800 x1462271803685186/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395256724 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:48:37 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 16 previous
similar messages
Mar 20 00:58:45 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with
-5
Mar 20 00:58:45 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499c00 x1462271803693596/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395257371 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 00:58:45 node22 kernel: LustreError:
3253:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:09:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with
-5
Mar 20 01:09:09 node22 kernel: LustreError: Skipped 13 previous similar
messages
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b499000 x1462271803701825/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395257993 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:09:09 node22 kernel: LustreError:
3259:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
similar messages
Mar 20 01:19:39 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 01:19:39 node22 kernel: LustreError: Skipped 18 previous similar
messages
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881060e57800 x1462271803711034/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395258587 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:19:39 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 18 previous
similar messages
Mar 20 01:29:40 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.45@o2ib. The ost_write operation failed with
-5
Mar 20 01:29:40 node22 kernel: LustreError: Skipped 21 previous similar
messages
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a0db000 x1462271803719047/t0(0)
o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395259225 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:29:40 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
similar messages
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
messages
Mar 20 01:40:09 node22 kernel: LustreError:
3255:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b272400 x1462271803727400/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881062400800 x1462271803727404/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:40:09 node22 kernel: LustreError:
3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 82 previous
similar messages
Mar 20 01:50:10 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 01:50:10 node22 kernel: LustreError: Skipped 190 previous similar
messages
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803735098/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395260417 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 01:50:10 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 189 previous
similar messages
Mar 20 02:00:17 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 02:00:17 node22 kernel: LustreError: Skipped 199 previous similar
messages
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb000 x1462271803743031/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395261061 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:00:17 node22 kernel: LustreError:
3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 198 previous
similar messages
Mar 20 02:10:20 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.44@o2ib. The ost_write operation failed with
-5
Mar 20 02:10:20 node22 kernel: LustreError: Skipped 210 previous similar
messages
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106b5c1800 x1462271803750283/t0(0)
o4->lustre-OST0005-osc-ffff880819776400@192.168.1.44@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395261665 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:10:20 node22 kernel: LustreError:
3249:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 213 previous
similar messages
Mar 20 02:20:23 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 02:20:23 node22 kernel: LustreError: Skipped 191 previous similar
messages
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803758875/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395262269 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:20:23 node22 kernel: LustreError:
3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 191 previous
similar messages
Mar 20 02:30:25 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 02:30:25 node22 kernel: LustreError: Skipped 149 previous similar
messages
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff88106a962400 x1462271803766929/t0(0)
o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395262833 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:30:25 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 148 previous
similar messages
Mar 20 02:40:26 node22 kernel: LustreError: 11-0: an error occurred while
communicating with 192.168.1.43@o2ib. The ost_write operation failed with
-5
Mar 20 02:40:26 node22 kernel: LustreError: Skipped 225 previous similar
messages
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
error -5 req@ffff881069aeb800 x1462271803774144/t0(0)
o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
488/416 e 0 to 0 dl 1395263434 ref 2 fl Interpret:R/0/0 rc -5/-5
Mar 20 02:40:26 node22 kernel: LustreError:
3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 225 previous
similar messages
[root@node22 ~]#
Thank you
Venkat Reddy
2:29 a.m.
Have you seen Robin Humble's reply to this list on March 28 about the IB
fabric?
Another thing causing such troubles might be another subnet manager
running somewhere,
maybe on a switch which was isolated for a while and then reconnected to
the fabric.
If that's the case you even can't ping all nodes on the IB level.
On 04/12/2014 04:05 AM, Venkat Reddy wrote:
Can anybody guide, what causes the lustre errors ?
Jobs are conning out with no such file or directory ?
Is stripe count causes lustreerror ?
On Mar 25, 2014 6:24 PM, "Venkat Reddy" <reddy281(a)gmail.com> wrote:
> Hi,
>
> I am getting the following error messages in Lustre Clients continuously,
> I am unable to find any clu on this, Please advoice.
>
> I am posting one client messages.
>
> Mar 20 00:05:20 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 00:05:20 node22 kernel: LustreError: Skipped 373 previous similar
> messages
> Mar 20 00:05:20 node22 kernel: LustreError:
> 3248:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881069aeb000 x1462271803650175/t0(0)
> o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395254164 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 00:05:20 node22 kernel: LustreError:
> 3248:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 374 previous
> similar messages
> Mar 20 00:15:24 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.45@o2ib. The ost_write operation failed with
> -5
> Mar 20 00:15:24 node22 kernel: LustreError: Skipped 367 previous similar
> messages
> Mar 20 00:15:24 node22 kernel: LustreError:
> 3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881069f61c00 x1462271803659320/t0(0)
> o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395254732 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 00:15:24 node22 kernel: LustreError:
> 3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 368 previous
> similar messages
> Mar 20 00:28:04 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.45@o2ib. The ost_write operation failed with
> -5
> Mar 20 00:28:04 node22 kernel: LustreError: Skipped 13 previous similar
> messages
> Mar 20 00:28:04 node22 kernel: LustreError:
> 3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106a0db400 x1462271803668389/t0(0)
> o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395255492 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 00:28:04 node22 kernel: LustreError:
> 3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
> similar messages
> Mar 20 00:38:34 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.45@o2ib. The ost_write operation failed with
> -5
> Mar 20 00:38:34 node22 kernel: LustreError: Skipped 13 previous similar
> messages
> Mar 20 00:38:34 node22 kernel: LustreError:
> 3250:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106b499000 x1462271803676412/t0(0)
> o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395256121 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 00:38:34 node22 kernel: LustreError:
> 3250:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
> similar messages
> Mar 20 00:48:37 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 00:48:37 node22 kernel: LustreError: Skipped 16 previous similar
> messages
> Mar 20 00:48:37 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881060f94800 x1462271803685186/t0(0)
> o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395256724 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 00:48:37 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 16 previous
> similar messages
> Mar 20 00:58:45 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.45@o2ib. The ost_write operation failed with
> -5
> Mar 20 00:58:45 node22 kernel: LustreError: Skipped 21 previous similar
> messages
> Mar 20 00:58:45 node22 kernel: LustreError:
> 3253:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106b499c00 x1462271803693596/t0(0)
> o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395257371 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 00:58:45 node22 kernel: LustreError:
> 3253:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
> similar messages
> Mar 20 01:09:09 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.45@o2ib. The ost_write operation failed with
> -5
> Mar 20 01:09:09 node22 kernel: LustreError: Skipped 13 previous similar
> messages
> Mar 20 01:09:09 node22 kernel: LustreError:
> 3259:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106b499000 x1462271803701825/t0(0)
> o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395257993 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 01:09:09 node22 kernel: LustreError:
> 3259:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 13 previous
> similar messages
> Mar 20 01:19:39 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 01:19:39 node22 kernel: LustreError: Skipped 18 previous similar
> messages
> Mar 20 01:19:39 node22 kernel: LustreError:
> 3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881060e57800 x1462271803711034/t0(0)
> o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395258587 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 01:19:39 node22 kernel: LustreError:
> 3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 18 previous
> similar messages
> Mar 20 01:29:40 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.45@o2ib. The ost_write operation failed with
> -5
> Mar 20 01:29:40 node22 kernel: LustreError: Skipped 21 previous similar
> messages
> Mar 20 01:29:40 node22 kernel: LustreError:
> 3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106a0db000 x1462271803719047/t0(0)
> o4->lustre-OST0006-osc-ffff880819776400@192.168.1.45@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395259225 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 01:29:40 node22 kernel: LustreError:
> 3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 21 previous
> similar messages
> Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 01:40:09 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
> messages
> Mar 20 01:40:09 node22 kernel: LustreError: Skipped 83 previous similar
> messages
> Mar 20 01:40:09 node22 kernel: LustreError:
> 3255:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106b272400 x1462271803727400/t0(0)
> o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 01:40:09 node22 kernel: LustreError:
> 3256:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881062400800 x1462271803727404/t0(0)
> o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395259855 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 01:40:09 node22 kernel: LustreError:
> 3256:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 82 previous
> similar messages
> Mar 20 01:50:10 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 01:50:10 node22 kernel: LustreError: Skipped 190 previous similar
> messages
> Mar 20 01:50:10 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881069aeb000 x1462271803735098/t0(0)
> o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395260417 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 01:50:10 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 189 previous
> similar messages
> Mar 20 02:00:17 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 02:00:17 node22 kernel: LustreError: Skipped 199 previous similar
> messages
> Mar 20 02:00:17 node22 kernel: LustreError:
> 3260:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881069aeb000 x1462271803743031/t0(0)
> o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395261061 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 02:00:17 node22 kernel: LustreError:
> 3260:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 198 previous
> similar messages
> Mar 20 02:10:20 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.44@o2ib. The ost_write operation failed with
> -5
> Mar 20 02:10:20 node22 kernel: LustreError: Skipped 210 previous similar
> messages
> Mar 20 02:10:20 node22 kernel: LustreError:
> 3249:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106b5c1800 x1462271803750283/t0(0)
> o4->lustre-OST0005-osc-ffff880819776400@192.168.1.44@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395261665 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 02:10:20 node22 kernel: LustreError:
> 3249:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 213 previous
> similar messages
> Mar 20 02:20:23 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 02:20:23 node22 kernel: LustreError: Skipped 191 previous similar
> messages
> Mar 20 02:20:23 node22 kernel: LustreError:
> 3251:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106a962400 x1462271803758875/t0(0)
> o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395262269 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 02:20:23 node22 kernel: LustreError:
> 3251:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 191 previous
> similar messages
> Mar 20 02:30:25 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 02:30:25 node22 kernel: LustreError: Skipped 149 previous similar
> messages
> Mar 20 02:30:25 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff88106a962400 x1462271803766929/t0(0)
> o4->lustre-OST0000-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395262833 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 02:30:25 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 148 previous
> similar messages
> Mar 20 02:40:26 node22 kernel: LustreError: 11-0: an error occurred while
> communicating with 192.168.1.43@o2ib. The ost_write operation failed with
> -5
> Mar 20 02:40:26 node22 kernel: LustreError: Skipped 225 previous similar
> messages
> Mar 20 02:40:26 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable
> error -5 req@ffff881069aeb800 x1462271803774144/t0(0)
> o4->lustre-OST0001-osc-ffff880819776400@192.168.1.43@o2ib:6/4 lens
> 488/416 e 0 to 0 dl 1395263434 ref 2 fl Interpret:R/0/0 rc -5/-5
> Mar 20 02:40:26 node22 kernel: LustreError:
> 3254:0:(osc_request.c:1689:osc_brw_redo_request()) Skipped 225 previous
> similar messages
> [root@node22 ~]#
>
> Thank you
>
> Venkat Reddy
>
>
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
3070
days inactive
3099
days old
5 comments
3 participants
participants (3)
-
Martin Hecht -
Robin Humble -
Venkat Reddy