6:06 a.m.
Dear All,
I am seeing a ton of these recently and based on reading this post
http://lists.lustre.org/pipermail/lustre-discuss/2011-February/015065.html
It seems it has to do with mmap IO. Although I do not know if any of our user/application
is performing mmap IO, my guess is mostly like they are.
Current version of our Lustre on MDS & OSS is: 1.8.5 and on Clients is 1.8.7;
Does these error indicate any kind of problem that needs to be addressed?
Please advise.
ON CLINET I SEE THE FOLLOWING,
CORRESPONDING SERVER MESSAGES AT THE END
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 9 previous similar
messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 8 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067dd00800 x1439211562817922/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376338917 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 8 previous similar
messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067fdf6400 x1439211562999631/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339732 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 4 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067eb13c00 x1439211562999918/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339747 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 4 previous similar
messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067dd00800 x1439211563037211/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339925 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 1 previous similar
message
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 8 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067fdf6000 x1439211563038445/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376339970 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 8 previous similar
messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 1 previous similar message
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067eb13400 x1439211563041360/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376340060 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 8438:0:(rw.c:198:ll_file_punch()) obd_truncate fails (-30) ino 29050451
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 10 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067fdf6400 x1439211691656856/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476163 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 9 previous similar
messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 5 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067fdf6c00 x1439211691662173/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476184 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 5 previous similar
messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 6 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067eb13800 x1439211691697906/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476223 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 5 previous similar
messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 15 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067eb13400 x1439211691832310/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476292 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 14 previous
similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 30 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067fdf6400 x1439211691998133/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476438 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 27 previous
similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 54 previous similar messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067eb13c00 x1439211692361454/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376476739 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 49 previous
similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 1 previous similar
message
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 2 previous similar
messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 4 previous similar
messages
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) @@@ redo for recoverable
error req@ffff81067fdf6c00 x1439211693129332/t0
o4->smuhpc-OST001a_UUID@10.1.1.51@tcp:6/4 lens 448/608 e 0 to 1 dl 1376477339 ref 2 fl
Interpret:R/0/0 rc -30/-30
LustreError: 5290:0:(osc_request.c:1629:osc_brw_redo_request()) Skipped 100 previous
similar messages
LustreError: 11-0: an error occurred while communicating with 10.1.1.51@tcp. The ost_write
operation failed with -30
LustreError: Skipped 111 previous similar messages
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) too many resent retries,
returning error
LustreError: 5290:0:(osc_request.c:1625:osc_brw_redo_request()) Skipped 8 previous similar
messages
SERVER MESSAGES:
LustreError: 10258:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8105f9e02000 x1442836047652704/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376496995 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10258:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar
messages
LustreError: 14875:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 14875:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 48 previous
similar messages
LustreError: 14875:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 14875:0:(filter.c:427:filter_client_add()) Skipped 43 previous similar
messages
LustreError: 14875:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19177: rc -30
LustreError: 14875:0:(filter.c:451:filter_client_add()) Skipped 43 previous similar
messages
LustreError: 14875:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8101551bb800 x1442836047654898/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376497598 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14875:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 43 previous similar
messages
LustreError: 10290:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10290:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous
similar messages
LustreError: 10290:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10290:0:(filter.c:427:filter_client_add()) Skipped 26 previous similar
messages
LustreError: 10290:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19204: rc -30
LustreError: 10290:0:(filter.c:451:filter_client_add()) Skipped 26 previous similar
messages
LustreError: 10290:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81058f257000 x1442836047657674/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376498207 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10290:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 26 previous similar
messages
LustreError: 14915:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 14915:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 48 previous
similar messages
LustreError: 14915:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 14915:0:(filter.c:427:filter_client_add()) Skipped 41 previous similar
messages
LustreError: 14915:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19246: rc -30
LustreError: 14915:0:(filter.c:451:filter_client_add()) Skipped 41 previous similar
messages
LustreError: 14915:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8101f2b6e000 x1443293932604564/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376498812 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14915:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 41 previous similar
messages
LustreError: 10123:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10123:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous
similar messages
LustreError: 10123:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10123:0:(filter.c:427:filter_client_add()) Skipped 25 previous similar
messages
LustreError: 10123:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19272: rc -30
LustreError: 10123:0:(filter.c:451:filter_client_add()) Skipped 25 previous similar
messages
LustreError: 10123:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8104154f5c00 x1443293932605656/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376499437 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10123:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 25 previous similar
messages
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 46 previous
similar messages
LustreError: 10144:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10144:0:(filter.c:427:filter_client_add()) Skipped 40 previous similar
messages
LustreError: 10144:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19313: rc -30
LustreError: 10144:0:(filter.c:451:filter_client_add()) Skipped 40 previous similar
messages
LustreError: 10144:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81032ae86c00 x1443293932607309/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376500052 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10144:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 40 previous similar
messages
LustreError: 14900:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 14900:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 37 previous
similar messages
LustreError: 14900:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 14900:0:(filter.c:427:filter_client_add()) Skipped 32 previous similar
messages
LustreError: 14900:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19346: rc -30
LustreError: 14900:0:(filter.c:451:filter_client_add()) Skipped 32 previous similar
messages
LustreError: 14900:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81027b87bc00 x1443293932619476/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376500668 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14900:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32 previous similar
messages
LustreError: 10180:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10180:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 53 previous
similar messages
LustreError: 10180:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10180:0:(filter.c:427:filter_client_add()) Skipped 46 previous similar
messages
LustreError: 10180:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19393: rc -30
LustreError: 10180:0:(filter.c:451:filter_client_add()) Skipped 46 previous similar
messages
LustreError: 10180:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81061f6db800 x1442836047673405/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376501270 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10180:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 46 previous similar
messages
LustreError: 10076:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10076:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 32 previous
similar messages
LustreError: 10076:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10076:0:(filter.c:427:filter_client_add()) Skipped 27 previous similar
messages
LustreError: 10076:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19421: rc -30
LustreError: 10076:0:(filter.c:451:filter_client_add()) Skipped 27 previous similar
messages
LustreError: 10076:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81010e007400 x1442836047675443/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376501870 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10076:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 27 previous similar
messages
LustreError: 10367:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10367:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 55 previous
similar messages
LustreError: 10367:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10367:0:(filter.c:427:filter_client_add()) Skipped 43 previous similar
messages
LustreError: 10367:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19465: rc -30
LustreError: 10367:0:(filter.c:451:filter_client_add()) Skipped 43 previous similar
messages
LustreError: 10367:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81009ff42800 x1442836047677670/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376502470 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10367:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 43 previous similar
messages
LustreError: 10084:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10084:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 36 previous
similar messages
LustreError: 10084:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10084:0:(filter.c:427:filter_client_add()) Skipped 28 previous similar
messages
LustreError: 10084:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19494: rc -30
LustreError: 10084:0:(filter.c:451:filter_client_add()) Skipped 28 previous similar
messages
LustreError: 10084:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff810170e92850 x1443293932753685/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376503079 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10084:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar
messages
LustreError: 10397:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10397:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous
similar messages
LustreError: 10397:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10397:0:(filter.c:427:filter_client_add()) Skipped 39 previous similar
messages
LustreError: 10397:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19534: rc -30
LustreError: 10397:0:(filter.c:451:filter_client_add()) Skipped 39 previous similar
messages
LustreError: 10397:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81029cedfc00 x1443293932886002/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376503687 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10397:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 39 previous similar
messages
LustreError: 14958:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 14958:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 39 previous
similar messages
LustreError: 14958:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 14958:0:(filter.c:427:filter_client_add()) Skipped 31 previous similar
messages
LustreError: 14958:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19566: rc -30
LustreError: 14958:0:(filter.c:451:filter_client_add()) Skipped 31 previous similar
messages
LustreError: 14958:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff810611441000 x1443293933016614/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376504294 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 14958:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31 previous similar
messages
LustreError: 10475:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10475:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 37 previous
similar messages
LustreError: 10475:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10475:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10475:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19602: rc -30
LustreError: 10475:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10475:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff810031d5dc00 x1442836047685269/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376504895 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10475:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar
messages
LustreError: 10347:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10347:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 41 previous
similar messages
LustreError: 10347:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10347:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10347:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19638: rc -30
LustreError: 10347:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10347:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff810574a2b400 x1442836047687442/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376505520 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10347:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar
messages
LustreError: 10165:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10165:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous
similar messages
LustreError: 10165:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10165:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10165:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19674: rc -30
LustreError: 10165:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10165:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8104380fd000 x1443293933392255/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376506144 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10165:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar
messages
LustreError: 10350:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10350:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 50 previous
similar messages
LustreError: 10350:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10350:0:(filter.c:427:filter_client_add()) Skipped 38 previous similar
messages
LustreError: 10350:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19713: rc -30
LustreError: 10350:0:(filter.c:451:filter_client_add()) Skipped 38 previous similar
messages
LustreError: 10350:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8102593f6c00 x1442836047692767/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376506752 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10350:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 38 previous similar
messages
LustreError: 10370:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10370:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 46 previous
similar messages
LustreError: 10370:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10370:0:(filter.c:427:filter_client_add()) Skipped 36 previous similar
messages
LustreError: 10370:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19750: rc -30
LustreError: 10370:0:(filter.c:451:filter_client_add()) Skipped 36 previous similar
messages
LustreError: 10370:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff810527481400 x1442836047694671/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376507370 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10370:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 36 previous similar
messages
LustreError: 10330:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10330:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 42 previous
similar messages
LustreError: 10330:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10330:0:(filter.c:427:filter_client_add()) Skipped 37 previous similar
messages
LustreError: 10330:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19788: rc -30
LustreError: 10330:0:(filter.c:451:filter_client_add()) Skipped 37 previous similar
messages
LustreError: 10330:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff81063a334c00 x1443293933800716/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376507987 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10330:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar
messages
LustreError: 10153:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10153:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous
similar messages
LustreError: 10153:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10153:0:(filter.c:427:filter_client_add()) Skipped 37 previous similar
messages
LustreError: 10153:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19826: rc -30
LustreError: 10153:0:(filter.c:451:filter_client_add()) Skipped 37 previous similar
messages
LustreError: 10153:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8102c7ebf800 x1442836047701333/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376508595 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10153:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 37 previous similar
messages
LustreError: 10256:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10256:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 35 previous
similar messages
LustreError: 10256:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10256:0:(filter.c:427:filter_client_add()) Skipped 31 previous similar
messages
LustreError: 10256:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19858: rc -30
LustreError: 10256:0:(filter.c:451:filter_client_add()) Skipped 31 previous similar
messages
LustreError: 10256:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8101f41db800 x1442836047703557/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376509195 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10256:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 31 previous similar
messages
LustreError: 10170:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) error starting handle
for op 8 (71 credits): rc -30
LustreError: 10170:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start()) Skipped 45 previous
similar messages
LustreError: 10170:0:(filter.c:427:filter_client_add()) unable to start transaction: rc
-30
LustreError: 10170:0:(filter.c:427:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10170:0:(filter.c:451:filter_client_add()) error writing last_rcvd client idx
19894: rc -30
LustreError: 10170:0:(filter.c:451:filter_client_add()) Skipped 35 previous similar
messages
LustreError: 10170:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-30)
req@ffff8101afc16c00 x1442836047705962/t0 o8-><?>@<?>:0/0 lens 368/264 e 0
to 0 dl 1376509795 ref 1 fl Interpret:/0/0 rc -30/0
LustreError: 10170:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 35 previous similar
messages
Thank you,
Amit H. Kumar
4:01 p.m.
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start())
error starting handle for op 8 (71 credits): rc -30
./asm-generic/errno-base.h:#define EROFS 30
mmap IO isn't your issue: It seems that (at least) one OST on your OSS
got automatically remounted read-only due to a filesystem error.
Older log entries on the OST should tell you what went wrong (grep remounting
/var/log/messages*).
The only way to recover from this is by:
- umounting the affected ost(s)
- run a full fsck on the volume(s) (using the latest e2fsprogs fsck)
- mount the volume again IF you are sure that your storage array didn't go bananas
and will not cause any new corruption (btw: are you running a 'snowbird-lustre'
with linux sw-raid6?)
Oh: and tell your users that they should check the return value of write() calls ;-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
11:58 p.m.
Hi Adrian,
Thank you for your quick response. NO I am not running snow-bird lustre with sw-raid6;
All of our arrays configured with hardware raid6.
I found the disk array in problem. Although I see when it remounted file system as
read-only, looking into the logs before remount does not reveal anything about what went
wrong. I see common processing errors but not more that I can understand to be critical
enough to result in read only mounts.
Remount happened a week back, attached is the copy of log file. Can you please look into
and see if you notice anything unusual.
***Another quick question.
My assumption is I can un-mount the OST in problem while the others are in production
and run the fsck on the ost volumes.
Any specific options I need to pass when I run fsck?
Version of e2fsprogs we have is: e2fsprogs-1.41.10.sun2-0redhat
Anything else do I need to watch out, before I bring this OST back online?
Please advise.
Amit
-----Original Message-----
From: Adrian Ulrich [mailto:adrian@blinkenlights.ch]
Sent: Thursday, August 15, 2013 1:02 AM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start())
error starting handle for op 8 (71 credits): rc -30
./asm-generic/errno-base.h:#define EROFS 30
mmap IO isn't your issue: It seems that (at least) one OST on your OSS got
automatically remounted read-only due to a filesystem error.
Older log entries on the OST should tell you what went wrong (grep remounting
/var/log/messages*).
The only way to recover from this is by:
- umounting the affected ost(s)
- run a full fsck on the volume(s) (using the latest e2fsprogs fsck)
- mount the volume again IF you are sure that your storage array didn't go bananas
and will not cause any new corruption (btw: are you running a 'snowbird-lustre'
with linux sw-raid6?)
Oh: and tell your users that they should check the return value of write() calls ;-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
12:31 a.m.
Forgot the attachment: here is the messages file: look for line number 3442
http://tinyurl.com/kagz38t
Thank you,
Amit
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Kumar, Amit
Sent: Thursday, August 15, 2013 8:59 AM
To: Adrian Ulrich
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Hi Adrian,
Thank you for your quick response. NO I am not running snow-bird lustre with sw-raid6;
All of our arrays configured with hardware raid6.
I found the disk array in problem. Although I see when it remounted file system as
read-only, looking into the logs before remount does not reveal anything about what went
wrong. I see common processing errors but not more that I can understand to be critical
enough to result in read only mounts.
Remount happened a week back, attached is the copy of log file. Can you please look into
and see if you notice anything unusual.
***Another quick question.
My assumption is I can un-mount the OST in problem while the others are in production
and run the fsck on the ost volumes.
Any specific options I need to pass when I run fsck?
Version of e2fsprogs we have is: e2fsprogs-1.41.10.sun2-0redhat
Anything else do I need to watch out, before I bring this OST back online?
Please advise.
Amit
-----Original Message-----
From: Adrian Ulrich [mailto:adrian@blinkenlights.ch]
Sent: Thursday, August 15, 2013 1:02 AM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start())
error starting handle for op 8 (71 credits): rc -30
./asm-generic/errno-base.h:#define EROFS 30
mmap IO isn't your issue: It seems that (at least) one OST on your OSS got
automatically remounted read-only due to a filesystem error.
Older log entries on the OST should tell you what went wrong (grep remounting
/var/log/messages*).
The only way to recover from this is by:
- umounting the affected ost(s)
- run a full fsck on the volume(s) (using the latest e2fsprogs fsck)
- mount the volume again IF you are sure that your storage array didn't go bananas
and will not cause any new corruption (btw: are you running a 'snowbird-lustre'
with linux sw-raid6?)
Oh: and tell your users that they should check the return value of write() calls ;-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
1:48 a.m.
Would it be a better idea to move the data off of the OST in problem, and then run fsck?
Regards,
Amit
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Kumar, Amit
Sent: Thursday, August 15, 2013 9:32 AM
To: Adrian Ulrich
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Forgot the attachment: here is the messages file: look for line number 3442
http://tinyurl.com/kagz38t
Thank you,
Amit
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Kumar, Amit
Sent: Thursday, August 15, 2013 8:59 AM
To: Adrian Ulrich
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Hi Adrian,
Thank you for your quick response. NO I am not running snow-bird lustre with sw-raid6;
All of our arrays configured with hardware raid6.
I found the disk array in problem. Although I see when it remounted file system as
read-only, looking into the logs before remount does not reveal anything about what went
wrong. I see common processing errors but not more that I can understand to be critical
enough to result in read only mounts.
Remount happened a week back, attached is the copy of log file. Can you please look into
and see if you notice anything unusual.
***Another quick question.
My assumption is I can un-mount the OST in problem while the others are in production
and run the fsck on the ost volumes.
Any specific options I need to pass when I run fsck?
Version of e2fsprogs we have is: e2fsprogs-1.41.10.sun2-0redhat
Anything else do I need to watch out, before I bring this OST back online?
Please advise.
Amit
-----Original Message-----
From: Adrian Ulrich [mailto:adrian@blinkenlights.ch]
Sent: Thursday, August 15, 2013 1:02 AM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start())
error starting handle for op 8 (71 credits): rc -30
./asm-generic/errno-base.h:#define EROFS 30
mmap IO isn't your issue: It seems that (at least) one OST on your OSS got
automatically remounted read-only due to a filesystem error.
Older log entries on the OST should tell you what went wrong (grep remounting
/var/log/messages*).
The only way to recover from this is by:
- umounting the affected ost(s)
- run a full fsck on the volume(s) (using the latest e2fsprogs fsck)
- mount the volume again IF you are sure that your storage array didn't go bananas
and will not cause any new corruption (btw: are you running a 'snowbird-lustre'
with linux sw-raid6?)
Oh: and tell your users that they should check the return value of write() calls ;-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
5:14 a.m.
I got the OST back to healthy state now, by running e2fsck and fixing 3 minor errors.
Thank you very much Adrian for your direction ... !!!
Regards,
Amit
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Kumar, Amit
Sent: Thursday, August 15, 2013 10:48 AM
To: Adrian Ulrich
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Would it be a better idea to move the data off of the OST in problem, and then run fsck?
Regards,
Amit
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Kumar, Amit
Sent: Thursday, August 15, 2013 9:32 AM
To: Adrian Ulrich
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Forgot the attachment: here is the messages file: look for line number 3442
http://tinyurl.com/kagz38t
Thank you,
Amit
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Kumar, Amit
Sent: Thursday, August 15, 2013 8:59 AM
To: Adrian Ulrich
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Hi Adrian,
Thank you for your quick response. NO I am not running snow-bird lustre with sw-raid6;
All of our arrays configured with hardware raid6.
I found the disk array in problem. Although I see when it remounted file system as
read-only, looking into the logs before remount does not reveal anything about what went
wrong. I see common processing errors but not more that I can understand to be critical
enough to result in read only mounts.
Remount happened a week back, attached is the copy of log file. Can you please look into
and see if you notice anything unusual.
***Another quick question.
My assumption is I can un-mount the OST in problem while the others are in production
and run the fsck on the ost volumes.
Any specific options I need to pass when I run fsck?
Version of e2fsprogs we have is: e2fsprogs-1.41.10.sun2-0redhat
Anything else do I need to watch out, before I bring this OST back online?
Please advise.
Amit
-----Original Message-----
From: Adrian Ulrich [mailto:adrian@blinkenlights.ch]
Sent: Thursday, August 15, 2013 1:02 AM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
LustreError: 10144:0:(fsfilt-ldiskfs.c:367:fsfilt_ldiskfs_start())
error starting handle for op 8 (71 credits): rc -30
./asm-generic/errno-base.h:#define EROFS 30
mmap IO isn't your issue: It seems that (at least) one OST on your OSS got
automatically remounted read-only due to a filesystem error.
Older log entries on the OST should tell you what went wrong (grep remounting
/var/log/messages*).
The only way to recover from this is by:
- umounting the affected ost(s)
- run a full fsck on the volume(s) (using the latest e2fsprogs fsck)
- mount the volume again IF you are sure that your storage array didn't go bananas
and will not cause any new corruption (btw: are you running a 'snowbird-lustre'
with linux sw-raid6?)
Oh: and tell your users that they should check the return value of write() calls ;-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
3:22 p.m.
Hello Amit,
Although I see when it remounted file system as read-only, looking
into the logs before remount does not reveal anything about what went
wrong.
No: The syslog clearly shows that ldiskfs found some filesystem corruption:
Aug 6 18:57:00 diskarray2 kernel: LDISKFS-fs error (device sdb):
ldiskfs_valid_block_bitmap: Invalid block bitmap - block_group = 30176, block = 988807170
Aug 6 18:57:00 diskarray2 kernel: LDISKFS-fs error (device sdb):
ldiskfs_valid_block_bitmap: Invalid block bitmap - block_group = 30176, block = 988807170
Aug 6 18:57:00 diskarray2 kernel: Aborting journal on device sdb-8.
Aug 6 18:57:00 diskarray2 kernel: LDISKFS-fs (sdb): Remounting
filesystem read-only
This corruption might be old, but you should still investigate why it happened.
A normal kernel crash shouldn't cause such an error. This is most likely a bug in your
storage controller.
I got the OST back to healthy state now, by running e2fsck and fixing
3
minor errors.
Glad it worked for you, but i would still be very sceptical about the sanity of your
storage device :-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
8:53 p.m.
Dear Adrian,
Kudos for your insight. I will check if there are firmware updates to storage
arrays/controller.
Please let me know if you found anything else in the syslog that is worth paying attention
to.
Best Regards,
Amit
-----Original Message-----
From: Adrian Ulrich [mailto:adrian@blinkenlights.ch]
Sent: Friday, August 16, 2013 12:23 AM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Redo for recoverable error ?
Hello Amit,
Although I see when it remounted file system as read-only, looking
into the logs before remount does not reveal anything about what went
wrong.
No: The syslog clearly shows that ldiskfs found some filesystem corruption:
Aug 6 18:57:00 diskarray2 kernel: LDISKFS-fs error (device sdb):
ldiskfs_valid_block_bitmap: Invalid block bitmap - block_group = 30176, block = 988807170
Aug 6 18:57:00 diskarray2 kernel: LDISKFS-fs error (device sdb):
ldiskfs_valid_block_bitmap: Invalid block bitmap - block_group = 30176, block = 988807170
Aug 6 18:57:00 diskarray2 kernel: Aborting journal on device sdb-8.
Aug 6 18:57:00 diskarray2 kernel: LDISKFS-fs (sdb): Remounting filesystem read-only
This corruption might be old, but you should still investigate why it happened.
A normal kernel crash shouldn't cause such an error. This is most likely a bug in your
storage controller.
I got the OST back to healthy state now, by running e2fsck and fixing
3 minor errors.
Glad it worked for you, but i would still be very sceptical about the sanity of your
storage device :-)
Regards,
Adrian
--
RFC 1925:
(11) Every old idea will be proposed again with a different name and
a different presentation, regardless of whether it works.
3320
days inactive
3322
days old
7 comments
2 participants
participants (2)
-
Adrian Ulrich -
Kumar, Amit