5:27 a.m.
Dear All,
Is this
link(http://wiki.old.lustre.org/manual/LustreManual20_HTML/LustreTroubles...
the most up to date list of error messages? I remember looking up a header file that had
these defined as well, but can't seem to recall which one it was now?
One of over 1000 clients is being stubborn on mounting a handful OST's(total of 4 out
of 77) .
We had a maintenance and this server was forced to shut down, as it wouldn't. Now that
rest of them have rebooted successfully I can't figure out why this is stuck. It
seems to me that MDS has set these OST's as read only for this specific client because
it was forced to shut down, not sure if this is that case?
Below is what I find:
Client Machine> #lfs df -h
.....
OST001b : inactive device
...
OST001d : inactive device
....
OST0027 : inactive device
....
OST002c : inactive device
They mount perfectly on other clients.
I have made sure that these devices are up/active on MDS.
Below are relevant messages when I mount the file system on the client. Below here I am
showing message logs from MDS,OSS and Client
mds# dmesg
LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send())
scratch-OST002c-osc-MDT0000: can't precreate: rc = -30
LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 240 previous
similar messages
LustreError: 9729:0:(osp_precreate.c:989:osp_precreate_thread())
scratch-OST002c-osc-MDT0000: cannot precreate objects: rc = -30
LustreError: 9729:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 240 previous
similar messages
Lustre: scratch-OST001d-osc-MDT0000: slow creates, last=[0x1001d0000:0xc1941:0x0],
next=[0x1001d0000:0xc1941:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-30
Lustre: Skipped 3 previous similar messages
Lustre: scratch-OST002c-osc-MDT0000: slow creates, last=[0x1002c0000:0x76f41:0x0],
next=[0x1002c0000:0x76f41:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-30
Lustre: scratch-OST001b-osc-MDT0000: slow creates, last=[0x1001b0000:0xc5361:0x0],
next=[0x1001b0000:0xc5361:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-30
LustreError: 9697:0:(osp_precreate.c:484:osp_precreate_send())
scratch-OST001b-osc-MDT0000: can't precreate: rc = -30
LustreError: 9697:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 241 previous
similar messages
LustreError: 9697:0:(osp_precreate.c:989:osp_precreate_thread())
scratch-OST001b-osc-MDT0000: cannot precreate objects: rc = -30
LustreError: 9697:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 241 previous
similar messages
Lustre: scratch-OST001b-osc-MDT0000: slow creates, last=[0x1001b0000:0xc5361:0x0],
next=[0x1001b0000:0xc5361:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-30
LustreError: 9721:0:(osp_precreate.c:484:osp_precreate_send())
scratch-OST0027-osc-MDT0000: can't precreate: rc = -30
LustreError: 9721:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 241 previous
similar messages
LustreError: 9721:0:(osp_precreate.c:989:osp_precreate_thread())
scratch-OST0027-osc-MDT0000: cannot precreate objects: rc = -30
LustreError: 9721:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 241 previous
similar messages
Lustre: scratch-OST002c-osc-MDT0000: slow creates, last=[0x1002c0000:0x76f41:0x0],
next=[0x1002c0000:0x76f41:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-30
Lustre: Skipped 4 previous similar messages
LustreError: 9697:0:(osp_precreate.c:484:osp_precreate_send())
scratch-OST001b-osc-MDT0000: can't precreate: rc = -30
LustreError: 9697:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 244 previous
similar messages
LustreError: 9697:0:(osp_precreate.c:989:osp_precreate_thread())
scratch-OST001b-osc-MDT0000: cannot precreate objects: rc = -30
LustreError: 9697:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 244 previous
similar messages
Lustre: scratch-OST001b-osc-MDT0000: slow creates, last=[0x1001b0000:0xc5361:0x0],
next=[0x1001b0000:0xc5361:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-30
Lustre: Skipped 4 previous similar messages
LustreError: 9697:0:(osp_precreate.c:484:osp_precreate_send())
scratch-OST001b-osc-MDT0000: can't precreate: rc = -30
LustreError: 9697:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 243 previous
similar messages
LustreError: 9697:0:(osp_precreate.c:989:osp_precreate_thread())
scratch-OST001b-osc-MDT0000: cannot precreate objects: rc = -30
LustreError: 9697:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 243 previous
similar messages
On the OSS with troublesome OST's for the client, I saw following messages:
oss# dmesg
LustreError: 17312:0:(ofd_obd.c:1348:ofd_create()) scratch-OST001b: unable to precreate:
rc = -30
LustreError: 17312:0:(ofd_obd.c:1348:ofd_create()) Skipped 120 previous similar messages
LustreError: 18874:0:(tgt_lastrcvd.c:577:tgt_client_new()) scratch-OST001b: Failed to
write client lcd at idx 368, rc -30
LustreError: 18874:0:(tgt_lastrcvd.c:577:tgt_client_new()) Skipped 35 previous similar
messages
On Client I see these messages:
Client #dmesg
LustreError: 5978:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 5978:0:(mdc_locks.c:848:mdc_enqueue()) Skipped 25 previous similar messages
LustreError: 5963:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 5963:0:(mdc_locks.c:848:mdc_enqueue()) Skipped 19 previous similar messages
Lustre: Unmounted scratch-client
LNet: Removed LNI 10.42.211.97@o2ib
LNet: HW CPU cores: 20, npartitions: 4
alg: No test for crc32 (crc32-table)
alg: No test for adler32 (adler32-zlib)
alg: No test for crc32 (crc32-pclmul)
Lustre: Lustre: Build Version: 2.4.3-RC1--PRISTINE-2.6.32-431.17.1.el6.x86_64
LNet: Added LNI 10.42.211.97@o2ib [8/256/0/180]
Lustre: Layout lock feature supported.
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: 11-0: scratch-OST001b-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
Lustre: Mounted scratch-client
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 2 previous similar messages
LustreError: 11-0: scratch-OST001b-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 3 previous similar messages
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 3 previous similar messages
LustreError: 11-0: scratch-OST001b-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 3 previous similar messages
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 3 previous similar messages
LustreError: 6538:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 6538:0:(file.c:2718:ll_inode_revalidate_fini()) scratch: revalidate FID
[0x200000007:0x1:0x0] error: rc = -13
LustreError: 6538:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 6538:0:(file.c:2718:ll_inode_revalidate_fini()) scratch: revalidate FID
[0x200000007:0x1:0x0] error: rc = -13
LustreError: 6524:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 6524:0:(mdc_locks.c:848:mdc_enqueue()) Skipped 18 previous similar messages
LustreError: 6524:0:(file.c:2718:ll_inode_revalidate_fini()) scratch: revalidate FID
[0x200000007:0x1:0x0] error: rc = -13
LustreError: 6524:0:(file.c:2718:ll_inode_revalidate_fini()) Skipped 18 previous similar
messages
LustreError: 11-0: scratch-OST001b-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 3 previous similar messages
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 7 previous similar messages
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 11 previous similar messages
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 23 previous similar messages
LustreError: 11-0: scratch-OST001b-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 43 previous similar messages
LustreError: 6909:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 6909:0:(mdc_locks.c:848:mdc_enqueue()) Skipped 5 previous similar messages
LustreError: 6909:0:(file.c:2718:ll_inode_revalidate_fini()) scratch: revalidate FID
[0x200000007:0x1:0x0] error: rc = -13
LustreError: 6909:0:(file.c:2718:ll_inode_revalidate_fini()) Skipped 5 previous similar
messages
LustreError: 6909:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 6909:0:(mdc_locks.c:848:mdc_enqueue()) Skipped 11 previous similar messages
LustreError: 6909:0:(file.c:2718:ll_inode_revalidate_fini()) scratch: revalidate FID
[0x200000007:0x1:0x0] error: rc = -13
LustreError: 6909:0:(file.c:2718:ll_inode_revalidate_fini()) Skipped 11 previous similar
messages
LustreError: 11-0: scratch-OST001d-osc-ffff880314c17c00: Communicating with
10.42.145.10@o2ib, operation ost_connect failed with -30.
LustreError: Skipped 83 previous similar messages
LustreError: 7153:0:(mdc_locks.c:848:mdc_enqueue()) ldlm_cli_enqueue: -13
LustreError: 7153:0:(mdc_locks.c:848:mdc_enqueue()) Skipped 13 previous similar messages
Thank you,
Amit
5:53 a.m.
On Jul 28, 2015, at 3:27 PM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
mds# dmesg
LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send())
scratch-OST002c-osc-MDT0000: can't precreate: rc = -30
LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 240 previous
similar messages
The “rc=-30” error usually means that an OST has gone read-only and the server cannot
write data to it for some reason.
Have you tried writing to one of those OSTs from a “working” client to make sure you can
actually send data to it?
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
6:07 a.m.
Hi Rick,
> mds# dmesg
>
> LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send())
> scratch-OST002c-osc-MDT0000: can't precreate: rc = -30
> LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send()) Skipped
> 240 previous similar messages
The “rc=-30” error usually means that an OST has gone read-only and the
server cannot write data to it for some reason.
Have you tried writing to one of those OSTs from a “working” client to make
sure you can actually send data to it?
Yes it is rather slow as you can see the file27 & file31 is still being written to
while others have finished:
I created a file with ost-index and wrote to them directly.
# ll
total 32634864
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file26
-rw-r--r-- 1 root root 1643765760 Jul 28 15:04 file27
-rw-r--r-- 1 root root 2654289920 Jul 28 15:04 file28
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file29
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file30
-rw-r--r-- 1 root root 2003828736 Jul 28 15:04 file31
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file32
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file33
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file34
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file35
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file36
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file37
What can we do about this? Run e2fsck and bring it back online? Or migrate the data off
and recreate OST?
Any ideas?
Thank you,
Amit
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
8:09 a.m.
I see a handful(5 OST's) with 'rc=30'. They all belong to the same OSS server.
Is this
(http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco... )
document still a good reference to recover from rc=30 for OST's I have done this in
the part bust would like to confirm.
Any input here is greatly appreciated.
Thank you,
Amit
-----Original Message-----
From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of
Kumar, Amit
Sent: Tuesday, July 28, 2015 3:08 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
OST's
Hi Rick,
>> mds# dmesg
>>
>> LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send())
>> scratch-OST002c-osc-MDT0000: can't precreate: rc = -30
>> LustreError: 9729:0:(osp_precreate.c:484:osp_precreate_send())
>> Skipped
>> 240 previous similar messages
>
>The “rc=-30” error usually means that an OST has gone read-only and the
>server cannot write data to it for some reason.
>
>Have you tried writing to one of those OSTs from a “working” client to
>make sure you can actually send data to it?
Yes it is rather slow as you can see the file27 & file31 is still being written to
while others have finished:
I created a file with ost-index and wrote to them directly.
# ll
total 32634864
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file26
-rw-r--r-- 1 root root 1643765760 Jul 28 15:04 file27
-rw-r--r-- 1 root root 2654289920 Jul 28 15:04 file28
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file29
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file30
-rw-r--r-- 1 root root 2003828736 Jul 28 15:04 file31
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file32
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file33
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file34
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file35
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file36
-rw-r--r-- 1 root root 3355443200 Jul 28 14:59 file37
What can we do about this? Run e2fsck and bring it back online? Or migrate
the data off and recreate OST?
Any ideas?
Thank you,
Amit
>
>--
>Rick Mohr
>Senior HPC System Administrator
>National Institute for Computational Sciences
>http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
8:13 a.m.
On Jul 28, 2015, at 6:09 PM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
I see a handful(5 OST's) with 'rc=30'. They all belong to the same OSS
server.
Is this
(http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco... )
document still a good reference to recover from rc=30 for OST's I have done this in
the part bust would like to confirm.
I think running the e2fsck is a reasonable step to make sure things are OK.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
12:59 p.m.
Running 'cat /proc/fs/lustre/health_check' on the problematic OSS should tell you
which volumes need fsck'd. We run this on all of our OSS servers as a sanity check
whenever we see recurrent problems with a client.
Hope that helps.
-Ed Walter
Carnegie Mellon University
Sent from my iPad
On Jul 28, 2015, at 6:13 PM, Mohr Jr, Richard Frank (Rick Mohr)
<rmohr(a)utk.edu> wrote:
> On Jul 28, 2015, at 6:09 PM, Kumar, Amit <ahkumar(a)mail.smu.edu> wrote:
>
> I see a handful(5 OST's) with 'rc=30'. They all belong to the same OSS
server.
> Is this
(http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco... )
document still a good reference to recover from rc=30 for OST's I have done this in
the part bust would like to confirm.
I think running the e2fsck is a reasonable step to make sure things are OK.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
11:44 p.m.
Dear All,
Ed, Thank you for the suggestion, yes i run a constant check on
/proc/fs/lustre/health_check to make sure the OSS is healthy, but as understand this may
fail to capture all of the underlying file system related issues with OST's.
Following instructions from here:
http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco...
Here is something that I found was interesting, it turned out OST had some quota
warnings/errors and fix to that resolved one of the OST's that I attempted to run
es2fsck on. Could it be I have some incorrect quota settings? We do not use quota at all.
Any thoughts here is greatly appreciated.
On my dry run of 'e2fsck -fn /dev/mapper/mpathc' I saw these to issues being
reported and no action was taken.
Update quota info for quota type 0? no
Update quota info for quota type 1? no
# time e2fsck -fp /dev/mapper/mpathc
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15367602176, 44622) != expected
(3130023936, 44)
[QUOTA WARNING] Usage inconsistent for ID 6007:actual (118618427392, 149782) != expected
(92410003456, 149576)
[QUOTA WARNING] Usage inconsistent for ID 6001:actual (12952391680, 199) != expected
(12948684800, 195)
[QUOTA WARNING] Usage inconsistent for ID 6014:actual (14362611712, 324) != expected
(9919168512, 313)
[QUOTA WARNING] Usage inconsistent for ID 6012:actual (4572880248832, 351) != expected
(4571084505088, 342)
[QUOTA WARNING] Usage inconsistent for ID 6015:actual (4512403456, 958) != expected
(4512399360, 957)
[QUOTA WARNING] Usage inconsistent for ID 6018:actual (25878560768, 320) != expected
(25876611072, 316)
[QUOTA WARNING] Usage inconsistent for ID 6017:actual (2435170304, 207) != expected
(2430341120, 205)
[QUOTA WARNING] Usage inconsistent for ID 6013:actual (577578684416, 9602) != expected
(577578577920, 9600)
[QUOTA WARNING] Usage inconsistent for ID 6020:actual (296636416, 21) != expected (0, 3)
[QUOTA WARNING] Usage inconsistent for ID 6016:actual (2532306944, 61) != expected
(1910861824, 58)
[QUOTA WARNING] Usage inconsistent for ID 6024:actual (84418273280, 23489) != expected
(84391985152, 23406)
[QUOTA WARNING] Usage inconsistent for ID 6025:actual (38466719744, 289) != expected
(28055818240, 115)
[QUOTA WARNING] Usage inconsistent for ID 6011:actual (10530041856, 208) != expected
(7811461120, 187)
[QUOTA WARNING] Usage inconsistent for ID 6021:actual (6542618624, 4408) != expected
(6521831424, 4401)
[QUOTA WARNING] Usage inconsistent for ID 6042:actual (1128153088, 1386) != expected
(157282304, 11)
[QUOTA WARNING] Usage inconsistent for ID 6006:actual (2205396992, 15468) != expected
(874708992, 11545)
scratch-OST001b: Update quota info for quota type 0.
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15367602176, 44622) != expected
(3130023936, 44)
[QUOTA WARNING] Usage inconsistent for ID 1007:actual (4874933129216, 176018) != expected
(3413366910976, 22886)
[QUOTA WARNING] Usage inconsistent for ID 501:actual (12533772288, 180) != expected
(12530065408, 176)
[QUOTA WARNING] Usage inconsistent for ID 1008:actual (577578684416, 9602) != expected
(577578577920, 9600)
[QUOTA WARNING] Usage inconsistent for ID 1005:actual (11552878592, 13517) != expected
(11532091392, 13510)
[QUOTA WARNING] Usage inconsistent for ID 1000:actual (1129291776, 1399) != expected
(113139712, 13)
[QUOTA WARNING] Usage inconsistent for ID 1001:actual (2982502400, 16036) != expected
(1651302400, 12104)
scratch-OST001b: Update quota info for quota type 1.
scratch-OST001b: 262297/22892544 files (9.0% non-contiguous), 1346700157/5860466688
blocks
All is well with this OST, now on to next one.
Thank you,
Amit
________________________________________
From: Edward Walter [ewalter(a)cs.cmu.edu]
Sent: Tuesday, July 28, 2015 9:59 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: Kumar, Amit; hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful OST's
Running 'cat /proc/fs/lustre/health_check' on the problematic OSS should tell you
which volumes need fsck'd. We run this on all of our OSS servers as a sanity check
whenever we see recurrent problems with a client.
Hope that helps.
-Ed Walter
Carnegie Mellon University
Sent from my iPad
On Jul 28, 2015, at 6:13 PM, Mohr Jr, Richard Frank (Rick Mohr)
<rmohr(a)utk.edu> wrote:
> On Jul 28, 2015, at 6:09 PM, Kumar, Amit <ahkumar(a)mail.smu.edu> wrote:
>
> I see a handful(5 OST's) with 'rc=30'. They all belong to the same OSS
server.
> Is this
(http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco... )
document still a good reference to recover from rc=30 for OST's I have done this in
the part bust would like to confirm.
I think running the e2fsck is a reasonable step to make sure things are OK.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2:22 a.m.
I have encountered similar errors on some of my OSTs when things did not shutdown cleanly.
I just disabled quotas on the OST and then re-enabled them so that they would be
recalculated. After that, e2fsck did not report any errors.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
On Jul 30, 2015, at 9:44 AM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
Dear All,
Ed, Thank you for the suggestion, yes i run a constant check on
/proc/fs/lustre/health_check to make sure the OSS is healthy, but as understand this may
fail to capture all of the underlying file system related issues with OST's.
Following instructions from here:
http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco...
Here is something that I found was interesting, it turned out OST had some quota
warnings/errors and fix to that resolved one of the OST's that I attempted to run
es2fsck on. Could it be I have some incorrect quota settings? We do not use quota at all.
Any thoughts here is greatly appreciated.
On my dry run of 'e2fsck -fn /dev/mapper/mpathc' I saw these to issues being
reported and no action was taken.
Update quota info for quota type 0? no
Update quota info for quota type 1? no
# time e2fsck -fp /dev/mapper/mpathc
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15367602176, 44622) != expected
(3130023936, 44)
[QUOTA WARNING] Usage inconsistent for ID 6007:actual (118618427392, 149782) != expected
(92410003456, 149576)
[QUOTA WARNING] Usage inconsistent for ID 6001:actual (12952391680, 199) != expected
(12948684800, 195)
[QUOTA WARNING] Usage inconsistent for ID 6014:actual (14362611712, 324) != expected
(9919168512, 313)
[QUOTA WARNING] Usage inconsistent for ID 6012:actual (4572880248832, 351) != expected
(4571084505088, 342)
[QUOTA WARNING] Usage inconsistent for ID 6015:actual (4512403456, 958) != expected
(4512399360, 957)
[QUOTA WARNING] Usage inconsistent for ID 6018:actual (25878560768, 320) != expected
(25876611072, 316)
[QUOTA WARNING] Usage inconsistent for ID 6017:actual (2435170304, 207) != expected
(2430341120, 205)
[QUOTA WARNING] Usage inconsistent for ID 6013:actual (577578684416, 9602) != expected
(577578577920, 9600)
[QUOTA WARNING] Usage inconsistent for ID 6020:actual (296636416, 21) != expected (0, 3)
[QUOTA WARNING] Usage inconsistent for ID 6016:actual (2532306944, 61) != expected
(1910861824, 58)
[QUOTA WARNING] Usage inconsistent for ID 6024:actual (84418273280, 23489) != expected
(84391985152, 23406)
[QUOTA WARNING] Usage inconsistent for ID 6025:actual (38466719744, 289) != expected
(28055818240, 115)
[QUOTA WARNING] Usage inconsistent for ID 6011:actual (10530041856, 208) != expected
(7811461120, 187)
[QUOTA WARNING] Usage inconsistent for ID 6021:actual (6542618624, 4408) != expected
(6521831424, 4401)
[QUOTA WARNING] Usage inconsistent for ID 6042:actual (1128153088, 1386) != expected
(157282304, 11)
[QUOTA WARNING] Usage inconsistent for ID 6006:actual (2205396992, 15468) != expected
(874708992, 11545)
scratch-OST001b: Update quota info for quota type 0.
[QUOTA WARNING] Usage inconsistent for ID 0:actual (15367602176, 44622) != expected
(3130023936, 44)
[QUOTA WARNING] Usage inconsistent for ID 1007:actual (4874933129216, 176018) != expected
(3413366910976, 22886)
[QUOTA WARNING] Usage inconsistent for ID 501:actual (12533772288, 180) != expected
(12530065408, 176)
[QUOTA WARNING] Usage inconsistent for ID 1008:actual (577578684416, 9602) != expected
(577578577920, 9600)
[QUOTA WARNING] Usage inconsistent for ID 1005:actual (11552878592, 13517) != expected
(11532091392, 13510)
[QUOTA WARNING] Usage inconsistent for ID 1000:actual (1129291776, 1399) != expected
(113139712, 13)
[QUOTA WARNING] Usage inconsistent for ID 1001:actual (2982502400, 16036) != expected
(1651302400, 12104)
scratch-OST001b: Update quota info for quota type 1.
scratch-OST001b: 262297/22892544 files (9.0% non-contiguous), 1346700157/5860466688
blocks
All is well with this OST, now on to next one.
Thank you,
Amit
________________________________________
From: Edward Walter [ewalter(a)cs.cmu.edu]
Sent: Tuesday, July 28, 2015 9:59 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: Kumar, Amit; hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful OST's
Running 'cat /proc/fs/lustre/health_check' on the problematic OSS should tell you
which volumes need fsck'd. We run this on all of our OSS servers as a sanity check
whenever we see recurrent problems with a client.
Hope that helps.
-Ed Walter
Carnegie Mellon University
Sent from my iPad
> On Jul 28, 2015, at 6:13 PM, Mohr Jr, Richard Frank (Rick Mohr) <rmohr(a)utk.edu>
wrote:
>
>
>> On Jul 28, 2015, at 6:09 PM, Kumar, Amit <ahkumar(a)mail.smu.edu> wrote:
>>
>> I see a handful(5 OST's) with 'rc=30'. They all belong to the same
OSS server.
>> Is this
(http://wiki.old.lustre.org/manual/LustreManual20_HTML/TroubleShootingReco... )
document still a good reference to recover from rc=30 for OST's I have done this in
the part bust would like to confirm.
>
> I think running the e2fsck is a reasonable step to make sure things are OK.
>
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
4:13 a.m.
Hi Rick,
I have encountered similar errors on some of my OSTs when things did not
shutdown cleanly. I just disabled quotas on the OST and then re-enabled
them so that they would be recalculated. After that, e2fsck did not report any
errors.
Version of my lustre: 2.4.3
I fail to see if see any difference in the quota settings, after I disable them as
follows
lctl conf_param scratch.quota.ost=none
lctl conf_param scratch.quota.mdt=none
Also looking at the troubled OST's parameters "before" and "after"
applying the above commands on MGS, I don't think it had any effect.
Please let me know if you did anything else to disable quota on OST's.
Thank you,
Amit
# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.scratch-OST001a.quota_slave.info=
target name: scratch-OST001a
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001b.quota_slave.info=
target name: scratch-OST001b
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001c.quota_slave.info=
target name: scratch-OST001c
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001d.quota_slave.info=
target name: scratch-OST001d
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001e.quota_slave.info=
target name: scratch-OST001e
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001f.quota_slave.info=
target name: scratch-OST001f
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
[root@mfoss05 scratch-OST001d]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.scratch-OST001a.quota_slave.info=
target name: scratch-OST001a
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001b.quota_slave.info=
target name: scratch-OST001b
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001c.quota_slave.info=
target name: scratch-OST001c
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001d.quota_slave.info=
target name: scratch-OST001d
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001e.quota_slave.info=
target name: scratch-OST001e
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001f.quota_slave.info=
target name: scratch-OST001f
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
7:39 a.m.
Upon e2fsck -fp /dev/...
These OST's recover from "rc = -30". But I see a trend with them running
into slow creates, it his part of the recovery cycle and should it subside eventually?
Lustre: scratch-OST001d-osc-MDT0000: slow creates, last=[0x1001d0000:0xc1941:0x0],
next=[0x1001d0000:0xc1941:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-19
I see similar entries in https://jira.hpdd.intel.com/browse/LU-2279, any thoughts on: if
this is an issue of concern will be very helpful.
Thank you,
Amit
-----Original Message-----
From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of
Kumar, Amit
Sent: Thursday, July 30, 2015 1:13 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
OST's
Hi Rick,
>
>I have encountered similar errors on some of my OSTs when things did
>not shutdown cleanly. I just disabled quotas on the OST and then
>re-enabled them so that they would be recalculated. After that, e2fsck
>did not report any errors.
>
Version of my lustre: 2.4.3
I fail to see if see any difference in the quota settings, after I disable them as
follows
lctl conf_param scratch.quota.ost=none
lctl conf_param scratch.quota.mdt=none
Also looking at the troubled OST's parameters "before" and "after"
applying
the above commands on MGS, I don't think it had any effect.
Please let me know if you did anything else to disable quota on OST's.
Thank you,
Amit
# lctl get_param osd-*.*.quota_slave.info osd-ldiskfs.scratch-
OST001a.quota_slave.info=
target name: scratch-OST001a
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001b.quota_slave.info=
target name: scratch-OST001b
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001c.quota_slave.info=
target name: scratch-OST001c
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001d.quota_slave.info=
target name: scratch-OST001d
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001e.quota_slave.info=
target name: scratch-OST001e
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001f.quota_slave.info=
target name: scratch-OST001f
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
[root@mfoss05 scratch-OST001d]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.scratch-OST001a.quota_slave.info=
target name: scratch-OST001a
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001b.quota_slave.info=
target name: scratch-OST001b
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001c.quota_slave.info=
target name: scratch-OST001c
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001d.quota_slave.info=
target name: scratch-OST001d
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001e.quota_slave.info=
target name: scratch-OST001e
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001f.quota_slave.info=
target name: scratch-OST001f
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
8:41 a.m.
Amit,
-ENODEV (status=-19) may be from sort of race related to reconnection/recovery, if I'm
reading that LU correctly. If those messages continue for a long time, it's cause for
concern. If they go away within, say, 10-30 minutes of startup (so all clients should be
reconnected), I'd it's nothing to be concerned about, unless they come back.
- Patrick
________________________________________
From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Kumar, Amit
[ahkumar(a)mail.smu.edu]
Sent: Thursday, July 30, 2015 4:39 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful OST's
Upon e2fsck -fp /dev/...
These OST's recover from "rc = -30". But I see a trend with them running
into slow creates, it his part of the recovery cycle and should it subside eventually?
Lustre: scratch-OST001d-osc-MDT0000: slow creates, last=[0x1001d0000:0xc1941:0x0],
next=[0x1001d0000:0xc1941:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0,
status=-19
I see similar entries in https://jira.hpdd.intel.com/browse/LU-2279, any thoughts on: if
this is an issue of concern will be very helpful.
Thank you,
Amit
-----Original Message-----
From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of
Kumar, Amit
Sent: Thursday, July 30, 2015 1:13 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
OST's
Hi Rick,
>
>I have encountered similar errors on some of my OSTs when things did
>not shutdown cleanly. I just disabled quotas on the OST and then
>re-enabled them so that they would be recalculated. After that, e2fsck
>did not report any errors.
>
Version of my lustre: 2.4.3
I fail to see if see any difference in the quota settings, after I disable them as
follows
lctl conf_param scratch.quota.ost=none
lctl conf_param scratch.quota.mdt=none
Also looking at the troubled OST's parameters "before" and "after"
applying
the above commands on MGS, I don't think it had any effect.
Please let me know if you did anything else to disable quota on OST's.
Thank you,
Amit
# lctl get_param osd-*.*.quota_slave.info osd-ldiskfs.scratch-
OST001a.quota_slave.info=
target name: scratch-OST001a
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001b.quota_slave.info=
target name: scratch-OST001b
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001c.quota_slave.info=
target name: scratch-OST001c
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001d.quota_slave.info=
target name: scratch-OST001d
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001e.quota_slave.info=
target name: scratch-OST001e
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001f.quota_slave.info=
target name: scratch-OST001f
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
[root@mfoss05 scratch-OST001d]# lctl get_param osd-*.*.quota_slave.info
osd-ldiskfs.scratch-OST001a.quota_slave.info=
target name: scratch-OST001a
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001b.quota_slave.info=
target name: scratch-OST001b
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001c.quota_slave.info=
target name: scratch-OST001c
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001d.quota_slave.info=
target name: scratch-OST001d
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001e.quota_slave.info=
target name: scratch-OST001e
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
osd-ldiskfs.scratch-OST001f.quota_slave.info=
target name: scratch-OST001f
pool ID: 0
type: dt
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
1:40 p.m.
HI Patrick,
That worries me. I am taking down the file system for another clean restart. Is there
anything different one should do in this case just to make sure this does not escalate to
worry some repetition?
Thank you for your response.
Amit
-----Original Message-----
From: Patrick Farrell [mailto:paf@cray.com]
Sent: Thursday, July 30, 2015 5:42 PM
To: Kumar, Amit; Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: RE: [HPDD-discuss] One of several Clients failing to mount handful
OST's
Amit,
-ENODEV (status=-19) may be from sort of race related to
reconnection/recovery, if I'm reading that LU correctly. If those messages
continue for a long time, it's cause for concern. If they go away within, say,
10-30 minutes of startup (so all clients should be reconnected), I'd it's nothing
to be concerned about, unless they come back.
- Patrick
________________________________________
From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Kumar,
Amit [ahkumar(a)mail.smu.edu]
Sent: Thursday, July 30, 2015 4:39 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
OST's
Upon e2fsck -fp /dev/...
These OST's recover from "rc = -30". But I see a trend with them running
into
slow creates, it his part of the recovery cycle and should it subside eventually?
Lustre: scratch-OST001d-osc-MDT0000: slow creates,
last=[0x1001d0000:0xc1941:0x0], next=[0x1001d0000:0xc1941:0x0],
reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=-19
I see similar entries in https://jira.hpdd.intel.com/browse/LU-2279, any
thoughts on: if this is an issue of concern will be very helpful.
Thank you,
Amit
>-----Original Message-----
>From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf
>Of Kumar, Amit
>Sent: Thursday, July 30, 2015 1:13 PM
>To: Mohr Jr, Richard Frank (Rick Mohr)
>Cc: hpdd-discuss(a)lists.01.org
>Subject: Re: [HPDD-discuss] One of several Clients failing to mount
>handful OST's
>
>Hi Rick,
>>
>>I have encountered similar errors on some of my OSTs when things did
>>not shutdown cleanly. I just disabled quotas on the OST and then
>>re-enabled them so that they would be recalculated. After that,
>>e2fsck did not report any errors.
>>
>Version of my lustre: 2.4.3
>
>I fail to see if see any difference in the quota settings, after I
>disable them as follows
>
>lctl conf_param scratch.quota.ost=none
>lctl conf_param scratch.quota.mdt=none
>
>Also looking at the troubled OST's parameters "before" and
"after"
>applying the above commands on MGS, I don't think it had any effect.
>Please let me know if you did anything else to disable quota on OST's.
>
>Thank you,
>Amit
>
># lctl get_param osd-*.*.quota_slave.info osd-ldiskfs.scratch-
>OST001a.quota_slave.info=
>target name: scratch-OST001a
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001b.quota_slave.info=
>target name: scratch-OST001b
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001c.quota_slave.info=
>target name: scratch-OST001c
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001d.quota_slave.info=
>target name: scratch-OST001d
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001e.quota_slave.info=
>target name: scratch-OST001e
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001f.quota_slave.info=
>target name: scratch-OST001f
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>[root@mfoss05 scratch-OST001d]# lctl get_param osd-*.*.quota_slave.info
>osd-ldiskfs.scratch-OST001a.quota_slave.info=
>target name: scratch-OST001a
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001b.quota_slave.info=
>target name: scratch-OST001b
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001c.quota_slave.info=
>target name: scratch-OST001c
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001d.quota_slave.info=
>target name: scratch-OST001d
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001e.quota_slave.info=
>target name: scratch-OST001e
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>osd-ldiskfs.scratch-OST001f.quota_slave.info=
>target name: scratch-OST001f
>pool ID: 0
>type: dt
>quota enabled: none
>conn to master: setup
>space acct: ug
>user uptodate: glb[0],slv[0],reint[0]
>group uptodate: glb[0],slv[0],reint[0]
>_______________________________________________
>HPDD-discuss mailing list
>HPDD-discuss(a)lists.01.org
>https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
7:27 p.m.
Hi,
LU-2279 is marked as resolved in 2.4.0 and you have 2.4.3.
Did you have a look at the hardware below the OSTs? Is it a RAID with a
disk slowly failing or currently in rebuild, or a RAID controller which
switched to write through because of an empty cache battery... things
like that may cause slow IO.
Martin
On 07/31/2015 05:40 AM, Kumar, Amit wrote:
HI Patrick,
That worries me. I am taking down the file system for another clean restart. Is there
anything different one should do in this case just to make sure this does not escalate to
worry some repetition?
Thank you for your response.
Amit
> -----Original Message-----
> From: Patrick Farrell [mailto:paf@cray.com]
> Sent: Thursday, July 30, 2015 5:42 PM
> To: Kumar, Amit; Mohr Jr, Richard Frank (Rick Mohr)
> Cc: hpdd-discuss(a)lists.01.org
> Subject: RE: [HPDD-discuss] One of several Clients failing to mount handful
> OST's
>
> Amit,
>
> -ENODEV (status=-19) may be from sort of race related to
> reconnection/recovery, if I'm reading that LU correctly. If those messages
> continue for a long time, it's cause for concern. If they go away within, say,
> 10-30 minutes of startup (so all clients should be reconnected), I'd it's
nothing
> to be concerned about, unless they come back.
>
> - Patrick
> ________________________________________
> From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Kumar,
> Amit [ahkumar(a)mail.smu.edu]
> Sent: Thursday, July 30, 2015 4:39 PM
> To: Mohr Jr, Richard Frank (Rick Mohr)
> Cc: hpdd-discuss(a)lists.01.org
> Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
> OST's
>
> Upon e2fsck -fp /dev/...
>
> These OST's recover from "rc = -30". But I see a trend with them
running into
> slow creates, it his part of the recovery cycle and should it subside eventually?
>
> Lustre: scratch-OST001d-osc-MDT0000: slow creates,
> last=[0x1001d0000:0xc1941:0x0], next=[0x1001d0000:0xc1941:0x0],
> reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=-19
>
> I see similar entries in https://jira.hpdd.intel.com/browse/LU-2279, any
> thoughts on: if this is an issue of concern will be very helpful.
>
> Thank you,
> Amit
>
>
>> -----Original Message-----
>> From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf
>> Of Kumar, Amit
>> Sent: Thursday, July 30, 2015 1:13 PM
>> To: Mohr Jr, Richard Frank (Rick Mohr)
>> Cc: hpdd-discuss(a)lists.01.org
>> Subject: Re: [HPDD-discuss] One of several Clients failing to mount
>> handful OST's
>>
>> Hi Rick,
>>> I have encountered similar errors on some of my OSTs when things did
>>> not shutdown cleanly. I just disabled quotas on the OST and then
>>> re-enabled them so that they would be recalculated. After that,
>>> e2fsck did not report any errors.
>>>
>> Version of my lustre: 2.4.3
>>
>> I fail to see if see any difference in the quota settings, after I
>> disable them as follows
>>
>> lctl conf_param scratch.quota.ost=none
>> lctl conf_param scratch.quota.mdt=none
>>
>> Also looking at the troubled OST's parameters "before" and
"after"
>> applying the above commands on MGS, I don't think it had any effect.
>> Please let me know if you did anything else to disable quota on OST's.
>>
>> Thank you,
>> Amit
>> [...]
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2:03 a.m.
It turns out that storage controller had flaky ports, taking that controller offline and
running e2fsck seems to have resorted it so far.
Hope it will not reappear when I boot up clients. Following helped as well.
Quota Disable: tune2fs -O ^quota ${DEV}
Quota Enable: tune2fs -O quota $DEV
One thing I found was running tune2fs warns if recovery flag was set on the ${DEV} hence
telling us we have to run e2fsck to resolve the problem.
Sincere thanks to all for very timely help!!
Amit
-----Original Message-----
From: Martin Hecht [mailto:hecht@hlrs.de]
Sent: Friday, July 31, 2015 4:27 AM
To: Kumar, Amit; Patrick Farrell; Mohr Jr, Richard Frank (Rick Mohr); hpdd-
discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
OST's
Hi,
LU-2279 is marked as resolved in 2.4.0 and you have 2.4.3.
Did you have a look at the hardware below the OSTs? Is it a RAID with a
disk slowly failing or currently in rebuild, or a RAID controller which
switched to write through because of an empty cache battery... things
like that may cause slow IO.
Martin
On 07/31/2015 05:40 AM, Kumar, Amit wrote:
> HI Patrick,
>
> That worries me. I am taking down the file system for another clean restart.
Is there anything different one should do in this case just to make sure this
does not escalate to worry some repetition?
>
> Thank you for your response.
> Amit
>
>
>
>> -----Original Message-----
>> From: Patrick Farrell [mailto:paf@cray.com]
>> Sent: Thursday, July 30, 2015 5:42 PM
>> To: Kumar, Amit; Mohr Jr, Richard Frank (Rick Mohr)
>> Cc: hpdd-discuss(a)lists.01.org
>> Subject: RE: [HPDD-discuss] One of several Clients failing to mount handful
>> OST's
>>
>> Amit,
>>
>> -ENODEV (status=-19) may be from sort of race related to
>> reconnection/recovery, if I'm reading that LU correctly. If those messages
>> continue for a long time, it's cause for concern. If they go away within,
say,
>> 10-30 minutes of startup (so all clients should be reconnected), I'd
it's
nothing
>> to be concerned about, unless they come back.
>>
>> - Patrick
>> ________________________________________
>> From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of
Kumar,
>> Amit [ahkumar(a)mail.smu.edu]
>> Sent: Thursday, July 30, 2015 4:39 PM
>> To: Mohr Jr, Richard Frank (Rick Mohr)
>> Cc: hpdd-discuss(a)lists.01.org
>> Subject: Re: [HPDD-discuss] One of several Clients failing to mount handful
>> OST's
>>
>> Upon e2fsck -fp /dev/...
>>
>> These OST's recover from "rc = -30". But I see a trend with them
running
into
>> slow creates, it his part of the recovery cycle and should it subside
eventually?
>>
>> Lustre: scratch-OST001d-osc-MDT0000: slow creates,
>> last=[0x1001d0000:0xc1941:0x0], next=[0x1001d0000:0xc1941:0x0],
>> reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=-19
>>
>> I see similar entries in https://jira.hpdd.intel.com/browse/LU-2279, any
>> thoughts on: if this is an issue of concern will be very helpful.
>>
>> Thank you,
>> Amit
>>
>>
>>> -----Original Message-----
>>> From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf
>>> Of Kumar, Amit
>>> Sent: Thursday, July 30, 2015 1:13 PM
>>> To: Mohr Jr, Richard Frank (Rick Mohr)
>>> Cc: hpdd-discuss(a)lists.01.org
>>> Subject: Re: [HPDD-discuss] One of several Clients failing to mount
>>> handful OST's
>>>
>>> Hi Rick,
>>>> I have encountered similar errors on some of my OSTs when things did
>>>> not shutdown cleanly. I just disabled quotas on the OST and then
>>>> re-enabled them so that they would be recalculated. After that,
>>>> e2fsck did not report any errors.
>>>>
>>> Version of my lustre: 2.4.3
>>>
>>> I fail to see if see any difference in the quota settings, after I
>>> disable them as follows
>>>
>>> lctl conf_param scratch.quota.ost=none
>>> lctl conf_param scratch.quota.mdt=none
>>>
>>> Also looking at the troubled OST's parameters "before" and
"after"
>>> applying the above commands on MGS, I don't think it had any effect.
>>> Please let me know if you did anything else to disable quota on OST's.
>>>
>>> Thank you,
>>> Amit
>>> [...]
>>> _______________________________________________
>>> HPDD-discuss mailing list
>>> HPDD-discuss(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
7:46 a.m.
On Jul 30, 2015, at 2:13 PM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
I fail to see if see any difference in the quota settings, after I disable them as
follows
lctl conf_param scratch.quota.ost=none
lctl conf_param scratch.quota.mdt=none
Sorry. I should have explained more. When I talked about disabling quotas, I meant
running this command:
tune2fs -O ^quota ${DEV}
(This of course needs to be done when the OST is unmounted.)
To re-enable them, run this command:
tune2fs -O quota $DEV
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
2606
days inactive
2609
days old
14 comments
5 participants
participants (5)
-
Edward Walter -
Kumar, Amit -
Martin Hecht -
Mohr Jr, Richard Frank (Rick Mohr) -
Patrick Farrell