3:26 a.m.
Dear All,
Lustre v2.4.3, RHEL6.4, files system: ldsikfs
We have seen these errors[(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on
lock cancel] on all OST's and after a while MDS gets very busy and takes forever to
respond to RPC requests.
I read similar messages being a jira request reported here:
https://jira.hpdd.intel.com/browse/LU-3421
https://jira.hpdd.intel.com/browse/LU-6664
Although I did not see any ENOSPC errors on OST's as reported in one of the above
request that is solved. But as of this past week I had to bring down the entire file
system to resolve from long lockups. I use ltop utility to monitor IOPS and I noticed that
LOCKS held by each OST were in the range of 6,000-10,000.
I am a bit lost if this is solved as per LU-3421 in version 2.4.1, then why I am seeing
similar messages in v2.4.3. Could be something else? I can provide logs if required.
Please let me know if you think I need to tag on to the jira request...
Thank you,
Amit
11:41 p.m.
I hope somebody can help me answer this. I see LOCKS in ltop in the numbers 25000 and
system is really slowing down. Any help here is greatly appreciated.
Thank you,
Amit
From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of Kumar, Amit
Sent: Thursday, August 13, 2015 12:26 PM
To: hpdd-discuss(a)lists.01.org
Subject: [HPDD-discuss] (ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on
lock cancel
Dear All,
Lustre v2.4.3, RHEL6.4, files system: ldsikfs
We have seen these errors[(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on
lock cancel] on all OST's and after a while MDS gets very busy and takes forever to
respond to RPC requests.
I read similar messages being a jira request reported here:
https://jira.hpdd.intel.com/browse/LU-3421
https://jira.hpdd.intel.com/browse/LU-6664
Although I did not see any ENOSPC errors on OST's as reported in one of the above
request that is solved. But as of this past week I had to bring down the entire file
system to resolve from long lockups. I use ltop utility to monitor IOPS and I noticed that
LOCKS held by each OST were in the range of 6,000-10,000.
I am a bit lost if this is solved as per LU-3421 in version 2.4.1, then why I am seeing
similar messages in v2.4.3. Could be something else? I can provide logs if required.
Please let me know if you think I need to tag on to the jira request...
Thank you,
Amit
5:30 a.m.
On Aug 13, 2015, at 1:26 PM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
Although I did not see any ENOSPC errors on OST’s as reported in one of the above request
that is solved. But as of this past week I had to bring down the entire file system to
resolve from long lockups. I use ltop utility to monitor IOPS and I noticed that LOCKS
held by each OST were in the range of 6,000-10,000.
How many clients do you have using the file system? If there are quite a few, you might
need to look into limiting the number of locks each client will cache. By default, Lustre
comes up with a number based on the amount of memory on the system, but this can
potentially be a big number for each client. If there are lots of clients, then the locks
on the server side can start using up quite a bit of memory.
I don’t know if this is actually causing the issue you are seeing, but I have had a couple
of cases of crashing (or slow performing) MDS servers caused by large numbers of locks, so
this is usually one of the first things I check. If you want to get an idea of how much
memory is being used by locks, just look at /proc/slabinfo on the server and find any
lines with “ldlm” in them.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
7:42 a.m.
Hi Rick,
I have about 1100 clients total but only about 800 are mounting it for now.
Looking at the /proc/slabinfo on MDS I see large numbers but as I interpret it is it is
not taking up huge memory though.
ldlm_locks 2980551 3301620 576 7 1 : tunables 54 27 8 : slabdata
471660 471660 0
ldlm_resources 2903161 3120192 320 12 1 : tunables 54 27 8 : slabdata
260016 260016 0
Also I see all of my OSS except 2, have used up memory 100%, although I can some to the
conclusion if locks are really using it based on the following numbers:
ldlm_locks 68298 75096 576 7 1 : tunables 54 27 8 : slabdata
10728 10728 0
ldlm_resources 64722 74844 320 12 1 : tunables 54 27 8 : slabdata
6237 6237 0
Also under the cat /proc/fs/lustre/ldlm/namespaces/*/*
I see lru_size = 36000000
While others are all "0" not sure these numbers need to be tuned?
And: # cat /proc/fs/lustre/ldlm/namespaces/scratch-MDT0000-lwp-OST0048/pool/limit
3313022
Are these numbers too high causing it to use high memory?
Thank you, Amit H. Kumar, https://wiki.smu.edu/display/smuhpc/SMU+HPC+Wiki
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, August 14, 2015 2:30 PM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] (ost_handler.c:1764:ost_blocking_ast()) Error -2
syncing data on lock cancel
> On Aug 13, 2015, at 1:26 PM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
>
> Although I did not see any ENOSPC errors on OST’s as reported in one of the
above request that is solved. But as of this past week I had to bring down the
entire file system to resolve from long lockups. I use ltop utility to monitor
IOPS and I noticed that LOCKS held by each OST were in the range of 6,000-
10,000.
How many clients do you have using the file system? If there are quite a few,
you might need to look into limiting the number of locks each client will
cache. By default, Lustre comes up with a number based on the amount of
memory on the system, but this can potentially be a big number for each
client. If there are lots of clients, then the locks on the server side can start
using up quite a bit of memory.
I don’t know if this is actually causing the issue you are seeing, but I have had
a couple of cases of crashing (or slow performing) MDS servers caused by
large numbers of locks, so this is usually one of the first things I check. If you
want to get an idea of how much memory is being used by locks, just look at
/proc/slabinfo on the server and find any lines with “ldlm” in them.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
7:59 a.m.
On Aug 14, 2015, at 5:42 PM, Kumar, Amit <ahkumar(a)mail.smu.edu>
wrote:
Hi Rick,
I have about 1100 clients total but only about 800 are mounting it for now.
Looking at the /proc/slabinfo on MDS I see large numbers but as I interpret it is it is
not taking up huge memory though.
ldlm_locks 2980551 3301620 576 7 1 : tunables 54 27 8 : slabdata
471660 471660 0
ldlm_resources 2903161 3120192 320 12 1 : tunables 54 27 8 : slabdata
260016 260016 0
That looks like it is only a couple of GB’s so it shouldn’t be a problem.
Also I see all of my OSS except 2, have used up memory 100%, although
I can some to the conclusion if locks are really using it based on the following
numbers:
ldlm_locks 68298 75096 576 7 1 : tunables 54 27 8 : slabdata
10728 10728 0
ldlm_resources 64722 74844 320 12 1 : tunables 54 27 8 : slabdata
6237 6237 0
Do you have Lustre read/write caching enabled? That could be consuming the memory. You
can check the values by running
lctl get_param obdfilter.*.read_cache_enable
lctl get_param obdfilter.*.writethrough_cache_enable
If they are enabled, you could try disabling them to see how that affects memory usage
(and if it has any effect on your problem).
Also under the cat /proc/fs/lustre/ldlm/namespaces/*/*
I see lru_size = 36000000
While others are all "0" not sure these numbers need to be tuned?
I think the “0” just means the limit is dynamically determined.
On our clients, we set lru_size=10000 and lru_max_age=172800 which is more than enough.
(The dynamically determined values are crazy big…)
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
3:44 a.m.
Hi Rick,
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Do you have Lustre read/write caching enabled? That could be
consuming
the memory. You can check the values by running
lctl get_param obdfilter.*.read_cache_enable lctl get_param
obdfilter.*.writethrough_cache_enable
If they are enabled, you could try disabling them to see how that affects
memory usage (and if it has any effect on your problem).
I have disabled them both to see the affect. %mem seems to be dropping!! I also read that
setting readcache_max_filesize can control the max file size to cache in memory. I will
probably try this as well when %mem drops to certain number.
> Also under the cat /proc/fs/lustre/ldlm/namespaces/*/*
> I see lru_size = 36000000
> While others are all "0" not sure these numbers need to be tuned?
I think the “0” just means the limit is dynamically determined.
On our clients, we set lru_size=10000 and lru_max_age=172800 which is
more than enough. (The dynamically determined values are crazy big…)
I have tweaked ours as well!!
Another outstanding issue I am having as a consequence of these events is my file creation
and opening times have shot up. When I strace it: I see time spent in ioctl/open is in
couple of minutes while creating files with lfs setstripe, and time spent in
"select/open" call while editing files is in couple of minutes.
Example strace time spent:
23994 open(".newfile_idx69_real.swp", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
<20.590901>
23994 open(".newfile_idx69_real.swp", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW, 0600) =
4 <100.007478>
Any hints on troubleshooting this will be big help....
Thank you,
Amit
9:17 a.m.
Dear All,
I seem to have resolved my issues and recovered from the problems I have been having.
Thank you Rick for pointing out the cache parameters disabling that seems to have helped
the situation.
I have to understand our load a bit more to make sense on how to use
readcache_max_filesize so that some small I/O can benefit from it.
Slowness was caused by error's on OST's recovering them with e2fsck restored all
my problems.
Regards,
Amit
-----Original Message-----
From: HPDD-discuss [mailto:hpdd-discuss-bounces@lists.01.org] On Behalf Of
Kumar, Amit
Sent: Monday, August 17, 2015 12:45 PM
To: Mohr Jr, Richard Frank (Rick Mohr)
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] (ost_handler.c:1764:ost_blocking_ast()) Error -2
syncing data on lock cancel
Hi Rick,
>-----Original Message-----
>From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
>Do you have Lustre read/write caching enabled? That could be consuming
>the memory. You can check the values by running
>
>lctl get_param obdfilter.*.read_cache_enable lctl get_param
>obdfilter.*.writethrough_cache_enable
>
>If they are enabled, you could try disabling them to see how that
>affects memory usage (and if it has any effect on your problem).
>
I have disabled them both to see the affect. %mem seems to be dropping!! I
also read that setting readcache_max_filesize can control the max file size to
cache in memory. I will probably try this as well when %mem drops to certain
number.
>> Also under the cat /proc/fs/lustre/ldlm/namespaces/*/*
>> I see lru_size = 36000000
>> While others are all "0" not sure these numbers need to be tuned?
>
>I think the “0” just means the limit is dynamically determined.
>
>On our clients, we set lru_size=10000 and lru_max_age=172800 which is
>more than enough. (The dynamically determined values are crazy big…)
>
I have tweaked ours as well!!
Another outstanding issue I am having as a consequence of these events is my
file creation and opening times have shot up. When I strace it: I see time spent
in ioctl/open is in couple of minutes while creating files with lfs setstripe, and
time spent in "select/open" call while editing files is in couple of minutes.
Example strace time spent:
23994 open(".newfile_idx69_real.swp", O_RDWR|O_CREAT|O_EXCL, 0600) =
4 <20.590901>
23994 open(".newfile_idx69_real.swp",
O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW, 0600) = 4 <100.007478>
Any hints on troubleshooting this will be big help....
Thank you,
Amit
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2589
days inactive
2593
days old
6 comments
2 participants
participants (2)
-
Kumar, Amit -
Mohr Jr, Richard Frank (Rick Mohr)