4:22 a.m.
All,
I'll start with the question:
Is there a Lustre config parameter that throttles the amount of system
memory that a given OSC can use when reading a file?
I am running an iozone job, single process, single thread. Writes an
80GB file twice, reads forwards twice, reads backwards twice. This is
run with varying stripe count values ( 2,3,4,5,6,7,8,12 and 16 ) with
1MB stripe. While each job is running I monitor the "Cached" value in
/proc/meminfo.
That "Cached" value is plotted against time for the duration of the
runs. The jobs are all run independently.
The client node has 128 GB of memory, so the file should easily fit in
the system buffers. The filesystem has 16 OSTs.
As indicated in the following plot, the maximum value for "Cached" when
stripe_count=2 is 17GB
stripe_count=3 is 25GB
stripe_count=4 is 33GB
stripe_count=5 is 42GB
stripe_count=6 is 50GB
stripe_count=7 is 59GB
stripe_count=8,12,16 are all 63GB
What is limiting the system buffer utilization? I have been grep'ing
around in the Lustre documentation and don't find many references to
system cache or buffer cache.
And none that explain this.
Just for completeness, here is a plot of how the file is being
written,read forward,read backward where it is easy to see when the data
is coming from the system buffers.
This is for OST=3
Thanks
John
5:22 a.m.
On Feb 3, 2015, at 12:22 PM, John Bauer <bauerj(a)iodoctors.com>
wrote:
Is there a Lustre config parameter that throttles the amount of system memory that a
given OSC can use when reading a file?
...
The client node has 128 GB of memory, so the file should easily fit in the system
buffers. The filesystem has 16 OSTs.
As indicated in the following plot, the maximum value for "Cached" when
stripe_count=2 is 17GB
stripe_count=3 is 25GB
stripe_count=4 is 33GB
stripe_count=5 is 42GB
stripe_count=6 is 50GB
stripe_count=7 is 59GB
stripe_count=8,12,16 are all 63GB
The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
There is also /proc/fs/lustre/osc/<object_name>/osc_cached_mb. From what I can
tell, you can read that file to see how much data is being cached. I have never tried
writing to that file to see if you can force a certain max cache for the osc, but a quick
look at the Lustre source seems to indicate that it can be (and it looks like it may be
tied to the LRU settings).
The file /proc/fs/lustre/osc/<object_name>/max_dirty_mb can be used to limit the
amount of dirty write data that can be cached, but I don’t think that has any effect on
the amount of read data cached.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
7:26 a.m.
Hello!
On Feb 3, 2015, at 1:22 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Feb 3, 2015, at 12:22 PM, John Bauer <bauerj(a)iodoctors.com> wrote:
> Is there a Lustre config parameter that throttles the amount of system memory that a
given OSC can use when reading a file?
...
>
> The client node has 128 GB of memory, so the file should easily fit in the system
buffers. The filesystem has 16 OSTs.
>
> As indicated in the following plot, the maximum value for "Cached" when
> stripe_count=2 is 17GB
> stripe_count=3 is 25GB
> stripe_count=4 is 33GB
> stripe_count=5 is 42GB
> stripe_count=6 is 50GB
> stripe_count=7 is 59GB
> stripe_count=8,12,16 are all 63GB
>
The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop that down to 1/2
indeed, because that shown some performance benefit.
Jinshan probably can elaborate on that in more details.
Bye,
Oleg
7:34 a.m.
On Feb 3, 2015, at 3:26 PM, Drokin, Oleg
<oleg.drokin(a)intel.com> wrote:
>
> The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop that down to 1/2
indeed, because that shown some performance benefit.
Jinshan probably can elaborate on that in more details.
Hmm. My Cray client is running 2.5.1 and has max_cached_mb set to 1/2 RAM. But I have
another client running 2.5.3 which sets the parameter to 3/4 of the RAM. Maybe Cray
patched their version to change the default value?
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
7:41 a.m.
On Feb 3, 2015, at 3:34 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Feb 3, 2015, at 3:26 PM, Drokin, Oleg <oleg.drokin(a)intel.com> wrote:
>
>>
>> The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
>
> Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop that down to 1/2
indeed, because that shown some performance benefit.
> Jinshan probably can elaborate on that in more details.
Hmm. My Cray client is running 2.5.1 and has max_cached_mb set to 1/2 RAM. But I have
another client running 2.5.3 which sets the parameter to 3/4 of the RAM. Maybe Cray
patched their version to change the default value?
I tracked the change to http://review.whamcloud.com/2514 which was landed in Oct 2012, but
in it's original form if you had less than certain amount of RAM, then it defaulted to
3/4.
Is your 2.5.3 client have relatively little RAM by any chance?
Bye,
Oleg
7:51 a.m.
Richard, I am running on at Cray system, at Cray. Version 2.5
John
Sent from my iPhone
On Feb 3, 2015, at 2:34 PM, "Mohr Jr, Richard Frank (Rick
Mohr)" <rmohr(a)utk.edu> wrote:
>> On Feb 3, 2015, at 3:26 PM, Drokin, Oleg <oleg.drokin(a)intel.com> wrote:
>>
>>
>> The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
>
> Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop that down to 1/2
indeed, because that shown some performance benefit.
> Jinshan probably can elaborate on that in more details.
Hmm. My Cray client is running 2.5.1 and has max_cached_mb set to 1/2 RAM. But I have
another client running 2.5.3 which sets the parameter to 3/4 of the RAM. Maybe Cray
patched their version to change the default value?
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
7:51 a.m.
Rick,
I can confirm we did.
Oleg, that change was made upstream in master here, as part of LU-3321:
http://review.whamcloud.com/7890
Which went in to 2.6.
- Patrick Farrell
On 02/03/2015 02:34 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Feb 3, 2015, at 3:26 PM, Drokin, Oleg
<oleg.drokin(a)intel.com> wrote:
>
>> The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
> Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop that down to 1/2
indeed, because that shown some performance benefit.
> Jinshan probably can elaborate on that in more details.
Hmm. My Cray client is running 2.5.1 and has max_cached_mb set to 1/2 RAM. But I have
another client running 2.5.3 which sets the parameter to 3/4 of the RAM. Maybe Cray
patched their version to change the default value?
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
7:58 a.m.
Oleg -
Ah, looking at your most recent note, I see the change I highlighted
made it non-conditional. It was conditionally true since the change you
noted.
The nodes John is testing on have a very large amount of memory, so they
should be at 1/2.
Interesting that Rick's seeing 3/4 on his system. The limit looks to be
< 512MB, if I'm reading correctly.
- Patrick
On 02/03/2015 02:51 PM, Patrick Farrell wrote:
Rick,
I can confirm we did.
Oleg, that change was made upstream in master here, as part of LU-3321:
http://review.whamcloud.com/7890
Which went in to 2.6.
- Patrick Farrell
On 02/03/2015 02:34 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
>> On Feb 3, 2015, at 3:26 PM, Drokin, Oleg <oleg.drokin(a)intel.com> wrote:
>>
>>> The total amount of cache used by lustre on a client should be
>>> controlled by /proc/fs/lustre/llite/<fsname>/max_cached_mb.
>>> According to the manual, this defaults to 3/4 of the client RAM.
>>> However, on my Cray system, it looks like it is defaulting to 1/2
>>> the RAM (not sure why). From your tests, it looks like the lustre
>>> cache is topping out around 1/2 the RAM. Are you running on a Cray
>>> client by any chance?
>> Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop
>> that down to 1/2 indeed, because that shown some performance benefit.
>> Jinshan probably can elaborate on that in more details.
> Hmm. My Cray client is running 2.5.1 and has max_cached_mb set to
> 1/2 RAM. But I have another client running 2.5.3 which sets the
> parameter to 3/4 of the RAM. Maybe Cray patched their version to
> change the default value?
>
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
8:18 a.m.
On Feb 3, 2015, at 3:41 PM, Drokin, Oleg
<oleg.drokin(a)intel.com> wrote:
I tracked the change to http://review.whamcloud.com/2514 which was landed in Oct 2012,
but in it's original form if you had less than certain amount of RAM, then it
defaulted to 3/4.
Is your 2.5.3 client have relatively little RAM by any chance?
Here are the stats from a few different client systems.
Client #1 (Linux cluster login node):
- Lustre 2.5.3
- 64 GB RAM
- max_cached_mb = 48 GB
Client #2 (External Cray login node):
- Lustre 2.5.1
- 256 GB RAM
- max_cached_mb = 128 GB
Client #3 (Internal Cray login node):
- Lustre 2.5.1
- 64 GB RAM
- max_cached_mb = 32 GB
Notice that client #1 and #3 have the same amount of RAM but different defaults for
max_cached_mb. (Not that it’s a problem. I just never noticed the difference before I
took a closer look today.)
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
9:47 a.m.
On Feb 3, 2015, at 3:58 PM, Patrick Farrell <paf(a)cray.com>
wrote:
Interesting that Rick's seeing 3/4 on his system. The limit looks to be < 512MB,
if I'm reading correctly.
I glanced at the Lustre source for my 2.5.3 client and found this:
pages = si.totalram - si.totalhigh;
if (pages >> (20 - PAGE_CACHE_SHIFT) < 512) {
lru_page_max = pages / 2;
} else {
lru_page_max = (pages / 4) * 3;
}
The way I am reading this is that if the system has <512MB of memory, the lru_page_max
is 1/2 the system RAM. Otherwise, it will be 3/4 of the system RAM.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
12:46 a.m.
Richard, Patrick
The more I look at this, the more bizarre it gets. Now when I run this
iozone test, I also track *cached_mb* for each OSC. This plot has the
file position activity plot overlayed
with the value of cached_mb for the 16 OST's that the file was stripped
across. Things are predictable until the size of the file being written
exceeds the amount of memory that
Lustre can use for caching ( during the first write). After that, the
competition for buffer memory by the OSC's starts. Further comment on
the plot image.
I still have not determined why *cached_mb* for any OSC never exceeds
8GB in the test cases where I stripe across 2,3,4,5,6,or 7 OSTs. In
those cases the sum of the cached_mb for
the used OSTs never exceeds the 50% of system memory limit.
John
On 2/3/2015 4:47 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Feb 3, 2015, at 3:58 PM, Patrick Farrell <paf(a)cray.com>
wrote:
>
> Interesting that Rick's seeing 3/4 on his system. The limit looks to be <
512MB, if I'm reading correctly.
I glanced at the Lustre source for my 2.5.3 client and found this:
pages = si.totalram - si.totalhigh;
if (pages >> (20 - PAGE_CACHE_SHIFT) < 512) {
lru_page_max = pages / 2;
} else {
lru_page_max = (pages / 4) * 3;
}
The way I am reading this is that if the system has <512MB of memory, the lru_page_max
is 1/2 the system RAM. Otherwise, it will be 3/4 of the system RAM.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:27 a.m.
John,
I don't have anything to add at the moment, but I am watching your
explorations with interest. Thanks for sharing this.
One question - The blue and red lines coming up the graph... What are
those? (Particularly, the one which peaks and then heads back down?)
- Patrick
On 02/05/2015 07:46 AM, John Bauer wrote:
Richard, Patrick
The more I look at this, the more bizarre it gets. Now when I run
this iozone test, I also track *cached_mb* for each OSC. This plot
has the file position activity plot overlayed
with the value of cached_mb for the 16 OST's that the file was
stripped across. Things are predictable until the size of the file
being written exceeds the amount of memory that
Lustre can use for caching ( during the first write). After that, the
competition for buffer memory by the OSC's starts. Further comment on
the plot image.
I still have not determined why *cached_mb* for any OSC never exceeds
8GB in the test cases where I stripe across 2,3,4,5,6,or 7 OSTs. In
those cases the sum of the cached_mb for
the used OSTs never exceeds the 50% of system memory limit.
John
On 2/3/2015 4:47 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
>> On Feb 3, 2015, at 3:58 PM, Patrick Farrell<paf(a)cray.com> wrote:
>>
>> Interesting that Rick's seeing 3/4 on his system. The limit looks to be <
512MB, if I'm reading correctly.
> I glanced at the Lustre source for my 2.5.3 client and found this:
>
> pages = si.totalram - si.totalhigh;
> if (pages >> (20 - PAGE_CACHE_SHIFT) < 512) {
> lru_page_max = pages / 2;
> } else {
> lru_page_max = (pages / 4) * 3;
> }
>
> The way I am reading this is that if the system has <512MB of memory, the
lru_page_max is 1/2 the system RAM. Otherwise, it will be 3/4 of the system RAM.
>
> --
> Rick Mohr
> Senior HPC System Administrator
> National Institute for Computational Sciences
> http://www.nics.tennessee.edu
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
5:55 a.m.
Patrick
Here are a few images that should clarify what the blue ( writes ) and
red ( reads ) lines are all about. When plotting the millions of these
for the entire iozone
run, each read and write plotted will end up on one pixel. But now you
get the aggregate view of the application's I/O pattern. The slopes of
the aggregate lines
give an indication of data delivery rate of the file system. When reads
are coming out of the system cache, the slope will be very steep, as
opposed to the shallow
slope when having to go to the OST.
With this info, now go back and look at the image in the previous email
with the file position activity overlayed with the OSC cached. Now you
can see how the
OSCs are responding to the iozone's writes and reads.
John
On 2/5/2015 11:27 AM, Patrick Farrell wrote:
John,
I don't have anything to add at the moment, but I am watching your
explorations with interest. Thanks for sharing this.
One question - The blue and red lines coming up the graph... What are
those? (Particularly, the one which peaks and then heads back down?)
- Patrick
On 02/05/2015 07:46 AM, John Bauer wrote:
> Richard, Patrick
>
> The more I look at this, the more bizarre it gets. Now when I run
> this iozone test, I also track *cached_mb* for each OSC. This plot
> has the file position activity plot overlayed
> with the value of cached_mb for the 16 OST's that the file was
> stripped across. Things are predictable until the size of the file
> being written exceeds the amount of memory that
> Lustre can use for caching ( during the first write). After that,
> the competition for buffer memory by the OSC's starts. Further
> comment on the plot image.
>
>
>
>
>
> I still have not determined why *cached_mb* for any OSC never exceeds
> 8GB in the test cases where I stripe across 2,3,4,5,6,or 7 OSTs. In
> those cases the sum of the cached_mb for
> the used OSTs never exceeds the 50% of system memory limit.
>
> John
>
>
> On 2/3/2015 4:47 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
>>> On Feb 3, 2015, at 3:58 PM, Patrick Farrell<paf(a)cray.com> wrote:
>>>
>>> Interesting that Rick's seeing 3/4 on his system. The limit looks to be
< 512MB, if I'm reading correctly.
>> I glanced at the Lustre source for my 2.5.3 client and found this:
>>
>> pages = si.totalram - si.totalhigh;
>> if (pages >> (20 - PAGE_CACHE_SHIFT) < 512) {
>> lru_page_max = pages / 2;
>> } else {
>> lru_page_max = (pages / 4) * 3;
>> }
>>
>> The way I am reading this is that if the system has <512MB of memory, the
lru_page_max is 1/2 the system RAM. Otherwise, it will be 3/4 of the system RAM.
>>
>> --
>> Rick Mohr
>> Senior HPC System Administrator
>> National Institute for Computational Sciences
>> http://www.nics.tennessee.edu
>>
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
7:39 a.m.
John,
I’m afraid that I am at a loss as well. I am not well enough versed with the Lustre
client caching code to be able to explain what is happening. I did a little digging in
the source code and was not able to identify where the 8GB limit was coming from either.
The only other thing I can think of to look at would be to map each OSC to its
corresponding OSS server to see if there is any correlation between the OSC’s with low
cache and a particular OSS server.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
On Feb 5, 2015, at 8:46 AM, John Bauer <bauerj(a)iodoctors.com>
wrote:
Richard, Patrick
The more I look at this, the more bizarre it gets. Now when I run this iozone test, I
also track cached_mb for each OSC. This plot has the file position activity plot
overlayed
with the value of cached_mb for the 16 OST's that the file was stripped across.
Things are predictable until the size of the file being written exceeds the amount of
memory that
Lustre can use for caching ( during the first write). After that, the competition for
buffer memory by the OSC's starts. Further comment on the plot image.
<cached.png>
I still have not determined why cached_mb for any OSC never exceeds 8GB in the test cases
where I stripe across 2,3,4,5,6,or 7 OSTs. In those cases the sum of the cached_mb for
the used OSTs never exceeds the 50% of system memory limit.
John
2781
days inactive
2784
days old
13 comments
4 participants
participants (4)
-
Drokin, Oleg -
John Bauer -
Mohr Jr, Richard Frank (Rick Mohr) -
Patrick Farrell