Re: [HPDD-discuss] OSC and buffer cache throttling
Hello!
On Feb 3, 2015, at 1:22 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Feb 3, 2015, at 12:22 PM, John Bauer <bauerj(a)iodoctors.com> wrote:
> Is there a Lustre config parameter that throttles the amount of system memory that a
given OSC can use when reading a file?
...
>
> The client node has 128 GB of memory, so the file should easily fit in the system
buffers. The filesystem has 16 OSTs.
>
> As indicated in the following plot, the maximum value for "Cached" when
> stripe_count=2 is 17GB
> stripe_count=3 is 25GB
> stripe_count=4 is 33GB
> stripe_count=5 is 42GB
> stripe_count=6 is 50GB
> stripe_count=7 is 59GB
> stripe_count=8,12,16 are all 63GB
>
The total amount of cache used by lustre on a client should be controlled by
/proc/fs/lustre/llite/<fsname>/max_cached_mb. According to the manual, this
defaults to 3/4 of the client RAM. However, on my Cray system, it looks like it is
defaulting to 1/2 the RAM (not sure why). From your tests, it looks like the lustre cache
is topping out around 1/2 the RAM. Are you running on a Cray client by any chance?
Recent lustre versions (2.4.0 may be? or was that 2.5.0?) did drop that down to 1/2
indeed, because that shown some performance benefit.
Jinshan probably can elaborate on that in more details.
Bye,
Oleg