All,
I'll start with the question:
Is there a Lustre config parameter that throttles the amount of system
memory that a given OSC can use when reading a file?
I am running an iozone job, single process, single thread. Writes an
80GB file twice, reads forwards twice, reads backwards twice. This is
run with varying stripe count values ( 2,3,4,5,6,7,8,12 and 16 ) with
1MB stripe. While each job is running I monitor the "Cached" value in
/proc/meminfo.
That "Cached" value is plotted against time for the duration of the
runs. The jobs are all run independently.
The client node has 128 GB of memory, so the file should easily fit in
the system buffers. The filesystem has 16 OSTs.
As indicated in the following plot, the maximum value for "Cached" when
stripe_count=2 is 17GB
stripe_count=3 is 25GB
stripe_count=4 is 33GB
stripe_count=5 is 42GB
stripe_count=6 is 50GB
stripe_count=7 is 59GB
stripe_count=8,12,16 are all 63GB
What is limiting the system buffer utilization? I have been grep'ing
around in the Lustre documentation and don't find many references to
system cache or buffer cache.
And none that explain this.
Just for completeness, here is a plot of how the file is being
written,read forward,read backward where it is easy to see when the data
is coming from the system buffers.
This is for OST=3
Thanks
John