All,
I'll start with the question:
Is there a Lustre config parameter that throttles the amount of
system memory that a given OSC can use when reading a file?
I am running an iozone job, single process, single thread. Writes an
80GB file twice, reads forwards twice, reads backwards twice. This
is
run with varying stripe count values ( 2,3,4,5,6,7,8,12 and 16 )
with 1MB stripe. While each job is running I monitor the "Cached"
value in /proc/meminfo.
That "Cached" value is plotted against time for the duration of the
runs. The jobs are all run independently.
The client node has 128 GB of memory, so the file should easily fit
in the system buffers. The filesystem has 16 OSTs.
As indicated in the following plot, the maximum value for "Cached"
when
stripe_count=2 is 17GB
stripe_count=3 is 25GB
stripe_count=4 is 33GB
stripe_count=5 is 42GB
stripe_count=6 is 50GB
stripe_count=7 is 59GB
stripe_count=8,12,16 are all 63GB
What is limiting the system buffer utilization? I have been
grep'ing around in the Lustre documentation and don't find many
references to system cache or buffer cache.
And none that explain this.

Just for completeness, here is a plot of how the file is being
written,read forward,read backward where it is easy to see when the
data is coming from the system buffers.
This is for OST=3

Thanks
John