Richard, Patrick
The more I look at this, the more bizarre it gets. Now when I run this
iozone test, I also track *cached_mb* for each OSC. This plot has the
file position activity plot overlayed
with the value of cached_mb for the 16 OST's that the file was stripped
across. Things are predictable until the size of the file being written
exceeds the amount of memory that
Lustre can use for caching ( during the first write). After that, the
competition for buffer memory by the OSC's starts. Further comment on
the plot image.
I still have not determined why *cached_mb* for any OSC never exceeds
8GB in the test cases where I stripe across 2,3,4,5,6,or 7 OSTs. In
those cases the sum of the cached_mb for
the used OSTs never exceeds the 50% of system memory limit.
John
On 2/3/2015 4:47 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Feb 3, 2015, at 3:58 PM, Patrick Farrell <paf(a)cray.com>
wrote:
>
> Interesting that Rick's seeing 3/4 on his system. The limit looks to be <
512MB, if I'm reading correctly.
I glanced at the Lustre source for my 2.5.3 client and found this:
pages = si.totalram - si.totalhigh;
if (pages >> (20 - PAGE_CACHE_SHIFT) < 512) {
lru_page_max = pages / 2;
} else {
lru_page_max = (pages / 4) * 3;
}
The way I am reading this is that if the system has <512MB of memory, the lru_page_max
is 1/2 the system RAM. Otherwise, it will be 3/4 of the system RAM.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss