John,

I don't have anything to add at the moment, but I am watching your explorations with interest.  Thanks for sharing this.

One question - The blue and red lines coming up the graph...  What are those?  (Particularly, the one which peaks and then heads back down?)

- Patrick

On 02/05/2015 07:46 AM, John Bauer wrote:
Richard, Patrick

The more I look at this, the more bizarre it gets.  Now when I run this iozone test, I also track cached_mb for each OSC.  This plot has the file position activity plot overlayed
with the value of cached_mb for the 16 OST's that the file was stripped across.  Things are predictable until the size of the file being written exceeds the amount of memory that
Lustre can use for caching ( during the first write).  After that, the competition for buffer memory by the OSC's starts.  Further comment on the plot image.





I still have not determined why cached_mb for any OSC never exceeds 8GB in the test cases where I stripe across 2,3,4,5,6,or 7 OSTs.  In those cases the sum of the cached_mb for
the used OSTs never exceeds the 50% of system memory limit.

John


On 2/3/2015 4:47 PM, Mohr Jr, Richard Frank (Rick Mohr) wrote:
On Feb 3, 2015, at 3:58 PM, Patrick Farrell <paf@cray.com> wrote:

Interesting that Rick's seeing 3/4 on his system.  The limit looks to be < 512MB, if I'm reading correctly.

I glanced at the Lustre source for my 2.5.3 client and found this:

pages = si.totalram - si.totalhigh;
if (pages >> (20 - PAGE_CACHE_SHIFT) < 512) {
                lru_page_max = pages / 2;
 } else {
                lru_page_max = (pages / 4) * 3;
 }

The way I am reading this is that if the system has <512MB of memory, the lru_page_max is 1/2 the system RAM.  Otherwise, it will be 3/4 of the system RAM.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss