3:29 p.m.
On Jan 18, 2015, at 17:19, John Bauer
<bauerj@iodoctors.com<mailto:bauerj@iodoctors.com>> wrote:
I have been observing what I would think is unexpected behavior. I will try to keep this
short, and start with the question.
Should it be expected, when sequentially reading a striped file multiple times, that the
data from some OST's remains in the system cache
while others does not?
This isn't something that I'm aware of myself, nor something I'd necessarily
expect. That said, this isn't actually a bad thing.
File is 80GB is size.
System has 64GB of memory.
File is striped 16 way, 1MB stripe size. Application is iozone.
File is written forwards twice, then read forwards twice, then read backwards twice.
There is 80GB / 16 stripes = 5GB of data per stripe. If the pages were handled in strict
LRU order, then one would expect the two forward reads to blow out the cache, and result
in 10GB of data read per stripe. Then, the first backwards read would access most of the
data from cache, maybe 60GB taking into account the OS, so 80GB - 60GB = 20GB read on the
first pass (1.25GB/stripe), and another full 5GB for the second backward read. That gives
16.25GB/stripe in the expected LRU case.
That you got 16-17GB read on many OSCs is expected. For the OSCs that had less read i
checked that the cached reads sum(16Gb - actual read) = 45GB or so, so it doesn't
exceed the amount that could have been cached.
I don't know why this might have happened, but there could be several causes. If one
of the LDLM locks was cancelled due to memory pressure, it would have allowed some data to
stay in cache for the first backward read, and by being accessed more than once it
wouldn't fall off the LRU for the second backward read.
Cheers, Andreas
Application request size is 1MB.
Run on the swan cluster at Cray, Inc.
lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51
The file is large enough to oversubscribe the system's memory. I would expect that
each OST would see uniform activity.
But that is far from the case. Here is the amount of data read by each OST during the
entire iozone job, ranges from 10G to 17G.
<afcjgffc.png>
When I look at how much data the OST's have read versus time, some have no activity
during the entire 2nd backwards read.
The OST's that have the low amount of data read also have very high application data
delivery rates during these same periods, indicating the data is in the system cache.
Is this to be expected?
Thanks
John
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
