12:59 a.m.
Andreas
Thank you for the reply. This investigation started with the
observation of slow backwards reads of file by an MSC.Nastran run doing
a Lanczos eigenvalue solve ( see image below ). I point that out so it
is known that I am not investigating an academic run of iozone.
It is far simpiler to work with iozone than MSC.Nastran.
If you care to read a bit more to see the observed behavior of Lustre,
please read on.
The following image depicts the access of the file over time, by the
iozone run. What is quite odd is that when the second backward read of
the file begins,
the reading of the file is at its fastest(steep slope) during this
backwards read. This is at at time when all of the end of the file
should have been LRU'd out of the system buffers by the previous
backwards read. The rate then slows down through the meat of the file
and then starts getting faster again toward the end of the second
backwards read.
I have run this job many times and the behavior, as depicted in the
first image, is always the same. The slopes vary some, but there is
always this
serpentine look to it. It is not the same OST's every time. If I run
this with iozone using 256K requests, the slopes for the backwards reads
gets much lower.
To me, it seems at though something is wrong with the LRU mechanism.
Note in the last image, when iozone is using 256k requests, that this
behavior starts during the
forward reads of the file. It is not just a backward read phenomenon.
It happens every time when reading backwards. Only occasionally during
the forward reads.
John
On 1/18/2015 10:29 PM, Dilger, Andreas wrote:
On Jan 18, 2015, at 17:19, John Bauer
<bauerj@iodoctors.com<mailto:bauerj@iodoctors.com>> wrote:
I have been observing what I would think is unexpected behavior. I will try to keep this
short, and start with the question.
Should it be expected, when sequentially reading a striped file multiple times, that the
data from some OST's remains in the system cache
while others does not?
This isn't something that I'm aware of myself, nor something I'd necessarily
expect. That said, this isn't actually a bad thing.
File is 80GB is size.
System has 64GB of memory.
File is striped 16 way, 1MB stripe size. Application is iozone.
File is written forwards twice, then read forwards twice, then read backwards twice.
There is 80GB / 16 stripes = 5GB of data per stripe. If the pages were handled in strict
LRU order, then one would expect the two forward reads to blow out the cache, and result
in 10GB of data read per stripe. Then, the first backwards read would access most of the
data from cache, maybe 60GB taking into account the OS, so 80GB - 60GB = 20GB read on the
first pass (1.25GB/stripe), and another full 5GB for the second backward read. That gives
16.25GB/stripe in the expected LRU case.
That you got 16-17GB read on many OSCs is expected. For the OSCs that had less read i
checked that the cached reads sum(16Gb - actual read) = 45GB or so, so it doesn't
exceed the amount that could have been cached.
I don't know why this might have happened, but there could be several causes. If one
of the LDLM locks was cancelled due to memory pressure, it would have allowed some data to
stay in cache for the first backward read, and by being accessed more than once it
wouldn't fall off the LRU for the second backward read.
Cheers, Andreas
Application request size is 1MB.
Run on the swan cluster at Cray, Inc.
lustre-cray_ari_s/2.5_3.0.101_0.31.1_1.0502.8394.10.1-1.0502.17198.8.51
The file is large enough to oversubscribe the system's memory. I would expect that
each OST would see uniform activity.
But that is far from the case. Here is the amount of data read by each OST during the
entire iozone job, ranges from 10G to 17G.
<afcjgffc.png>
When I look at how much data the OST's have read versus time, some have no activity
during the entire 2nd backwards read.
The OST's that have the low amount of data read also have very high application data
delivery rates during these same periods, indicating the data is in the system cache.
Is this to be expected?
Thanks
John
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
