1:11 p.m.
Jeremy,
thanks for pointing this out. I believe the readahead code currently has support for
linear and strided readahead, but as you write it does NOT have support for backward
readahead (strided or linear). It may or may not be easy to add this, it hasn't come
up as a use case for applications in the past.
I recall when doing code inspections on the readahead code that it deliberately avoids the
case where the readahead stride comes out negative, so it _may_ just be a case of allowing
the stride to be negative. I'd encourage someone to take a look into this. Probably
lustre/llite/vvp_io.c::ll_readahead() is the right place to start.
It may also be worthwhile to look at the "lctl get_param
llite.*.read_ahead_stats" to see what is going on there.
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
On 2015/01/19, 9:55 AM, "Jeremy Filizetti"
<jeremy.filizetti@gmail.com<mailto:jeremy.filizetti@gmail.com>> wrote:
I can't answer your question about the LRU but I don't believe Lustre has
readahead/prefetching support for backwards reads so you're probably doing 1 page per
RPC which will be fairly slow. You can look at your RPC stats to see what size RPCs are
being issued on the client:
lctl get_param osc.*.rpc_stats
To reset them before a test:
lctl set_param osc.*.rpc_stats=0
Jeremy
On Mon, Jan 19, 2015 at 8:59 AM, John Bauer
<bauerj@iodoctors.com<mailto:bauerj@iodoctors.com>> wrote:
Andreas
Thank you for the reply. This investigation started with the observation of slow
backwards reads of file by an MSC.Nastran run doing
a Lanczos eigenvalue solve ( see image below ). I point that out so it is known that I am
not investigating an academic run of iozone.
It is far simpiler to work with iozone than MSC.Nastran.
If you care to read a bit more to see the observed behavior of Lustre, please read on.
The following image depicts the access of the file over time, by the iozone run. What is
quite odd is that when the second backward read of the file begins,
the reading of the file is at its fastest(steep slope) during this backwards read. This
is at at time when all of the end of the file should have been LRU'd out of the system
buffers by the previous backwards read. The rate then slows down through the meat of the
file and then starts getting faster again toward the end of the second backwards read.
I have run this job many times and the behavior, as depicted in the first image, is always
the same. The slopes vary some, but there is always this
serpentine look to it. It is not the same OST's every time. If I run this with
iozone using 256K requests, the slopes for the backwards reads gets much lower.
To me, it seems at though something is wrong with the LRU mechanism. Note in the last
image, when iozone is using 256k requests, that this behavior starts during the
forward reads of the file. It is not just a backward read phenomenon. It happens every
time when reading backwards. Only occasionally during the forward reads.
John