There's no concept or expectation of staleness in Lustre. It's likely something else is going on. What do the actual files look like? ________________________________________ From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Kumar, Amit [ahkumar(a)mail.smu.edu] Sent: Friday, September 04, 2015 10:32 AM To: hpdd-discuss(a)lists.01.org Subject: [HPDD-discuss] Stale File is it possible? Dear All, We are running lot of Gaussian(I am sure everyone knows this but for clarity, it is a computational chemistry package) jobs that run for month or 2 and we have noticed that these jobs write to open files today and then next time it would write to this file will be in 1-2-3 weeks, on our diskless cluster-compute nodes. Theoretically this should not be a problem. But since we are running into a problem where Gaussian is complaining about: Erroneous read on file. I am thinking could the luster client/server evict any of information preserved after sometime because on no activity? Is there a timeout setting on the client that I could tweak to make the application resilient to getting access to the files and not behave as if it were stale. Just trying to understand any help here is greatly appreciated. Best regards, Amit

-----Original Message----- From: Martin Hecht [mailto:hecht@hlrs.de] Sent: Friday, September 04, 2015 11:15 AM To: Patrick Farrell; Kumar, Amit; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? maybe a flaky network connection, or the network over which lustre is mounted gets flooded by Gaussians own communication? Are there any lustre or lnet messages in dmesg and syslog which could be related to such an event? On 09/04/2015 06:08 PM, Patrick Farrell wrote: > There's no concept or expectation of staleness in Lustre. It's likely something else is going on. > > What do the actual files look like? > ________________________________________ > From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Kumar, Amit [ahkumar(a)mail.smu.edu] > Sent: Friday, September 04, 2015 10:32 AM > To: hpdd-discuss(a)lists.01.org > Subject: [HPDD-discuss] Stale File is it possible? > > Dear All, > > We are running lot of Gaussian(I am sure everyone knows this but for clarity, it is a computational chemistry package) jobs that run for month or 2 and we have noticed that these jobs write to open files today and then next time it would write to this file will be in 1-2-3 weeks, on our diskless cluster- compute nodes. > Theoretically this should not be a problem. But since we are running into a problem where Gaussian is complaining about: Erroneous read on file. I am thinking could the luster client/server evict any of information preserved after sometime because on no activity? Is there a timeout setting on the client that I could tweak to make the application resilient to getting access to the files and not behave as if it were stale. Just trying to understand any help here is greatly appreciated. > > Best regards, > Amit >

Sent this once already, but in reply to the wrong message... Martin might know about that short read thing, since his site has a nice wiki page on it: https://wickie.hlrs.de/platforms/index.php/Lustre_short_read

Technically Lustre is allowed to return fewer bytes than requested, as it says on that page. But it doesn't normally - LU-6389 is a bug where that can happen kind of often. (Again, it's technically allowed as that page says... But it shouldn't really happen in practice, which is why LU-6389 is a bug.) So perhaps Gaussian does not retry short reads? If memory serves, it's closed source, so you can't check - but perhaps you could ask the vendor?

________________________________________ From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Patrick Farrell [paf(a)cray.com] Sent: Friday, September 04, 2015 4:03 PM To: Kumar, Amit; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? What's your Lustre version? If the file looks OK when you look at it yourself (ie, no gaps), you might be running in to this bug: https://jira.hpdd.intel.com/browse/LU-6389 Lustre 2.5 and newer will sometimes return fewer than expected bytes on a read or write, without giving an error. - Patrick ________________________________________ From: Kumar, Amit [ahkumar(a)mail.smu.edu] Sent: Friday, September 04, 2015 2:51 PM To: Martin Hecht; Patrick Farrell; hpdd-discuss(a)lists.01.org Subject: RE: [HPDD-discuss] Stale File is it possible? Patrick & Matin, Thank you for responses. Files are Gau-*.rwf files, they are data files created by Gaussian and a bunch of text or log files. Gaussian does what they call it as logically random-access. Error that we run into often is: Erroneous read. Read 1 instead of 8 I don't seem to find any lustre/lnet errors in dmesg or syslogs. Although I have seen " Error -2 syncing data on lock cancel " here and there but they tend to be also on other clients so I don't think that is of significance. We are mounting Lustre over IB, not sure if that has any significance. I am looking at runing another test with additional debugging enabled, if you have any hints here that would be helpful. Best Regards, Amit

Martin, Interesting find on the quota stuff. About file locking... Flock (the mount option I think you're talking about) refers specifically to POSIX file locks. It does not change the normal Lustre locking, it only controls POSIX file locking, which is voluntary in any case (in the sense that an application can request and wait for such a lock if it is held elsewhere, but it can also or read write without asking for the lock). - Patrick ________________________________________ From: Martin Hecht [hecht(a)hlrs.de] Sent: Monday, September 07, 2015 4:46 AM To: Patrick Farrell; Kumar, Amit; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? Hi Amit and Patrick, On 09/04/2015 11:09 PM, Patrick Farrell wrote: > Sent this once already, but in reply to the wrong message... > > Martin might know about that short read thing, since his site has a nice wiki page on it: > https://wickie.hlrs.de/platforms/index.php/Lustre_short_read sure. At first I didn't relate this question to short reads, but "Erroneous read. Read 1 instead of 8" sounds a bit like short reads. On the other hand, are these numbers in bytes? If that's the case I would doubt it's a short read of a single byte. I would expect that at least the block size is read in one IOP (unless the application really reads each byte separately). At biowulf.nih.gov/apps/gaussian/#errors I have found the "translation to English": Disk quota of disk size exceeded. Could also be disk failure or NFS timeout. If you are really close to the quota limit, it might happen that the client thinks that it could write the data, and until the next read happens, the quota mechanism has wiped away a few bytes. At least I have seen such a behavior on NFS. Full OSTs could perhaps cause similar behavior Another issue comes to my mind: Which kind of locking is used? It can be specified in the mount options. Locking might slow down lustre a bit (also depending on the version you are using). No mount option at all means there is no locking and thread A does not notice that thread B has the file still open. But the error message you wrote suggests that at some locking is active. localflock handles locking on each client locally, so the threads notice that the file is still in use as long as the owner is on the same node, and flock does global locking. Maybe you have to switch to global flock if you use several nodes within one job. > Technically Lustre is allowed to return fewer bytes than requested, as it says on that page. But it doesn't normally - LU-6389 is a bug where that can happen kind of often. (Again, it's technically allowed as that page says... But it shouldn't really happen in practice, which is why LU-6389 is a bug.) > > So perhaps Gaussian does not retry short reads? If memory serves, it's closed source, so you can't check - but perhaps you could ask the vendor? maybe you can see something with strace? But I'm not sure if you are allowed to do such kind of debugging Martin > ________________________________________ > From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Patrick Farrell [paf(a)cray.com] > Sent: Friday, September 04, 2015 4:03 PM > To: Kumar, Amit; Martin Hecht; hpdd-discuss(a)lists.01.org > Subject: Re: [HPDD-discuss] Stale File is it possible? > > What's your Lustre version? > > If the file looks OK when you look at it yourself (ie, no gaps), you might be running in to this bug: > https://jira.hpdd.intel.com/browse/LU-6389 > > Lustre 2.5 and newer will sometimes return fewer than expected bytes on a read or write, without giving an error. > > - Patrick > ________________________________________ > From: Kumar, Amit [ahkumar(a)mail.smu.edu] > Sent: Friday, September 04, 2015 2:51 PM > To: Martin Hecht; Patrick Farrell; hpdd-discuss(a)lists.01.org > Subject: RE: [HPDD-discuss] Stale File is it possible? > > Patrick & Matin, > > Thank you for responses. Files are Gau-*.rwf files, they are data files created by Gaussian and a bunch of text or log files. Gaussian does what they call it as logically random-access. > > Error that we run into often is: Erroneous read. Read 1 instead of 8 > > I don't seem to find any lustre/lnet errors in dmesg or syslogs. Although I have seen " Error -2 syncing data on lock cancel " here and there but they tend to be also on other clients so I don't think that is of significance. > > We are mounting Lustre over IB, not sure if that has any significance. > > I am looking at runing another test with additional debugging enabled, if you have any hints here that would be helpful. > > Best Regards, > Amit

-----Original Message----- From: Bernd Schubert [mailto:bernd.schubert@fastmail.fm] Sent: Monday, September 07, 2015 3:14 PM To: Patrick Farrell; Kumar, Amit; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? On 09/04/2015 11:09 PM, Patrick Farrell wrote: > Sent this once already, but in reply to the wrong message... > > Martin might know about that short read thing, since his site has a nice wiki page on it: > https://wickie.hlrs.de/platforms/index.php/Lustre_short_read > > Technically Lustre is allowed to return fewer bytes than requested, as > it says on that page. But it doesn't normally - LU-6389 is a bug > where that can happen kind of often. (Again, it's technically allowed > as that page says... But it shouldn't really happen in practice, > which is why LU-6389 is a bug.) > > So perhaps Gaussian does not retry short reads? If memory serves, it's closed source, so you can't check - but perhaps you could ask the vendor? Depends on the license. At least in the past it was common for to get the Gaussian source code to be able to compile yourself or to modify it (have been doing that myself a long time ago). Short reads should be solvable using LD_PRELOAD. Although one might argue that Gaussian as well as Lustre should be able to handle this any better on their own. Attached are rather untested read-preload files. Compile it and then run the gaussian binary with something like LD_PRELOAD=<path/to/file>/read-preload.so <binary> Btw, the wiki code is not ideal, assuming already the first read returns -1, it is going to use ptr -1, which might be outside of valid address space. Similar if there would be one successful (short) read, but several -1 read results after. Cheers, Bernd -- DataDirect Networks

-----Original Message----- From: Bernd Schubert [mailto:bernd.schubert@fastmail.fm] Sent: Monday, September 07, 2015 3:14 PM To: Patrick Farrell; Kumar, Amit; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? On 09/04/2015 11:09 PM, Patrick Farrell wrote: > Sent this once already, but in reply to the wrong message... > > Martin might know about that short read thing, since his site has a nice wiki page on it: > https://wickie.hlrs.de/platforms/index.php/Lustre_short_read > > Technically Lustre is allowed to return fewer bytes than requested, as > it says on that page. But it doesn't normally - LU-6389 is a bug > where that can happen kind of often. (Again, it's technically allowed > as that page says... But it shouldn't really happen in practice, > which is why LU-6389 is a bug.) > > So perhaps Gaussian does not retry short reads? If memory serves, it's closed source, so you can't check - but perhaps you could ask the vendor? Depends on the license. At least in the past it was common for to get the Gaussian source code to be able to compile yourself or to modify it (have been doing that myself a long time ago). Short reads should be solvable using LD_PRELOAD. Although one might argue that Gaussian as well as Lustre should be able to handle this any better on their own. Attached are rather untested read-preload files. Compile it and then run the gaussian binary with something like LD_PRELOAD=<path/to/file>/read-preload.so <binary> Btw, the wiki code is not ideal, assuming already the first read returns -1, it is going to use ptr -1, which might be outside of valid address space. Similar if there would be one successful (short) read, but several -1 read results after. Cheers, Bernd -- DataDirect Networks

In all the errors that I have seen it always says Read -1 instead of 8 This message is printed by Gaussian afte: if ((istat = read (fd, bufadr, len)) < 0) {

-----Original Message----- From: Bernd Schubert [mailto:bernd.schubert@fastmail.fm] Sent: Monday, September 07, 2015 3:14 PM To: Patrick Farrell; Kumar, Amit; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? On 09/04/2015 11:09 PM, Patrick Farrell wrote: > Sent this once already, but in reply to the wrong message... > > Martin might know about that short read thing, since his site has a nice wiki page on it: > https://wickie.hlrs.de/platforms/index.php/Lustre_short_read > > Technically Lustre is allowed to return fewer bytes than requested, as > it says on that page. But it doesn't normally - LU-6389 is a bug > where that can happen kind of often. (Again, it's technically allowed > as that page says... But it shouldn't really happen in practice, > which is why LU-6389 is a bug.) > > So perhaps Gaussian does not retry short reads? If memory serves, it's closed source, so you can't check - but perhaps you could ask the vendor? Depends on the license. At least in the past it was common for to get the Gaussian source code to be able to compile yourself or to modify it (have been doing that myself a long time ago). Short reads should be solvable using LD_PRELOAD. Although one might argue that Gaussian as well as Lustre should be able to handle this any better on their own. Attached are rather untested read-preload files. Compile it and then run the gaussian binary with something like LD_PRELOAD=<path/to/file>/read-preload.so <binary> Btw, the wiki code is not ideal, assuming already the first read returns -1, it is going to use ptr -1, which might be outside of valid address space. Similar if there would be one successful (short) read, but several -1 read results after. Cheers, Bernd -- DataDirect Networks

So just to confirm, if it's saying -1, that pretty much rules out short reads. You might check dmesg or the Lustre logs for any errors around that time. ________________________________________ From: Kumar, Amit [ahkumar(a)mail.smu.edu] Sent: Thursday, September 10, 2015 6:58 PM To: Faaland, Olaf P.; Patrick Farrell; Bernd Schubert; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: RE: [HPDD-discuss] Stale File is it possible? HI Olaf, Yes it does, It has it's own gperror which then calls perror. Although it only prints the function name which I already know where it is. I will look at the perror and strerror and see if I can update the gperror function to print additional information. Strangely this happens to a special set of long running jobs rest of the long running jobs seem to work fine. So let's see if I can dig into this further. Unfortunately each run takes about 7 hours before it fails so hopefully I will have more info on this in the morning. Thank you for your input!!! Regards, Amit ________________________________________ From: Faaland, Olaf P. [faaland1(a)llnl.gov] Sent: Thursday, September 10, 2015 3:39 PM To: Kumar, Amit; Patrick Farrell; Bernd Schubert; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: RE: [HPDD-discuss] Stale File is it possible? Amit, > In all the errors that I have seen it always says Read -1 instead of 8 > > This message is printed by Gaussian afte: > if ((istat = read (fd, bufadr, len)) < 0) { So if read() returned -1, then the read failed, and no bytes were returned at all. It isn't necessarily a Lustre issue, it could be a bug in Gaussian or a misconfiguration of some kind. However the -1 just tells you the read failed, not the mode of failure. You need to know the value of errno to find that out. After this, does Gaussian call perror() and/or log the actual value of errno and its corresponding string? It should. See man 3 perror and man 3 strerror. Then report back what it says. Good luck! Olaf P. Faaland Livermore Computing phone : 925-422-2263 ________________________________________ From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Kumar, Amit [ahkumar(a)mail.smu.edu] Sent: Thursday, September 10, 2015 1:12 PM To: Patrick Farrell; Bernd Schubert; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Stale File is it possible? Dear All, Bernd: I ran with LD_PRELOAD for read and do not get any far, gaussian programs just sit there spinning on read. In fact I think I may have to tweak the read function. I also ran the re-producer functions attached by Bill Barth to the bug report, to see if I run into this issue. And I could not reproduce the bug errors. So I am not sure if I am running in right directions. Interestingly when I ran Bill Barth's re-producer code with LD_PRELOAD function I get to see that BUG error. I looked at the read man pages and it seems that it would return -1 or the length of bytes read in situations that fall out of the defined errors. In all the errors that I have seen it always says Read -1 instead of 8 This message is printed by Gaussian afte: if ((istat = read (fd, bufadr, len)) < 0) { Any hints to further debug this will be greatly appreciated. I am trying recompiling of the code and see if I can add debug statements. Best Regards, Amit ________________________________________ From: Patrick Farrell [paf(a)cray.com] Sent: Wednesday, September 09, 2015 10:05 AM To: Kumar, Amit; Bernd Schubert; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: RE: [HPDD-discuss] Stale File is it possible? Since the layout lock is in Lustre 2.4, I think LU-6389 is a possibility in 2.4. Good luck. ________________________________________ From: Kumar, Amit [ahkumar(a)mail.smu.edu] Sent: Wednesday, September 09, 2015 9:27 AM To: Bernd Schubert; Patrick Farrell; Martin Hecht; hpdd-discuss(a)lists.01.org Subject: RE: [HPDD-discuss] Stale File is it possible? Awesome you all!! And sorry for the void in my response. Was out for extended long weekend and catching up on things. My version of Lustre is: 2.4.3 assuming LU-6389 also affects this version, this already known fact could solve our mystery. As Bernd pointed out, we compiled our own from source since we have license for that. I will try the read-preload solution and see if we can relieve our users from this. Exciting ..thank you for this hope ... Best Regards, Amit > -----Original Message----- > From: Bernd Schubert [mailto:bernd.schubert@fastmail.fm] > Sent: Monday, September 07, 2015 3:14 PM > To: Patrick Farrell; Kumar, Amit; Martin Hecht; hpdd-discuss(a)lists.01.org > Subject: Re: [HPDD-discuss] Stale File is it possible? > > On 09/04/2015 11:09 PM, Patrick Farrell wrote: >> Sent this once already, but in reply to the wrong message... >> >> Martin might know about that short read thing, since his site has a nice wiki > page on it: >> https://wickie.hlrs.de/platforms/index.php/Lustre_short_read >> >> Technically Lustre is allowed to return fewer bytes than requested, as >> it says on that page. But it doesn't normally - LU-6389 is a bug >> where that can happen kind of often. (Again, it's technically allowed >> as that page says... But it shouldn't really happen in practice, >> which is why LU-6389 is a bug.) >> >> So perhaps Gaussian does not retry short reads? If memory serves, it's > closed source, so you can't check - but perhaps you could ask the vendor? > > Depends on the license. At least in the past it was common for to get the > Gaussian source code to be able to compile yourself or to modify it (have > been doing that myself a long time ago). > > Short reads should be solvable using LD_PRELOAD. Although one might argue > that Gaussian as well as Lustre should be able to handle this any better on > their own. > Attached are rather untested read-preload files. Compile it and then run the > gaussian binary with something like > > LD_PRELOAD=<path/to/file>/read-preload.so <binary> > > > Btw, the wiki code is not ideal, assuming already the first read returns -1, it is > going to use ptr -1, which might be outside of valid address space. Similar if > there would be one successful (short) read, but several -1 read results after. > > Cheers, > Bernd > > -- > DataDirect Networks