7:24 p.m.
Dear all,
a user has noticed a weird problem with our Lustre setup. The servers
are running Lustre 2.1.3 and the client is 1.8.9, both on SL6.
When I run the command 'lfs find .' several times in a row, each time
the result is different. Some files are removed from the listing, some
are added, no regular pattern to be seen.
There are no unusual syslog messages on the servers or the client and
the file system is not very busy.
Do you have any idea how to resolve this problem?
Regards, Götz Waschk
7:43 p.m.
It seems LU-3029 (https://jira.hpdd.intel.com/browse/LU-3029). We are trouble with how to
reproduce it locally. If you can check the comment and give more debug info as LU-3029
suggested, that will be helpful.
--
Best Regards,
Nasf
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Götz Waschk
Sent: Friday, April 19, 2013 5:24 PM
To: hpdd-discuss(a)lists.01.org
Subject: [HPDD-discuss] Missing files between adjacent lfs file runs
Dear all,
a user has noticed a weird problem with our Lustre setup. The servers are running Lustre
2.1.3 and the client is 1.8.9, both on SL6.
When I run the command 'lfs find .' several times in a row, each time the result
is different. Some files are removed from the listing, some are added, no regular pattern
to be seen.
There are no unusual syslog messages on the servers or the client and the file system is
not very busy.
Do you have any idea how to resolve this problem?
Regards, Götz Waschk
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
3:14 a.m.
Can you please share the upgrade history of your servers? What version
of Lustre were they formatted with? Our servers affected by LU-3029
were originally formatted with 1.6, so we wonder if that is a factor.
Thanks,
Ned
On Fri, Apr 19, 2013 at 09:43:43AM +0000, Yong, Fan wrote:
It seems LU-3029 (https://jira.hpdd.intel.com/browse/LU-3029). We are
trouble with how to reproduce it locally. If you can check the comment and give more debug
info as LU-3029 suggested, that will be helpful.
--
Best Regards,
Nasf
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of G?tz Waschk
Sent: Friday, April 19, 2013 5:24 PM
To: hpdd-discuss(a)lists.01.org
Subject: [HPDD-discuss] Missing files between adjacent lfs file runs
Dear all,
a user has noticed a weird problem with our Lustre setup. The servers are running Lustre
2.1.3 and the client is 1.8.9, both on SL6.
When I run the command 'lfs find .' several times in a row, each time the result
is different. Some files are removed from the listing, some are added, no regular pattern
to be seen.
There are no unusual syslog messages on the servers or the client and the file system is
not very busy.
Do you have any idea how to resolve this problem?
Regards, G?tz Waschk
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:59 p.m.
On Mon, Apr 22, 2013 at 7:14 PM, Ned Bass <bass6(a)llnl.gov> wrote:
Can you please share the upgrade history of your servers? What
version
of Lustre were they formatted with? Our servers affected by LU-3029
were originally formatted with 1.6, so we wonder if that is a factor.
Dear Ned,
the situation is different for my servers. They have been formatted
with Lustre 2.1.3 and have never been upgraded. I can give you the
complete mkfs command lines if needed.
Regards, Götz
7:18 a.m.
Agreed, sounds like LU-3029.
How easy is it for you to reproduce this issue? We've been struggling to
find a root cause for this due to it being difficult to reproduce. I'd
love to work with you to gather some debugging info, if you have an easy
way to trigger the problem.
--
Cheers, Prakash
On Fri, Apr 19, 2013 at 09:43:43AM +0000, Yong, Fan wrote:
It seems LU-3029 (https://jira.hpdd.intel.com/browse/LU-3029). We are
trouble with how to reproduce it locally. If you can check the comment and give more debug
info as LU-3029 suggested, that will be helpful.
--
Best Regards,
Nasf
-----Original Message-----
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Götz Waschk
Sent: Friday, April 19, 2013 5:24 PM
To: hpdd-discuss(a)lists.01.org
Subject: [HPDD-discuss] Missing files between adjacent lfs file runs
Dear all,
a user has noticed a weird problem with our Lustre setup. The servers are running Lustre
2.1.3 and the client is 1.8.9, both on SL6.
When I run the command 'lfs find .' several times in a row, each time the result
is different. Some files are removed from the listing, some are added, no regular pattern
to be seen.
There are no unusual syslog messages on the servers or the client and the file system is
not very busy.
Do you have any idea how to resolve this problem?
Regards, Götz Waschk
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
5:01 p.m.
On Mon, Apr 22, 2013 at 11:18 PM, Prakash Surya <surya1(a)llnl.gov> wrote:
How easy is it for you to reproduce this issue? We've been
struggling to
find a root cause for this due to it being difficult to reproduce. I'd
love to work with you to gather some debugging info, if you have an easy
way to trigger the problem.
Dear Prakash,
reproducing the problem is easy, it happens practically all the time
for the reporter's user directory. Debugging on the other hand will be
harder as the file system is in production and I cannot simply umount
it.
Regards, Götz
2:26 a.m.
Understood. Are you willing to run a systemtap script on the machine?
Since I can't umount our production FS easily, I've been using systemtap
to get additional information out of the system.
--
Cheers, Prakash
On Tue, Apr 23, 2013 at 09:01:05AM +0200, Götz Waschk wrote:
On Mon, Apr 22, 2013 at 11:18 PM, Prakash Surya
<surya1(a)llnl.gov> wrote:
> How easy is it for you to reproduce this issue? We've been struggling to
> find a root cause for this due to it being difficult to reproduce. I'd
> love to work with you to gather some debugging info, if you have an easy
> way to trigger the problem.
Dear Prakash,
reproducing the problem is easy, it happens practically all the time
for the reporter's user directory. Debugging on the other hand will be
harder as the file system is in production and I cannot simply umount
it.
Regards, Götz
9:22 p.m.
On Tue, Apr 23, 2013 at 6:26 PM, Prakash Surya <surya1(a)llnl.gov> wrote:
Understood. Are you willing to run a systemtap script on the
machine?
Since I can't umount our production FS easily, I've been using systemtap
to get additional information out of the system.
Dear Prakash,
yes, please send me the script.
Regards, Götz
1:23 a.m.
I think we've gotten to the root cause for the issue we've been seeing
at LLNL, so before I waste your time with gathering debug data I'd like
to point you at the discussion on the JIRA ticket. There's a good
explanation of the issue in this comment:
https://jira.hpdd.intel.com/browse/LU-3029?focusedCommentId=56903&pag...
Are you running old 1.8 clients which do *not* support FMODE_32BITHASH?
My guess is you are running a mix of clients with FMODE_32BITHASH, and
FMODE_64BITHASH. If that is the case, you are likely seeing the same
issue as us. A fix will be proposed soon which would require an MDS
downtime.
--
Cheers, Prakash
On Wed, Apr 24, 2013 at 01:22:43PM +0200, Götz Waschk wrote:
On Tue, Apr 23, 2013 at 6:26 PM, Prakash Surya
<surya1(a)llnl.gov> wrote:
> Understood. Are you willing to run a systemtap script on the machine?
> Since I can't umount our production FS easily, I've been using systemtap
> to get additional information out of the system.
Dear Prakash,
yes, please send me the script.
Regards, Götz
6:02 p.m.
On Wed, Apr 24, 2013 at 5:23 PM, Prakash Surya <surya1(a)llnl.gov> wrote:
Are you running old 1.8 clients which do *not* support
FMODE_32BITHASH?
My guess is you are running a mix of clients with FMODE_32BITHASH, and
FMODE_64BITHASH. If that is the case, you are likely seeing the same
issue as us. A fix will be proposed soon which would require an MDS
downtime.
Dear Prakash,
I had no clients that didn't request 64 bit hashes running but I might
have in the past, the server had an uptime of 231 days. I have
rebooted it and now the oldest connected client is Lustre 1.8.8. It
seems this problem has gone away. Thanks for your help.
Regards, Götz Waschk
2:37 a.m.
Great! Sounds like you were hitting the same issue as us. A patch has
landed on master (or will soon) to fix this, and will hit b2_1 soon as
well. But if you no longer have 32-bit hash clients, a simple reboot
will suffice for you.
--
Cheers, Prakash
On Thu, May 02, 2013 at 10:02:31AM +0200, Götz Waschk wrote:
On Wed, Apr 24, 2013 at 5:23 PM, Prakash Surya
<surya1(a)llnl.gov> wrote:
> Are you running old 1.8 clients which do *not* support FMODE_32BITHASH?
> My guess is you are running a mix of clients with FMODE_32BITHASH, and
> FMODE_64BITHASH. If that is the case, you are likely seeing the same
> issue as us. A fix will be proposed soon which would require an MDS
> downtime.
Dear Prakash,
I had no clients that didn't request 64 bit hashes running but I might
have in the past, the server had an uptime of 231 days. I have
rebooted it and now the oldest connected client is Lustre 1.8.8. It
seems this problem has gone away. Thanks for your help.
Regards, Götz Waschk
3426
days inactive
3439
days old
10 comments
4 participants
participants (4)
-
Götz Waschk -
Ned Bass -
Prakash Surya -
Yong, Fan