3:58 a.m.
Did you mount lustre client with the flock option?
On Feb 15, 2018 9:56 AM, "Prentice Bisbal" <pbisbal(a)pppl.gov> wrote:
Hi.
I'm an experience HPC system admin, but I know almost nothing about Lustre
administration. The system admin who administered our small Lustre
filesystem recently retired, and no one has filled that gap yet. I user
recently reported they are now getting file-locking errors from a program
they've run repeatedly on Lustre in the past. When the run the same program
on an NFS filesystem, the error goes away. I've cut-and-pasted the error
messages below.
Since I have real experience as a Lustre admin, I turned to google, and it
looks like it might be that the file-locking daemon died (if Lustre has a
separate file-lock daemon), or somehow file-locking was recently disabled.
If that is possible, how do I check this, and restart or re-enable if
necessary? I skimmed the user manual, and could not find anything on
either of these issues.
Any and all help will be greatly appreciated.
Some of the error messages:
HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) MPI-process 9:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or
initialize file structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file,
errno = 38, error message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
Error: couldn't open file HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1)
MPI-process 13:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or
initialize file structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file,
errno = 38, error message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
--
Prentice
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:10 a.m.
New subject: Lustre filelocking issue
How can I check that? The output of mount shows the following:
172.16.2.110@tcp:/lustre on /pfs/nobackup type lustre (rw)
which is about as basic mount options as you can get. I am the only
person who makes config changes to the clients, so if filelocking was
enabled before, and it's not now, I'm at a loss to explain how/why that
changed.
Prentice
On 02/15/2018 11:58 AM, Colin Faber wrote:
Did you mount lustre client with the flock option?
On Feb 15, 2018 9:56 AM, "Prentice Bisbal" <pbisbal(a)pppl.gov
<mailto:pbisbal@pppl.gov>> wrote:
Hi.
I'm an experience HPC system admin, but I know almost nothing
about Lustre administration. The system admin who administered our
small Lustre filesystem recently retired, and no one has filled
that gap yet. I user recently reported they are now getting
file-locking errors from a program they've run repeatedly on
Lustre in the past. When the run the same program on an NFS
filesystem, the error goes away. I've cut-and-pasted the error
messages below.
Since I have real experience as a Lustre admin, I turned to
google, and it looks like it might be that the file-locking daemon
died (if Lustre has a separate file-lock daemon), or somehow
file-locking was recently disabled. If that is possible, how do I
check this, and restart or re-enable if necessary? I skimmed the
user manual, and could not find anything on either of these issues.
Any and all help will be greatly appreciated.
Some of the error messages:
HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) MPI-process 9:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file
or initialize file structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock
file, errno = 38, error message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
Error: couldn't open file HDF5-DIAG: Error detected in HDF5
(1.10.0-patch1) MPI-process 13:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file
or initialize file structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock
file, errno = 38, error message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
--
Prentice
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org <mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
<https://lists.01.org/mailman/listinfo/hpdd-discuss>
4:42 a.m.
New subject: Lustre filelocking issue
If you have user_xattr enabled, output should look something like this:
# mount |grep lustre
192.168.1.109@o2ib:/lustretest on /mnt/lustre type lustre (rw,user_xattr,lazystatfs)
#
So, I would say you do not have the option enabled in your mount.
Your client mount should be something like “mount -t lustre -o user_xattr <FS
name/NID> <mountpoint> “
For example: “mount -t lustre -o user_xattr 192.168.1.109@o2ib:/lustretest /mnt/lustre”
Cliffw
From: HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org> on behalf of Prentice Bisbal
<pbisbal(a)pppl.gov>
Date: Thursday, February 15, 2018 at 9:11 AM
Cc: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
Subject: Re: [HPDD-discuss] Lustre filelocking issue
How can I check that? The output of mount shows the following:
172.16.2.110@tcp:/lustre on /pfs/nobackup type lustre (rw)
which is about as basic mount options as you can get. I am the only person who makes
config changes to the clients, so if filelocking was enabled before, and it's not now,
I'm at a loss to explain how/why that changed.
Prentice
On 02/15/2018 11:58 AM, Colin Faber wrote:
Did you mount lustre client with the flock option?
On Feb 15, 2018 9:56 AM, "Prentice Bisbal"
<pbisbal@pppl.gov<mailto:pbisbal@pppl.gov>> wrote:
Hi.
I'm an experience HPC system admin, but I know almost nothing about Lustre
administration. The system admin who administered our small Lustre filesystem recently
retired, and no one has filled that gap yet. I user recently reported they are now getting
file-locking errors from a program they've run repeatedly on Lustre in the past. When
the run the same program on an NFS filesystem, the error goes away. I've
cut-and-pasted the error messages below.
Since I have real experience as a Lustre admin, I turned to google, and it looks like it
might be that the file-locking daemon died (if Lustre has a separate file-lock daemon), or
somehow file-locking was recently disabled. If that is possible, how do I check this, and
restart or re-enable if necessary? I skimmed the user manual, and could not find anything
on either of these issues.
Any and all help will be greatly appreciated.
Some of the error messages:
HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) MPI-process 9:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or initialize file
structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file, errno = 38, error
message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
Error: couldn't open file HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1)
MPI-process 13:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or initialize file
structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file, errno = 38, error
message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
--
Prentice
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:51 a.m.
New subject: Lustre filelocking issue
User_xattr is different from flock. If this is indeed POSIX file locking (rather than, I
guess, something using xattrs?), you’ll need to add “flock”, like “-o flock” or “-o
flock,user_xattr” if you wanted to have both.
* Patrick
From: HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org> on behalf of "White,
Cliff" <cliff.white(a)intel.com>
Date: Thursday, February 15, 2018 at 11:42 AM
To: Prentice Bisbal <pbisbal(a)pppl.gov>
Cc: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
Subject: Re: [HPDD-discuss] Lustre filelocking issue
If you have user_xattr enabled, output should look something like this:
# mount |grep lustre
192.168.1.109@o2ib:/lustretest on /mnt/lustre type lustre (rw,user_xattr,lazystatfs)
#
So, I would say you do not have the option enabled in your mount.
Your client mount should be something like “mount -t lustre -o user_xattr <FS
name/NID> <mountpoint> “
For example: “mount -t lustre -o user_xattr 192.168.1.109@o2ib:/lustretest /mnt/lustre”
Cliffw
From: HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org> on behalf of Prentice Bisbal
<pbisbal(a)pppl.gov>
Date: Thursday, February 15, 2018 at 9:11 AM
Cc: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
Subject: Re: [HPDD-discuss] Lustre filelocking issue
How can I check that? The output of mount shows the following:
172.16.2.110@tcp:/lustre on /pfs/nobackup type lustre (rw)
which is about as basic mount options as you can get. I am the only person who makes
config changes to the clients, so if filelocking was enabled before, and it's not now,
I'm at a loss to explain how/why that changed.
Prentice
On 02/15/2018 11:58 AM, Colin Faber wrote:
Did you mount lustre client with the flock option?
On Feb 15, 2018 9:56 AM, "Prentice Bisbal"
<pbisbal@pppl.gov<mailto:pbisbal@pppl.gov>> wrote:
Hi.
I'm an experience HPC system admin, but I know almost nothing about Lustre
administration. The system admin who administered our small Lustre filesystem recently
retired, and no one has filled that gap yet. I user recently reported they are now getting
file-locking errors from a program they've run repeatedly on Lustre in the past. When
the run the same program on an NFS filesystem, the error goes away. I've
cut-and-pasted the error messages below.
Since I have real experience as a Lustre admin, I turned to google, and it looks like it
might be that the file-locking daemon died (if Lustre has a separate file-lock daemon), or
somehow file-locking was recently disabled. If that is possible, how do I check this, and
restart or re-enable if necessary? I skimmed the user manual, and could not find anything
on either of these issues.
Any and all help will be greatly appreciated.
Some of the error messages:
HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) MPI-process 9:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or initialize file
structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file, errno = 38, error
message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
Error: couldn't open file HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1)
MPI-process 13:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or initialize file
structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file, errno = 38, error
message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
--
Prentice
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
6:17 p.m.
New subject: Lustre filelocking issue
Hi Prentice,
You can find the options to enable file locking on the lustre filesystem
Manual:
flock ->
Enables advisory file locking support between participating applications
using the flock(2) system call.
This causes file locking to be coherent across all client nodes also using
this mount option.
This is useful if applications need coherent userspace file locking across
multiple client nodes, but also imposes communications overhead in order to
maintain locking consistency between client nodes.
localflock ->
Enables client-local flock(2) support, using only client-local advisory
file locking.
This is faster than using the global flock option, and can be used for
applications that depend on functioning flock(2) but run only on a single
node. It has minimal overhead using only the Linux kernel's locks.
Keep it in mind to avoid affect to the performance.
BR
2018-02-15 18:51 GMT+01:00 Patrick Farrell <paf(a)cray.com>:
User_xattr is different from flock. If this is indeed POSIX file
locking
(rather than, I guess, something using xattrs?), you’ll need to add
“flock”, like “-o flock” or “-o flock,user_xattr” if you wanted to have
both.
- Patrick
*From: *HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org> on behalf of
"White, Cliff" <cliff.white(a)intel.com>
*Date: *Thursday, February 15, 2018 at 11:42 AM
*To: *Prentice Bisbal <pbisbal(a)pppl.gov>
*Cc: *"hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
*Subject: *Re: [HPDD-discuss] Lustre filelocking issue
If you have user_xattr enabled, output should look something like this:
# mount |grep lustre
192.168.1.109@o2ib:/lustretest on /mnt/lustre type lustre
(rw,user_xattr,lazystatfs)
#
So, I would say you do not have the option enabled in your mount.
Your client mount should be something like “mount -t lustre -o user_xattr
<FS name/NID> <mountpoint> “
For example: “mount -t lustre -o user_xattr 192.168.1.109@o2ib:/lustretest
/mnt/lustre”
Cliffw
*From: *HPDD-discuss <hpdd-discuss-bounces(a)lists.01.org> on behalf of
Prentice Bisbal <pbisbal(a)pppl.gov>
*Date: *Thursday, February 15, 2018 at 9:11 AM
*Cc: *"hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
*Subject: *Re: [HPDD-discuss] Lustre filelocking issue
How can I check that? The output of mount shows the following:
172.16.2.110@tcp:/lustre on /pfs/nobackup type lustre (rw)
which is about as basic mount options as you can get. I am the only person
who makes config changes to the clients, so if filelocking was enabled
before, and it's not now, I'm at a loss to explain how/why that changed.
Prentice
On 02/15/2018 11:58 AM, Colin Faber wrote:
Did you mount lustre client with the flock option?
On Feb 15, 2018 9:56 AM, "Prentice Bisbal" <pbisbal(a)pppl.gov> wrote:
Hi.
I'm an experience HPC system admin, but I know almost nothing about Lustre
administration. The system admin who administered our small Lustre
filesystem recently retired, and no one has filled that gap yet. I user
recently reported they are now getting file-locking errors from a program
they've run repeatedly on Lustre in the past. When the run the same program
on an NFS filesystem, the error goes away. I've cut-and-pasted the error
messages below.
Since I have real experience as a Lustre admin, I turned to google, and it
looks like it might be that the file-locking daemon died (if Lustre has a
separate file-lock daemon), or somehow file-locking was recently disabled.
If that is possible, how do I check this, and restart or re-enable if
necessary? I skimmed the user manual, and could not find anything on
either of these issues.
Any and all help will be greatly appreciated.
Some of the error messages:
HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) MPI-process 9:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or
initialize file structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file,
errno = 38, error message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
Error: couldn't open file HDF5-DIAG: Error detected in HDF5
(1.10.0-patch1) MPI-process 13:
#000: H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: H5Fint.c line 1168 in H5F_open(): unable to lock the file or
initialize file structure
major: File accessibilty
minor: Unable to open file
#002: H5FD.c line 1821 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Can't update object
#003: H5FDsec2.c line 939 in H5FD_sec2_lock(): unable to flock file,
errno = 38, error message = 'Function not implemented'
major: File accessibilty
minor: Bad file ID accessed
--
Prentice
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
--
Saludos,
Aníbal
1675
days inactive
1676
days old
4 comments
5 participants
participants (5)
-
Anibal Moreno -
Colin Faber -
Patrick Farrell -
Prentice Bisbal -
White, Cliff