7:01 a.m.
Dear All,
I have deactivated an OST yet I see the size of the OST is increasing constantly?
Any thoughts what could be causing this?
Thank you,
Amit
7:13 a.m.
Hi,
temp. deactivated OSTs via 'lctl --device devno deactivate' will be marked as
inactive. No new objects are assigned to the deactivated OST, but reads and writes of
existing objects are still no problem.
Lustre_Manual Chapter: Removing an OST from the File System
Hope that helps
- --
"la lykken gro, som gresset bak do"
Nico Budewitz
High Performance Computing
Max-Planck-Institute for Gravitational Physics /
Albert-Einstein-Institute
Am Muehlenberg 1, 14476 Golm
Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
http://supercomputers.aei.mpg.de
On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu> wrote:
Dear All,
I have deactivated an OST yet I see the size of the OST is increasing constantly?
Any thoughts what could be causing this?
Thank you,
Amit
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:56 a.m.
Hi Nico,
I Just saw your message. Even though I have it deactivated, something is still writing
significantly. Hope you can help me understand this better. Even though new objects are
not assigned can the disk usage increase for the existing objects?
On the other hand, my migration script is not able to locate any files on the OST to move,
which is odd. One thing I know is some files names are very odd, like may be 20-30 spaces
in a file of length about at least 100 characters. Some users program is fanatic in file
name creation and I have no control over it.
Hope you can shed some more light. I am planning on running fsck to identify any issues
with the ost.
Thank you,
Amit
-----Original Message-----
From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
Sent: Thursday, October 10, 2013 3:14 PM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] OST deactivated but still objects being written to it?
Hi,
temp. deactivated OSTs via 'lctl --device devno deactivate' will be marked as
inactive. No new objects are assigned to the deactivated OST, but reads and writes of
existing objects are still no problem.
Lustre_Manual Chapter: Removing an OST from the File System Hope that helps
- --
"la lykken gro, som gresset bak do"
Nico Budewitz
High Performance Computing
Max-Planck-Institute for Gravitational Physics / Albert-Einstein-Institute Am Muehlenberg
1, 14476 Golm
Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284 http://supercomputers.aei.mpg.de
On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu> wrote:
Dear All,
I have deactivated an OST yet I see the size of the OST is increasing constantly?
Any thoughts what could be causing this?
Thank you,
Amit
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
10:19 a.m.
On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu> wrote:
I Just saw your message. Even though I have it deactivated, something
is
still writing significantly. Hope you can help me understand this better.
Even though new objects are not assigned can the disk usage increase for
the existing objects?
On the other hand, my migration script is not able to locate any files on
the OST to move, which is odd. One thing I know is some files names are
very odd, like may be 20-30 spaces in a file of length about at least
100 characters. Some users program is fanatic in file name creation and I
have no control over it.
Hope you can shed some more light. I am planning on running fsck to
identify any issues with the ost.
I don't think fsck is needed. It seems like you have some process on a
client that
is writing to an open-unlinked file on those OSTs.
To find out which client, run on the affected OSS:
# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
ost.OSS.ost_io.req_history=
1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Com
plete:1382050012:0s(-43s) opc 4
1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Com
plete:1382050012:0s(-43s) opc 4
This will print out all clients (NID "192.168.20.1@tcp" in this case) that
are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to see
what processes have files open on the filesystem (probably marked
"(deleted)")
and kill those processes.
This should immediately free up all of the space on those OSTs.
Cheers, Andreas
-----Original Message-----
From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
Sent: Thursday, October 10, 2013 3:14 PM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] OST deactivated but still objects being
written to it?
Hi,
temp. deactivated OSTs via 'lctl --device devno deactivate' will be
marked as inactive. No new objects are assigned to the deactivated OST,
but reads and writes of existing objects are still no problem.
Lustre_Manual Chapter: Removing an OST from the File System Hope that
helps
- --
"la lykken gro, som gresset bak do"
Nico Budewitz
High Performance Computing
Max-Planck-Institute for Gravitational Physics /
Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
http://supercomputers.aei.mpg.de
On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu> wrote:
> Dear All,
>
> I have deactivated an OST yet I see the size of the OST is increasing
>constantly?
>
> Any thoughts what could be causing this?
>
> Thank you,
> Amit
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
4:14 p.m.
Awesome Andreas sounds promising, I will give this a shot in the morning....
Regards
Amit
Sent from my iPhone
On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas" <andreas.dilger(a)intel.com>
wrote:
On 2013/10/14 11:56 AM, "Kumar, Amit"
<ahkumar(a)mail.smu.edu> wrote:
> I Just saw your message. Even though I have it deactivated, something is
> still writing significantly. Hope you can help me understand this better.
> Even though new objects are not assigned can the disk usage increase for
> the existing objects?
>
> On the other hand, my migration script is not able to locate any files on
> the OST to move, which is odd. One thing I know is some files names are
> very odd, like may be 20-30 spaces in a file of length about at least
> 100 characters. Some users program is fanatic in file name creation and I
> have no control over it.
>
> Hope you can shed some more light. I am planning on running fsck to
> identify any issues with the ost.
I don't think fsck is needed. It seems like you have some process on a
client that
is writing to an open-unlinked file on those OSTs.
To find out which client, run on the affected OSS:
# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
ost.OSS.ost_io.req_history=
1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Com
plete:1382050012:0s(-43s) opc 4
1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Com
plete:1382050012:0s(-43s) opc 4
This will print out all clients (NID "192.168.20.1@tcp" in this case) that
are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to see
what processes have files open on the filesystem (probably marked
"(deleted)")
and kill those processes.
This should immediately free up all of the space on those OSTs.
Cheers, Andreas
> -----Original Message-----
> From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
> Sent: Thursday, October 10, 2013 3:14 PM
> To: Kumar, Amit
> Cc: hpdd-discuss(a)lists.01.org
> Subject: Re: [HPDD-discuss] OST deactivated but still objects being
> written to it?
>
> Hi,
>
> temp. deactivated OSTs via 'lctl --device devno deactivate' will be
> marked as inactive. No new objects are assigned to the deactivated OST,
> but reads and writes of existing objects are still no problem.
>
> Lustre_Manual Chapter: Removing an OST from the File System Hope that
> helps
> - --
> "la lykken gro, som gresset bak do"
> Nico Budewitz
> High Performance Computing
> Max-Planck-Institute for Gravitational Physics /
> Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
> Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
> http://supercomputers.aei.mpg.de
>
> On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
wrote:
>
>> Dear All,
>>
>> I have deactivated an OST yet I see the size of the OST is increasing
>> constantly?
>>
>> Any thoughts what could be causing this?
>>
>> Thank you,
>> Amit
>>
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
6:53 p.m.
Very interesting.
I run the lctl command and here is the output:
*65838502*:10.149.254.2@o2ib:12345-10.149.1.171@o2ib:x1448244599539665:448:*Interpret*:1382081942:-1382081942s(-1382081952s)
opc 4
For my understanding, could you please give more details on the field of
this line ?
Thank you,
Pascal.
On 18/10/2013 07:14, Kumar, Amit wrote:
Awesome Andreas sounds promising, I will give this a shot in the
morning....
Regards
Amit
Sent from my iPhone
On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas" <andreas.dilger(a)intel.com>
wrote:
> On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu> wrote:
>> I Just saw your message. Even though I have it deactivated, something is
>> still writing significantly. Hope you can help me understand this better.
>> Even though new objects are not assigned can the disk usage increase for
>> the existing objects?
>>
>> On the other hand, my migration script is not able to locate any files on
>> the OST to move, which is odd. One thing I know is some files names are
>> very odd, like may be 20-30 spaces in a file of length about at least
>> 100 characters. Some users program is fanatic in file name creation and I
>> have no control over it.
>>
>> Hope you can shed some more light. I am planning on running fsck to
>> identify any issues with the ost.
> I don't think fsck is needed. It seems like you have some process on a
> client that
> is writing to an open-unlinked file on those OSTs.
>
> To find out which client, run on the affected OSS:
> # lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
> ost.OSS.ost_io.req_history=
> 1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Com
> plete:1382050012:0s(-43s) opc 4
> 1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Com
> plete:1382050012:0s(-43s) opc 4
>
> This will print out all clients (NID "192.168.20.1@tcp" in this case) that
> are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to
see
> what processes have files open on the filesystem (probably marked
> "(deleted)")
> and kill those processes.
>
> This should immediately free up all of the space on those OSTs.
>
> Cheers, Andreas
>
>> -----Original Message-----
>> From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
>> Sent: Thursday, October 10, 2013 3:14 PM
>> To: Kumar, Amit
>> Cc: hpdd-discuss(a)lists.01.org
>> Subject: Re: [HPDD-discuss] OST deactivated but still objects being
>> written to it?
>>
>> Hi,
>>
>> temp. deactivated OSTs via 'lctl --device devno deactivate' will be
>> marked as inactive. No new objects are assigned to the deactivated OST,
>> but reads and writes of existing objects are still no problem.
>>
>> Lustre_Manual Chapter: Removing an OST from the File System Hope that
>> helps
>> - --
>> "la lykken gro, som gresset bak do"
>> Nico Budewitz
>> High Performance Computing
>> Max-Planck-Institute for Gravitational Physics /
>> Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
>> Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
>> http://supercomputers.aei.mpg.de
>>
>> On Oct 10, 2013, at 10:01 PM, "Kumar, Amit"
<ahkumar(a)mail.smu.edu> wrote:
>>
>>> Dear All,
>>>
>>> I have deactivated an OST yet I see the size of the OST is increasing
>>> constantly?
>>>
>>> Any thoughts what could be causing this?
>>>
>>> Thank you,
>>> Amit
>>>
>>> _______________________________________________
>>> HPDD-discuss mailing list
>>> HPDD-discuss(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>>
>
> Cheers, Andreas
> --
> Andreas Dilger
>
> Lustre Software Architect
> Intel High Performance Data Division
>
>
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
--
*Pascal Decourt
*System Support Engineer Europe | SGI France
+33 1 34 88 80 31 | +33 6 80 45 24 77
pdecourt(a)sgi.com
SGI
Silicon Graphics SA
21 rue Albert Calmette
78350 Jouy en Josas RCS Versailles B 344 515 291
8:46 p.m.
On 2013/10/18 1:53 AM, "Pascal Decourt" <pdecourt(a)sgi.com> wrote:
Very interesting.
I run the lctl command and here is the output:
65838502:10.149.254.2@o2ib:12345-10.149.1.171@o2ib:x1448244599539665:448:I
nterpret:1382081942:-1382081942s(-1382081952s) opc 4
The size of the history buffer is controlled via the
"req_buffer_history_max"
tunable. By default it only has a few entries in it.
For my understanding, could you please give more details on the field
of
this line ?
Fields are separated by colons.
1st field: a sequence counter, so it is possible to determine if a
record has already been processed.
2nd field: target NID (i.e. server)
3rd field: source NID (i.e. client)
4th field: client RPC XID number (per-client unique identifier)
5th field: RPC request buffer size
6th field: state of the RPC processing
7th field: time (unix epoch) when RPC arrived
8th field: total processing time (seconds)
9th field: time before client would have timed out request
10th field: RPC opcode number (see lustre_idl.h)
This is all described in the user manual if you search for req_history:
https://wiki.hpdd.intel.com/display/PUB/Documentation
There is also a tool "lustre_req_history" that processes these files a bit.
Cheers, Andreas
On 18/10/2013 07:14, Kumar, Amit wrote:
Awesome Andreas sounds promising, I will give this a shot in the
morning....
Regards
Amit
Sent from my iPhone
On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas" <andreas.dilger(a)intel.com>
<mailto:andreas.dilger@intel.com> wrote:
On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
<mailto:ahkumar@mail.smu.edu> wrote:
I Just saw your message. Even though I have it deactivated, something is
still writing significantly. Hope you can help me understand this better.
Even though new objects are not assigned can the disk usage increase for
the existing objects?
On the other hand, my migration script is not able to locate any files on
the OST to move, which is odd. One thing I know is some files names are
very odd, like may be 20-30 spaces in a file of length about at least
100 characters. Some users program is fanatic in file name creation and I
have no control over it.
Hope you can shed some more light. I am planning on running fsck to
identify any issues with the ost.
I don't think fsck is needed. It seems like you have some process on a
client that
is writing to an open-unlinked file on those OSTs.
To find out which client, run on the affected OSS:
# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
ost.OSS.ost_io.req_history=
1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Co
m
plete:1382050012:0s(-43s) opc 4
1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Co
m
plete:1382050012:0s(-43s) opc 4
This will print out all clients (NID "192.168.20.159@tcp" in this case)
that
are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to
see
what processes have files open on the filesystem (probably marked
"(deleted)")
and kill those processes.
This should immediately free up all of the space on those OSTs.
Cheers, Andreas
-----Original Message-----
From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
Sent: Thursday, October 10, 2013 3:14 PM
To: Kumar, Amit
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] OST deactivated but still objects being
written to it?
Hi,
temp. deactivated OSTs via 'lctl --device devno deactivate' will be
marked as inactive. No new objects are assigned to the deactivated OST,
but reads and writes of existing objects are still no problem.
Lustre_Manual Chapter: Removing an OST from the File System Hope that
helps
- --
"la lykken gro, som gresset bak do"
Nico Budewitz
High Performance Computing
Max-Planck-Institute for Gravitational Physics /
Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
http://supercomputers.aei.mpg.de
On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
<mailto:ahkumar@mail.smu.edu> wrote:
Dear All,
I have deactivated an OST yet I see the size of the OST is increasing
constantly?
Any thoughts what could be causing this?
Thank you,
Amit
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
9:35 p.m.
Thx a lot Andreas.
Pascal.
On 18/10/2013 11:46, Dilger, Andreas wrote:
On 2013/10/18 1:53 AM, "Pascal Decourt"
<pdecourt(a)sgi.com> wrote:
> Very interesting.
> I run the lctl command and here is the output:
>
>
> 65838502:10.149.254.2@o2ib:12345-10.149.1.171@o2ib:x1448244599539665:448:I
> nterpret:1382081942:-1382081942s(-1382081952s) opc 4
The size of the history buffer is controlled via the
"req_buffer_history_max"
tunable. By default it only has a few entries in it.
> For my understanding, could you please give more details on the field of
> this line ?
Fields are separated by colons.
1st field: a sequence counter, so it is possible to determine if a
record has already been processed.
2nd field: target NID (i.e. server)
3rd field: source NID (i.e. client)
4th field: client RPC XID number (per-client unique identifier)
5th field: RPC request buffer size
6th field: state of the RPC processing
7th field: time (unix epoch) when RPC arrived
8th field: total processing time (seconds)
9th field: time before client would have timed out request
10th field: RPC opcode number (see lustre_idl.h)
This is all described in the user manual if you search for req_history:
https://wiki.hpdd.intel.com/display/PUB/Documentation
There is also a tool "lustre_req_history" that processes these files a bit.
Cheers, Andreas
> On 18/10/2013 07:14, Kumar, Amit wrote:
>
>
> Awesome Andreas sounds promising, I will give this a shot in the
> morning....
> Regards
> Amit
>
> Sent from my iPhone
>
> On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas"
<andreas.dilger(a)intel.com>
> <mailto:andreas.dilger@intel.com> wrote:
>
> On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
> <mailto:ahkumar@mail.smu.edu> wrote:
> I Just saw your message. Even though I have it deactivated, something is
> still writing significantly. Hope you can help me understand this better.
> Even though new objects are not assigned can the disk usage increase for
> the existing objects?
>
> On the other hand, my migration script is not able to locate any files on
> the OST to move, which is odd. One thing I know is some files names are
> very odd, like may be 20-30 spaces in a file of length about at least
> 100 characters. Some users program is fanatic in file name creation and I
> have no control over it.
>
> Hope you can shed some more light. I am planning on running fsck to
> identify any issues with the ost.
>
>
> I don't think fsck is needed. It seems like you have some process on a
> client that
> is writing to an open-unlinked file on those OSTs.
>
> To find out which client, run on the affected OSS:
> # lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
> ost.OSS.ost_io.req_history=
> 1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Co
> m
> plete:1382050012:0s(-43s) opc 4
> 1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Co
> m
> plete:1382050012:0s(-43s) opc 4
>
> This will print out all clients (NID "192.168.20.159@tcp" in this case)
> that
> are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to
> see
> what processes have files open on the filesystem (probably marked
> "(deleted)")
> and kill those processes.
>
> This should immediately free up all of the space on those OSTs.
>
> Cheers, Andreas
>
> -----Original Message-----
> From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
> Sent: Thursday, October 10, 2013 3:14 PM
> To: Kumar, Amit
> Cc: hpdd-discuss(a)lists.01.org
> Subject: Re: [HPDD-discuss] OST deactivated but still objects being
> written to it?
>
> Hi,
>
> temp. deactivated OSTs via 'lctl --device devno deactivate' will be
> marked as inactive. No new objects are assigned to the deactivated OST,
> but reads and writes of existing objects are still no problem.
>
> Lustre_Manual Chapter: Removing an OST from the File System Hope that
> helps
> - --
> "la lykken gro, som gresset bak do"
> Nico Budewitz
> High Performance Computing
> Max-Planck-Institute for Gravitational Physics /
> Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
> Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
> http://supercomputers.aei.mpg.de
>
> On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
> <mailto:ahkumar@mail.smu.edu> wrote:
>
> Dear All,
>
> I have deactivated an OST yet I see the size of the OST is increasing
> constantly?
>
> Any thoughts what could be causing this?
>
> Thank you,
> Amit
>
Cheers, Andreas
--
*Pascal Decourt
*System Support Engineer Europe | SGI France
+33 1 34 88 80 31 | +33 6 80 45 24 77
pdecourt(a)sgi.com
SGI
Silicon Graphics SA
21 rue Albert Calmette
78350 Jouy en Josas RCS Versailles B 344 515 291
12:45 a.m.
Andreas,
Your summary here is incredibly valuable. Although I aspire to someday know and
understand the entire lustre documentation, in the meantime this translation of fact into
application is wonderful.
Thanks for your time on this list.
John Richards
john.richards(a)icecube.wisc.edu
On Oct 18, 2013, at 04:46 , "Dilger, Andreas" <andreas.dilger(a)intel.com>
wrote:
On 2013/10/18 1:53 AM, "Pascal Decourt"
<pdecourt(a)sgi.com> wrote:
> Very interesting.
> I run the lctl command and here is the output:
>
>
> 65838502:10.149.254.2@o2ib:12345-10.149.1.171@o2ib:x1448244599539665:448:I
> nterpret:1382081942:-1382081942s(-1382081952s) opc 4
The size of the history buffer is controlled via the
"req_buffer_history_max"
tunable. By default it only has a few entries in it.
> For my understanding, could you please give more details on the field of
> this line ?
Fields are separated by colons.
1st field: a sequence counter, so it is possible to determine if a
record has already been processed.
2nd field: target NID (i.e. server)
3rd field: source NID (i.e. client)
4th field: client RPC XID number (per-client unique identifier)
5th field: RPC request buffer size
6th field: state of the RPC processing
7th field: time (unix epoch) when RPC arrived
8th field: total processing time (seconds)
9th field: time before client would have timed out request
10th field: RPC opcode number (see lustre_idl.h)
This is all described in the user manual if you search for req_history:
https://wiki.hpdd.intel.com/display/PUB/Documentation
There is also a tool "lustre_req_history" that processes these files a bit.
Cheers, Andreas
> On 18/10/2013 07:14, Kumar, Amit wrote:
>
>
> Awesome Andreas sounds promising, I will give this a shot in the
> morning....
> Regards
> Amit
>
> Sent from my iPhone
>
> On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas"
<andreas.dilger(a)intel.com>
> <mailto:andreas.dilger@intel.com> wrote:
>
> On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
> <mailto:ahkumar@mail.smu.edu> wrote:
> I Just saw your message. Even though I have it deactivated, something is
> still writing significantly. Hope you can help me understand this better.
> Even though new objects are not assigned can the disk usage increase for
> the existing objects?
>
> On the other hand, my migration script is not able to locate any files on
> the OST to move, which is odd. One thing I know is some files names are
> very odd, like may be 20-30 spaces in a file of length about at least
> 100 characters. Some users program is fanatic in file name creation and I
> have no control over it.
>
> Hope you can shed some more light. I am planning on running fsck to
> identify any issues with the ost.
>
>
> I don't think fsck is needed. It seems like you have some process on a
> client that
> is writing to an open-unlinked file on those OSTs.
>
> To find out which client, run on the affected OSS:
> # lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
> ost.OSS.ost_io.req_history=
> 1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Co
> m
> plete:1382050012:0s(-43s) opc 4
> 1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Co
> m
> plete:1382050012:0s(-43s) opc 4
>
> This will print out all clients (NID "192.168.20.159@tcp" in this case)
> that
> are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to
> see
> what processes have files open on the filesystem (probably marked
> "(deleted)")
> and kill those processes.
>
> This should immediately free up all of the space on those OSTs.
>
> Cheers, Andreas
>
> -----Original Message-----
> From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
> Sent: Thursday, October 10, 2013 3:14 PM
> To: Kumar, Amit
> Cc: hpdd-discuss(a)lists.01.org
> Subject: Re: [HPDD-discuss] OST deactivated but still objects being
> written to it?
>
> Hi,
>
> temp. deactivated OSTs via 'lctl --device devno deactivate' will be
> marked as inactive. No new objects are assigned to the deactivated OST,
> but reads and writes of existing objects are still no problem.
>
> Lustre_Manual Chapter: Removing an OST from the File System Hope that
> helps
> - --
> "la lykken gro, som gresset bak do"
> Nico Budewitz
> High Performance Computing
> Max-Planck-Institute for Gravitational Physics /
> Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
> Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
> http://supercomputers.aei.mpg.de
>
> On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu>
> <mailto:ahkumar@mail.smu.edu> wrote:
>
> Dear All,
>
> I have deactivated an OST yet I see the size of the OST is increasing
> constantly?
>
> Any thoughts what could be causing this?
>
> Thank you,
> Amit
>
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2:05 a.m.
Good Morning Andreas,
Very valuable insight one gets with this command.
I see quite a few of these requests coming in and completing ever few seconds. So this
confirms something is still writing to it based on the opc 4 = OST_WRITE (I just found out
from one of your older posts :) )
And this is not just one client, many clients are writing to it. I have quite a number of
jobs running for weeks now.
I also looked at the clients and since they are all running several jobs there are over
2500 files open from /luster, and I could not find any deleted files from lsof's
output, except on local file system.
My guess is that there are active jobs in our cluster that still have a file handle open
with objects on OST's before I deactivated them? Is this possible? Any thoughts?
.....
[root@diskarray6 ~]# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
[root@diskarray6 ~]# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
1222327203:10.1.1.59@tcp:12345-10.1.11.54@tcp:x1421713825905502:448:Complete:1382101014:0s(-6s)
opc 4
1222327204:10.1.1.59@tcp:12345-10.1.13.17@tcp:x1421714339953255:448:Complete:1382101014:0s(-6s)
opc 4
1222327205:10.1.1.59@tcp:12345-10.1.13.32@tcp:x1431400659252398:448:Complete:1382101014:0s(-6s)
opc 4
[root@diskarray6 ~]# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
[root@diskarray6 ~]# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
1222327210:10.1.1.59@tcp:12345-10.1.11.58@tcp:x1421713794727740:448:Complete:1382101019:0s(-6s)
opc 4
1222327211:10.1.1.59@tcp:12345-10.1.11.77@tcp:x1421713757657163:448:Complete:1382101019:0s(-6s)
opc 4
1222327212:10.1.1.59@tcp:12345-10.1.13.43@tcp:x1421714872793796:448:Complete:1382101019:0s(-6s)
opc 4
1222327213:10.1.1.59@tcp:12345-10.1.11.49@tcp:x1421713827559437:448:Complete:1382101019:0s(-6s)
opc 4
1222327214:10.1.1.59@tcp:12345-10.1.13.34@tcp:x1421714086296893:448:Complete:1382101020:0s(-6s)
opc 4
.....
Best Regards,
Amit
From: hpdd-discuss-bounces(a)lists.01.org [mailto:hpdd-discuss-bounces@lists.01.org] On
Behalf Of Pascal Decourt
Sent: Friday, October 18, 2013 5:35 AM
To: Dilger, Andreas
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] OST deactivated but still objects being written to it?
Thx a lot Andreas.
Pascal.
On 18/10/2013 11:46, Dilger, Andreas wrote:
On 2013/10/18 1:53 AM, "Pascal Decourt"
<pdecourt@sgi.com><mailto:pdecourt@sgi.com> wrote:
Very interesting.
I run the lctl command and here is the output:
65838502:10.149.254.2@o2ib:12345-10.149.1.171@o2ib:x1448244599539665:448:I<mailto:65838502:10.149.254.2@o2ib:12345-10.149.1.171@o2ib:x1448244599539665:448:I>
nterpret:1382081942:-1382081942s(-1382081952s) opc 4
The size of the history buffer is controlled via the
"req_buffer_history_max"
tunable. By default it only has a few entries in it.
For my understanding, could you please give more details on the field of
this line ?
Fields are separated by colons.
1st field: a sequence counter, so it is possible to determine if a
record has already been processed.
2nd field: target NID (i.e. server)
3rd field: source NID (i.e. client)
4th field: client RPC XID number (per-client unique identifier)
5th field: RPC request buffer size
6th field: state of the RPC processing
7th field: time (unix epoch) when RPC arrived
8th field: total processing time (seconds)
9th field: time before client would have timed out request
10th field: RPC opcode number (see lustre_idl.h)
This is all described in the user manual if you search for req_history:
https://wiki.hpdd.intel.com/display/PUB/Documentation
There is also a tool "lustre_req_history" that processes these files a bit.
Cheers, Andreas
On 18/10/2013 07:14, Kumar, Amit wrote:
Awesome Andreas sounds promising, I will give this a shot in the
morning....
Regards
Amit
Sent from my iPhone
On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas"
<andreas.dilger@intel.com><mailto:andreas.dilger@intel.com>
<mailto:andreas.dilger@intel.com><mailto:andreas.dilger@intel.com> wrote:
On 2013/10/14 11:56 AM, "Kumar, Amit"
<ahkumar@mail.smu.edu><mailto:ahkumar@mail.smu.edu>
<mailto:ahkumar@mail.smu.edu><mailto:ahkumar@mail.smu.edu> wrote:
I Just saw your message. Even though I have it deactivated, something is
still writing significantly. Hope you can help me understand this better.
Even though new objects are not assigned can the disk usage increase for
the existing objects?
On the other hand, my migration script is not able to locate any files on
the OST to move, which is odd. One thing I know is some files names are
very odd, like may be 20-30 spaces in a file of length about at least
100 characters. Some users program is fanatic in file name creation and I
have no control over it.
Hope you can shed some more light. I am planning on running fsck to
identify any issues with the ost.
I don't think fsck is needed. It seems like you have some process on a
client that
is writing to an open-unlinked file on those OSTs.
To find out which client, run on the affected OSS:
# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
ost.OSS.ost_io.req_history=
1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Co<mailto:1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Co>
m
plete:1382050012:0s(-43s) opc 4
1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Co<mailto:1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Co>
m
plete:1382050012:0s(-43s) opc 4
This will print out all clients (NID
"192.168.20.159@tcp<mailto:192.168.20.159@tcp>" in this case)
that
are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to
see
what processes have files open on the filesystem (probably marked
"(deleted)")
and kill those processes.
This should immediately free up all of the space on those OSTs.
Cheers, Andreas
-----Original Message-----
From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
Sent: Thursday, October 10, 2013 3:14 PM
To: Kumar, Amit
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] OST deactivated but still objects being
written to it?
Hi,
temp. deactivated OSTs via 'lctl --device devno deactivate' will be
marked as inactive. No new objects are assigned to the deactivated OST,
but reads and writes of existing objects are still no problem.
Lustre_Manual Chapter: Removing an OST from the File System Hope that
helps
- --
"la lykken gro, som gresset bak do"
Nico Budewitz
High Performance Computing
Max-Planck-Institute for Gravitational Physics /
Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
http://supercomputers.aei.mpg.de
On Oct 10, 2013, at 10:01 PM, "Kumar, Amit"
<ahkumar@mail.smu.edu><mailto:ahkumar@mail.smu.edu>
<mailto:ahkumar@mail.smu.edu><mailto:ahkumar@mail.smu.edu> wrote:
Dear All,
I have deactivated an OST yet I see the size of the OST is increasing
constantly?
Any thoughts what could be causing this?
Thank you,
Amit
Cheers, Andreas
--
Pascal Decourt
System Support Engineer Europe | SGI France
+33 1 34 88 80 31 | +33 6 80 45 24 77
pdecourt@sgi.com<mailto:pdecourt@sgi.com>
[SGI]
Silicon Graphics SA
21 rue Albert Calmette
78350 Jouy en Josas RCS Versailles B 344 515 291
3257
days inactive
3265
days old
9 comments
5 participants
participants (5)
-
Dilger, Andreas -
John Richards -
Kumar, Amit -
Nico Budewitz -
Pascal Decourt