Awesome Andreas sounds promising, I will give this a shot in the morning....
Regards 
Amit

Sent from my iPhone

On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas" <andreas.dilger@intel.com> wrote:

On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar@mail.smu.edu> wrote:

I Just saw your message. Even though I have it deactivated,  something is
still writing significantly. Hope you can help me understand this better.
Even though new objects are not assigned can the disk usage increase for
the existing objects?

On the other hand, my migration script is not able to locate any files on
the OST to move, which is odd. One thing I know is some files names are
very odd, like may be 20-30 spaces in a file of length about at least
100 characters. Some users program is fanatic in file name creation and I
have no control over it.

Hope you can shed some more light. I am planning on running fsck to
identify any issues with the ost.

I don't think fsck is needed.  It seems like you have some process on a
client that
is writing to an open-unlinked file on those OSTs.

To find out which client, run on the affected OSS:
# lctl get_param ost.OSS.ost_io.req_history | grep "opc 4"
ost.OSS.ost_io.req_history=
1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Com
plete:1382050012:0s(-43s) opc 4
1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Com
plete:1382050012:0s(-43s) opc 4

This will print out all clients (NID "192.168.20.1@tcp" in this case) that
are writing.  On the client(s), use "lsof | grep /lustre/mountpoint" to see
what processes have files open on the filesystem (probably marked
"(deleted)")
and kill those processes.

This should immediately free up all of the space on those OSTs.

Cheers, Andreas 

-----Original Message-----
From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de]
Sent: Thursday, October 10, 2013 3:14 PM
To: Kumar, Amit
Cc: hpdd-discuss@lists.01.org
Subject: Re: [HPDD-discuss] OST deactivated but still objects being
written to it?

Hi,

temp. deactivated OSTs via 'lctl --device devno deactivate' will be
marked as inactive. No new objects are assigned to the deactivated OST,
but reads and writes of existing objects are still no problem.

Lustre_Manual Chapter: Removing an OST from the File System Hope that
helps
- --
"la lykken gro, som gresset bak do"
Nico Budewitz
High Performance Computing
Max-Planck-Institute for Gravitational Physics /
Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm
Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284
http://supercomputers.aei.mpg.de

On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar@mail.smu.edu> wrote:

Dear All,

I have deactivated an OST yet I see the size of the OST is increasing
constantly?

Any thoughts what could be causing this?

Thank you,
Amit

_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss

_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss

Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division

_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss