Re: [HPDD-discuss] OST deactivated but still objects being written to it?

On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu&gt; wrote: > I Just saw your message. Even though I have it deactivated, something is > still writing significantly. Hope you can help me understand this better. > Even though new objects are not assigned can the disk usage increase for > the existing objects? > > On the other hand, my migration script is not able to locate any files on > the OST to move, which is odd. One thing I know is some files names are > very odd, like may be 20-30 spaces in a file of length about at least > 100 characters. Some users program is fanatic in file name creation and I > have no control over it. > > Hope you can shed some more light. I am planning on running fsck to > identify any issues with the ost. I don't think fsck is needed. It seems like you have some process on a client that is writing to an open-unlinked file on those OSTs. To find out which client, run on the affected OSS: # lctl get_param ost.OSS.ost_io.req_history | grep "opc 4" ost.OSS.ost_io.req_history= 1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Com plete:1382050012:0s(-43s) opc 4 1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Com plete:1382050012:0s(-43s) opc 4 This will print out all clients (NID "192.168.20.1@tcp" in this case) that are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to see what processes have files open on the filesystem (probably marked "(deleted)") and kill those processes. This should immediately free up all of the space on those OSTs. Cheers, Andreas > -----Original Message----- > From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de] > Sent: Thursday, October 10, 2013 3:14 PM > To: Kumar, Amit > Cc: hpdd-discuss(a)lists.01.org > Subject: Re: [HPDD-discuss] OST deactivated but still objects being > written to it? > > Hi, > > temp. deactivated OSTs via 'lctl --device devno deactivate' will be > marked as inactive. No new objects are assigned to the deactivated OST, > but reads and writes of existing objects are still no problem. > > Lustre_Manual Chapter: Removing an OST from the File System Hope that > helps > - -- > "la lykken gro, som gresset bak do" > Nico Budewitz > High Performance Computing > Max-Planck-Institute for Gravitational Physics / > Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm > Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284 > http://supercomputers.aei.mpg.de > > On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu&gt; wrote: > >> Dear All, >> >> I have deactivated an OST yet I see the size of the OST is increasing >> constantly? >> >> Any thoughts what could be causing this? >> >> Thank you, >> Amit >> >> _______________________________________________ >> HPDD-discuss mailing list >> HPDD-discuss(a)lists.01.org >> https://lists.01.org/mailman/listinfo/hpdd-discuss > > _______________________________________________ > HPDD-discuss mailing list > HPDD-discuss(a)lists.01.org > https://lists.01.org/mailman/listinfo/hpdd-discuss > Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division