Re: [HPDD-discuss] OST deactivated but still objects being written to it?

Awesome Andreas sounds promising, I will give this a shot in the morning.... Regards Amit Sent from my iPhone On Oct 17, 2013, at 6:19 PM, "Dilger, Andreas" <andreas.dilger(a)intel.com&gt; wrote: > On 2013/10/14 11:56 AM, "Kumar, Amit" <ahkumar(a)mail.smu.edu&gt; wrote: >> I Just saw your message. Even though I have it deactivated, something is >> still writing significantly. Hope you can help me understand this better. >> Even though new objects are not assigned can the disk usage increase for >> the existing objects? >> >> On the other hand, my migration script is not able to locate any files on >> the OST to move, which is odd. One thing I know is some files names are >> very odd, like may be 20-30 spaces in a file of length about at least >> 100 characters. Some users program is fanatic in file name creation and I >> have no control over it. >> >> Hope you can shed some more light. I am planning on running fsck to >> identify any issues with the ost. > I don't think fsck is needed. It seems like you have some process on a > client that > is writing to an open-unlinked file on those OSTs. > > To find out which client, run on the affected OSS: > # lctl get_param ost.OSS.ost_io.req_history | grep "opc 4" > ost.OSS.ost_io.req_history= > 1431689:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689984:488:Com > plete:1382050012:0s(-43s) opc 4 > 1431690:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1448908035689988:488:Com > plete:1382050012:0s(-43s) opc 4 > > This will print out all clients (NID "192.168.20.1@tcp" in this case) that > are writing. On the client(s), use "lsof | grep /lustre/mountpoint" to see > what processes have files open on the filesystem (probably marked > "(deleted)") > and kill those processes. > > This should immediately free up all of the space on those OSTs. > > Cheers, Andreas > >> -----Original Message----- >> From: Nico Budewitz [mailto:Nico.Budewitz@aei.mpg.de] >> Sent: Thursday, October 10, 2013 3:14 PM >> To: Kumar, Amit >> Cc: hpdd-discuss(a)lists.01.org >> Subject: Re: [HPDD-discuss] OST deactivated but still objects being >> written to it? >> >> Hi, >> >> temp. deactivated OSTs via 'lctl --device devno deactivate' will be >> marked as inactive. No new objects are assigned to the deactivated OST, >> but reads and writes of existing objects are still no problem. >> >> Lustre_Manual Chapter: Removing an OST from the File System Hope that >> helps >> - -- >> "la lykken gro, som gresset bak do" >> Nico Budewitz >> High Performance Computing >> Max-Planck-Institute for Gravitational Physics / >> Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm >> Tel.: +49 (0)331 567 7364 Fax: +49 (0)331 567 7284 >> http://supercomputers.aei.mpg.de >> >> On Oct 10, 2013, at 10:01 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu&gt; wrote: >> >>> Dear All, >>> >>> I have deactivated an OST yet I see the size of the OST is increasing >>> constantly? >>> >>> Any thoughts what could be causing this? >>> >>> Thank you, >>> Amit >>> >>> _______________________________________________ >>> HPDD-discuss mailing list >>> HPDD-discuss(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/hpdd-discuss >> _______________________________________________ >> HPDD-discuss mailing list >> HPDD-discuss(a)lists.01.org >> https://lists.01.org/mailman/listinfo/hpdd-discuss >> > > Cheers, Andreas > -- > Andreas Dilger > > Lustre Software Architect > Intel High Performance Data Division > > _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss