I have used lfs migrate --block in the past to move files that were in use. It works for files that are being read, but will fail with a message if they are being written to (as expected). Seems to be safe for files in either case. I'd limit the # of migrates per OST to a traditional number, optimally 1, but 2 or 4 at most. I generally used it when trying to increase the stripe width on a file that was being read heavily, and it's a great tool for that. It seems to do a pretty slow copy, single threaded, so it's not the fastest method. I've written scripts that will launch a job per file and use this to rebalance file systems, and it seemed to work well for a decent set of data, but not nearly 500 TB. For that amount of data you might want to consider weighing the costs of a lfs migrate (time spent burning up compute nodes/data mover nodes) vs. the time you'd spend with a high speed parallel copy (dcp) + a resync at a downtime. I assume you don't have a purge policy where you could just let the file system rebalance naturally over time? Shawn On Thu, 2014-09-04 at 12:59 -0400, Robin Humble wrote:

On 2014/09/04, 10:59 AM, "Robin Humble" <rjh+lustre(a)cita.utoronto.ca&gt; wrote: Note that there were some bugs in "lfs migrate" that leaked inodes on the MDS (see LU-3969) that I doubt is fixed in the version you have. Note that the "lfs_migrate" script does a before and after checksum pass, but "lfs migrate" (which it calls internally on systems that support it) does not. I agree that the single-threaded copy in lfs_migrate may be sub-optimal performance wise, so if people are interested to improve this it would be great if they did it in a manner that could be included back into lfs_migrate. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division