On Mon, Jul 21, 2014 at 02:15:31PM +0000, Kumar, Amit wrote: I suspect you would get ~2x the iops if you used 2 fs's and 2 MDS's. if you're already running an active/passive MDS pair then this would mean no extra hardware. cheers, robin

Dear Andreas, Rick, & Robin Thank you for your input on this. Our main motivation to go with OST pools versus multiple file system was to allow non-savvy end users to simply point to a directory and write to a faster performing disk as opposed to SATA, in the hope that everything else is aligned well for them to gain the bandwidth and performance out of our storage. If I understand this correctly our current storage setup is such that if we were to exclusively use SAS vs SATA we have enough storage/bandwidth of each type to max out the performance on our DDN controllers below the 12 OSS. Please let me know if there are any flaws in what I say. On the other hand we wanted to have an easy way to manage the entire file-system as opposed to multiple files system, and also the ability to scale across when we fill up. To add to this as you point out savvy users could use all OST's if required. A follow up question based on your input: I was hoping to use quota just so we could keep tabs on the usage: Using OST pools, do we have to rule out the possibility of using quota feature all together ? I was hoping to accomplish the following, would quota's work at any level of the hierarchy? /lustre/{sas,sata}/<user> Please advise. Thank you, Amit H. Kumar -----Original Message----- From: Dilger, Andreas [mailto:andreas.dilger@intel.com] Sent: Monday, July 21, 2014 1:01 PM To: Robin Humble; Kumar, Amit Cc: hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Lustre Pools vs Multiple Lustre File System recommendation On 2014/07/21, 11:27 AM, "Robin Humble" <rjh+lustre(a)cita.utoronto.ca&gt; wrote: That depends on where your bottleneck is. Yes, you would increase the metadata ops/second, but you would only get 1/2 of your IO bandwidth for each filesystem if you split the OSTs across two filesystems. I think the one major limitation to using OST pools today is that they are "advisory" only, and savvy users could specify any OST pool that they want. That may be ok, but it depends on your users and environment. We've discussed implementing usage quotas/limits on OST pools, but this has not been implemented yet. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division

Makes sense! I will take this as a discussion item and see what we finalize on this. Thank youall for your inputs. Amit -----Original Message----- From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu] Sent: Monday, July 21, 2014 4:15 PM To: Kumar, Amit Cc: Dilger, Andreas; Robin Humble; hpdd-discuss(a)lists.01.org Subject: Re: [HPDD-discuss] Lustre Pools vs Multiple Lustre File System recommendation On Jul 21, 2014, at 4:04 PM, "Kumar, Amit" <ahkumar(a)mail.smu.edu&gt; wrote: You can still use quotas with ost pools, but the quotas will apply to the aggregate usage across both the sas and sata pools. If you want to enforce separate quotas for sas and sata, you would need different file systems. -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu

On Jul 22, 2014, at 4:28 PM, Marcus Nilsson <marcus.nilsson(a)atea.se&gt; wrote: That is a good point. On our Lustre 1.8 file system, I have seen cases where hundreds of clients issue read requests to a single OST. When this is happening, all the OSS service threads get tied up serving requests to that one OST and the throughput to other OSTs on the same host is very low. (I don't know if the Lustre 2.x behavior is much different.) If this becomes a problem, it might be worthwhile looking into the Network Request Scheduler (NRS) feature in Lustre 2.4. This could be used to apply a policy that helps mitigate that kind of uneven behavior. -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu