Re: [HPDD-discuss] Same performance Infiniband and Ethernet

Alfonso, Based on my attempts to benchmark single client Lustre performance, here is some advice/comments that I have. (YMMV) 1) On the IB client, I recommend disabling checksums (lctl set_param osc.*.checksums=0). Having checksums enabled sometimes results in a significant performance hit. 2) Single-threaded tests (like dd) will usually bottleneck before you can max out the total client performance. You need to use a multi-threaded tool (like xdd) and have several threads perform IO at the same time in order to measure aggregate single client performance. 3) When using a tool like xdd, set up the test to run for a fixed amount of time rather than having each thread write a fixed amount of data. If all threads write a fixed amount of data (say 1 GB), and if any of the threads run slower than others, you might get skewed results for the aggregate throughput because of the stragglers. 4) In order to avoid contention at the ost level among the multiple threads on a single client, precreate the output files with stripe_count=1 and statically assign them evenly to the different osts. Have each thread write to a different file so that no two processes write to the same ost. If you don't have enough osts to saturate the client, you can always have two files per ost. Going beyond that will likely hurt more than help, at least for an ldiskfs backend. 5) In my testing, I seem to get worse results using direct I/O for write tests, so I usually just use buffered I/O. Based on my understanding, the max_dirty_mb parameter on the client (which defaults to 32 MB) limits the amount of dirty written data than can be cached on each ost. Unless you have increased this to a very large number, that parameter will likely mitigate any effects of client caching on the test results. (NOTE: This reasoning only applies to write tests. Any written data can still be cached by the client, and a subsequent read test might very well pull data from cache unless you have taken steps to flush the cached data.) If you have 10 oss nodes and 20 osts in your file system, I would start by running a test with 10 threads and have each thread write to a single ost on different servers. You can increase/decrease the number of threads as needed to see if the aggregate performance gets better/worse. On my clients with QDR IB, I typically see aggregate write speeds in the range of 2.5-3.0 GB/s. You are probably already aware of this, but just in case, make sure that the IB clients you use for testing don't also have ethernet connections to your OSS servers. If the client has an ethernet and an IB path to the same server, it will choose one of the paths to use. It could end up choosing ethernet instead of IB and mess up your results. -- Rick Mohr Senior HPC System Administrator National Institute for Computational Sciences http://www.nics.tennessee.edu On May 19, 2014, at 6:33 AM, "Pardo Diaz, Alfonso" <alfonso.pardo(a)ciemat.es&gt; wrote: