Re: [HPDD-discuss] slow write performance on 2.4.1-RC1

On 2013-10-11, at 12:32, "Michael Bloom" <michael.bloom@trd2inc.com<mailto:michael.bloom@trd2inc.com>> wrote: I've modified my iozone testing for now to just "-i0", i.e., write/rewrite. Here's the command: iozone -e -M -m -r 1m -s 1g -0 -+n -+A 2 -+u -C -t -P 0 -+d -F $TESTDIR/iozone-file_1GB.a_001.$$ 1>$LOG.10gb.write_00 & where $TESTDIR points to a directory in my Lustre FS whose striping config are (the files are created new on each test run so they inherit the striping config from the directory): lfs getstripe iozone-file_1GB.a_005.28844 iozone-file_1GB.a_005.28844 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 617201 0x96af1 0 0 616584 0x96888 0 The file size isn't quite the 1GB I was looking for either: -rw-r-----. 1 root root 471859200 Oct 11 01:33 iozone-file_1GB.a_005.28844 Only reason we're running pre-release stuff is that's what we downloaded. We could upgrade but it didn't seem necessary to chase the releases. Of course, if there are bugs hampering our efforts then of course we'd upgrade. Just hasn't seem to be necessary yet. What I meant was that you are running bleeding edge code, not that it was too old. 2.4.1 is the most recent maintenance release, though 2.5.0 (the next maintenance release is a few weeks away). I ran some more experiments after I posted my question. When I saw that osc_cached_mb's associated busy_cnt was also pegged at some high number, I also noticed that max_dirty_mb was set to the 32MB default (as are most/all of our config). Thinking our write rate far exceeded the ability of 32MB to efficiently buffer the async writes, When I bumped osc_cached_mb up to 2GB(!) I no longer saw busy_cnt pegged either. However, the test's throughput was still in the 50MB/sec range. I would suspect either your OST is broken or the network. However, you wrote you got 5GB/s for reads. Was that from the OST, or might it have been from the client cache? You should unmount your broken OST and mount it with "-t ldiskfs" instead of "-t lustre" and then run iozone against it directly. If that works, you need to run LNET selftest on the network (this is described in the Lustre user manual) to see if there is a network problem. Then I noticed these in the single client's /var/log/messages: kernel: LustreError: 28855:0:(osc_request.c:854:osc_announce_cached()) dirty 0 - dirty_max 2147483648<tel:2147483648> too big??? That is a direct result of your changing osc_cached_mb and max_dirty_mb to 2GB, which is per OST, and too high. We probably should fix that limit and message, since it isn't outrageously high anymore, but in the meantime you need to make it a bit smaller ... Oct 11 01:21:07 hadoop23 kernel: LustreError: 28855:0:(osc_request.c:854:osc_announce_cached()) Skipped 21370 previous similar messages. I see that osc_announce_cached logs this message when there are too many dirty pages. Effectively, the underlying issue is still there: Either the OSS is out to lunch, not committing these writes back to the client, or they aren't being drained by the OSS. Lustre clients send write RPCs as soon as 1MB of dirty data, so it would only be stuck on the client if the OST isn't accepting it fast enough. Saw this on the associated OSS: kernel: LustreError: 20316:0:(ofd_grant.c:607:ofd_grant()) lustrewt-OST0000: client f4cc2e8f-022a-a927-e9d 1-b8540d7ad1a9/ffff880464abf800 requesting > 2GB grant 2147483648<tel:2147483648>. I don't know why the grant is so big! Same problem. Keep the clien tunables below 2GB. Is there a knob on the OSS that's equivalent to the client's osc_cached_mb? That is, is there a knob that defines the size of the OSS's buffer for receiving the client's traffic? Thanks, Michael On Fri, Oct 11, 2013 at 2:00 AM, Dilger, Andreas <andreas.dilger@intel.com<mailto:andreas.dilger@intel.com>> wrote: On 2013-10-10, at 12:16, "Michael Bloom" <michael.bloom@trd2inc.com<mailto:michael.bloom@trd2inc.com>> wrote: We've gotten multi-GB/s with 2.4, so 50 MB/s is definitely not expected. It isn't really possible to make any sensible guesses about your performance problem without knowing what kind of writes you are doing. It is also possible that your underlying storage is having problems. Did you try mounting it locally and running iozone directly? Is there any reason to be running the pre-release code on the clients? Testing is great, and the 2.5 client should have some improved performance over 2.4, but that isn't necessarily production ready yet. Cheers, Andreas