1:14 a.m.
Well the bottleneck was Adaptec Controller. It was not able to write as
quick as possible. My cpu on brick was under the heavy load.
Solution probably will be to replace single controller to multiple.
a.
On Thu, Aug 1, 2013 at 12:16 PM, Arman Khalatyan <arm2arm(a)gmail.com> wrote:
Hello,
I am little bit puzzled my Reading is faster than Writing, should not be
other way around? Writing on single OST is about 400MiB/s using Lustre.
If I write directly to raid it brings 1200MiB/s and random 800MiB/s
Could you please advise where to dig or tuneup?
Thanks in advance
Arman.
tuned-admin
My recent IOR test shows:
mpirun -np 4 src/ior -a POSIX -r -w -b 32g -t 1m -o
/lustre/arm2arm/IORFile -O lustreStripeCount=-1 -v -i 3
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test#
#Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize
aggsize API RefNum
write 343.27 177.23 287.06 77.67 502.04247 0 4 4 3 0
0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0
read 788.68 768.17 776.53 8.79 168.81345 0 4 4 3 0
0 1 0 0 1 34359738368 1048576 137438953472 POSIX 0
the mds-survey shows good performance:
thrhi=16 dir_count=16 file_count=200000 mds-survey
Thu Aug 1 12:05:12 CEST 2013 /usr/bin/mds-survey from araid062
mdt 1 file 200000 dir 16 thr 16 create 33506.27 [25998.91,36998.41]
lookup 790273.32 [790273.32,790273.32] md_getattr 474972.51
[474972.51,474972.51] setxattr 11483.39 [ 0.00,21999.03] destroy 73063.39
[63998.59,63998.59]
done!
My hardware is following:
Raid box has a 24 SATA Disks(7200rpm with adaptec 6805controller raid6)
,DDR IB, 32GB Ram, 8core CPU.
RaidBox1: mds/mdt+oss2
drbd |
RaidBox2: mds/mdt(slave)+oss1
Each box has a sigle port DDR IBs.
oss1, oss2- 34T - used 75%
mdt: 92G - used 3%
Software SL6.4, Lustre 2.4
kernel 2.6.32-358.6.2
tuned-adm active
Current active profile: latency-performance
Service tuned: enabled, running
Service ktune: enabled, running