[HPDD-discuss] OST Tuning sgpdd-survey question

Dear All, I read the following note on tuning storage devices from: http://wiki.lustre.org/manual/LustreManual20_HTML/BenchmarkingTests.html <snip> 24.2.1 Tuning Linux Storage Devices To get large I/O transfers (1 MB) to disk, it may be necessary to tune several kernel parameters as specified: /sys/block/sdN/queue/max_sectors_kb = 4096 /sys/block/sdN/queue/max_phys_segments = 256 /proc/scsi/sg/allow_dio = 1 /sys/module/ib_srp/parameters/srp_sg_tablesize = 255 /sys/block/sdN/queue/scheduler </snip> I have a question to help me understand the performance numbers I notice below: Setup: LUN = (8 data disks + 2 Parity) Block size on RAID format is 4096; Storage array DDN; Q1. When a LUN which is formatted with 4096 block size, And max_sectors_kb set to 4096 on the host side, why do I see minor poorer performance as seen below between the default value and the recommended value to align with the IO size and stripe width. I was expecting it to be the other way around as I have taken care of formatting the OSTs as per <stripe_width_blocks> = <chunk_blocks> * <number_of_data_disks> = 1 MB In my case: -E stride=32,stripe_width=256 Although I am not sure if I am comparing appels to appels, because sgpdd runs on the scsi device and I am not sure if the ldiskfs does some magic in between before the IO goes to the scsi device to achieve better performance. BTW I am not seeing any performance issue but I was taking time to tune on the outset if it made sense to do so. Any thoughts on this will be very helpful. Thank you, Amit # cat /tmp/*.summary with: /sys/block/sdN/queue/max_sectors_kb = 4096 Tue Aug 12 06:52:36 CDT 2014 sgpdd-survey on /dev/sdb dev 1 sz 8388608K rsz 1024K crg 1 thr 1 write 115.21 MB/s 1 x 115.21 = 115.21 MB/s read 582.06 MB/s 1 x 582.28 = 582.28 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 2 write 235.84 MB/s 1 x 235.88 = 235.88 MB/s read 838.81 MB/s 1 x 839.26 = 839.26 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 4 write 418.66 MB/s 1 x 418.78 = 418.78 MB/s read 1417.23 MB/s 1 x 1418.61 = 1418.61 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 8 write 902.29 MB/s 1 x 902.84 = 902.84 MB/s read 1439.37 MB/s 1 x 1440.78 = 1440.78 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 16 write 1386.26 MB/s 1 x 1387.55 = 1387.55 MB/s read 1440.14 MB/s 1 x 1441.51 = 1441.51 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 2 write 156.03 MB/s 2 x 78.02 = 156.04 MB/s read 807.73 MB/s 2 x 404.05 = 808.11 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 4 write 297.36 MB/s 2 x 148.71 = 297.41 MB/s read 1120.12 MB/s 2 x 560.50 = 1121.01 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 8 write 528.16 MB/s 2 x 264.17 = 528.34 MB/s read 1188.89 MB/s 2 x 594.92 = 1189.84 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 16 write 817.80 MB/s 2 x 409.13 = 818.25 MB/s read 1293.56 MB/s 2 x 647.37 = 1294.75 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 32 write 789.20 MB/s 2 x 394.81 = 789.62 MB/s read 1295.69 MB/s 2 x 648.41 = 1296.83 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 4 write 162.58 MB/s 4 x 40.65 = 162.58 MB/s read 1099.55 MB/s 4 x 275.08 = 1100.31 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 8 write 314.89 MB/s 4 x 78.74 = 314.94 MB/s read 1139.88 MB/s 4 x 285.21 = 1140.82 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 16 write 586.44 MB/s 4 x 146.67 = 586.66 MB/s read 1162.53 MB/s 4 x 290.93 = 1163.71 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 32 write 550.92 MB/s 4 x 137.78 = 551.11 MB/s read 1172.37 MB/s 4 x 293.32 = 1173.29 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 64 write 563.08 MB/s 4 x 140.83 = 563.32 MB/s read 1164.33 MB/s 4 x 291.31 = 1165.24 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 8 write 165.33 MB/s 8 x 20.67 = 165.33 MB/s read 973.23 MB/s 8 x 121.74 = 973.89 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 16 write 324.42 MB/s 8 x 40.56 = 324.48 MB/s read 960.74 MB/s 8 x 120.17 = 961.38 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 32 write 461.81 MB/s 8 x 57.74 = 461.96 MB/s read 1145.76 MB/s 8 x 143.33 = 1146.62 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 64 write 494.97 MB/s 8 x 61.89 = 495.15 MB/s read 1083.78 MB/s 8 x 135.57 = 1084.59 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 128 write 486.25 MB/s 8 x 60.80 = 486.37 MB/s read 1121.30 MB/s 8 x 140.27 = 1122.13 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 16 write 179.30 MB/s 16 x 11.21 = 179.29 MB/s read 952.32 MB/s 16 x 59.57 = 953.06 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 32 write 282.92 MB/s 16 x 17.69 = 283.05 MB/s read 1025.33 MB/s 16 x 64.13 = 1026.15 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 64 write 376.39 MB/s 16 x 23.53 = 376.43 MB/s read 1030.90 MB/s 16 x 64.48 = 1031.65 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 128 write 397.60 MB/s 16 x 24.86 = 397.80 MB/s read 1113.20 MB/s 16 x 69.64 = 1114.20 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 256 write 434.59 MB/s 16 x 27.17 = 434.72 MB/s read 1049.02 MB/s 16 x 65.61 = 1049.80 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 32 write 171.02 MB/s 32 x 5.34 = 170.90 MB/s read 902.83 MB/s 32 x 28.24 = 903.63 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 64 write 300.66 MB/s 32 x 9.39 = 300.60 MB/s read 966.62 MB/s 32 x 30.23 = 967.41 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 128 write 378.49 MB/s 32 x 11.84 = 378.72 MB/s read 967.73 MB/s 32 x 30.27 = 968.63 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 256 write 385.06 MB/s 32 x 12.04 = 385.13 MB/s read 1067.40 MB/s 32 x 33.39 = 1068.42 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 512 write 455.00 MB/s 32 x 14.23 = 455.32 MB/s read 1017.93 MB/s 32 x 31.84 = 1018.98 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 64 write 199.82 MB/s 64 x 3.12 = 199.58 MB/s read 887.81 MB/s 64 x 13.89 = 888.67 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 128 write 296.58 MB/s 64 x 4.63 = 296.63 MB/s read 925.64 MB/s 64 x 14.48 = 926.51 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 256 write 402.48 MB/s 64 x 6.29 = 402.83 MB/s read 937.37 MB/s 64 x 14.66 = 938.11 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 512 write 427.50 MB/s 64 x 6.69 = 427.86 MB/s read 966.39 MB/s 64 x 15.12 = 967.41 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 1024 write 428.08 MB/s 64 x 6.69 = 428.47 MB/s read 946.22 MB/s 64 x 14.80 = 947.27 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 128 write 204.15 MB/s 128 x 1.59 = 203.86 MB/s read 876.41 MB/s 128 x 6.86 = 877.69 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 256 write 323.54 MB/s 128 x 2.53 = 323.49 MB/s read 911.02 MB/s 128 x 7.12 = 911.87 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 512 write 421.51 MB/s 128 x 3.29 = 421.14 MB/s read 920.23 MB/s 128 x 7.20 = 921.63 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 1024 write 442.79 MB/s 128 x 3.46 = 443.12 MB/s read 887.19 MB/s 128 x 6.93 = 887.45 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 2048 write 413.75 MB/s 128 x 3.23 = 413.82 MB/s read 849.72 MB/s 128 x 6.65 = 850.83 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 256 write 228.51 MB/s 256 x 0.90 = 229.49 MB/s read 879.51 MB/s 256 x 3.44 = 881.35 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 512 write 331.11 MB/s 256 x 1.30 = 332.03 MB/s read 878.87 MB/s 256 x 3.43 = 878.91 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 1024 write 420.35 MB/s 256 x 1.64 = 419.92 MB/s read 839.53 MB/s 256 x 3.28 = 839.84 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 2048 write 407.09 MB/s 256 x 1.59 = 407.71 MB/s read 805.58 MB/s 256 x 3.15 = 805.66 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 4096 write 423.09 MB/s 256 x 1.65 = 422.36 MB/s read 692.53 MB/s 256 x 2.71 = 693.36 MB/s # cat /root/sgpdd_runs/sdb-ost9-sas1/*.summary with /sys/block/sdN/queue/max_sectors_kb = default(32767) Tue Aug 12 06:02:51 CDT 2014 sgpdd-survey on /dev/sdb dev 1 sz 8388608K rsz 1024K crg 1 thr 1 write 115.99 MB/s 1 x 116.03 = 116.03 MB/s read 580.83 MB/s 1 x 581.04 = 581.04 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 2 write 237.90 MB/s 1 x 237.94 = 237.94 MB/s read 845.28 MB/s 1 x 845.72 = 845.72 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 4 write 389.79 MB/s 1 x 389.89 = 389.89 MB/s read 1436.95 MB/s 1 x 1438.23 = 1438.23 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 8 write 909.36 MB/s 1 x 909.88 = 909.88 MB/s read 1439.63 MB/s 1 x 1440.94 = 1440.94 MB/s dev 1 sz 8388608K rsz 1024K crg 1 thr 16 write 1390.04 MB/s 1 x 1391.24 = 1391.24 MB/s read 1440.02 MB/s 1 x 1441.32 = 1441.32 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 2 write 157.83 MB/s 2 x 78.92 = 157.83 MB/s read 812.07 MB/s 2 x 406.24 = 812.47 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 4 write 298.19 MB/s 2 x 149.13 = 298.25 MB/s read 1127.00 MB/s 2 x 563.94 = 1127.87 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 8 write 513.79 MB/s 2 x 256.99 = 513.97 MB/s read 1206.77 MB/s 2 x 603.86 = 1207.71 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 16 write 821.34 MB/s 2 x 410.90 = 821.80 MB/s read 1278.98 MB/s 2 x 640.04 = 1280.08 MB/s dev 1 sz 8388608K rsz 1024K crg 2 thr 32 write 841.46 MB/s 2 x 420.96 = 841.92 MB/s read 1288.13 MB/s 2 x 644.60 = 1289.20 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 4 write 161.84 MB/s 4 x 40.46 = 161.86 MB/s read 1113.69 MB/s 4 x 278.63 = 1114.50 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 8 write 315.10 MB/s 4 x 78.79 = 315.17 MB/s read 1137.07 MB/s 4 x 284.47 = 1137.89 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 16 write 588.28 MB/s 4 x 147.13 = 588.53 MB/s read 1165.16 MB/s 4 x 291.52 = 1166.08 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 32 write 545.81 MB/s 4 x 136.50 = 546.00 MB/s read 1150.96 MB/s 4 x 287.95 = 1151.81 MB/s dev 1 sz 8388608K rsz 1024K crg 4 thr 64 write 573.67 MB/s 4 x 143.47 = 573.88 MB/s read 1153.93 MB/s 4 x 288.71 = 1154.82 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 8 write 168.87 MB/s 8 x 21.11 = 168.91 MB/s read 946.06 MB/s 8 x 118.34 = 946.73 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 16 write 327.68 MB/s 8 x 40.97 = 327.76 MB/s read 970.74 MB/s 8 x 121.42 = 971.37 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 32 write 478.45 MB/s 8 x 59.82 = 478.59 MB/s read 1117.55 MB/s 8 x 139.80 = 1118.39 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 64 write 499.48 MB/s 8 x 62.46 = 499.65 MB/s read 1125.30 MB/s 8 x 140.77 = 1126.17 MB/s dev 1 sz 8388608K rsz 1024K crg 8 thr 128 write 468.83 MB/s 8 x 58.62 = 468.98 MB/s read 1083.02 MB/s 8 x 135.48 = 1083.83 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 16 write 200.05 MB/s 16 x 12.50 = 200.04 MB/s read 931.16 MB/s 16 x 58.24 = 931.85 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 32 write 264.30 MB/s 16 x 16.52 = 264.28 MB/s read 1025.74 MB/s 16 x 64.16 = 1026.61 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 64 write 360.96 MB/s 16 x 22.56 = 361.02 MB/s read 1024.78 MB/s 16 x 64.10 = 1025.54 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 128 write 353.03 MB/s 16 x 22.07 = 353.09 MB/s read 1046.71 MB/s 16 x 65.48 = 1047.67 MB/s dev 1 sz 8388608K rsz 1024K crg 16 thr 256 write 387.39 MB/s 16 x 24.21 = 387.42 MB/s read 1049.81 MB/s 16 x 65.66 = 1050.57 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 32 write 183.08 MB/s 32 x 5.72 = 183.11 MB/s read 898.33 MB/s 32 x 28.10 = 899.05 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 64 write 300.33 MB/s 32 x 9.38 = 300.29 MB/s read 935.15 MB/s 32 x 29.25 = 935.97 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 128 write 402.90 MB/s 32 x 12.60 = 403.14 MB/s read 994.12 MB/s 32 x 31.09 = 994.87 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 256 write 431.51 MB/s 32 x 13.49 = 431.82 MB/s read 1027.41 MB/s 32 x 32.14 = 1028.44 MB/s dev 1 sz 8388608K rsz 1024K crg 32 thr 512 write 472.92 MB/s 32 x 14.78 = 473.02 MB/s read 978.77 MB/s 32 x 30.61 = 979.61 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 64 write 174.25 MB/s 64 x 2.73 = 174.56 MB/s read 887.77 MB/s 64 x 13.89 = 888.67 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 128 write 304.43 MB/s 64 x 4.76 = 304.57 MB/s read 915.25 MB/s 64 x 14.31 = 916.14 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 256 write 411.21 MB/s 64 x 6.43 = 411.38 MB/s read 945.09 MB/s 64 x 14.78 = 946.04 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 512 write 450.88 MB/s 64 x 7.05 = 451.05 MB/s read 965.13 MB/s 64 x 15.10 = 966.19 MB/s dev 1 sz 8388608K rsz 1024K crg 64 thr 1024 write 417.15 MB/s 64 x 6.52 = 417.48 MB/s read 942.48 MB/s 64 x 14.74 = 943.60 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 128 write 225.25 MB/s 128 x 1.76 = 225.83 MB/s read 896.70 MB/s 128 x 7.01 = 897.22 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 256 write 325.95 MB/s 128 x 2.55 = 325.93 MB/s read 914.43 MB/s 128 x 7.15 = 915.53 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 512 write 437.91 MB/s 128 x 3.42 = 438.23 MB/s read 922.04 MB/s 128 x 7.21 = 922.85 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 1024 write 449.51 MB/s 128 x 3.51 = 449.22 MB/s read 902.09 MB/s 128 x 7.06 = 903.32 MB/s dev 1 sz 8388608K rsz 1024K crg 128 thr 2048 write 400.69 MB/s 128 x 3.13 = 400.39 MB/s read 825.51 MB/s 128 x 6.46 = 826.42 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 256 write 224.48 MB/s 256 x 0.88 = 224.61 MB/s read 860.48 MB/s 256 x 3.37 = 861.82 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 512 write 332.51 MB/s 256 x 1.30 = 332.03 MB/s read 887.48 MB/s 256 x 3.47 = 888.67 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 1024 write 408.58 MB/s 256 x 1.59 = 407.71 MB/s read 866.96 MB/s 256 x 3.39 = 866.70 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 2048 write 413.03 MB/s 256 x 1.61 = 412.60 MB/s read 803.31 MB/s 256 x 3.14 = 803.22 MB/s dev 1 sz 8388608K rsz 1024K crg 256 thr 4096 write 424.10 MB/s 256 x 1.66 = 424.80 MB/s read 685.05 MB/s 256 x 2.68 = 686.04 MB/s Thank you, Amit H. Kumar