[HPDD-discuss] The ost_connect operation failed with -30

Hi, I am having problems installing a new system with Lustre 2.1.6 client talking to Lustre 2.1.4 servers. Dmesg on the client says: LustreError: 11-0: an error occurred while communicating with 192.168.2.5@o2ib. The ost_connect operation failed with -30 LustreError: 11-0: an error occurred while communicating with 192.168.2.3@o2ib. The ost_connect operation failed with -30 It sees that two OST's are involved, one on "192.168.2.3" and one on "192.168.2.5". There are a total of 16 OST's in the setup. Dmesg on the MDS says: ………. LustreError: 10713:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x3582 sub-object on OST idx 9/1: rc = -30 LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x46fd sub-object on OST idx 9/1: rc = -30 LustreError: 24530:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x49a2 sub-object on OST idx 9/1: rc = -30 LustreError: 21647:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x5bb7 sub-object on OST idx 9/1: rc = -30 LustreError: 21647:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x5ddc sub-object on OST idx 9/1: rc = -30 LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x6c0b sub-object on OST idx 9/1: rc = -30 LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x7f87 sub-object on OST idx 9/1: rc = -30 LustreError: 24532:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x8237 sub-object on OST idx 9/1: rc = -30 LustreError: 24528:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x981b sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xaddd sub-object on OST idx 9/1: rc = -30 LustreError: 24532:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xc419 sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xc8db sub-object on OST idx 9/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xd4cc sub-object on OST idx 9/1: rc = -30 LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xd720 sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xe7b3 sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xea01 sub-object on OST idx 9/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xfd5f sub-object on OST idx 9/1: rc = -30 LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x10cb1 sub-object on OST idx 9/1: rc = -30 LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x10f16 sub-object on OST idx 21/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x10f17 sub-object on OST idx 9/1: rc = -30 LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x124fe sub-object on OST idx 9/1: rc = -30 LustreError: 10713:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1273a sub-object on OST idx 9/1: rc = -30 LustreError: 24528:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1298b sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1376f sub-object on OST idx 9/1: rc = -30 LustreError: 17092:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x139e0 sub-object on OST idx 9/1: rc = -30 LustreError: 27480:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x13c3e sub-object on OST idx 9/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x15474 sub-object on OST idx 9/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x15681 sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x169b5 sub-object on OST idx 9/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x16bf4 sub-object on OST idx 9/1: rc = -30 LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x16e4f sub-object on OST idx 9/1: rc = -30 LustreError: 31484:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x17f08 sub-object on OST idx 9/1: rc = -30 LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1814e sub-object on OST idx 9/1: rc = -30 …………….. Dmesg on one of the OSS’es says: …………….. Lustre: 2053:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages LustreError: 2036:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 2036:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) Skipped 51 previous similar messages LustreError: 2040:0:(filter.c:383:filter_client_add()) unable to start transaction: rc -30 LustreError: 2040:0:(filter.c:383:filter_client_add()) Skipped 48 previous similar messages LustreError: 2040:0:(filter.c:407:filter_client_add()) error writing last_rcvd client idx 32: rc -30 LustreError: 2040:0:(filter.c:407:filter_client_add()) Skipped 47 previous similar messages Lustre: 2091:0:(ldlm_lib.c:952:target_handle_connect()) mtlustre-OST0009: connection from 24a25fe2-bb1f-f864-780f-842270fddd12@192.168.1.102@o2ib t0 exp (null) cur 1389788167 last 0 Lustre: 2091:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages LustreError: 2142:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 2142:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) Skipped 49 previous similar messages LustreError: 2106:0:(filter.c:383:filter_client_add()) unable to start transaction: rc -30 LustreError: 2106:0:(filter.c:383:filter_client_add()) Skipped 48 previous similar messages LustreError: 2106:0:(filter.c:407:filter_client_add()) error writing last_rcvd client idx 32: rc -30 LustreError: 2106:0:(filter.c:407:filter_client_add()) Skipped 48 previous similar messages Lustre: 2028:0:(ldlm_lib.c:952:target_handle_connect()) mtlustre-OST0009: connection from 930bfd88-c41e-c38c-567f-6e326fb7680a@192.168.0.16@o2ib t0 exp (null) cur 1389788779 last 0 Lustre: 2028:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages LustreError: 2106:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30 LustreError: 2106:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) Skipped 48 previous similar messages ……………… Here is some more info: Lustre server: # rpm -qa | grep lustre kernel-2.6.32-279.14.1.el6_lustre.x86_64 lustre-tests-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64 kernel-firmware-2.6.32-279.14.1.el6_lustre.x86_64 lustre-ldiskfs-3.3.0-2.6.32_279.14.1.el6_lustre.x86_64.x86_64 lustre-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64 lustre-modules-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64 Lustre client: # rpm -qa | grep lustre lustre-client-modules-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64 lustre-client-tests-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64 lustre-client-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64 [root@ansjos ~]# uname -a Linux xxxx.yyy.zz 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux On the MDS: lctl dl 0 UP mgc MGC192.168.3.1@o2ib eb992c1c-b363-a877-ab65-88c804d08706 5 1 UP lov mtlustre-clilov-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 4 2 UP lmv mtlustre-clilmv-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 4 3 UP mdc mtlustre-MDT0000-mdc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 4 UP osc mtlustre-OST0000-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 5 UP osc mtlustre-OST0001-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 6 UP osc mtlustre-OST0002-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 7 UP osc mtlustre-OST0003-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 8 UP osc mtlustre-OST0004-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 9 UP osc mtlustre-OST0005-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 10 UP osc mtlustre-OST0006-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 11 UP osc mtlustre-OST0007-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 12 UP osc mtlustre-OST0008-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 13 UP osc mtlustre-OST0009-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 14 UP osc mtlustre-OST000a-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 15 UP osc mtlustre-OST000b-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 16 UP osc mtlustre-OST000c-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 17 UP osc mtlustre-OST000d-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 18 UP osc mtlustre-OST000e-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 19 UP osc mtlustre-OST000f-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 20 UP osc mtlustre-OST0010-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 21 UP osc mtlustre-OST0011-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 22 UP osc mtlustre-OST0012-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 23 UP osc mtlustre-OST0013-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 24 UP osc mtlustre-OST0014-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 25 UP osc mtlustre-OST0015-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5 On "192.168.2.3": lctl dl 0 UP mgc MGC192.168.3.1@o2ib fe9a97e3-5101-d1cb-95a6-6e304ba4f4e3 5 1 UP ost OSS OSS_uuid 3 2 UP obdfilter mtlustre-OST0007 mtlustre-OST0007_UUID 75 3 UP obdfilter mtlustre-OST0011 mtlustre-OST0011_UUID 75 4 UP obdfilter mtlustre-OST0008 mtlustre-OST0008_UUID 75 5 UP obdfilter mtlustre-OST0009 mtlustre-OST0009_UUID 71 On "192.168.2.5": lctl dl 0 UP mgc MGC192.168.3.1@o2ib e6602327-9396-962d-3c82-5e2770358050 5 1 UP ost OSS OSS_uuid 3 2 UP obdfilter mtlustre-OST000d mtlustre-OST000d_UUID 75 3 UP obdfilter mtlustre-OST0015 mtlustre-OST0015_UUID 71 4 UP obdfilter mtlustre-OST0013 mtlustre-OST0013_UUID 75 5 UP obdfilter mtlustre-OST000e mtlustre-OST000e_UUID 75 6 UP obdfilter mtlustre-OST000f mtlustre-OST000f_UUID 75 Any clues? What to look for? Best regards, Hallstein Lohre Alpha System AS