Hi,
I am having problems installing a new system with Lustre 2.1.6 client talking to Lustre 2.1.4 servers.
Dmesg on the client says:
LustreError: 11-0: an error occurred while communicating with 192.168.2.5@o2ib. The ost_connect operation failed with -30
LustreError: 11-0: an error occurred while communicating with 192.168.2.3@o2ib. The ost_connect operation failed with -30
It sees that two OST's are involved, one on "192.168.2.3" and one on "192.168.2.5". There are a total of 16 OST's in the setup.
Dmesg on the MDS says:
……….
LustreError: 10713:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x3582 sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x46fd sub-object on OST idx 9/1: rc = -30
LustreError: 24530:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x49a2 sub-object on OST idx 9/1: rc = -30
LustreError: 21647:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x5bb7 sub-object on OST idx 9/1: rc = -30
LustreError: 21647:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x5ddc sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x6c0b sub-object on OST idx 9/1: rc = -30
LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x7f87 sub-object on OST idx 9/1: rc = -30
LustreError: 24532:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x8237 sub-object on OST idx 9/1: rc = -30
LustreError: 24528:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x981b sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xaddd sub-object on OST idx 9/1: rc = -30
LustreError: 24532:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xc419 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xc8db sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xd4cc sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xd720 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xe7b3 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xea01 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0xfd5f sub-object on OST idx 9/1: rc = -30
LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x10cb1 sub-object on OST idx 9/1: rc = -30
LustreError: 2089:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x10f16 sub-object on OST idx 21/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x10f17 sub-object on OST idx 9/1: rc = -30
LustreError: 17091:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x124fe sub-object on OST idx 9/1: rc = -30
LustreError: 10713:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1273a sub-object on OST idx 9/1: rc = -30
LustreError: 24528:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1298b sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1376f sub-object on OST idx 9/1: rc = -30
LustreError: 17092:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x139e0 sub-object on OST idx 9/1: rc = -30
LustreError: 27480:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x13c3e sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x15474 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x15681 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x169b5 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x16bf4 sub-object on OST idx 9/1: rc = -30
LustreError: 9040:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x16e4f sub-object on OST idx 9/1: rc = -30
LustreError: 31484:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x17f08 sub-object on OST idx 9/1: rc = -30
LustreError: 6511:0:(lov_request.c:579:lov_update_create_set()) error creating fid 0x1814e sub-object on OST idx 9/1: rc = -30
……………..
Dmesg on one of the OSS’es says:
……………..
Lustre: 2053:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages
LustreError: 2036:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 2036:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) Skipped 51 previous similar messages
LustreError: 2040:0:(filter.c:383:filter_client_add()) unable to start transaction: rc -30
LustreError: 2040:0:(filter.c:383:filter_client_add()) Skipped 48 previous similar messages
LustreError: 2040:0:(filter.c:407:filter_client_add()) error writing last_rcvd client idx 32: rc -30
LustreError: 2040:0:(filter.c:407:filter_client_add()) Skipped 47 previous similar messages
Lustre: 2091:0:(ldlm_lib.c:952:target_handle_connect()) mtlustre-OST0009: connection from 24a25fe2-bb1f-f864-780f-842270fddd12@192.168.1.102@o2ib t0 exp (null) cur 1389788167 last 0
Lustre: 2091:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages
LustreError: 2142:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 2142:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) Skipped 49 previous similar messages
LustreError: 2106:0:(filter.c:383:filter_client_add()) unable to start transaction: rc -30
LustreError: 2106:0:(filter.c:383:filter_client_add()) Skipped 48 previous similar messages
LustreError: 2106:0:(filter.c:407:filter_client_add()) error writing last_rcvd client idx 32: rc -30
LustreError: 2106:0:(filter.c:407:filter_client_add()) Skipped 48 previous similar messages
Lustre: 2028:0:(ldlm_lib.c:952:target_handle_connect()) mtlustre-OST0009: connection from 930bfd88-c41e-c38c-567f-6e326fb7680a@192.168.0.16@o2ib t0 exp (null) cur 1389788779 last 0
Lustre: 2028:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages
LustreError: 2106:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) error starting handle for op 8 (71 credits): rc -30
LustreError: 2106:0:(fsfilt-ldiskfs.c:379:fsfilt_ldiskfs_start()) Skipped 48 previous similar messages
………………
Here is some more info:
Lustre server:
# rpm -qa | grep lustre
kernel-2.6.32-279.14.1.el6_lustre.x86_64
lustre-tests-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
kernel-firmware-2.6.32-279.14.1.el6_lustre.x86_64
lustre-ldiskfs-3.3.0-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
lustre-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
lustre-modules-2.1.4-2.6.32_279.14.1.el6_lustre.x86_64.x86_64
Lustre client:
# rpm -qa | grep lustre
lustre-client-modules-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64
lustre-client-tests-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64
lustre-client-2.1.6-2.6.32_358.11.1.el6.x86_64.x86_64
[root@ansjos ~]# uname -a
Linux xxxx.yyy.zz 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
On the MDS:
lctl dl
0 UP mgc MGC192.168.3.1@o2ib eb992c1c-b363-a877-ab65-88c804d08706 5
1 UP lov mtlustre-clilov-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 4
2 UP lmv mtlustre-clilmv-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 4
3 UP mdc mtlustre-MDT0000-mdc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
4 UP osc mtlustre-OST0000-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
5 UP osc mtlustre-OST0001-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
6 UP osc mtlustre-OST0002-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
7 UP osc mtlustre-OST0003-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
8 UP osc mtlustre-OST0004-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
9 UP osc mtlustre-OST0005-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
10 UP osc mtlustre-OST0006-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
11 UP osc mtlustre-OST0007-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
12 UP osc mtlustre-OST0008-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
13 UP osc mtlustre-OST0009-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
14 UP osc mtlustre-OST000a-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
15 UP osc mtlustre-OST000b-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
16 UP osc mtlustre-OST000c-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
17 UP osc mtlustre-OST000d-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
18 UP osc mtlustre-OST000e-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
19 UP osc mtlustre-OST000f-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
20 UP osc mtlustre-OST0010-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
21 UP osc mtlustre-OST0011-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
22 UP osc mtlustre-OST0012-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
23 UP osc mtlustre-OST0013-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
24 UP osc mtlustre-OST0014-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
25 UP osc mtlustre-OST0015-osc-ffff88018fcfac00 24a25fe2-bb1f-f864-780f-842270fddd12 5
On "192.168.2.3":
lctl dl
0 UP mgc MGC192.168.3.1@o2ib fe9a97e3-5101-d1cb-95a6-6e304ba4f4e3 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter mtlustre-OST0007 mtlustre-OST0007_UUID 75
3 UP obdfilter mtlustre-OST0011 mtlustre-OST0011_UUID 75
4 UP obdfilter mtlustre-OST0008 mtlustre-OST0008_UUID 75
5 UP obdfilter mtlustre-OST0009 mtlustre-OST0009_UUID 71
On "192.168.2.5":
lctl dl
0 UP mgc MGC192.168.3.1@o2ib e6602327-9396-962d-3c82-5e2770358050 5
1 UP ost OSS OSS_uuid 3
2 UP obdfilter mtlustre-OST000d mtlustre-OST000d_UUID 75
3 UP obdfilter mtlustre-OST0015 mtlustre-OST0015_UUID 71
4 UP obdfilter mtlustre-OST0013 mtlustre-OST0013_UUID 75
5 UP obdfilter mtlustre-OST000e mtlustre-OST000e_UUID 75
6 UP obdfilter mtlustre-OST000f mtlustre-OST000f_UUID 75
Any clues? What to look for?
Best regards,
Hallstein Lohre
Alpha System AS