[HPDD-discuss] OST not activating for write following upgrade

I am just finishing an upgrade from Lustre 1.8.4 to 2.1.6, on SL5.8 to SL6.4. All OSS, MGS and MDT machines were upgraded. Following the upgrade I find that none of the OST will change to RW state. Could I get some advice from someone on this? We have separate mgs/mdt machines, and 8 OSS with ~70 OST attached. All OSS were upgraded and the OST were mounted. This morning, I started the MGS then the MDT. The MGS log entries in /var/log/messages shown here first, followed by MDT log entries, and the "lctl dl" output on the mdt. I made no file system changes on the OSS nodes. I rebuilt the mdt to a smaller size, using the setfattr/tar method described in the Lustre 1.8 manuals, with the mkfs.lustre issued from the newly built, 2.1.6 mdt. Have I missed anything here? Can someone tell me what is wrong with my system? Thanks, bob ===================== mgs log ================== 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162354.732167] Lustre: Lustre: Build Version: RC2--PRISTINE-2.6.32-358.11.1.el6_lustre.x86_64 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162355.043533] Lustre: Added LNI 10.10.1.140@tcp [8/256/0/180] 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162355.043545] Lustre: Added LNI 141.213.133.196@tcp2 [8/256/0/180] 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162355.043593] Lustre: Accept secure, port 988 2013-07-16T08:28:16-04:00 mgs.aglt2.org kernel: [162355.398195] Lustre: Lustre OSC module (ffffffffa08cc2a0). 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.433725] Lustre: Lustre LOV module (ffffffffa094a280). 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.544341] Lustre: Lustre client module (ffffffffa0a34920). 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.557307] LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.557998] LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts: 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.590277] LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck is recommended 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.591095] LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts: 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.690541] Lustre: MGS MGS started 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.690795] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 51233eb1-32bc-cb6f-3bbe-5008265553b7@0@lo t0 exp (null) cur 1373977697 last 0 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.697639] Lustre: MGC10.10.1.140@tcp: Reactivating import 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162357.003523] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 2f75f773-faaf-d242-de3e-18443c3c45db@10.10.2.23@tcp t0 exp (null) cur 1373977697 last 0 2013-07-16T08:28:26-04:00 mgs.aglt2.org kernel: [162365.602114] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from f49a6fb6-030d-b53b-f515-6df58cceb09d@10.10.2.21@tcp t0 exp (null) cur 1373977706 last 0 2013-07-16T08:28:54-04:00 mgs.aglt2.org kernel: [162394.024874] Lustre: 438424:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from bc883142-33e6-cef3-630e-0e3322f19033@10.10.2.24@tcp t0 exp (null) cur 1373977734 last 0 [ The 10.10.1.157 was an old, 1.8.4 machine that I missed. I shut it down ] 2013-07-16T08:28:58-04:00 mgs.aglt2.org kernel: [162397.566576] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 847c3782-0835-5695-93e9-4276f7d7152a@10.10.1.157@tcp t0 exp (null) cur 1373977738 last 0 2013-07-16T08:28:58-04:00 mgs.aglt2.org kernel: [162397.566580] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message 2013-07-16T08:29:05-04:00 mgs.aglt2.org kernel: [162405.195835] Lustre: 438424:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 118c0806-972c-3e9a-5de2-8bc902817bfe@10.10.2.27@tcp t0 exp (null) cur 1373977745 last 0 2013-07-16T08:29:14-04:00 mgs.aglt2.org kernel: [162413.238112] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from dbae7713-3f95-057a-28e1-7b6519506f64@10.10.2.22@tcp t0 exp (null) cur 1373977754 last 0 [ Now I start the mdt machine, and these log ] 2013-07-16T08:46:23-04:00 mgs.aglt2.org kernel: [163439.535234] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 12f9f864-0ad6-624f-115d-8801b67ca538@10.10.1.49@tcp t0 exp (null) cur 1373978783 last 0 2013-07-16T08:46:23-04:00 mgs.aglt2.org kernel: [163439.535258] Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages 2013-07-16T08:46:23-04:00 mgs.aglt2.org kernel: [163439.590237] Lustre: 440552:0:(mgs_llog.c:249:mgs_fsdb_handler()) MDT using 1.8 OSC name scheme ======================= mdt log from /var/log/messages ================== [ First mount attempt failed, added that unsupported "-o update" I found ] 2013-07-16T08:45:43-04:00 lmd02.aglt2.org kernel: [61935.936127] Lustre: Lustre: Build Version: RC2--PRISTINE-2.6.32-358.11.1.el6_lustre.x86_64 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61936.409749] Lustre: Added L NI 10.10.1.49@tcp [8/256/0/180] 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61936.409845] Lustre: Accept secure, port 988 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61936.833435] Lustre: Lustre OSC module (ffffffffa0bd22a0). 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61937.003602] Lustre: Lustre LOV module (ffffffffa0c64280). 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.195301] Lustre: Lustre client module (ffffffffa0d4e920). 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.344529] LDISKFS-fs (dm- 7): Unrecognized mount option "upgrade" or missing value 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.360076] LustreError: 210489:0:(obd_mount.c:1360:server_kernel_mount()) premount /dev/mapper/sun-mdt3:0x0 ldiskfs failed: -22 Is the ldiskfs module available? 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.389113] LustreError: 210489:0:(obd_mount.c:1700:server_fill_super()) Unable to mount device /dev/mapper /sun-mdt3: -22 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.411161] LustreError: 210489:0:(obd_mount.c:2203:lustre_fill_super()) Unable to mount (-22) [ end failed mount attempt ] 2013-07-16T08:46:22-04:00 lmd02.aglt2.org kernel: [61974.840288] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.189666] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.239509] Lustre: MGC10.10.1.140@tcp: Reactivating import 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.466165] Lustre: 210764:0:(obd_config.c:1396:class_config_llog_handler()) For 1.8 interoperability, rename obd type from mds to mdt 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.764633] Lustre: Enabling ACL 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.319567] Lustre: umt3-MDT0000: used disk, loading 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.343261] Lustre: Mounting umt3-MDT0000 at first time on 1.8 FS, remove all clients for interop needs 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.408147] LustreError: 210764:0:(llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile 0x1f0fc001:0x45460a07: rc -116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.431151] LustreError: 210764:0:(llog_obd.c:220:llog_setup_named()) obd umt3-OST0000-osc ctxt 2 lop_setup =ffffffffa0777ca0 failed -116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.455805] LustreError: 210764:0:(osc_request.c:4278:__osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.474526] LustreError: 210764:0:(osc_request.c:4295:__osc_llog_init()) osc 'umt3-OST0000-osc' tgt 'mdd_obd-umt3-MDT0000' catid ffff8803ab8678b0 rc=-116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.502137] LustreError: 210764:0:(osc_request.c:4297:__osc_llog_init()) logid 0x1f0fc001:0x45460a07 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.520514] LustreError: 210764:0:(osc_request.c:4325:osc_llog_init()) rc: -116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.535211] LustreError: 210764:0:(lov_log.c:248:lov_llog_init()) error osc_llog_init idx 0 osc 'umt3-OST0000-osc' tgt 'mdd_obd-umt3-MDT0000' (rc=-116) 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.562854] LustreError: 210764:0:(llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile 0x1f0fc002:0x45460a08: rc -116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.585795] LustreError: 210764:0:(llog_obd.c:220:llog_setup_named()) obd umt3-OST0001-osc ctxt 2 lop_setup=ffffffffa0777ca0 failed -116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610490] LustreError: 210764:0:(osc_request.c:4278:__osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610779] LustreError: 210793:0:(lov_log.c:160:lov_llog_origin_connect()) error osc_llog_connect tgt 0 (- 107) 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610785] LustreError: 210793:0:(mds_lov.c:832:__mds_lov_synchronize()) umt3-OST0000_UUID failed at llog_origin_connect: -107 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610790] LustreError: 210793:0:(mds_lov.c:860:__mds_lov_synchronize()) sync umt3-OST0000_UUID failed -107 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610794] LustreError: 210793:0:(mds_lov.c:864:__mds_lov_synchronize()) deactivating umt3-OST0000_UUID 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.711378] LustreError: 210764:0:(osc_request.c:4295:__osc_llog_init()) osc 'umt3-OST0001-osc' tgt 'mdd_obd-umt3-MDT0000' catid ffff8803ab8678b0 rc=-116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.739028] LustreError: 210764:0:(osc_request.c:4297:__osc_llog_init()) logid 0x1f0fc002:0x45460a08 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.757420] LustreError: 210764:0:(osc_request.c:4325:osc_llog_init()) rc: -116 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.772127] LustreError: 210764:0:(lov_log.c:248:lov_llog_init()) error osc_llog_init idx 1 osc 'umt3-OST0001-osc' tgt 'mdd_obd-umt3-MDT0000' (rc=-116) 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.800849] LustreError: 210794:0:(lov_log.c:160:lov_llog_origin_connect()) error osc_llog_connect tgt 1 (-107) 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.801867] LustreError: 21 0796:0:(mds_lov.c:832:__mds_lov_synchronize()) umt3-OST0003_UUID failed at llog_origin_connect: -107 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.801872] LustreError: 210796:0:(mds_lov.c:860:__mds_lov_synchronize()) sync umt3-OST0003_UUID failed -107 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.801877] LustreError: 210796:0:(mds_lov.c:864:__mds_lov_synchronize()) deactivating umt3-OST0003_UUID 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.883376] LustreError: 210794:0:(lov_log.c:160:lov_llog_origin_connect()) Skipped 2 previous similar messages 2013-07-16T08:48:23-04:00 lmd02.aglt2.org kernel: [62095.558232] Lustre: 210777:0:(ldlm_lib.c:952:target_handle_connect()) umt3-MDT0000: connection from f1e54b83-bfa4-2f99-dfd2-c9a903f97369@141.213.133.211@tcp2 t533204841976 exp (null) cur 1373978903 last 0 [ Noticed that all mounts were RO, try to activate as subset as RW ] [ eg, /usr/sbin/lctl --device umt3-OSG0022-osc activate ] 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.496490] Lustre: umt3-OST0022-osc: Connection to umt3-OST0022 (at 10.10.2.21@tcp) was lost; in progress operations using this service will wait for recovery to complete 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.496953] Lustre: umt3-OST0022-osc: Connection restored to umt3-OST0022 (at 10.10.2.21@tcp) 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.497680] LustreError: 213631:0:(lov_log.c:160:lov_llog_origin_connect()) error osc_llog_connect tgt 34 ( -107) 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.498190] Lustre: umt3-OST0023-osc: Connection to umt3-OST0023 (at 10.10.2.21@tcp) was lost; in progress operations using this service will wait for recovery to complete 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.518189] LustreError: 213631:0:(lov_log.c:160:lov_llog_origin_connect()) Skipped 62 previous similar messages 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.518270] Lustre: umt3-OST0023-osc: Connection restored to umt3-OST0023 (at 10.10.2.21@tcp) 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519058] LustreError: 213633:0:(mds_lov.c:832:__mds_lov_synchronize()) umt3-OST0023_UUID failed at llog_origin_connect: -107 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519062] LustreError: 213633:0:(mds_lov.c:832:__mds_lov_synchronize()) Skipped 68 previous similar messages 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519067] LustreError: 213633:0:(mds_lov.c:860:__mds_lov_synchronize()) sync umt3-OST0023_UUID failed -107 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519070] LustreError: 213633:0:(mds_lov.c:860:__mds_lov_synchronize()) Skipped 68 previous similar messages 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519075] LustreError: 213633:0:(mds_lov.c:864:__mds_lov_synchronize()) deactivating umt3-OST0023_UUID 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519078] LustreError: 213633:0:(mds_lov.c:864:__mds_lov_synchronize()) Skipped 68 previous similar messages 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.660863] LustreError: 213631:0:(mds_lov.c:832:__mds_lov_synchronize()) umt3-OST0022_UUID failed at llog_origin_connect: -107 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.661660] LustreError: 213635:0:(lov_log.c:160:lov_llog_origin_connect()) error osc_llog_connect tgt 36 (-107) 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.661669] LustreError: 213635:0:(mds_lov.c:860:__mds_lov_synchronize()) sync umt3-OST0024_UUID failed -107 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.661674] LustreError: 213635:0:(mds_lov.c:864:__mds_lov_synchronize()) deactivating umt3-OST0024_UUID 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.743206] LustreError: 213631:0:(mds_lov.c:832:__mds_lov_synchronize()) Skipped 1 previous similar message ===================================== On the mdt node, I see this. [root@lmd02 ~]# lctl dl 0 UP mgc MGC10.10.1.140@tcp 12f9f864-0ad6-624f-115d-8801b67ca538 5 1 UP lov umt3-mdtlov umt3-mdtlov_UUID 4 2 UP mdt umt3-MDT0000 umt3-MDT0000_UUID 5 3 UP mds mdd_obd-umt3-MDT0000 mdd_obd_uuid-umt3-MDT0000 3 4 IN osc umt3-OST0000-osc umt3-mdtlov_UUID 5 5 IN osc umt3-OST0001-osc umt3-mdtlov_UUID 5 6 IN osc umt3-OST0002-osc umt3-mdtlov_UUID 5 7 IN osc umt3-OST0003-osc umt3-mdtlov_UUID 5 8 IN osc umt3-OST0004-osc umt3-mdtlov_UUID 5 9 IN osc umt3-OST0005-osc umt3-mdtlov_UUID 5 10 IN osc umt3-OST0006-osc umt3-mdtlov_UUID 5 11 IN osc umt3-OST0007-osc umt3-mdtlov_UUID 5 12 IN osc umt3-OST0010-osc umt3-mdtlov_UUID 5 13 IN osc umt3-OST0011-osc umt3-mdtlov_UUID 5 14 IN osc umt3-OST0012-osc umt3-mdtlov_UUID 5 15 IN osc umt3-OST0013-osc umt3-mdtlov_UUID 5 16 IN osc umt3-OST0014-osc umt3-mdtlov_UUID 5 17 IN osc umt3-OST0015-osc umt3-mdtlov_UUID 5 18 IN osc umt3-OST0016-osc umt3-mdtlov_UUID 5 19 IN osc umt3-OST0017-osc umt3-mdtlov_UUID 5 20 IN osc umt3-OST0018-osc umt3-mdtlov_UUID 5 21 IN osc umt3-OST0019-osc umt3-mdtlov_UUID 5 22 IN osc umt3-OST001a-osc umt3-mdtlov_UUID 5 23 IN osc umt3-OST001b-osc umt3-mdtlov_UUID 5 24 IN osc umt3-OST0008-osc umt3-mdtlov_UUID 5 25 IN osc umt3-OST0009-osc umt3-mdtlov_UUID 5 26 IN osc umt3-OST000a-osc umt3-mdtlov_UUID 5 27 IN osc umt3-OST000b-osc umt3-mdtlov_UUID 5 28 IN osc umt3-OST000c-osc umt3-mdtlov_UUID 5 29 IN osc umt3-OST000d-osc umt3-mdtlov_UUID 5 30 IN osc umt3-OST000e-osc umt3-mdtlov_UUID 5 31 IN osc umt3-OST000f-osc umt3-mdtlov_UUID 5 32 IN osc umt3-OST001c-osc umt3-mdtlov_UUID 5 33 IN osc umt3-OST001d-osc umt3-mdtlov_UUID 5 34 IN osc umt3-OST001e-osc umt3-mdtlov_UUID 5 35 IN osc umt3-OST001f-osc umt3-mdtlov_UUID 5 36 IN osc umt3-OST0020-osc umt3-mdtlov_UUID 5 37 IN osc umt3-OST0021-osc umt3-mdtlov_UUID 5 38 IN osc umt3-OST0022-osc umt3-mdtlov_UUID 5 39 IN osc umt3-OST0023-osc umt3-mdtlov_UUID 5 40 IN osc umt3-OST0024-osc umt3-mdtlov_UUID 5 41 IN osc umt3-OST0025-osc umt3-mdtlov_UUID 5 42 IN osc umt3-OST0026-osc umt3-mdtlov_UUID 5 43 IN osc umt3-OST0027-osc umt3-mdtlov_UUID 5 44 IN osc umt3-OST0028-osc umt3-mdtlov_UUID 5 45 IN osc umt3-OST0029-osc umt3-mdtlov_UUID 5 46 IN osc umt3-OST002a-osc umt3-mdtlov_UUID 5 47 IN osc umt3-OST002b-osc umt3-mdtlov_UUID 5 48 IN osc umt3-OST002c-osc umt3-mdtlov_UUID 5 49 IN osc umt3-OST002d-osc umt3-mdtlov_UUID 5 50 IN osc umt3-OST002e-osc umt3-mdtlov_UUID 5 51 IN osc umt3-OST002f-osc umt3-mdtlov_UUID 5 52 IN osc umt3-OST0030-osc umt3-mdtlov_UUID 5 53 IN osc umt3-OST0031-osc umt3-mdtlov_UUID 5 54 IN osc umt3-OST0032-osc umt3-mdtlov_UUID 5 55 IN osc umt3-OST0033-osc umt3-mdtlov_UUID 5 56 IN osc umt3-OST0034-osc umt3-mdtlov_UUID 5 57 IN osc umt3-OST0035-osc umt3-mdtlov_UUID 5 58 IN osc umt3-OST0036-osc umt3-mdtlov_UUID 5 59 IN osc umt3-OST0037-osc umt3-mdtlov_UUID 5 60 IN osc umt3-OST0038-osc umt3-mdtlov_UUID 5 61 IN osc umt3-OST0039-osc umt3-mdtlov_UUID 5 62 IN osc umt3-OST003a-osc umt3-mdtlov_UUID 5 63 IN osc umt3-OST003b-osc umt3-mdtlov_UUID 5 64 IN osc umt3-OST003c-osc umt3-mdtlov_UUID 5 65 IN osc umt3-OST003d-osc umt3-mdtlov_UUID 5 66 IN osc umt3-OST003e-osc umt3-mdtlov_UUID 5 67 IN osc umt3-OST003f-osc umt3-mdtlov_UUID 5 68 IN osc umt3-OST0040-osc umt3-mdtlov_UUID 5 69 IN osc umt3-OST0041-osc umt3-mdtlov_UUID 5 70 IN osc umt3-OST0042-osc umt3-mdtlov_UUID 5 71 IN osc umt3-OST0043-osc umt3-mdtlov_UUID 5 72 IN osc umt3-OST0044-osc umt3-mdtlov_UUID 5 73 IN osc umt3-OST0045-osc umt3-mdtlov_UUID 5