Sorry for the noise. This request can be cancelled. I left out one
very important step in restoring the mdt, namely
rm OBJECTS/* CATALOGS
DOH! Umounted, removed via ldiskfs mount, remounted, and now all OST
are showing UP instead of IN.
bob
On 7/16/2013 9:22 AM, Bob Ball wrote:
I am just finishing an upgrade from Lustre 1.8.4 to 2.1.6, on SL5.8
to
SL6.4. All OSS, MGS and MDT machines were upgraded. Following the
upgrade I find that none of the OST will change to RW state. Could I
get some advice from someone on this?
We have separate mgs/mdt machines, and 8 OSS with ~70 OST attached.
All OSS were upgraded and the OST were mounted. This morning, I
started the MGS then the MDT. The MGS log entries in
/var/log/messages shown here first, followed by MDT log entries, and
the "lctl dl" output on the mdt.
I made no file system changes on the OSS nodes. I rebuilt the mdt to
a smaller size, using the setfattr/tar method described in the Lustre
1.8 manuals, with the mkfs.lustre issued from the newly built, 2.1.6 mdt.
Have I missed anything here? Can someone tell me what is wrong with
my system?
Thanks,
bob
===================== mgs log ==================
2013-07-16T08:28:15-04:00
mgs.aglt2.org kernel: [162354.732167]
Lustre: Lustre: Build Version:
RC2--PRISTINE-2.6.32-358.11.1.el6_lustre.x86_64
2013-07-16T08:28:15-04:00
mgs.aglt2.org kernel: [162355.043533]
Lustre: Added LNI 10.10.1.140@tcp [8/256/0/180]
2013-07-16T08:28:15-04:00
mgs.aglt2.org kernel: [162355.043545]
Lustre: Added LNI 141.213.133.196@tcp2 [8/256/0/180]
2013-07-16T08:28:15-04:00
mgs.aglt2.org kernel: [162355.043593]
Lustre: Accept secure, port 988
2013-07-16T08:28:16-04:00
mgs.aglt2.org kernel: [162355.398195]
Lustre: Lustre OSC module (ffffffffa08cc2a0).
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.433725]
Lustre: Lustre LOV module (ffffffffa094a280).
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.544341]
Lustre: Lustre client module (ffffffffa0a34920).
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.557307]
LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck
is recommended
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.557998]
LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts:
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.590277]
LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck
is recommended
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.591095]
LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts:
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.690541]
Lustre: MGS MGS started
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.690795]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from 51233eb1-32bc-cb6f-3bbe-5008265553b7@0@lo t0 exp
(null) cur 1373977697 last 0
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162356.697639]
Lustre: MGC10.10.1.140@tcp: Reactivating import
2013-07-16T08:28:17-04:00
mgs.aglt2.org kernel: [162357.003523]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from 2f75f773-faaf-d242-de3e-18443c3c45db@10.10.2.23@tcp t0
exp (null) cur 1373977697 last 0
2013-07-16T08:28:26-04:00
mgs.aglt2.org kernel: [162365.602114]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from f49a6fb6-030d-b53b-f515-6df58cceb09d@10.10.2.21@tcp t0
exp (null) cur 1373977706 last 0
2013-07-16T08:28:54-04:00
mgs.aglt2.org kernel: [162394.024874]
Lustre: 438424:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from bc883142-33e6-cef3-630e-0e3322f19033@10.10.2.24@tcp t0
exp (null) cur 1373977734 last 0
[ The 10.10.1.157 was an old, 1.8.4 machine that I
missed. I shut it down ]
2013-07-16T08:28:58-04:00
mgs.aglt2.org kernel: [162397.566576]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from 847c3782-0835-5695-93e9-4276f7d7152a@10.10.1.157@tcp
t0 exp (null) cur 1373977738 last 0
2013-07-16T08:28:58-04:00
mgs.aglt2.org kernel: [162397.566580]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1
previous similar message
2013-07-16T08:29:05-04:00
mgs.aglt2.org kernel: [162405.195835]
Lustre: 438424:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from 118c0806-972c-3e9a-5de2-8bc902817bfe@10.10.2.27@tcp t0
exp (null) cur 1373977745 last 0
2013-07-16T08:29:14-04:00
mgs.aglt2.org kernel: [162413.238112]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from dbae7713-3f95-057a-28e1-7b6519506f64@10.10.2.22@tcp t0
exp (null) cur 1373977754 last 0
[ Now I start the mdt machine, and these log ]
2013-07-16T08:46:23-04:00
mgs.aglt2.org kernel: [163439.535234]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
connection from 12f9f864-0ad6-624f-115d-8801b67ca538@10.10.1.49@tcp t0
exp (null) cur 1373978783 last 0
2013-07-16T08:46:23-04:00
mgs.aglt2.org kernel: [163439.535258]
Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3
previous similar messages
2013-07-16T08:46:23-04:00
mgs.aglt2.org kernel: [163439.590237]
Lustre: 440552:0:(mgs_llog.c:249:mgs_fsdb_handler()) MDT using 1.8 OSC
name scheme
======================= mdt log from /var/log/messages ==================
[ First mount attempt failed, added that unsupported "-o
update" I found ]
2013-07-16T08:45:43-04:00
lmd02.aglt2.org kernel: [61935.936127]
Lustre: Lustre:
Build Version: RC2--PRISTINE-2.6.32-358.11.1.el6_lustre.x86_64
2013-07-16T08:45:44-04:00
lmd02.aglt2.org kernel: [61936.409749]
Lustre: Added L
NI 10.10.1.49@tcp [8/256/0/180]
2013-07-16T08:45:44-04:00
lmd02.aglt2.org kernel: [61936.409845]
Lustre: Accept
secure, port 988
2013-07-16T08:45:44-04:00
lmd02.aglt2.org kernel: [61936.833435]
Lustre: Lustre
OSC module (ffffffffa0bd22a0).
2013-07-16T08:45:44-04:00
lmd02.aglt2.org kernel: [61937.003602]
Lustre: Lustre
LOV module (ffffffffa0c64280).
2013-07-16T08:45:45-04:00
lmd02.aglt2.org kernel: [61937.195301]
Lustre: Lustre
client module (ffffffffa0d4e920).
2013-07-16T08:45:45-04:00
lmd02.aglt2.org kernel: [61937.344529]
LDISKFS-fs (dm-
7): Unrecognized mount option "upgrade" or missing value
2013-07-16T08:45:45-04:00
lmd02.aglt2.org kernel: [61937.360076]
LustreError: 210489:0:(obd_mount.c:1360:server_kernel_mount())
premount /dev/mapper/sun-mdt3:0x0 ldiskfs failed: -22 Is the ldiskfs
module available?
2013-07-16T08:45:45-04:00
lmd02.aglt2.org kernel: [61937.389113]
LustreError: 210489:0:(obd_mount.c:1700:server_fill_super()) Unable to
mount device /dev/mapper
/sun-mdt3: -22
2013-07-16T08:45:45-04:00
lmd02.aglt2.org kernel: [61937.411161]
LustreError: 210489:0:(obd_mount.c:2203:lustre_fill_super()) Unable to
mount (-22)
[ end failed mount attempt ]
2013-07-16T08:46:22-04:00
lmd02.aglt2.org kernel: [61974.840288]
LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts:
2013-07-16T08:46:23-04:00
lmd02.aglt2.org kernel: [61975.189666]
LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts:
2013-07-16T08:46:23-04:00
lmd02.aglt2.org kernel: [61975.239509]
Lustre: MGC10.10.1.140@tcp: Reactivating import
2013-07-16T08:46:23-04:00
lmd02.aglt2.org kernel: [61975.466165]
Lustre: 210764:0:(obd_config.c:1396:class_config_llog_handler()) For
1.8 interoperability, rename obd type from mds to mdt
2013-07-16T08:46:23-04:00
lmd02.aglt2.org kernel: [61975.764633]
Lustre: Enabling ACL
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.319567]
Lustre: umt3-MDT0000: used disk, loading
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.343261]
Lustre: Mounting umt3-MDT0000 at first time on 1.8 FS, remove all
clients for interop needs
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.408147]
LustreError: 210764:0:(llog_lvfs.c:616:llog_lvfs_create()) error
looking up logfile 0x1f0fc001:0x45460a07: rc -116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.431151]
LustreError: 210764:0:(llog_obd.c:220:llog_setup_named()) obd
umt3-OST0000-osc ctxt 2 lop_setup
=ffffffffa0777ca0 failed -116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.455805]
LustreError: 210764:0:(osc_request.c:4278:__osc_llog_init()) failed
LLOG_MDS_OST_ORIG_CTXT
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.474526]
LustreError: 210764:0:(osc_request.c:4295:__osc_llog_init()) osc
'umt3-OST0000-osc' tgt 'mdd_obd-umt3-MDT0000' catid ffff8803ab8678b0
rc=-116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.502137]
LustreError: 210764:0:(osc_request.c:4297:__osc_llog_init()) logid
0x1f0fc001:0x45460a07
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.520514]
LustreError: 210764:0:(osc_request.c:4325:osc_llog_init()) rc: -116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.535211]
LustreError: 210764:0:(lov_log.c:248:lov_llog_init()) error
osc_llog_init idx 0 osc 'umt3-OST0000-osc' tgt 'mdd_obd-umt3-MDT0000'
(rc=-116)
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.562854]
LustreError: 210764:0:(llog_lvfs.c:616:llog_lvfs_create()) error
looking up logfile 0x1f0fc002:0x45460a08: rc -116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.585795]
LustreError: 210764:0:(llog_obd.c:220:llog_setup_named()) obd
umt3-OST0001-osc ctxt 2 lop_setup=ffffffffa0777ca0 failed -116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.610490]
LustreError: 210764:0:(osc_request.c:4278:__osc_llog_init()) failed
LLOG_MDS_OST_ORIG_CTXT
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.610779]
LustreError: 210793:0:(lov_log.c:160:lov_llog_origin_connect()) error
osc_llog_connect tgt 0 (-
107)
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.610785]
LustreError: 210793:0:(mds_lov.c:832:__mds_lov_synchronize())
umt3-OST0000_UUID failed at llog_origin_connect: -107
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.610790]
LustreError: 210793:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
umt3-OST0000_UUID failed -107
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.610794]
LustreError: 210793:0:(mds_lov.c:864:__mds_lov_synchronize())
deactivating umt3-OST0000_UUID
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.711378]
LustreError: 210764:0:(osc_request.c:4295:__osc_llog_init()) osc
'umt3-OST0001-osc' tgt 'mdd_obd-umt3-MDT0000' catid ffff8803ab8678b0
rc=-116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.739028]
LustreError: 210764:0:(osc_request.c:4297:__osc_llog_init()) logid
0x1f0fc002:0x45460a08
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.757420]
LustreError: 210764:0:(osc_request.c:4325:osc_llog_init()) rc: -116
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.772127]
LustreError: 210764:0:(lov_log.c:248:lov_llog_init()) error
osc_llog_init idx 1 osc 'umt3-OST0001-osc' tgt 'mdd_obd-umt3-MDT0000'
(rc=-116)
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.800849]
LustreError: 210794:0:(lov_log.c:160:lov_llog_origin_connect()) error
osc_llog_connect tgt 1 (-107)
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.801867]
LustreError: 21
0796:0:(mds_lov.c:832:__mds_lov_synchronize()) umt3-OST0003_UUID
failed at llog_origin_connect: -107
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.801872]
LustreError: 210796:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
umt3-OST0003_UUID failed -107
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.801877]
LustreError: 210796:0:(mds_lov.c:864:__mds_lov_synchronize())
deactivating umt3-OST0003_UUID
2013-07-16T08:46:24-04:00
lmd02.aglt2.org kernel: [61976.883376]
LustreError: 210794:0:(lov_log.c:160:lov_llog_origin_connect())
Skipped 2 previous similar messages
2013-07-16T08:48:23-04:00
lmd02.aglt2.org kernel: [62095.558232]
Lustre: 210777:0:(ldlm_lib.c:952:target_handle_connect())
umt3-MDT0000: connection from
f1e54b83-bfa4-2f99-dfd2-c9a903f97369@141.213.133.211@tcp2
t533204841976 exp (null) cur 1373978903 last 0
[ Noticed that all mounts were RO, try to activate as subset
as RW ]
[ eg, /usr/sbin/lctl --device umt3-OSG0022-osc activate ]
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.496490]
Lustre: umt3-OST0022-osc: Connection to umt3-OST0022 (at
10.10.2.21@tcp) was lost; in progress operations using this service
will wait for recovery to complete
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.496953]
Lustre: umt3-OST0022-osc: Connection restored to umt3-OST0022 (at
10.10.2.21@tcp)
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.497680]
LustreError: 213631:0:(lov_log.c:160:lov_llog_origin_connect()) error
osc_llog_connect tgt 34 (
-107)
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.498190]
Lustre: umt3-OST0023-osc: Connection to umt3-OST0023 (at
10.10.2.21@tcp) was lost; in progress operations using this service
will wait for recovery to complete
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.518189]
LustreError: 213631:0:(lov_log.c:160:lov_llog_origin_connect())
Skipped 62 previous similar messages
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.518270]
Lustre: umt3-OST0023-osc: Connection restored to umt3-OST0023 (at
10.10.2.21@tcp)
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.519058]
LustreError: 213633:0:(mds_lov.c:832:__mds_lov_synchronize())
umt3-OST0023_UUID failed at llog_origin_connect: -107
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.519062]
LustreError: 213633:0:(mds_lov.c:832:__mds_lov_synchronize()) Skipped
68 previous similar messages
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.519067]
LustreError: 213633:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
umt3-OST0023_UUID failed -107
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.519070]
LustreError: 213633:0:(mds_lov.c:860:__mds_lov_synchronize()) Skipped
68 previous similar messages
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.519075]
LustreError: 213633:0:(mds_lov.c:864:__mds_lov_synchronize())
deactivating umt3-OST0023_UUID
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.519078]
LustreError: 213633:0:(mds_lov.c:864:__mds_lov_synchronize()) Skipped
68 previous similar messages
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.660863]
LustreError: 213631:0:(mds_lov.c:832:__mds_lov_synchronize())
umt3-OST0022_UUID failed at llog_origin_connect: -107
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.661660]
LustreError: 213635:0:(lov_log.c:160:lov_llog_origin_connect()) error
osc_llog_connect tgt 36 (-107)
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.661669]
LustreError: 213635:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
umt3-OST0024_UUID failed -107
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.661674]
LustreError: 213635:0:(mds_lov.c:864:__mds_lov_synchronize())
deactivating umt3-OST0024_UUID
2013-07-16T08:59:04-04:00
lmd02.aglt2.org kernel: [62736.743206]
LustreError: 213631:0:(mds_lov.c:832:__mds_lov_synchronize()) Skipped
1 previous similar message
=====================================
On the mdt node, I see this.
[root@lmd02 ~]# lctl dl
0 UP mgc MGC10.10.1.140@tcp 12f9f864-0ad6-624f-115d-8801b67ca538 5
1 UP lov umt3-mdtlov umt3-mdtlov_UUID 4
2 UP mdt umt3-MDT0000 umt3-MDT0000_UUID 5
3 UP mds mdd_obd-umt3-MDT0000 mdd_obd_uuid-umt3-MDT0000 3
4 IN osc umt3-OST0000-osc umt3-mdtlov_UUID 5
5 IN osc umt3-OST0001-osc umt3-mdtlov_UUID 5
6 IN osc umt3-OST0002-osc umt3-mdtlov_UUID 5
7 IN osc umt3-OST0003-osc umt3-mdtlov_UUID 5
8 IN osc umt3-OST0004-osc umt3-mdtlov_UUID 5
9 IN osc umt3-OST0005-osc umt3-mdtlov_UUID 5
10 IN osc umt3-OST0006-osc umt3-mdtlov_UUID 5
11 IN osc umt3-OST0007-osc umt3-mdtlov_UUID 5
12 IN osc umt3-OST0010-osc umt3-mdtlov_UUID 5
13 IN osc umt3-OST0011-osc umt3-mdtlov_UUID 5
14 IN osc umt3-OST0012-osc umt3-mdtlov_UUID 5
15 IN osc umt3-OST0013-osc umt3-mdtlov_UUID 5
16 IN osc umt3-OST0014-osc umt3-mdtlov_UUID 5
17 IN osc umt3-OST0015-osc umt3-mdtlov_UUID 5
18 IN osc umt3-OST0016-osc umt3-mdtlov_UUID 5
19 IN osc umt3-OST0017-osc umt3-mdtlov_UUID 5
20 IN osc umt3-OST0018-osc umt3-mdtlov_UUID 5
21 IN osc umt3-OST0019-osc umt3-mdtlov_UUID 5
22 IN osc umt3-OST001a-osc umt3-mdtlov_UUID 5
23 IN osc umt3-OST001b-osc umt3-mdtlov_UUID 5
24 IN osc umt3-OST0008-osc umt3-mdtlov_UUID 5
25 IN osc umt3-OST0009-osc umt3-mdtlov_UUID 5
26 IN osc umt3-OST000a-osc umt3-mdtlov_UUID 5
27 IN osc umt3-OST000b-osc umt3-mdtlov_UUID 5
28 IN osc umt3-OST000c-osc umt3-mdtlov_UUID 5
29 IN osc umt3-OST000d-osc umt3-mdtlov_UUID 5
30 IN osc umt3-OST000e-osc umt3-mdtlov_UUID 5
31 IN osc umt3-OST000f-osc umt3-mdtlov_UUID 5
32 IN osc umt3-OST001c-osc umt3-mdtlov_UUID 5
33 IN osc umt3-OST001d-osc umt3-mdtlov_UUID 5
34 IN osc umt3-OST001e-osc umt3-mdtlov_UUID 5
35 IN osc umt3-OST001f-osc umt3-mdtlov_UUID 5
36 IN osc umt3-OST0020-osc umt3-mdtlov_UUID 5
37 IN osc umt3-OST0021-osc umt3-mdtlov_UUID 5
38 IN osc umt3-OST0022-osc umt3-mdtlov_UUID 5
39 IN osc umt3-OST0023-osc umt3-mdtlov_UUID 5
40 IN osc umt3-OST0024-osc umt3-mdtlov_UUID 5
41 IN osc umt3-OST0025-osc umt3-mdtlov_UUID 5
42 IN osc umt3-OST0026-osc umt3-mdtlov_UUID 5
43 IN osc umt3-OST0027-osc umt3-mdtlov_UUID 5
44 IN osc umt3-OST0028-osc umt3-mdtlov_UUID 5
45 IN osc umt3-OST0029-osc umt3-mdtlov_UUID 5
46 IN osc umt3-OST002a-osc umt3-mdtlov_UUID 5
47 IN osc umt3-OST002b-osc umt3-mdtlov_UUID 5
48 IN osc umt3-OST002c-osc umt3-mdtlov_UUID 5
49 IN osc umt3-OST002d-osc umt3-mdtlov_UUID 5
50 IN osc umt3-OST002e-osc umt3-mdtlov_UUID 5
51 IN osc umt3-OST002f-osc umt3-mdtlov_UUID 5
52 IN osc umt3-OST0030-osc umt3-mdtlov_UUID 5
53 IN osc umt3-OST0031-osc umt3-mdtlov_UUID 5
54 IN osc umt3-OST0032-osc umt3-mdtlov_UUID 5
55 IN osc umt3-OST0033-osc umt3-mdtlov_UUID 5
56 IN osc umt3-OST0034-osc umt3-mdtlov_UUID 5
57 IN osc umt3-OST0035-osc umt3-mdtlov_UUID 5
58 IN osc umt3-OST0036-osc umt3-mdtlov_UUID 5
59 IN osc umt3-OST0037-osc umt3-mdtlov_UUID 5
60 IN osc umt3-OST0038-osc umt3-mdtlov_UUID 5
61 IN osc umt3-OST0039-osc umt3-mdtlov_UUID 5
62 IN osc umt3-OST003a-osc umt3-mdtlov_UUID 5
63 IN osc umt3-OST003b-osc umt3-mdtlov_UUID 5
64 IN osc umt3-OST003c-osc umt3-mdtlov_UUID 5
65 IN osc umt3-OST003d-osc umt3-mdtlov_UUID 5
66 IN osc umt3-OST003e-osc umt3-mdtlov_UUID 5
67 IN osc umt3-OST003f-osc umt3-mdtlov_UUID 5
68 IN osc umt3-OST0040-osc umt3-mdtlov_UUID 5
69 IN osc umt3-OST0041-osc umt3-mdtlov_UUID 5
70 IN osc umt3-OST0042-osc umt3-mdtlov_UUID 5
71 IN osc umt3-OST0043-osc umt3-mdtlov_UUID 5
72 IN osc umt3-OST0044-osc umt3-mdtlov_UUID 5
73 IN osc umt3-OST0045-osc umt3-mdtlov_UUID 5