3:29 p.m.
I have split an MDS from an MGS using 2.4.2; running on the same server
using
CentOS 2.6.32-358.23.2.el6_lustre + ldiskfs only.
I have done a writeconf; now the MDS won't "register" with the MGS.
On the split MDS/MGS server the MGT and MDT devices mount but the logs show:
Process recover log testfs-mtdir error -22.
However, if I now try to mount the OSTs, they get stuck in "AT".
One symptom is that on the split MGS/MDS the lustre modules cannot unload,
as some process is using osp.o. If I don't start the OSTs the lustre stack
can be unloaded.
Any suggestions on getting the MDS/MGS to play nice?
Thanks
Anthony
[root@mds1 ~]# lctl dl
0 UP osd-ldiskfs MGS-osd MGS-osd_UUID 5
1 UP mgs MGS MGS 5
2 UP mgc MGC5.5.200.5@o2ib 9ce8a82a-136e-616e-f6f3-f4570fbd364e 5
3 UP osd-ldiskfs testfs-MDT0000-osd testfs-MDT0000-osd_UUID 7
4 UP mds MDS MDS_uuid 3
5 UP lod testfs-MDT0000-mdtlov testfs-MDT0000-mdtlov_UUID 4
6 UP mdt testfs-MDT0000 testfs-MDT0000_UUID 3
7 UP mdd testfs-MDD0000 testfs-MDD0000_UUID 4
8 UP qmt testfs-QMT0000 testfs-QMT0000_UUID 4
Feb 4 12:12:10 mds1 kernel: Lustre: MGS: Logs for fs testfs were removed
by user request.
All servers must be restarted in order to regenerate the logs.
Feb 4 12:12:10 mds1 kernel: Lustre: testfs-MDT0000: used disk, loading
Feb 4 12:12:10 mds1 kernel: LustreError:
8412:0:(osd_io.c:1000:osd_ldiskfs_read()) testfs=M
DT0000: can't read 128@8192 on ino 21: rc = 0
Feb 4 12:12:10 mds1 kernel: LustreError:
8412:0:(mdt_recovery.c:112:mdt_clients_data_init()
) error reading MDS last_rcvd idx 0, off 8192: rc -14
Feb 4 12:12:18 mds1 kernel: Lustre:
6987:0:(mgc_request.c:1564:mgc_process_recover_log()) P
rocess recover log testfs-mdtir error -22
Now attempt to mount 2 x OST:
9 AT osp testfs-OST0000-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 1
Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
by user request.
Feb 4 12:22:58 mds1 kernel: Lustre:
8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
rocess recover log testfs-mdtir error -22
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
dd initial connection
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
osc-MDT0000: can't setup obd: -2
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:572:class_setup()) setup testfs-OST0000-osc-MDT0000
failed (-2)
Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
by user request.
Feb 4 12:22:58 mds1 kernel: Lustre:
8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
rocess recover log testfs-mdtir error -22
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
dd initial connection
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
osc-MDT0000: can't setup obd: -2
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:572:class_setup()) setup mant
le-OST0000-osc-MDT0000 failed (-2)
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:1553:class_config_llog_handle
r()) MGC5.5.200.5@o2ib: cfg command failed: rc = -2
Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
0:testfs-OST0000-osc-MDT0000 1:testfs-OST
0000_UUID 2:0@<0:0>
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:1553:class_config_llog_handler())
MGC5.5.200.5@o2ib: cfg command failed: rc = -2
Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
0:testfs-OST0000-osc-MDT0000
1:testfs-OST0000_UUID 2:0@<0:0>
6:35 a.m.
On 2014/02/03, 9:29 PM, "Anthony Alba" <ascanio.alba7(a)gmail.com> wrote:
I have split an MDS from an MGS using 2.4.2; running on the same
server
using
CentOS 2.6.32-358.23.2.el6_lustre + ldiskfs only.
What steps did you take to do this?
I have done a writeconf; now the MDS won't "register"
with the MGS.
On the split MDS/MGS server the MGT and MDT devices mount but the logs
show:
Process recover log testfs-mtdir error -22.
Did you copy this file over to the new MGS?
However, if I now try to mount the OSTs, they get stuck in
"AT".
One symptom is that on the split MGS/MDS the lustre modules cannot unload,
as some process is using osp.o. If I don't start the OSTs the lustre
stack can be unloaded.
Any suggestions on getting the MDS/MGS to play nice?
Thanks
Anthony
[root@mds1 ~]# lctl dl
0 UP osd-ldiskfs MGS-osd MGS-osd_UUID 5
1 UP mgs MGS MGS 5
2 UP mgc MGC5.5.200.5@o2ib 9ce8a82a-136e-616e-f6f3-f4570fbd364e 5
3 UP osd-ldiskfs testfs-MDT0000-osd testfs-MDT0000-osd_UUID 7
4 UP mds MDS MDS_uuid 3
5 UP lod testfs-MDT0000-mdtlov testfs-MDT0000-mdtlov_UUID 4
6 UP mdt testfs-MDT0000 testfs-MDT0000_UUID 3
7 UP mdd testfs-MDD0000 testfs-MDD0000_UUID 4
8 UP qmt testfs-QMT0000 testfs-QMT0000_UUID 4
Feb 4 12:12:10 mds1 kernel: Lustre: MGS: Logs for fs testfs were removed
by user request.
All servers must be restarted in order to regenerate the logs.
Feb 4 12:12:10 mds1 kernel: Lustre: testfs-MDT0000: used disk, loading
Feb 4 12:12:10 mds1 kernel: LustreError:
8412:0:(osd_io.c:1000:osd_ldiskfs_read()) testfs=M
DT0000: can't read 128@8192 on ino 21: rc = 0
Feb 4 12:12:10 mds1 kernel: LustreError:
8412:0:(mdt_recovery.c:112:mdt_clients_data_init()
) error reading MDS last_rcvd idx 0, off 8192: rc -14
Feb 4 12:12:18 mds1 kernel: Lustre:
6987:0:(mgc_request.c:1564:mgc_process_recover_log()) P
rocess recover log testfs-mdtir error -22
Now attempt to mount 2 x OST:
9 AT osp testfs-OST0000-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 1
Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
by user request.
Feb 4 12:22:58 mds1 kernel: Lustre:
8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
rocess recover log testfs-mdtir error -22
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
dd initial connection
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
osc-MDT0000: can't setup obd: -2
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:572:class_setup()) setup testfs-OST0000-osc-MDT0000
failed (-2)
Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
by user request.
Feb 4 12:22:58 mds1 kernel: Lustre:
8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
rocess recover log testfs-mdtir error -22
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
dd initial connection
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
osc-MDT0000: can't setup obd: -2
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:572:class_setup()) setup mant
le-OST0000-osc-MDT0000 failed (-2)
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:1553:class_config_llog_handle
r()) MGC5.5.200.5@o2ib: cfg command failed: rc = -2
Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
0:testfs-OST0000-osc-MDT0000 1:testfs-OST
0000_UUID 2:0@<0:0>
Feb 4 12:22:58 mds1 kernel: LustreError:
8803:0:(obd_config.c:1553:class_config_llog_handler())
MGC5.5.200.5@o2ib: cfg command failed: rc = -2
Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
0:testfs-OST0000-osc-MDT0000
1:testfs-OST0000_UUID 2:0@<0:0>
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
2:32 p.m.
What steps did you take to do this?
This was an existing 2.1.6, upgraded to 2.4.2 while stil using combined
MDS/MGS.
MGS was split using:
mkfs.lustre --mgs <new-mgs-device>
mount -t ldiskfs <old-mgs-device> /mnt/1
mount -t ldiskfs <new-mgs-device> /mnt/2
cp -r /mnt/1/CONFIGS/testfs-* /mnt/2/CONFIGS
then I did --nomgs on the previous combo device.
Actually the entire filesystem came up correctly after that: MGT MDT 4 x
OSTs.
The system was taken down for hardware maintenance and I encountered this
"AT" osp issue on restart.
The system was cleanly unmounted (or so I thought).
Thanks.
3:46 p.m.
Another observation:
With my stuck OST in state "AT", the lustre modules on the OSS end can be
cleanly removed
when the OST is umounted.
On the MDS/MGS side there is a stuck zombie thread [obd_zombid] even after
unmounting
the MDT & MGT.
lctl dl will still show "9 AT osp ...." as the only remaining lustre device
A reboot of the server is required to clean up this process.
On Wed, Feb 5, 2014 at 3:35 AM, Dilger, Andreas <andreas.dilger(a)intel.com>wrote:
On 2014/02/03, 9:29 PM, "Anthony Alba"
<ascanio.alba7(a)gmail.com> wrote:
>I have split an MDS from an MGS using 2.4.2; running on the same server
>using
>CentOS 2.6.32-358.23.2.el6_lustre + ldiskfs only.
What steps did you take to do this?
>I have done a writeconf; now the MDS won't "register" with the MGS.
>
>
>On the split MDS/MGS server the MGT and MDT devices mount but the logs
>show:
>
>
>Process recover log testfs-mtdir error -22.
Did you copy this file over to the new MGS?
>However, if I now try to mount the OSTs, they get stuck in "AT".
>
>
>One symptom is that on the split MGS/MDS the lustre modules cannot unload,
>as some process is using osp.o. If I don't start the OSTs the lustre
>stack can be unloaded.
>
>
>Any suggestions on getting the MDS/MGS to play nice?
>
>
>Thanks
>Anthony
>
>
>
>
>[root@mds1 ~]# lctl dl
> 0 UP osd-ldiskfs MGS-osd MGS-osd_UUID 5
> 1 UP mgs MGS MGS 5
> 2 UP mgc MGC5.5.200.5@o2ib 9ce8a82a-136e-616e-f6f3-f4570fbd364e 5
> 3 UP osd-ldiskfs testfs-MDT0000-osd testfs-MDT0000-osd_UUID 7
> 4 UP mds MDS MDS_uuid 3
> 5 UP lod testfs-MDT0000-mdtlov testfs-MDT0000-mdtlov_UUID 4
> 6 UP mdt testfs-MDT0000 testfs-MDT0000_UUID 3
> 7 UP mdd testfs-MDD0000 testfs-MDD0000_UUID 4
> 8 UP qmt testfs-QMT0000 testfs-QMT0000_UUID 4
>
>
>Feb 4 12:12:10 mds1 kernel: Lustre: MGS: Logs for fs testfs were removed
>by user request.
>All servers must be restarted in order to regenerate the logs.
>
>Feb 4 12:12:10 mds1 kernel: Lustre: testfs-MDT0000: used disk, loading
>
>Feb 4 12:12:10 mds1 kernel: LustreError:
>8412:0:(osd_io.c:1000:osd_ldiskfs_read()) testfs=M
>DT0000: can't read 128@8192 on ino 21: rc = 0
>
>Feb 4 12:12:10 mds1 kernel: LustreError:
>8412:0:(mdt_recovery.c:112:mdt_clients_data_init()
>) error reading MDS last_rcvd idx 0, off 8192: rc -14
>
>Feb 4 12:12:18 mds1 kernel: Lustre:
>6987:0:(mgc_request.c:1564:mgc_process_recover_log()) P
>rocess recover log testfs-mdtir error -22
>
>
>
>Now attempt to mount 2 x OST:
>
>
> 9 AT osp testfs-OST0000-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 1
>
>
>
>
>
>Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
>by user request.
>Feb 4 12:22:58 mds1 kernel: Lustre:
>8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
>rocess recover log testfs-mdtir error -22
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
>dd initial connection
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
>osc-MDT0000: can't setup obd: -2
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:572:class_setup()) setup testfs-OST0000-osc-MDT0000
>failed (-2)
>
>
>Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
>by user request.
>Feb 4 12:22:58 mds1 kernel: Lustre:
>8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
>rocess recover log testfs-mdtir error -22
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
>dd initial connection
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
>osc-MDT0000: can't setup obd: -2
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:572:class_setup()) setup mant
>le-OST0000-osc-MDT0000 failed (-2)
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:1553:class_config_llog_handle
>r()) MGC5.5.200.5@o2ib: cfg command failed: rc = -2
>
>Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
>0:testfs-OST0000-osc-MDT0000 1:testfs-OST
>0000_UUID 2:0@<0:0>
>
>
>
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:1553:class_config_llog_handler())
>MGC5.5.200.5@o2ib: cfg command failed: rc = -2
>
>Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
>0:testfs-OST0000-osc-MDT0000
>1:testfs-OST0000_UUID 2:0@<0:0>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
5:25 p.m.
I have also noticed, when only MDS/MGS are up I do not have an "UP" lwp
device.
I suspect this may have something to do with the connection problems.
Is there a way to clear the process recover log and get lwp up again?
On Wed, Feb 5, 2014 at 3:35 AM, Dilger, Andreas <andreas.dilger(a)intel.com>wrote:
On 2014/02/03, 9:29 PM, "Anthony Alba"
<ascanio.alba7(a)gmail.com> wrote:
>I have split an MDS from an MGS using 2.4.2; running on the same server
>using
>CentOS 2.6.32-358.23.2.el6_lustre + ldiskfs only.
What steps did you take to do this?
>I have done a writeconf; now the MDS won't "register" with the MGS.
>
>
>On the split MDS/MGS server the MGT and MDT devices mount but the logs
>show:
>
>
>Process recover log testfs-mtdir error -22.
Did you copy this file over to the new MGS?
>However, if I now try to mount the OSTs, they get stuck in "AT".
>
>
>One symptom is that on the split MGS/MDS the lustre modules cannot unload,
>as some process is using osp.o. If I don't start the OSTs the lustre
>stack can be unloaded.
>
>
>Any suggestions on getting the MDS/MGS to play nice?
>
>
>Thanks
>Anthony
>
>
>
>
>[root@mds1 ~]# lctl dl
> 0 UP osd-ldiskfs MGS-osd MGS-osd_UUID 5
> 1 UP mgs MGS MGS 5
> 2 UP mgc MGC5.5.200.5@o2ib 9ce8a82a-136e-616e-f6f3-f4570fbd364e 5
> 3 UP osd-ldiskfs testfs-MDT0000-osd testfs-MDT0000-osd_UUID 7
> 4 UP mds MDS MDS_uuid 3
> 5 UP lod testfs-MDT0000-mdtlov testfs-MDT0000-mdtlov_UUID 4
> 6 UP mdt testfs-MDT0000 testfs-MDT0000_UUID 3
> 7 UP mdd testfs-MDD0000 testfs-MDD0000_UUID 4
> 8 UP qmt testfs-QMT0000 testfs-QMT0000_UUID 4
>
>
>Feb 4 12:12:10 mds1 kernel: Lustre: MGS: Logs for fs testfs were removed
>by user request.
>All servers must be restarted in order to regenerate the logs.
>
>Feb 4 12:12:10 mds1 kernel: Lustre: testfs-MDT0000: used disk, loading
>
>Feb 4 12:12:10 mds1 kernel: LustreError:
>8412:0:(osd_io.c:1000:osd_ldiskfs_read()) testfs=M
>DT0000: can't read 128@8192 on ino 21: rc = 0
>
>Feb 4 12:12:10 mds1 kernel: LustreError:
>8412:0:(mdt_recovery.c:112:mdt_clients_data_init()
>) error reading MDS last_rcvd idx 0, off 8192: rc -14
>
>Feb 4 12:12:18 mds1 kernel: Lustre:
>6987:0:(mgc_request.c:1564:mgc_process_recover_log()) P
>rocess recover log testfs-mdtir error -22
>
>
>
>Now attempt to mount 2 x OST:
>
>
> 9 AT osp testfs-OST0000-osc-MDT0000 testfs-MDT0000-mdtlov_UUID 1
>
>
>
>
>
>Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
>by user request.
>Feb 4 12:22:58 mds1 kernel: Lustre:
>8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
>rocess recover log testfs-mdtir error -22
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
>dd initial connection
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
>osc-MDT0000: can't setup obd: -2
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:572:class_setup()) setup testfs-OST0000-osc-MDT0000
>failed (-2)
>
>
>Feb 4 12:22:48 mds1 kernel: Lustre: MGS: Regenerating testfs-OST0000 log
>by user request.
>Feb 4 12:22:58 mds1 kernel: Lustre:
>8697:0:(mgc_request.c:1564:mgc_process_recover_log()) P
>rocess recover log testfs-mdtir error -22
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(ldlm_lib.c:429:client_obd_setup()) can't a
>dd initial connection
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(osp_dev.c:686:osp_init0()) testfs-OST0000-
>osc-MDT0000: can't setup obd: -2
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:572:class_setup()) setup mant
>le-OST0000-osc-MDT0000 failed (-2)
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:1553:class_config_llog_handle
>r()) MGC5.5.200.5@o2ib: cfg command failed: rc = -2
>
>Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
>0:testfs-OST0000-osc-MDT0000 1:testfs-OST
>0000_UUID 2:0@<0:0>
>
>
>
>
>Feb 4 12:22:58 mds1 kernel: LustreError:
>8803:0:(obd_config.c:1553:class_config_llog_handler())
>MGC5.5.200.5@o2ib: cfg command failed: rc = -2
>
>Feb 4 12:22:58 mds1 kernel: Lustre: cmd=cf003
>0:testfs-OST0000-osc-MDT0000
>1:testfs-OST0000_UUID 2:0@<0:0>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel High Performance Data Division
8:03 p.m.
Further debugging through the logs
Comparing the start up of this MDS/MGS pair with a know good system,
On the known good system:
mgc_copy_llog()
lustre_start_simple() starting obd goodfs-MDT0000-lwp-MDT0000 (type=lwp)
..etc etc
On the bad system:
mgc_copy_llog()
/* no lwp stuff at all */
For some reason, my MDS/MGS pair are not invoking the lwp OBD.
1:52 a.m.
Anthony,
If you have another volume available and don’t mind losing any settings stored on the
MGS/MGT (No data – just anything you set as a conf_param), you could try formatting a
different, new, volume as the MGS/MGT. Use the usual process from the manual, then try to
start as normal.
(Note – The manual doesn’t really distinguish between the management server (MGS) and
management target (MGT).)
I suggest trying a new MGT because it seems likely your copy of the MGS/MGT didn’t get
some things, and often, replacing the MGT is fairly painless. And if it’s missing
something important, as long as you used a new volume, you’re just back where you were
before.
About the file referenced – I’m not really familiar with the on-disk contents of the MGT,
but you should be able to see that if you mount the MGT as ldiskfs. The thing that’s
worrying is that if you’re missing that, what else are you missing/didn’t get copied,
which is why I suggested trying a new MGS/MGT if that’s an option.
- Patrick
From: Anthony Alba <ascanio.alba7@gmail.com<mailto:ascanio.alba7@gmail.com>>
Date: Wednesday, February 5, 2014 at 3:03 AM
To: "Dilger, Andreas"
<andreas.dilger@intel.com<mailto:andreas.dilger@intel.com>>
Cc: "hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: Re: [HPDD-discuss] Split MDS/MGS - process recover log testfs-mtdir error -22
Further debugging through the logs
Comparing the start up of this MDS/MGS pair with a know good system,
On the known good system:
mgc_copy_llog()
lustre_start_simple() starting obd goodfs-MDT0000-lwp-MDT0000 (type=lwp)
..etc etc
On the bad system:
mgc_copy_llog()
/* no lwp stuff at all */
For some reason, my MDS/MGS pair are not invoking the lwp OBD.
2:30 a.m.
Patrick,
Thanks for the suggestion but no luck.
I recreated a clean MGT with just CONFIGS/testfs-* stuff.
When I bring up the MDS, I encounter the same issue. I get 8 OBDs UP and
I'm missing the 'UP lwp ' dev 9.
(On a newly created filesystem with only MDS and MGS I see I should have 9
OBDS in lctl dl).
My mgc_process_recover_log() returning -EINVAL may be a red herring; I see
that it is just a warning.
Ultimately mgc_process_log() returns 0. At this stage, I expect
lustre_simple_start() or something equivalent to bring up the lwp, but in
this case nothing further happens after mgc_copy_llog() returns 0.
- Richard
On Wed, Feb 5, 2014 at 10:52 PM, Patrick Farrell <paf(a)cray.com> wrote:
Anthony,
If you have another volume available and don't mind losing any settings
stored on the MGS/MGT (No data - just anything you set as a conf_param),
you could try formatting a different, new, volume as the MGS/MGT. Use the
usual process from the manual, then try to start as normal.
(Note - The manual doesn't really distinguish between the management
server (MGS) and management target (MGT).)
I suggest trying a new MGT because it seems likely your copy of the
MGS/MGT didn't get some things, and often, replacing the MGT is fairly
painless. And if it's missing something important, as long as you used a
new volume, you're just back where you were before.
About the file referenced - I'm not really familiar with the on-disk
contents of the MGT, but you should be able to see that if you mount the
MGT as ldiskfs. The thing that's worrying is that if you're missing that,
what else are you missing/didn't get copied, which is why I suggested
trying a new MGS/MGT if that's an option.
- Patrick
From: Anthony Alba <ascanio.alba7(a)gmail.com>
Date: Wednesday, February 5, 2014 at 3:03 AM
To: "Dilger, Andreas" <andreas.dilger(a)intel.com>
Cc: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
Subject: Re: [HPDD-discuss] Split MDS/MGS - process recover log
testfs-mtdir error -22
Further debugging through the logs
Comparing the start up of this MDS/MGS pair with a know good system,
On the known good system:
mgc_copy_llog()
lustre_start_simple() starting obd goodfs-MDT0000-lwp-MDT0000 (type=lwp)
..etc etc
On the bad system:
mgc_copy_llog()
/* no lwp stuff at all */
For some reason, my MDS/MGS pair are not invoking the lwp OBD.
3:25 a.m.
Hmm. When you say with just "CONFIGS/testfs-* stuff", what do you mean?
I was suggesting you just a new MGT with the method in the manual and
then try to start the file system with that - No attempts to save any
settings or move anything to the new MGT.
Without digging in, the next thing I'd suggest is a write conf on all
the volumes of the FS and try starting again with the new MGT.
- Patrick
On 02/05/2014 09:30 AM, Anthony Alba wrote:
Patrick,
Thanks for the suggestion but no luck.
I recreated a clean MGT with just CONFIGS/testfs-* stuff.
When I bring up the MDS, I encounter the same issue. I get 8 OBDs UP
and I'm missing the 'UP lwp ' dev 9.
(On a newly created filesystem with only MDS and MGS I see I should
have 9 OBDS in lctl dl).
My mgc_process_recover_log() returning -EINVAL may be a red herring;
I see that it is just a warning.
Ultimately mgc_process_log() returns 0. At this stage, I expect
lustre_simple_start() or something equivalent to bring up the lwp, but
in this case nothing further happens after mgc_copy_llog() returns 0.
- Richard
On Wed, Feb 5, 2014 at 10:52 PM, Patrick Farrell <paf(a)cray.com
<mailto:paf@cray.com>> wrote:
Anthony,
If you have another volume available and don't mind losing any
settings stored on the MGS/MGT (No data -- just anything you set
as a conf_param), you could try formatting a different, new,
volume as the MGS/MGT. Use the usual process from the manual,
then try to start as normal.
(Note -- The manual doesn't really distinguish between the
management server (MGS) and management target (MGT).)
I suggest trying a new MGT because it seems likely your copy of
the MGS/MGT didn't get some things, and often, replacing the MGT
is fairly painless. And if it's missing something important, as
long as you used a new volume, you're just back where you were before.
About the file referenced -- I'm not really familiar with the
on-disk contents of the MGT, but you should be able to see that if
you mount the MGT as ldiskfs. The thing that's worrying is that
if you're missing that, what else are you missing/didn't get
copied, which is why I suggested trying a new MGS/MGT if that's an
option.
- Patrick
From: Anthony Alba <ascanio.alba7(a)gmail.com
<mailto:ascanio.alba7@gmail.com>>
Date: Wednesday, February 5, 2014 at 3:03 AM
To: "Dilger, Andreas" <andreas.dilger(a)intel.com
<mailto:andreas.dilger@intel.com>>
Cc: "hpdd-discuss(a)lists.01.org <mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss(a)lists.01.org <mailto:hpdd-discuss@lists.01.org>>
Subject: Re: [HPDD-discuss] Split MDS/MGS - process recover log
testfs-mtdir error -22
Further debugging through the logs
Comparing the start up of this MDS/MGS pair with a know good system,
On the known good system:
mgc_copy_llog()
lustre_start_simple() starting obd goodfs-MDT0000-lwp-MDT0000
(type=lwp)
..etc etc
On the bad system:
mgc_copy_llog()
/* no lwp stuff at all */
For some reason, my MDS/MGS pair are not invoking the lwp OBD.
3:34 a.m.
I misunderstood; I think I see what you mean now: create an empty MGS and
let the MGD/OSSes reconnect from scratch rather than try to reproduce my
existing MGS.
Ok, gotta try that and see.
On Thu, Feb 6, 2014 at 12:25 AM, Patrick Farrell <paf(a)cray.com> wrote:
Hmm. When you say with just "CONFIGS/testfs-* stuff",
what do you mean?
I was suggesting you just a new MGT with the method in the manual and then
try to start the file system with that - No attempts to save any settings
or move anything to the new MGT.
Without digging in, the next thing I'd suggest is a write conf on all the
volumes of the FS and try starting again with the new MGT.
- Patrick
On 02/05/2014 09:30 AM, Anthony Alba wrote:
Patrick,
Thanks for the suggestion but no luck.
I recreated a clean MGT with just CONFIGS/testfs-* stuff.
When I bring up the MDS, I encounter the same issue. I get 8 OBDs UP and
I'm missing the 'UP lwp ' dev 9.
(On a newly created filesystem with only MDS and MGS I see I should have 9
OBDS in lctl dl).
My mgc_process_recover_log() returning -EINVAL may be a red herring; I
see that it is just a warning.
Ultimately mgc_process_log() returns 0. At this stage, I expect
lustre_simple_start() or something equivalent to bring up the lwp, but in
this case nothing further happens after mgc_copy_llog() returns 0.
- Richard
On Wed, Feb 5, 2014 at 10:52 PM, Patrick Farrell <paf(a)cray.com> wrote:
> Anthony,
>
> If you have another volume available and don't mind losing any settings
> stored on the MGS/MGT (No data - just anything you set as a conf_param),
> you could try formatting a different, new, volume as the MGS/MGT. Use the
> usual process from the manual, then try to start as normal.
>
> (Note - The manual doesn't really distinguish between the management
> server (MGS) and management target (MGT).)
>
> I suggest trying a new MGT because it seems likely your copy of the
> MGS/MGT didn't get some things, and often, replacing the MGT is fairly
> painless. And if it's missing something important, as long as you used a
> new volume, you're just back where you were before.
>
> About the file referenced - I'm not really familiar with the on-disk
> contents of the MGT, but you should be able to see that if you mount the
> MGT as ldiskfs. The thing that's worrying is that if you're missing that,
> what else are you missing/didn't get copied, which is why I suggested
> trying a new MGS/MGT if that's an option.
>
> - Patrick
>
> From: Anthony Alba <ascanio.alba7(a)gmail.com>
> Date: Wednesday, February 5, 2014 at 3:03 AM
> To: "Dilger, Andreas" <andreas.dilger(a)intel.com>
> Cc: "hpdd-discuss(a)lists.01.org" <hpdd-discuss(a)lists.01.org>
> Subject: Re: [HPDD-discuss] Split MDS/MGS - process recover log
> testfs-mtdir error -22
>
> Further debugging through the logs
>
> Comparing the start up of this MDS/MGS pair with a know good system,
>
> On the known good system:
>
> mgc_copy_llog()
> lustre_start_simple() starting obd goodfs-MDT0000-lwp-MDT0000 (type=lwp)
> ..etc etc
>
> On the bad system:
> mgc_copy_llog()
> /* no lwp stuff at all */
>
>
> For some reason, my MDS/MGS pair are not invoking the lwp OBD.
>
3147
days inactive
3148
days old
10 comments
3 participants
participants (3)
-
Anthony Alba -
Dilger, Andreas -
Patrick Farrell