Hi Dan,

I found the “Dilger Procedure” on lustre-discuss. In this case, it was applied to an MDT.

Here’s what I did:

mount -t ldiskfs /dev/sdc1 /mnt/ost

cp /mnt/ost/last_rcvd  /mnt/ost/last_rcvd.sav

cp /mnt/ost/last_rcvd  /tmp/last_rcvd.sav

dd if=/mnt/ost/last_rcvd.sav of=/mnt/ost/last_rcvd bs=8k count=1

umount /mnt/ost

mount -t lustre /dev/sdc1 /mnt/ost

The OSS crashed again. The message in the system log is different:

Apr  4 13:57:16 fn3 kernel: kjournald starting.  Commit interval 5 seconds

Apr  4 13:57:16 fn3 kernel: LDISKFS FS on sdc1, internal journal

Apr  4 13:57:16 fn3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.

Apr  4 13:57:16 fn3 kernel: kjournald starting.  Commit interval 5 seconds

Apr  4 13:57:16 fn3 kernel: LDISKFS FS on sdc1, internal journal

Apr  4 13:57:16 fn3 kernel: LDISKFS-fs: mounted filesystem with ordered data mode.

Apr  4 13:57:16 fn3 kernel: LDISKFS-fs: file extents enabled

Apr  4 13:57:16 fn3 kernel: LDISKFS-fs: mballoc enabled

Apr  4 13:57:16 fn3 kernel: Lustre: ost_num_threads module parameter is deprecated, use oss_num_threads instead or unset both for dynamic thread startup

Apr  4 13:57:16 fn3 kernel: Lustre: Found index 0 for home1fs-OST0000, updating log

After that, the OSS crashed. Had to reset it to reboot.

On the other side, after rebooting, fsck /dev/sdc1 now runs correctly and recognizes the filesystem as an ext3.

I suspect that last_recv file is corrupted. Here’s the output when I cat this file:

xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4mX4xV4mX4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4x

V4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4xV4

This is a lot different than what I get on the OST of another Lustre server. Is there a way to rebuild the last_recv file?

Denis