Clocks are synchronized, /etc/modprobe.d/lustre.conf is identical through out all the machine, Iptables and Ip6tables are turned off, SELinux is turned off, lctl ping is responsive through out all the machines, lnet module is loaded and all hosts are resolvable.

Mar 19 09:00:43 hms-05 kernel: Lustre: Lustre: Build Version: 2.4.3-RC1--PRISTINE-2.6.32-358.23.2.el6.x86_64

Mar 19 09:00:44 hms-05 kernel: LustreError: 1202:0:(mgc_request.c:1403:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 132.65.56.203@tcp

Mar 19 09:00:44 hms-05 kernel: Lustre: 1202:0:(mgc_request.c:1564:mgc_process_recover_log()) Process recover log fs01-cliir error -2

Mar 19 09:00:44 hms-05 kernel: LustreError: 11-0: fs01-MDT0000-mdc-ffff88001a12a000: Communicating with 132.65.56.201@tcp, operation mds_connect failed with -11.

Mar 19 09:01:03 hms-05 kernel: LustreError: 1202:0:(lmv_obd.c:1289:lmv_statfs()) can't stat MDS #0 (fs01-MDT0000-mdc-ffff88001a12a000), error -11

Mar 19 09:01:03 hms-05 kernel: LustreError: 1205:0:(lov_obd.c:937:lov_cleanup()) lov tgt 1 not cleaned! deathrow=0, lovrc=1

Mar 19 09:01:03 hms-05 kernel: Lustre: Unmounted fs01-client

Mar 19 09:01:03 hms-05 kernel: LustreError: 1202:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount  (-11)

Mar 19 09:03:46 hms-05 kernel: LustreError: 1248:0:(mgc_request.c:1403:mgc_apply_recover_logs()) mgc: cannot find uuid by nid 132.65.56.203@tcp

Mar 19 09:03:46 hms-05 kernel: Lustre: 1248:0:(mgc_request.c:1564:mgc_process_recover_log()) Process recover log fs01-cliir error -2

Mar 19 09:03:46 hms-05 kernel: LustreError: 11-0: fs01-MDT0000-mdc-ffff88001d6f5c00: Communicating with 132.65.56.201@tcp, operation mds_connect failed with -11.

Mar 19 09:04:11 hms-05 kernel: LustreError: 11-0: fs01-MDT0000-mdc-ffff88001d6f5c00: Communicating with 132.65.56.201@tcp, operation mds_connect failed with -11.

Mar 19 09:04:15 hms-05 init: tty (/dev/tty1) main process (1171) killed by INT signal

Mar 19 09:04:15 hms-05 init: tty (/dev/tty1) main process ended, respawning

Mar 19 09:04:16 hms-05 kernel: LustreError: 1248:0:(lmv_obd.c:1289:lmv_statfs()) can't stat MDS #0 (fs01-MDT0000-mdc-ffff88001d6f5c00), error -11

Mar 19 09:04:16 hms-05 kernel: LustreError: 1205:0:(lov_obd.c:937:lov_cleanup()) lov tgt 1 not cleaned! deathrow=0, lovrc=1

Mar 19 09:04:16 hms-05 kernel: Lustre: Unmounted fs01-client

Mar 19 09:04:16 hms-05 kernel: LustreError: 1248:0:(obd_mount.c:1289:lustre_fill_super()) Unable to mount  (-11)