hello we add a problem on our lustre 2.5.3 infrastructure small cluster 24 nodes mds rebooted alone and now some client refused to mount again the filesystem client are in 1.8.9wc1 since we're in migration phase all clients could mount the 1.8 version 6 client can't mount again the 2.5.3 filesystem Jun 12 09:52:52 ciclad11 kernel: Lustre: Server MGS version (2.5.3.0) is much newer than client version (1.8.9) Jun 12 09:52:52 ciclad11 kernel: Lustre: MGC172.20.3.74@o2ib: Reactivating import Jun 12 09:52:52 ciclad11 kernel: Lustre: MGC172.20.3.74@o2ib: Connection restored to service MGS using nid 172.20.3.74@o2ib. Jun 12 09:52:52 ciclad11 kernel: Lustre: client etherfs-client(ffff88040ef2f800) umount complete Jun 12 09:52:52 ciclad11 kernel: LustreError: 4754:0:(obd_mount.c:2067:lustre_fill_super()) Unable to mount (-4) log from mds Jun 12 09:52:52 mds2-ipsl kernel: Lustre: MGS: Client e26b1313-d901-a410-7c8b-6c6148b6bd92 (at 172.20.3.243@o2ib) reconnecting Jun 12 09:53:45 mds2-ipsl kernel: Lustre: MGS: Client a3cd5035-35d2-4f23-e337-73d0e7192047 (at 172.20.3.243@o2ib) reconnecting Jun 12 09:54:38 mds2-ipsl kernel: Lustre: MGS: Client b7896a48-b23d-1651-4b3f-fa5c90cceab7 (at 172.20.3.243@o2ib) reconnecting Jun 12 09:55:41 mds2-ipsl kernel: Lustre: MGS: haven't heard from client b25d7b77-22b5-c391-b883-7ae8f2044d09 (at 172.20.3.243@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8811c86af400, cur 1434095741 expire 1434095591 last 1434095513 I try to change the client version on not working client Jun 12 12:01:35 ciclad19 kernel: Lustre: 13789:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1434103288/real 1434103288] req@ffff88301856b800 x1503749299241316/t0(0) o503->MGC172.20.3.74@o2ib@172.20.3.74@o2ib:26/25 lens 272/8416 e 0 to 1 dl 1434103295 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 12 12:01:35 ciclad19 kernel: LustreError: 166-1: MGC172.20.3.74@o2ib: Connection to MGS (at 172.20.3.74@o2ib) was lost; in progress operations using this service will fail Jun 12 12:01:41 ciclad19 kernel: Lustre: 3851:0:(client.c:1918:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1434103295/real 1434103295] req@ffff88381ba7fc00 x1503749299241320/t0(0) o250->MGC172.20.3.74@o2ib@172.20.3.74@o2ib:26/25 lens 400/544 e 0 to 1 dl 1434103301 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 12 12:01:48 ciclad19 kernel: LustreError: 15c-8: MGC172.20.3.74@o2ib: The configuration from log 'etherfs-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. Jun 12 12:01:48 ciclad19 kernel: LustreError: 13789:0:(llite_lib.c:1046:ll_fill_super()) Unable to process log: -5 Jun 12 12:01:48 ciclad19 kernel: Lustre: Unmounted etherfs-client Jun 12 12:01:48 ciclad19 kernel: LustreError: 13789:0:(obd_mount.c:1325:lustre_fill_super()) Unable to mount (-5) from mds Jun 12 12:01:35 mds2-ipsl kernel: Lustre: MGS: Client 5c623fa9-1cae-6b75-5e15-acb8add53042 (at 172.20.3.235@o2ib) reconnecting Jun 12 12:05:25 mds2-ipsl kernel: Lustre: MGS: haven't heard from client 5c623fa9-1cae-6b75-5e15-acb8add53042 (at 172.20.3.235@o2ib) in 230 seconds. I think it's dead, and I am evicting it. exp ffff88203ddda000, cur 1434103525 expire 1434103375 last 1434103295 any idea -- Weill Philippe - Administrateur Systeme et Reseaux CNRS/UPMC/IPSL LATMOS (UMR 8190)