May 9 18:39:25 ctaoss9 kernel: Lustre: 14967:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1368117558/real 0] req@ffff8801c47eec00 x1412988625229273/t0(0) o105->fs13-OST0004@xxx.xxx.15.60@tcp:15/16 lens 344/192 e 0 to 1 dl 1368117565 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 May 9 18:39:25 ctaoss9 kernel: LustreError: 138-a: fs13-OST0004: A client on nid xxx.xxx.15.60@tcp was evicted due to a lock completion callback time out: rc -107 May 9 18:39:25 ctaoss9 kernel: LustreError: 17766:0:(ldlm_lib.c:2652:target_bulk_io()) @@@ Eviction on bulk PUT req@ffff8801c6a69400 x1432649487261875/t0(0) o3->32d1ef37-5f4d-ad21-a3c8-8fce80f52f85@xxx.xxx.15.60@tcp:0/0 lens 448/400 e 0 to 0 dl 1368118303 ref 1 fl Interpret:/0/0 rc 0/0 May 9 18:39:25 ctaoss9 kernel: LustreError: 27927:0:(ldlm_lib.c:2652:target_bulk_io()) @@@ Eviction on bulk PUT req@ffff880289c73c00 x1432649487261876/t0(0) o3->32d1ef37-5f4d-ad21-a3c8-8fce80f52f85@xxx.xxx.15.60@tcp:0/0 lens 448/400 e 0 to 0 dl 1368118303 ref 1 fl Interpret:/0/0 rc 0/0 May 9 18:39:26 bladeed kernel: Lustre: 842:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1432649487261926 sent from fs13-OST0004-osc-ffff880812c01000 to NID xxx.xxx.22.98@tcp 8s ago has timed out (8s prior to deadline). May 9 18:39:26 bladeed kernel: req@ffff880b0c3d0800 x1432649487261926/t0 o101->fs13-OST0004_UUID@xxx.xxx.22.98@tcp:28/4 lens 296/544 e 0 to 1 dl 1368117566 ref 1 fl Rpc:/0/0 rc 0/0 May 9 18:39:26 bladeed kernel: Lustre: fs13-OST0004-osc-ffff880812c01000: Connection to service fs13-OST0004 via nid xxx.xxx.22.98@tcp was lost; in progress operations using this service will wait for recovery to complete. May 9 18:39:26 ctaoss9 kernel: Lustre: 27844:0:(ldlm_lib.c:946:target_handle_connect()) fs13-OST0004: connection from 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85@xxx.xxx.15.60@tcp t0 exp (null) cur 1368117566 last 0 May 9 18:39:26 bladeed kernel: Lustre: 26167:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1432649487261927 sent from fs13-OST0004-osc-ffff880812c01000 to NID xxx.xxx.22.98@tcp 8s ago has timed out (8s prior to deadline). May 9 18:39:26 bladeed kernel: req@ffff88101f9d4c00 x1432649487261927/t0 o103->fs13-OST0004_UUID@xxx.xxx.22.98@tcp:17/18 lens 296/384 e 0 to 1 dl 1368117566 ref 1 fl Rpc:N/0/0 rc 0/0 May 9 18:39:26 bladeed kernel: LustreError: 26167:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway May 9 18:39:26 bladeed kernel: LustreError: 26167:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11 May 9 18:39:33 bladeed kernel: Lustre: 6133:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1432649487261947 sent from fs13-OST0004-osc-ffff880812c01000 to NID xxx.xxx.22.98@tcp 7s ago has timed out (7s prior to deadline). May 9 18:39:33 bladeed kernel: req@ffff8809ff01b800 x1432649487261947/t0 o8->fs13-OST0004_UUID@xxx.xxx.22.98@tcp:28/4 lens 368/584 e 0 to 1 dl 1368117573 ref 1 fl Rpc:N/0/0 rc 0/0 May 9 18:39:34 bladeed kernel: Lustre: 6134:0:(import.c:517:import_select_connection()) fs13-OST0004-osc-ffff880812c01000: tried all connections, increasing latency to 3s May 9 18:39:34 ctaoss9 kernel: Lustre: fs13-OST0004: Client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) reconnecting May 9 18:39:36 ctaoss9 kernel: Lustre: fs13-OST0004: Bulk IO read error with 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp), client will retry: rc -107 May 9 18:39:43 ctamds2 kernel: Lustre: 17230:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1412813885590459 sent from fs9-MDT0000 to NID xxx.xxx.15.60@tcp 7s ago has timed out (7s prior to deadline). May 9 18:39:43 ctamds2 kernel: LustreError: 138-a: fs9-MDT0000: A client on nid xxx.xxx.15.60@tcp was evicted due to a lock blocking callback to xxx.xxx.15.60@tcp timed out: rc -107 May 9 18:39:52 ctamds3 kernel: Lustre: 30650:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1368117585/real 1368117585] req@ffff8805fd930c00 x1433904862686942/t0(0) o104->fs13-MDT0000@xxx.xxx.15.60@tcp:15/16 lens 296/192 e 0 to 1 dl 1368117592 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 May 9 18:39:52 ctamds3 kernel: LustreError: 138-a: fs13-MDT0000: A client on nid xxx.xxx.15.60@tcp was evicted due to a lock blocking callback time out: rc -107 May 9 18:40:02 ctaoss9 kernel: LustreError: 21146:0:(socklnd_cb.c:2518:ksocknal_check_peer_timeouts()) Total 1 stale ZC_REQs for peer xxx.xxx.15.60@tcp detected; the oldest(ffff880075018000) timed out 3 secs ago, resid: 0, wmem: 4259672 May 9 18:40:05 ctaoss6 kernel: Lustre: There was an unexpected network error while writing to xxx.xxx.15.60: -110. May 9 18:43:13 ctaoss9 kernel: Lustre: fs13-OST0004: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c014d000, cur 1368117793 expire 1368117643 last 1368117566 May 9 18:43:17 ctaoss8 kernel: Lustre: fs13-OST0003: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805ff8f2400, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 ctaoss11 kernel: Lustre: fs13-OST000b: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081a23a000, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 ctaoss12 kernel: Lustre: fs13-OST000f: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081747b000, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 ctaoss9 kernel: Lustre: fs13-OST0009: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039fc62800, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 i3oss2 kernel: Lustre: fs2-OST0001: haven't heard from client ef940b44-76fa-5f77-247f-167deb096818 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 i3oss3 kernel: Lustre: fs2-OST0002: haven't heard from client ef940b44-76fa-5f77-247f-167deb096818 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 i3oss4 kernel: Lustre: fs3-OST0003: haven't heard from client d21dc799-9f4b-77e9-b73b-0707834fa426 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 plus1 kernel: Lustre: fs4-OST0004: haven't heard from client 4edfa18f-0a7a-c254-53f6-c7356da66a6e (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 plus4 kernel: Lustre: fs4-OST000b: haven't heard from client 4edfa18f-0a7a-c254-53f6-c7356da66a6e (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 plus2 kernel: Lustre: fs4-OST0009: haven't heard from client 4edfa18f-0a7a-c254-53f6-c7356da66a6e (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst16 kernel: Lustre: fs5-OST000d: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst18 kernel: Lustre: fs5-OST0014: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst19 kernel: Lustre: fs5-OST0010: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ssu46 kernel: Lustre: fs5-OST0005: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ssu45 kernel: Lustre: fs5-OST0000: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst20 kernel: Lustre: fs5-OST0011: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst13 kernel: Lustre: fs6-OST0007: haven't heard from client 9bb2534c-0f18-4a1f-060b-f1557253282d (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst14 kernel: Lustre: fs6-OST000e: haven't heard from client 9bb2534c-0f18-4a1f-060b-f1557253282d (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst22 kernel: Lustre: fs6-OST000b: haven't heard from client 9bb2534c-0f18-4a1f-060b-f1557253282d (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst21 kernel: Lustre: fs6-OST0010: haven't heard from client 9bb2534c-0f18-4a1f-060b-f1557253282d (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst15 kernel: Lustre: fs6-OST0009: haven't heard from client 9bb2534c-0f18-4a1f-060b-f1557253282d (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 hessoss2 kernel: Lustre: fs8-OST0004: haven't heard from client 1d94345a-ba98-1f2c-bf68-71501b36ab4a (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 hessoss1 kernel: Lustre: fs8-OST0003: haven't heard from client 1d94345a-ba98-1f2c-bf68-71501b36ab4a (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss10 kernel: Lustre: fs13-OST000a: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804bb081000, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 atoss3 kernel: Lustre: fs7-OST000a: haven't heard from client 7282bbd6-1520-51dd-6937-82b256810461 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 atoss1 kernel: Lustre: fs7-OST0008: haven't heard from client 7282bbd6-1520-51dd-6937-82b256810461 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 hessoss3 kernel: Lustre: fs8-OST0005: haven't heard from client 1d94345a-ba98-1f2c-bf68-71501b36ab4a (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss1 kernel: Lustre: fs9-OST0004: haven't heard from client 54bf3917-588c-c5cf-fd2c-350cf80f3b50 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss4 kernel: Lustre: fs9-OST0007: haven't heard from client 54bf3917-588c-c5cf-fd2c-350cf80f3b50 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 atoss4 kernel: Lustre: fs7-OST000b: haven't heard from client 7282bbd6-1520-51dd-6937-82b256810461 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss2 kernel: Lustre: fs9-OST0001: haven't heard from client 54bf3917-588c-c5cf-fd2c-350cf80f3b50 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 atlasoss1 kernel: Lustre: fs14-OST0000: haven't heard from client 9264b270-265e-6097-7ff3-048b1cfaa34f (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800a11ddc00, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 atlasoss2 kernel: Lustre: fs14-OST0001: haven't heard from client 9264b270-265e-6097-7ff3-048b1cfaa34f (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806dbe08000, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 atlasoss3 kernel: Lustre: fs14-OST0006: haven't heard from client 9264b270-265e-6097-7ff3-048b1cfaa34f (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a1ce8400, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 atlasoss4 kernel: Lustre: fs14-OST0003: haven't heard from client 9264b270-265e-6097-7ff3-048b1cfaa34f (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803772b7400, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 obst17 kernel: Lustre: fs5-OST0013: haven't heard from client 508bec4e-330c-08a7-069a-17d3cf02eec4 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 plus3 kernel: Lustre: fs4-OST000e: haven't heard from client 4edfa18f-0a7a-c254-53f6-c7356da66a6e (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss3 kernel: Lustre: fs9-OST0006: haven't heard from client 54bf3917-588c-c5cf-fd2c-350cf80f3b50 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 obst12 kernel: Lustre: fs6-OST000c: haven't heard from client 9bb2534c-0f18-4a1f-060b-f1557253282d (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 atoss2 kernel: Lustre: fs7-OST000d: haven't heard from client 7282bbd6-1520-51dd-6937-82b256810461 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 i3oss1 kernel: Lustre: fs3-OST0000: haven't heard from client d21dc799-9f4b-77e9-b73b-0707834fa426 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss5 kernel: Lustre: fs13-OST0000: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880316903000, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 ctaoss6 kernel: Lustre: fs13-OST0006: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88025b269800, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:17 miscoss1 kernel: Lustre: fs10-OST0000: haven't heard from client 3d51b1ce-2a56-9000-32f5-25ab1bb05cda (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss4 kernel: Lustre: fs10-OST0003: haven't heard from client 3d51b1ce-2a56-9000-32f5-25ab1bb05cda (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss2 kernel: Lustre: fs10-OST0001: haven't heard from client 3d51b1ce-2a56-9000-32f5-25ab1bb05cda (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss5 kernel: Lustre: fs11-OST0004: haven't heard from client de118cf7-2baf-2f93-fe9b-d49985eade0a (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss3 kernel: Lustre: fs12-OST0002: haven't heard from client 23ab6a9f-f2b2-b9ac-59fe-0fb2aba446b6 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss1 kernel: Lustre: fs12-OST0000: haven't heard from client 23ab6a9f-f2b2-b9ac-59fe-0fb2aba446b6 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss2 kernel: Lustre: fs12-OST0001: haven't heard from client 23ab6a9f-f2b2-b9ac-59fe-0fb2aba446b6 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss4 kernel: Lustre: fs12-OST0003: haven't heard from client 23ab6a9f-f2b2-b9ac-59fe-0fb2aba446b6 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss5 kernel: Lustre: fs12-OST0004: haven't heard from client 23ab6a9f-f2b2-b9ac-59fe-0fb2aba446b6 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 miscoss3 kernel: Lustre: fs11-OST0002: haven't heard from client de118cf7-2baf-2f93-fe9b-d49985eade0a (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:17 ctaoss7 kernel: Lustre: fs13-OST0007: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800b25ae000, cur 1368117797 expire 1368117647 last 1368117570 May 9 18:43:20 ctaoss3 kernel: Lustre: fs9-OST0002: haven't heard from client 54bf3917-588c-c5cf-fd2c-350cf80f3b50 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. May 9 18:43:23 ctaoss5 kernel: Lustre: fs13-OST0005: haven't heard from client 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88025c97f800, cur 1368117803 expire 1368117653 last 1368117576 May 9 18:43:23 ctaoss5 kernel: LustreError: 5359:0:(ldlm_lib.c:2652:target_bulk_io()) @@@ Eviction on bulk PUT req@ffff880308604850 x1432649487262160/t0(0) o3->32d1ef37-5f4d-ad21-a3c8-8fce80f52f85@xxx.xxx.15.60@tcp:0/0 lens 448/400 e 0 to 0 dl 1368118331 ref 1 fl Interpret:/0/0 rc 0/0 May 9 18:43:23 ctaoss5 kernel: Lustre: fs13-OST0005: Bulk IO read error with 32d1ef37-5f4d-ad21-a3c8-8fce80f52f85 (at xxx.xxx.15.60@tcp), client will retry: rc -107