Re: [HPDD-discuss] Mellanox OFED MLNX_OFED_LINUX-1.5.3-3 for Lustre not working

** Hi, your procedure is correct, but you first need to recompile Mellanox OFED against your kernel using mlnx_add_kernel_support.sh from the .iso. This will generate a new iso which will install correctly against the lustre kernel. After that you have to recompile (at least) lustre-modules using --with-o2ib=/usr/src/ofa_kernel configure switch as you did. I don't want to question your choice, but since this procedure is a pain [you have to redo everything again for security updates] you should consider again if you can work with the stock IB stack. On my side I never saw a performance difference between the two, but if someone did, feel free to share :). Kind regards Enrico Tagliavini On Tuesday 07 May 2013 16:34:41 linux freaker wrote: Hi, I installed RHEL 6.3 which by default comes with openib (Infiniband support software). I needed MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64.iso to be installed and so I extracted and ran ./mlnxofedinstall script. It removed the old Mellanox installables and drivers for a while but couldnt install due to lustre kernel. In curious, I installed it on default kernel. It went fine on default RHEL kernel. and While I rebooted and tried to get lustre up through lctl network up it threw error on MDS: [root@oss2 ~]# modprobe lustre WARNING: Error inserting fld (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/fld.ko): Input/output error WARNING: Error inserting fid (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/fid.ko): Input/output error WARNING: Error inserting mdc (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/mdc.ko): Input/output error WARNING: Error inserting osc (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/osc.ko): Input/output error WARNING: Error inserting lov (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/lov.ko): Input/output error FATAL: Error inserting lustre (/lib/modules/2.6.32-279.14.1.el6_lustre.x86_64/updates/kernel/fs/lustre/lustre.ko): Input/output error [root@oss2 ~]# modprobe lnet [root@oss2 ~]# lctl network up LNET configure error 100: Network is down Now it seemed I need to build OFED for lustre and I got this thread: http://thr3ads.net/lustre-discuss/2012/12/2164117-problem-with-installing... It suggested to do the following steps and I followed it line by line: 2. boot into the lustre kernel 3. in our /usr/src/lustre-2.1.2 directory built lustre against the Mellanox "Module.symvers" information (which is why you see the "Input/Output" errors on fid.ko, mdc.ko, osc.ko, lov.ko and because of the aforementioned items, the lustre.ko. The MLNX version 1.8.5 that we needed was in the /usr/src/ofa_kernel directory (with the Module.symvers etc....) We used the defaults other than the o2ib so our command in the /usr/src/lustre-2.1.2 directory looked like "./configure --with-o2ib=/usr/src/ofa_kernel" 4. next we issued "make" 5. next we chose to run a "make rpms" command so that we could have rpms for our system for cluster re-building But even this failed to get my lustre up.modprobe lnet work but lctl network up doesnt. lctl list_nids IOC_LIBCFS_GET_NI error 100: Network is down My ifconfig shows: ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::202:c903:b:8b85/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:2628 errors:0 dropped:0 overruns:0 frame:0 TX packets:20 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1024 RX bytes:155819 (152.1 KiB) TX bytes:3503 (3.4 KiB) uname -arn Linux oss2 2.6.32-279.14.1.el6_lustre.x86_64 #1 SMP Fri Dec 14 23:22:17 PST 2012 x86_64 x86_64 x86_64 GNU/Linux I did tried running kernel support script under MLNX directory and it did installed RPM but no luck with lctl list_nids . Can anyone suggest how to fix it?