3:35 a.m.
Hi,
I am trying to compile a Lustre 2.6 client on ubuntu 14.04, kernel
3.13.0 and Mellanox OFED 2.2-1. I saw that developpers work hard to get
it supported, what is the status of this? Is it feasible? I did not
succeed yet.
Thanks.
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
Dorval, QC
Gouvernement du Canada | Government of Canada
7:56 a.m.
Hi,
I am trying to compile a Lustre 2.6 client on ubuntu 14.04, kernel
3.13.0 and Mellanox OFED 2.2-1. I saw that developpers work hard to get
it supported, what is the status of this? Is it feasible? I did not
succeed yet.
What errors are you seeing? It should work for the latest 2.6 code.
8:35 p.m.
Hi James,
Here is what I got:
git clone git://git.whamcloud.com/fs/lustre-release.git
ib8-bc1l42-be01:~# uname -a
Linux ib8-bc1l42-be01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15
03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
ib8-bc1l42-be01:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
ib8-bc1l42-be01:~/lustre-release# ./configure --disable-server
--with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --enable-quota
ib8-bc1l42-be01:~/lustre-release# make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
make: *** [all] Error 2
Thanks.
On 07/18/14 21:56, Simmons, James A. wrote:
> Hi,
>
> I am trying to compile a Lustre 2.6 client on ubuntu 14.04, kernel
> 3.13.0 and Mellanox OFED 2.2-1. I saw that developpers work hard to get
> it supported, what is the status of this? Is it feasible? I did not
> succeed yet.
What errors are you seeing? It should work for the latest 2.6 code.
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
8:57 p.m.
On debian we build like this:
./configure --disable-modules --disable-server --enable-client
--prefix=/usr/local/lustre/2.4.2
The modules we build as a part of the staging kernel (currently we use
3.14) that we build in house.
I don't know if the Ubuntu/Debian kernels ship with the lustre module
compiled, but I can imagine that they don't compile it yet...
Regards,
Eliyahu - אליהו
On Mon, Jul 21, 2014 at 1:35 PM, Patrice Hamelin <
patrice.hamelin(a)ssc-spc.gc.ca> wrote:
Hi James,
Here is what I got:
git clone git://git.whamcloud.com/fs/lustre-release.git
ib8-bc1l42-be01:~# uname -a
Linux ib8-bc1l42-be01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
ib8-bc1l42-be01:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
ib8-bc1l42-be01:~/lustre-release# ./configure --disable-server
--with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --enable-quota
ib8-bc1l42-be01:~/lustre-release# make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
Why are you building as root, only the install stage you would need to be
root/sudo.....
make: *** [all] Error 2
Thanks.
On 07/18/14 21:56, Simmons, James A. wrote:
> Hi,
>>
>> I am trying to compile a Lustre 2.6 client on ubuntu 14.04, kernel
>> 3.13.0 and Mellanox OFED 2.2-1. I saw that developpers work hard to get
>> it supported, what is the status of this? Is it feasible? I did not
>> succeed yet.
>>
> What errors are you seeing? It should work for the latest 2.6 code.
>
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
12:56 a.m.
On debian we build like this:
./configure --disable-modules --disable-server --enable-client
>--prefix=/usr/local/lustre/2.4.2
The modules we build as a part of the staging kernel (currently we use
3.14) that we build in house.
I don't know if the Ubuntu/Debian kernels ship with the lustre module compiled, but I
can imagine that >they don't compile it yet...
For 3.14 support you will need the following patch.
http://review.whamcloud.com/#/c/10995
If you want ldiskfs supprot for 3.14 you will need the following patches.
http://review.whamcloud.com/#/c/11088
http://review.whamcloud.com/#/c/10165
http://review.whamcloud.com/#/c/11000
Please note this is all code in development. Its for testing and not a production system.
On Mon, Jul 21, 2014 at 1:35 PM, Patrice Hamelin
<patrice.hamelin@ssc-spc.gc.ca<mailto:patrice.hamelin@ssc-spc.gc.ca>> wrote:
Hi James,
Here is what I got:
git clone
git://git.whamcloud.com/fs/lustre-release.git<http://git.whamcloud.com...
ib8-bc1l42-be01:~# uname -a
Linux ib8-bc1l42-be01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
ib8-bc1l42-be01:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
ib8-bc1l42-be01:~/lustre-release# ./configure --disable-server
--with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --enable-quota
ib8-bc1l42-be01:~/lustre-release# make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
Why are you building as root, only the install stage you would need to be root/sudo.....
make: *** [all] Error 2
Did you do a make oldconfig and make prepare in your kernel source tree?
8:47 p.m.
Hi James,
I am running 3.13 so I did not installed the patches. After
commenting out a symbol redefinition in autoconf.h from Mellanox:
include/generated/autoconf.h:3886:0: note: this is the location of the
previous definition
#define CONFIG_LNET_MAX_PAYLOAD 1048576
I am getting:
./configure --disable-modules --disable-server --enable-client
--prefix=/usr/local/lustre/2.4.2 --with-o2ib=/usr/src/mlnx-ofed-kernel-2.2
make
util/parser.c: In function 'init_input':
util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in this
function)
rl_attempted_completion_function = (CPPFunction
*)command_completion;
^
Mr. Rosenberg is right, the lnet module is a part of Trusty and it is
compiled and loading.
root:~/lustre-release# modinfo lnet
filename:
/lib/modules/3.13.0-32-generic/kernel/drivers/staging/lustre/lnet/lnet/lnet.ko
version: 1.0.0
license: GPL
description: Portals v3.1
author: Peter J. Braam <braam(a)clusterfs.com>
srcversion: 3441E33D76FAFAECF683EBF
root:~/lustre-release# cat /etc/modprobe.d/lustre.conf
options lnet networks=o2ib(bond1)
root:~/lustre-release# modprobe lnet
root:~/lustre-release# lsmod | grep lnet
lnet 345673 0
libcfs 331973 1 lnet
dmesg
[311197.918239] libcfs: module is from the staging directory, the
quality is unknown, you have been warned.
[311197.921832] LNet: HW CPU cores: 32, npartitions: 4
[311205.929725] lnet: module is from the staging directory, the quality
is unknown, you have been warned.
root:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
Where should I go from there? I found some solutions to this on the net
but it did not produce a viable Lustre.
Thanks for help!
On 07/21/14 14:56, Simmons, James A. wrote:
On debian we build like this:
> ./configure --disable-modules --disable-server
--enable-client>--prefix=/usr/local/lustre/2.4.2
> The modules we build as a part of the staging kernel (currently we use 3.14) that we
build in house.
> I don't know if the Ubuntu/Debian kernels ship with the lustre module compiled,
but I can imagine that>they don't compile it yet...
For 3.14 support you will need the following patch.
http://review.whamcloud.com/#/c/10995
If you want ldiskfs supprot for 3.14 you will need the following patches.
http://review.whamcloud.com/#/c/11088
http://review.whamcloud.com/#/c/10165
http://review.whamcloud.com/#/c/11000
Please note this is all code in development. Its for testing and not a production
system.
On Mon, Jul 21, 2014 at 1:35 PM, Patrice
Hamelin<patrice.hamelin@ssc-spc.gc.ca<mailto:patrice.hamelin@ssc-spc.gc.ca>>
wrote:
Hi James,
Here is what I got:
git clone
git://git.whamcloud.com/fs/lustre-release.git<http://git.whamcloud.com...
ib8-bc1l42-be01:~# uname -a
Linux ib8-bc1l42-be01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
ib8-bc1l42-be01:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
ib8-bc1l42-be01:~/lustre-release# ./configure --disable-server
--with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --enable-quota
ib8-bc1l42-be01:~/lustre-release# make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
Why are you building as root, only the install stage you would need to be root/sudo.....
make: *** [all] Error 2
Did you do a make oldconfig and make prepare in your kernel source tree?
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
11:45 p.m.
Please note that we are still using lustre 2.4.x and not 2.6.x or 2.5.x
We have not used any special patches except for aufs and mosix and all
seems to work well.
We hope to switch to lustre 2.5.x somewhere in the not too far future.
Regards,
Eliyahu - אליהו
On Tue, Jul 22, 2014 at 1:47 PM, Patrice Hamelin <
patrice.hamelin(a)ssc-spc.gc.ca> wrote:
Hi James,
I am running 3.13 so I did not installed the patches. After commenting
out a symbol redefinition in autoconf.h from Mellanox:
include/generated/autoconf.h:3886:0: note: this is the location of the
previous definition
#define CONFIG_LNET_MAX_PAYLOAD 1048576
I am getting:
./configure --disable-modules --disable-server --enable-client
--prefix=/usr/local/lustre/2.4.2 --with-o2ib=/usr/src/mlnx-ofed-kernel-2.2
make
util/parser.c: In function 'init_input':
util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in this
function)
rl_attempted_completion_function = (CPPFunction
*)command_completion;
^
Mr. Rosenberg is right, the lnet module is a part of Trusty and it is
compiled and loading.
root:~/lustre-release# modinfo lnet
filename: /lib/modules/3.13.0-32-generic/kernel/drivers/
staging/lustre/lnet/lnet/lnet.ko
version: 1.0.0
license: GPL
description: Portals v3.1
author: Peter J. Braam <braam(a)clusterfs.com>
srcversion: 3441E33D76FAFAECF683EBF
root:~/lustre-release# cat /etc/modprobe.d/lustre.conf
options lnet networks=o2ib(bond1)
root:~/lustre-release# modprobe lnet
root:~/lustre-release# lsmod | grep lnet
lnet 345673 0
libcfs 331973 1 lnet
dmesg
[311197.918239] libcfs: module is from the staging directory, the quality
is unknown, you have been warned.
[311197.921832] LNet: HW CPU cores: 32, npartitions: 4
[311205.929725] lnet: module is from the staging directory, the quality is
unknown, you have been warned.
root:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
Where should I go from there? I found some solutions to this on the net
but it did not produce a viable Lustre.
Thanks for help!
On 07/21/14 14:56, Simmons, James A. wrote:
> On debian we build like this:
>
> ./configure --disable-modules --disable-server
>> --enable-client>--prefix=/usr/local/lustre/2.4.2
>> The modules we build as a part of the staging kernel (currently we use
>> 3.14) that we build in house.
>> I don't know if the Ubuntu/Debian kernels ship with the lustre module
>> compiled, but I can imagine that>they don't compile it yet...
>>
> For 3.14 support you will need the following patch.
>
> http://review.whamcloud.com/#/c/10995
>
> If you want ldiskfs supprot for 3.14 you will need the following patches.
>
> http://review.whamcloud.com/#/c/11088
> http://review.whamcloud.com/#/c/10165
> http://review.whamcloud.com/#/c/11000
>
> Please note this is all code in development. Its for testing and not a
> production system.
>
> On Mon, Jul 21, 2014 at 1:35 PM, Patrice Hamelin<patrice.hamelin@ssc-
> spc.gc.ca<mailto:patrice.hamelin@ssc-spc.gc.ca>> wrote:
> Hi James,
>
> Here is what I got:
>
> git clone git://git.whamcloud.com/fs/lustre-release.git<http://git.
> whamcloud.com/fs/lustre-release.git>
> ib8-bc1l42-be01:~# uname -a
> Linux ib8-bc1l42-be01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15
> 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> ib8-bc1l42-be01:~/lustre-release# git describe
> v2_6_50_0-29-gfa4a67d
>
> ib8-bc1l42-be01:~/lustre-release# ./configure --disable-server
> --with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --enable-quota
> ib8-bc1l42-be01:~/lustre-release# make
> ...
> Building modules, stage 2.
> MODPOST 18 modules
> FATAL: parse error in symbol dump file
> make[4]: *** [__modpost] Error 1
> make[3]: *** [modules] Error 2
> make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
> make[2]: *** [modules] Error 2
> make[2]: Leaving directory `/root/lustre-release'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/root/lustre-release'
> Why are you building as root, only the install stage you would need to be
> root/sudo.....
> make: *** [all] Error 2
>
>
> Did you do a make oldconfig and make prepare in your kernel source tree?
>
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
12:11 a.m.
Hi,
I am installing a brand new cluster and would like to run the latest
Lustre 2.6. I am almost there. Here is what I am doing:
cd /usr/src
tar xvf linux-source-3.13.0.tar.bz2
cd linux-source-3.13.0
make oldconfig
make prepare
# Recipe from Mellanox engineering to avoid the following configure error
#configure:error:can't compile with OpenIB gen2 headers
cd /usr/src/mlnx-ofed-kernel-2.2
ofed_scripts/gen-compat-autoconf.sh include/linux/compat-3.5.h
include/linux/compat_autoconf.h
exportMODULES_DIR=/lib/modules
ofed_scripts/create_Module.symvers.sh
cd /root
rm -rf lustre-release
cp -rv ~aspgpah/data/Lustre/lustre-release .
cd lustre-release
sed -i 's/1.10 tar-ustar -Wno-portability/1.10 tar-ustar
-Wno-portability subdir-objects/g' configure.ac
# remove duplicated variable definition
sed -i '3886d'
/usr/src/linux-headers-3.13.0-32-generic/include/generated/autoconf.h
sh autogen.sh
./configure --with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --disable-server
--enable-quota
make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
make: *** [all] Error 2
I am stucked with that error for a while now. What is the symbol
dump file? Module.symvers?
I am continuing to pull my hair on this.
Thanks.
On 07/24/14 13:45, E.S. Rosenberg wrote:
Please note that we are still using lustre 2.4.x and not 2.6.x or
2.5.x
We have not used any special patches except for aufs and mosix and all
seems to work well.
We hope to switch to lustre 2.5.x somewhere in the not too far future.
Regards,
Eliyahu - אליהו
On Tue, Jul 22, 2014 at 1:47 PM, Patrice Hamelin
<patrice.hamelin(a)ssc-spc.gc.ca <mailto:patrice.hamelin@ssc-spc.gc.ca>>
wrote:
Hi James,
I am running 3.13 so I did not installed the patches. After
commenting out a symbol redefinition in autoconf.h from Mellanox:
include/generated/autoconf.h:3886:0: note: this is the location of
the previous definition
#define CONFIG_LNET_MAX_PAYLOAD 1048576
I am getting:
./configure --disable-modules --disable-server --enable-client
--prefix=/usr/local/lustre/2.4.2
--with-o2ib=/usr/src/mlnx-ofed-kernel-2.2
make
util/parser.c: In function 'init_input':
util/parser.c:308:45: error: 'CPPFunction' undeclared (first use
in this function)
rl_attempted_completion_function = (CPPFunction
*)command_completion;
^
Mr. Rosenberg is right, the lnet module is a part of Trusty and it
is compiled and loading.
root:~/lustre-release# modinfo lnet
filename:
/lib/modules/3.13.0-32-generic/kernel/drivers/staging/lustre/lnet/lnet/lnet.ko
version: 1.0.0
license: GPL
description: Portals v3.1
author: Peter J. Braam <braam(a)clusterfs.com
<mailto:braam@clusterfs.com>>
srcversion: 3441E33D76FAFAECF683EBF
root:~/lustre-release# cat /etc/modprobe.d/lustre.conf
options lnet networks=o2ib(bond1)
root:~/lustre-release# modprobe lnet
root:~/lustre-release# lsmod | grep lnet
lnet 345673 0
libcfs 331973 1 lnet
dmesg
[311197.918239] libcfs: module is from the staging directory, the
quality is unknown, you have been warned.
[311197.921832] LNet: HW CPU cores: 32, npartitions: 4
[311205.929725] lnet: module is from the staging directory, the
quality is unknown, you have been warned.
root:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
Where should I go from there? I found some solutions to this on
the net but it did not produce a viable Lustre.
Thanks for help!
On 07/21/14 14:56, Simmons, James A. wrote:
On debian we build like this:
./configure --disable-modules --disable-server
--enable-client>--prefix=/usr/local/lustre/2.4.2
The modules we build as a part of the staging kernel
(currently we use 3.14) that we build in house.
I don't know if the Ubuntu/Debian kernels ship with the
lustre module compiled, but I can imagine that>they don't
compile it yet...
For 3.14 support you will need the following patch.
http://review.whamcloud.com/#/c/10995
If you want ldiskfs supprot for 3.14 you will need the
following patches.
http://review.whamcloud.com/#/c/11088
http://review.whamcloud.com/#/c/10165
http://review.whamcloud.com/#/c/11000
Please note this is all code in development. Its for testing
and not a production system.
On Mon, Jul 21, 2014 at 1:35 PM, Patrice
Hamelin<patrice.hamelin(a)ssc-spc.gc.ca
<mailto:patrice.hamelin@ssc-spc.gc.ca><mailto:patrice.hamelin@ssc-spc.gc.ca
<mailto:patrice.hamelin@ssc-spc.gc.ca>>> wrote:
Hi James,
Here is what I got:
git clone git://git.whamcloud.com/fs/lustre-release.git
<http://git.whamcloud.com/fs/lustre-release.git><http://git.whamclou...
ib8-bc1l42-be01:~# uname -a
Linux ib8-bc1l42-be01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul
15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
ib8-bc1l42-be01:~/lustre-release# git describe
v2_6_50_0-29-gfa4a67d
ib8-bc1l42-be01:~/lustre-release# ./configure --disable-server
--with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --enable-quota
ib8-bc1l42-be01:~/lustre-release# make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory
`/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
Why are you building as root, only the install stage you would
need to be root/sudo.....
make: *** [all] Error 2
Did you do a make oldconfig and make prepare in your kernel
source tree?
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303 <tel:514-421-5303>
Télécopieur | Facsimile 514-421-7231 <tel:514-421-7231>
Gouvernement du Canada | Government of Canada
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
1:15 a.m.
Hi,
I am installing a brand new cluster and would like to run the latest Lustre 2.6. I am
almost there. Here is what I am doing:
cd /usr/src
tar xvf linux-source-3.13.0.tar.bz2
cd linux-source-3.13.0
make oldconfig
make prepare
# Recipe from Mellanox engineering to avoid the following configure error
# configure: error: can't compile with OpenIB gen2 headers
cd /usr/src/mlnx-ofed-kernel-2.2
ofed_scripts/gen-compat-autoconf.sh include/linux/compat-3.5.h >
include/linux/compat_autoconf.h
export MODULES_DIR=/lib/modules
ofed_scripts/create_Module.symvers.sh
Are 3.13+ kernels supported by Mellanox 2.2? Is their a reason why you are not using OFED
3.12 stack.
TItan at ORNL is also using an Mellanox 2.0 stack since we are using Connect-IB cards. The
plan is to go
to the OFED 3.12 stack very soon. We asked Mellanox to push their work upstream which
ended up in the
OFED 3.12 version. In fact with using such a new kernel you might not even need to use the
kernel drivers
from any external OFED stack.
cd /root
rm -rf lustre-release
cp -rv ~aspgpah/data/Lustre/lustre-release .
cd lustre-release
sed -i 's/1.10 tar-ustar -Wno-portability/1.10 tar-ustar -Wno-portability
subdir-objects/g' configure.ac
# remove duplicated variable definition
sed -i '3886d'
/usr/src/linux-headers-3.13.0-32-generic/include/generated/autoconf.h
What is this for? Never saw this issue.
sh autogen.sh
./configure --with-o2ib=/usr/src/mlnx-ofed-kernel-2.2 --disable-server --enable-quota
make
...
Building modules, stage 2.
MODPOST 18 modules
FATAL: parse error in symbol dump file
make[4]: *** [__modpost] Error 1
make[3]: *** [modules] Error 2
make[3]: Leaving directory `/usr/src/linux-headers-3.13.0-32-generic'
make[2]: *** [modules] Error 2
make[2]: Leaving directory `/root/lustre-release'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/lustre-release'
make: *** [all] Error 2
I am stucked with that error for a while now. What is the symbol dump file?
Module.symvers?
I am continuing to pull my hair on this.
Module.symvers contains a list of all exported symbols from a kernel build. The reason you
are having
problems is that the kernel has it own OFED stack and you are using building with a
external OFED stack.
Both provide module symbols for the same thing so they are conflicting. You need to fix
up Module.symvers
with symbols from external OFED stack, Here is a recipe for you.
cd linux-source-3.13.0
mv Module.symvers Module.symvers.orig
grep -v drivers/infiniband Module.symvers.orig > Module.symvers
cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers > Module.symvers
1:37 a.m.
On Jul 24, 2014, at 11:15 AM, "Simmons, James A." <simmonsja(a)ornl.gov>
wrote:
cd linux-source-3.13.0
mv Module.symvers Module.symvers.orig
grep -v drivers/infiniband Module.symvers.orig > Module.symvers
cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers > Module.symvers
Should that last line be:
cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers >> Module.symvers
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
1:38 a.m.
yes indeed!
On 07/24/14 15:37, Mohr Jr, Richard Frank (Rick Mohr) wrote:
On Jul 24, 2014, at 11:15 AM, "Simmons, James
A."<simmonsja(a)ornl.gov>
wrote:
> > cd linux-source-3.13.0
> > mv Module.symvers Module.symvers.orig
> > grep -v drivers/infiniband Module.symvers.orig> Module.symvers
> > cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers> Module.symvers
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
1:40 a.m.
yes indeed!
So it worked for you?
On 07/24/14 15:37, Mohr Jr, Richard Frank (Rick Mohr) wrote:
On Jul 24, 2014, at 11:15 AM, "Simmons, James
A."<simmonsja(a)ornl.gov>
wrote:
> > cd linux-source-3.13.0
> > mv Module.symvers Module.symvers.orig
> > grep -v drivers/infiniband Module.symvers.orig> Module.symvers
> > cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers> Module.symvers
Note as Rick pointed out it is >> Module.symvers. That was a typo on my part.
2:08 a.m.
One step at a time, now I am getting this:
util/parser.c: In function 'init_input':
util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in this
function)
rl_attempted_completion_function = (CPPFunction
*)command_completion;
And losts of warnings just before
WARNING: "backport_dependency_symbol"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_fmr_pool_map_phys"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_dealloc_pd"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_accept"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_destroy_id"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_destroy_fmr_pool"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_modify_qp"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_connect"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_set_reuseaddr"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_alloc_pd"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_get_dma_mr"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_query_device"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_destroy_qp"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_listen"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_create_id"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_destroy_cq"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_create_qp"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_bind_addr"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_resolve_route"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_disconnect"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_reject"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_dereg_mr"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_flush_fmr_pool"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_create_fmr_pool"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_reg_phys_mr"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "rdma_resolve_addr"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_create_cq"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
WARNING: "ib_fmr_pool_unmap"
[/root/lustre-release/lnet/klnds/o2iblnd/ko2iblnd.ko] undefined!
Of course we would prefer the Mellanox stack since we are using
partitions which am not sure are supported by generic OFED, and other
reasons which I am not sure of. I have to talk with the networking
groups here.
But I will try to compile Lustre over OFED if I cannot get it to run
on the Mellanox stack. BTW, I successfully compiled Lustre server on
Mellanox OFED/Centos 6.4.
Thanks.
On 07/24/14 15:40, Simmons, James A. wrote:
> yes indeed!
So it worked for you?
On 07/24/14 15:37, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Jul 24, 2014, at 11:15 AM, "Simmons, James
A."<simmonsja(a)ornl.gov>
> wrote:
>
>>> cd linux-source-3.13.0
>>> mv Module.symvers Module.symvers.orig
>>> grep -v drivers/infiniband Module.symvers.orig> Module.symvers
>>> cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers> Module.symvers
Note as Rick pointed out it is>> Module.symvers. That was a typo on my part.
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
2:14 a.m.
On 07/24/2014 11:08 AM, Patrice Hamelin wrote:
One step at a time, now I am getting this:
util/parser.c: In function 'init_input':
util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in this
function)
rl_attempted_completion_function = (CPPFunction
*)command_completion;
Replace CPPFunction with rl_completion_func_t.
Frank.
2:46 a.m.
Thanks Frank,
You make me do another step forward.
Regards!
On 07/24/14 16:14, Frank Zago wrote:
On 07/24/2014 11:08 AM, Patrice Hamelin wrote:
> One step at a time, now I am getting this:
>
> util/parser.c: In function 'init_input':
> util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in this
> function)
> rl_attempted_completion_function = (CPPFunction
> *)command_completion;
Replace CPPFunction with rl_completion_func_t.
Frank.
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
3:29 a.m.
The make/make debs process went fine but did not produce a viable kernel
object.
ib8-bc1l42-be01:~/lustre-release/debs# dpkg -i *.deb
ib8-bc1l42-be01:~/lustre-release/debs# dpkg -l | grep lustre
ii linux-patch-lustre
2.6.50-1 all Linux kernel patch
for the Lustre Filesystem
ii lustre-client-modules-3.13.0-32-generic
2.6.50-1 amd64 Lustre Linux kernel
module (kernel 3.13.0-32-generic)
ii lustre-dev
2.6.50-1 amd64 Development files for
the Lustre filesystem
ii lustre-source
2.6.50-1 all source for Lustre
filesystem client kernel modules
ii lustre-tests
2.6.50-1 amd64 Test suite for the
Lustre filesystem
ii lustre-utils
2.6.50-1 amd64 Userspace utilities
for the Lustre filesystem
depmod -a
ib8-bc1l42-be01:~/lustre-release/debs# modprobe lnet
ib8-bc1l42-be01:~/lustre-release/debs# lctl network configure
LNET configure error 100: Network is down
ib8-bc1l42-be01:~/lustre-release/debs# dmesg
...
[77604.074733] LNet: HW CPU cores: 16, npartitions: 4
[77618.345899] ko2iblnd: no symbol version for ib_fmr_pool_unmap
[77618.345902] ko2iblnd: Unknown symbol ib_fmr_pool_unmap (err -22)
[77618.345906] ko2iblnd: no symbol version for ib_create_cq
[77618.345907] ko2iblnd: Unknown symbol ib_create_cq (err -22)
[77618.345914] ko2iblnd: no symbol version for rdma_resolve_addr
[77618.345915] ko2iblnd: Unknown symbol rdma_resolve_addr (err -22)
[77618.345919] ko2iblnd: no symbol version for ib_reg_phys_mr
[77618.345920] ko2iblnd: Unknown symbol ib_reg_phys_mr (err -22)
[77618.345924] ko2iblnd: no symbol version for ib_create_fmr_pool
[77618.345925] ko2iblnd: Unknown symbol ib_create_fmr_pool (err -22)
[77618.345931] ko2iblnd: no symbol version for ib_flush_fmr_pool
[77618.345932] ko2iblnd: Unknown symbol ib_flush_fmr_pool (err -22)
[77618.345943] ko2iblnd: no symbol version for ib_dereg_mr
[77618.345944] ko2iblnd: Unknown symbol ib_dereg_mr (err -22)
[77618.345947] ko2iblnd: no symbol version for rdma_reject
[77618.345948] ko2iblnd: Unknown symbol rdma_reject (err -22)
[77618.345953] ko2iblnd: no symbol version for rdma_disconnect
[77618.345954] ko2iblnd: Unknown symbol rdma_disconnect (err -22)
[77618.345971] ko2iblnd: no symbol version for rdma_resolve_route
[77618.345973] ko2iblnd: Unknown symbol rdma_resolve_route (err -22)
[77618.345977] ko2iblnd: no symbol version for rdma_bind_addr
[77618.345978] ko2iblnd: Unknown symbol rdma_bind_addr (err -22)
[77618.345980] ko2iblnd: no symbol version for rdma_create_qp
[77618.345982] ko2iblnd: Unknown symbol rdma_create_qp (err -22)
[77618.345986] ko2iblnd: no symbol version for ib_destroy_cq
[77618.345987] ko2iblnd: Unknown symbol ib_destroy_cq (err -22)
[77618.345989] ko2iblnd: no symbol version for rdma_create_id
[77618.345990] ko2iblnd: Unknown symbol rdma_create_id (err -22)
[77618.346001] ko2iblnd: no symbol version for rdma_listen
[77618.346002] ko2iblnd: Unknown symbol rdma_listen (err -22)
[77618.346004] ko2iblnd: no symbol version for rdma_destroy_qp
[77618.346005] ko2iblnd: Unknown symbol rdma_destroy_qp (err -22)
[77618.346010] ko2iblnd: no symbol version for ib_query_device
[77618.346011] ko2iblnd: Unknown symbol ib_query_device (err -22)
[77618.346014] ko2iblnd: no symbol version for ib_get_dma_mr
[77618.346015] ko2iblnd: Unknown symbol ib_get_dma_mr (err -22)
[77618.346022] ko2iblnd: no symbol version for ib_alloc_pd
[77618.346023] ko2iblnd: Unknown symbol ib_alloc_pd (err -22)
[77618.346030] ko2iblnd: no symbol version for rdma_set_reuseaddr
[77618.346031] ko2iblnd: Unknown symbol rdma_set_reuseaddr (err -22)
[77618.346034] ko2iblnd: no symbol version for rdma_connect
[77618.346035] ko2iblnd: Unknown symbol rdma_connect (err -22)
[77618.346039] ko2iblnd: no symbol version for ib_modify_qp
[77618.346040] ko2iblnd: Unknown symbol ib_modify_qp (err -22)
[77618.346047] ko2iblnd: no symbol version for ib_destroy_fmr_pool
[77618.346048] ko2iblnd: Unknown symbol ib_destroy_fmr_pool (err -22)
[77618.346050] ko2iblnd: no symbol version for rdma_destroy_id
[77618.346051] ko2iblnd: Unknown symbol rdma_destroy_id (err -22)
[77618.346056] ko2iblnd: no symbol version for rdma_accept
[77618.346057] ko2iblnd: Unknown symbol rdma_accept (err -22)
[77618.346065] ko2iblnd: no symbol version for ib_dealloc_pd
[77618.346066] ko2iblnd: Unknown symbol ib_dealloc_pd (err -22)
[77618.346070] ko2iblnd: no symbol version for ib_fmr_pool_map_phys
[77618.346072] ko2iblnd: Unknown symbol ib_fmr_pool_map_phys (err -22)
[77618.346347] LNetError: 653079:0:(api-ni.c:1205:lnet_startup_lndnis())
Can't load LND o2ib, module ko2iblnd, rc=256
But I founf inconsistencies in the Mellanox Module.symvers file like:
ib8-bc1l42-be01:/usr/src/mlnx-ofed-kernel-2.2# grep rdma_accept
Module.symvers
0x000000008c72c652 __crc_rdma_accept
/limodules/3.13.0-32-generic/updates/dkms/rdma_cm
0x0000000026d04f94 __crc_rdma_accept
/limodules/3.13.0-32-generic/kernel/drivers/infiniband/core/rdma_cm
I think the /create_Module.symvers.sh from Mellanox is broken,
Stay tuned! To be continued tomorrow.
On 07/24/14 16:46, Patrice Hamelin wrote:
Thanks Frank,
You make me do another step forward.
Regards!
On 07/24/14 16:14, Frank Zago wrote:
> On 07/24/2014 11:08 AM, Patrice Hamelin wrote:
>> One step at a time, now I am getting this:
>>
>> util/parser.c: In function 'init_input':
>> util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in
>> this
>> function)
>> rl_attempted_completion_function = (CPPFunction
>> *)command_completion;
> Replace CPPFunction with rl_completion_func_t.
>
> Frank.
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
2:23 a.m.
>On 07/24/2014 11:08 AM, Patrice Hamelin wrote:
> One step at a time, now I am getting this:
>
> util/parser.c: In function 'init_input':
> util/parser.c:308:45: error: 'CPPFunction' undeclared (first use in this
> function)
> rl_attempted_completion_function = (CPPFunction
> *)command_completion;
Replace CPPFunction with rl_completion_func_t.
Time for a JIRA ticket :-)
2:11 a.m.
I forgot to mention that the error "FATAL: parse error in symbol dump
file" was coming from entries in the module.symvers that were starting
with "0xU" instead of 0x12345678. I just removed those entries. Not
sure where it was coming from!
On 07/24/14 15:40, Simmons, James A. wrote:
> yes indeed!
So it worked for you?
On 07/24/14 15:37, Mohr Jr, Richard Frank (Rick Mohr) wrote:
> On Jul 24, 2014, at 11:15 AM, "Simmons, James
A."<simmonsja(a)ornl.gov>
> wrote:
>
>>> cd linux-source-3.13.0
>>> mv Module.symvers Module.symvers.orig
>>> grep -v drivers/infiniband Module.symvers.orig> Module.symvers
>>> cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers> Module.symvers
Note as Rick pointed out it is>> Module.symvers. That was a typo on my part.
--
Patrice Hamelin
Specialiste sénior en systèmes d'exploitation | Senior OS specialist
Environnement Canada | Environment Canada
2121, route Transcanadienne | 2121 Transcanada Highway
Dorval, QC H9P 1J3
Téléphone | Telephone 514-421-5303
Télécopieur | Facsimile 514-421-7231
Gouvernement du Canada | Government of Canada
1:39 a.m.
________________________________________
From: Mohr Jr, Richard Frank (Rick Mohr) [rmohr(a)utk.edu]
Sent: Thursday, July 24, 2014 11:37 AM
To: Simmons, James A.
Cc: Patrice Hamelin; E.S. Rosenberg; hpdd-discuss(a)ml01.01.org
Subject: Re: [HPDD-discuss] Lustre 2.6 on Ubuntu 14
On Jul 24, 2014, at 11:15 AM, "Simmons, James A." <simmonsja(a)ornl.gov>
wrote:
cd linux-source-3.13.0
mv Module.symvers Module.symvers.orig
grep -v drivers/infiniband Module.symvers.orig > Module.symvers
cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers > Module.symvers
Should that last line be:
cat /usr/src/mlnx-ofed-kernel-2.2/Module.symvers >> Module.symvers
Oops. Yes its should be. My bad with the typo.
2977
days inactive
2984
days old
18 comments
5 participants
participants (5)
-
E.S. Rosenberg -
Frank Zago -
Mohr Jr, Richard Frank (Rick Mohr) -
Patrice Hamelin -
Simmons, James A.