7:12 a.m.
Hi, Ned,
So, it seems that you are saying, for OSS that utilize zfs, not ldiskfs,
then I can use a stock kernel and just add the Lustre rpms on top? That
is not something that I had ever considered, but then, we've only gone
to zfs since Lustre 2.7.0 came out. I will try to devise a test of this.
It is frustrating though, to follow these Twiki procedures, and end up
with a kernel that does not seem to understand the LVM disks underlying,
even though it was built on that same machine. It just seems wrong.
bob
On 7/22/2015 1:10 PM, Ned Bass wrote:
Hi Bob,
On Wed, Jul 22, 2015 at 10:21:41AM -0400, Bob Ball wrote:
> Hi,
>
> I'm looking for someone who can give me advice on a problem I am
> having rebuilding the full set (server and client) of Lustre rpms,
> including zfs support on the server side. The 2.7.0 rpms as
> distributed were built with zfs 0.6.3, and do not work with 0.6.4.
It's not clear to me from your description how below the boot failure is
related to Lustre or ZFS. But you may be able to avoid this whole issue
by using the stock kernel packaged by your distribution rather than
building your own. That _should_ be possible if you're not using
ldiskfs.
Ned
> So, I follow the directions here, fetching the current git source
> which is something like 2.7.56:
> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
> I modified this as follows to include zfs:
>
> cd /usr/src/spl-0.6.4.2/
> ./configure --with-config=kernel
> make all
> cd ../zfs-0.6.4.2/
> ./configure --with-config=kernel
> make all
>
> Then, when I reach the stage of building the Lustre rpms, I prepare by
> ./configure
--with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.504.16.2.el6_lustre/
> --with-zfs=/usr/src/zfs-0.6.4.2 --with-spl=/usr/src/spl-0.6.4.2
>
> I install the kernel rpm, create the initramfs, and reboot.
>
> This is where it all goes horribly wrong. Our systems are built
> with LVM partitions, with the exception of the /boot partition on
> /dev/sda1. grub.conf is seen there in sda1, the kernel choice is
> made, it starts going. Then nothing. The console goes silent. It
> appears that the kernel cannot access the LVM partitions. From
> everything I can see, dracut knows about them, but I am at a loss on
> how to proceed.
>
> /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod
> --install 2.6.32.504.16.2.el6_lustre
>
> title Scientific Linux (2.6.32.504.16.2.el6_lustre)
> root (hd0,0)
> kernel /vmlinuz-2.6.32.504.16.2.el6_lustre ro
> root=/dev/mapper/vg0-lv_root rd_NO_LUKS rd_LVM_LV=vg0/lv_root
> LANG=en_US.UTF-8 rd_NO_MD rhgb quiet selinux=0 rd_LVM_LV=vg0/lv_swap
> SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc
> KEYTABLE=us rd_NO_DM rhgb quiet printk.time=1 console=tty0
> console=ttyS1,57600n8
> initrd /initramfs-2.6.32.504.16.2.el6_lustre.img
>
> Can anyone advise me on how to proceed with this? I'd really like
> to get the upgraded zfs in use over the 0.6.3 that is running now.
>
> Thanks,
> bob
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
12:41 p.m.
New subject: Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and failing
You can't just use *any* kernel, but yes lustre on zfs uses a patchless
kernel, since 2.4. I also found this not clear in any documentation.
Scott
On 7/22/2015 4:12 PM, Bob Ball wrote:
Hi, Ned,
So, it seems that you are saying, for OSS that utilize zfs, not ldiskfs,
then I can use a stock kernel and just add the Lustre rpms on top? That
is not something that I had ever considered, but then, we've only gone
to zfs since Lustre 2.7.0 came out. I will try to devise a test of this.
It is frustrating though, to follow these Twiki procedures, and end up
with a kernel that does not seem to understand the LVM disks underlying,
even though it was built on that same machine. It just seems wrong.
bob
On 7/22/2015 1:10 PM, Ned Bass wrote:
> Hi Bob,
>
>
> On Wed, Jul 22, 2015 at 10:21:41AM -0400, Bob Ball wrote:
>> Hi,
>>
>> I'm looking for someone who can give me advice on a problem I am
>> having rebuilding the full set (server and client) of Lustre rpms,
>> including zfs support on the server side. The 2.7.0 rpms as
>> distributed were built with zfs 0.6.3, and do not work with 0.6.4.
> It's not clear to me from your description how below the boot failure is
> related to Lustre or ZFS. But you may be able to avoid this whole issue
> by using the stock kernel packaged by your distribution rather than
> building your own. That _should_ be possible if you're not using
> ldiskfs.
>
> Ned
>
>> So, I follow the directions here, fetching the current git source
>> which is something like 2.7.56:
>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>> I modified this as follows to include zfs:
>>
>> cd /usr/src/spl-0.6.4.2/
>> ./configure --with-config=kernel
>> make all
>> cd ../zfs-0.6.4.2/
>> ./configure --with-config=kernel
>> make all
>>
>> Then, when I reach the stage of building the Lustre rpms, I prepare by
>> ./configure
>>
--with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.504.16.2.el6_lustre/
>>
>> --with-zfs=/usr/src/zfs-0.6.4.2 --with-spl=/usr/src/spl-0.6.4.2
>>
>> I install the kernel rpm, create the initramfs, and reboot.
>>
>> This is where it all goes horribly wrong. Our systems are built
>> with LVM partitions, with the exception of the /boot partition on
>> /dev/sda1. grub.conf is seen there in sda1, the kernel choice is
>> made, it starts going. Then nothing. The console goes silent. It
>> appears that the kernel cannot access the LVM partitions. From
>> everything I can see, dracut knows about them, but I am at a loss on
>> how to proceed.
>>
>> /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod
>> --install 2.6.32.504.16.2.el6_lustre
>>
>> title Scientific Linux (2.6.32.504.16.2.el6_lustre)
>> root (hd0,0)
>> kernel /vmlinuz-2.6.32.504.16.2.el6_lustre ro
>> root=/dev/mapper/vg0-lv_root rd_NO_LUKS rd_LVM_LV=vg0/lv_root
>> LANG=en_US.UTF-8 rd_NO_MD rhgb quiet selinux=0 rd_LVM_LV=vg0/lv_swap
>> SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc
>> KEYTABLE=us rd_NO_DM rhgb quiet printk.time=1 console=tty0
>> console=ttyS1,57600n8
>> initrd /initramfs-2.6.32.504.16.2.el6_lustre.img
>>
>> Can anyone advise me on how to proceed with this? I'd really like
>> to get the upgraded zfs in use over the 0.6.3 that is running now.
>>
>> Thanks,
>> bob
>>
>> _______________________________________________
>> HPDD-discuss mailing list
>> HPDD-discuss(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
12:52 p.m.
New subject: Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and failing
So, given that, it seems I can use the stock kernel I have on the build
machine (2.6.32-504.16.2.el6.x86_64) with the rpms I built for this
newer zfs. Interesting. I'll give this a try.
All this makes it sound though, that ldiskfs will be another tale? There
are ldiskfs rpms also built during this process.
bob
On 7/22/2015 10:41 PM, Scott Nolin wrote:
You can't just use *any* kernel, but yes lustre on zfs uses a
patchless kernel, since 2.4. I also found this not clear in any
documentation.
Scott
On 7/22/2015 4:12 PM, Bob Ball wrote:
> Hi, Ned,
>
> So, it seems that you are saying, for OSS that utilize zfs, not ldiskfs,
> then I can use a stock kernel and just add the Lustre rpms on top? That
> is not something that I had ever considered, but then, we've only gone
> to zfs since Lustre 2.7.0 came out. I will try to devise a test of
> this.
>
> It is frustrating though, to follow these Twiki procedures, and end up
> with a kernel that does not seem to understand the LVM disks underlying,
> even though it was built on that same machine. It just seems wrong.
>
> bob
>
> On 7/22/2015 1:10 PM, Ned Bass wrote:
>> Hi Bob,
>>
>>
>> On Wed, Jul 22, 2015 at 10:21:41AM -0400, Bob Ball wrote:
>>> Hi,
>>>
>>> I'm looking for someone who can give me advice on a problem I am
>>> having rebuilding the full set (server and client) of Lustre rpms,
>>> including zfs support on the server side. The 2.7.0 rpms as
>>> distributed were built with zfs 0.6.3, and do not work with 0.6.4.
>> It's not clear to me from your description how below the boot
>> failure is
>> related to Lustre or ZFS. But you may be able to avoid this whole issue
>> by using the stock kernel packaged by your distribution rather than
>> building your own. That _should_ be possible if you're not using
>> ldiskfs.
>>
>> Ned
>>
>>> So, I follow the directions here, fetching the current git source
>>> which is something like 2.7.56:
>>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>> I modified this as follows to include zfs:
>>>
>>> cd /usr/src/spl-0.6.4.2/
>>> ./configure --with-config=kernel
>>> make all
>>> cd ../zfs-0.6.4.2/
>>> ./configure --with-config=kernel
>>> make all
>>>
>>> Then, when I reach the stage of building the Lustre rpms, I prepare by
>>> ./configure
>>>
--with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.504.16.2.el6_lustre/
>>>
>>>
>>> --with-zfs=/usr/src/zfs-0.6.4.2 --with-spl=/usr/src/spl-0.6.4.2
>>>
>>> I install the kernel rpm, create the initramfs, and reboot.
>>>
>>> This is where it all goes horribly wrong. Our systems are built
>>> with LVM partitions, with the exception of the /boot partition on
>>> /dev/sda1. grub.conf is seen there in sda1, the kernel choice is
>>> made, it starts going. Then nothing. The console goes silent. It
>>> appears that the kernel cannot access the LVM partitions. From
>>> everything I can see, dracut knows about them, but I am at a loss on
>>> how to proceed.
>>>
>>> /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod
>>> --install 2.6.32.504.16.2.el6_lustre
>>>
>>> title Scientific Linux (2.6.32.504.16.2.el6_lustre)
>>> root (hd0,0)
>>> kernel /vmlinuz-2.6.32.504.16.2.el6_lustre ro
>>> root=/dev/mapper/vg0-lv_root rd_NO_LUKS rd_LVM_LV=vg0/lv_root
>>> LANG=en_US.UTF-8 rd_NO_MD rhgb quiet selinux=0 rd_LVM_LV=vg0/lv_swap
>>> SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc
>>> KEYTABLE=us rd_NO_DM rhgb quiet printk.time=1 console=tty0
>>> console=ttyS1,57600n8
>>> initrd /initramfs-2.6.32.504.16.2.el6_lustre.img
>>>
>>> Can anyone advise me on how to proceed with this? I'd really like
>>> to get the upgraded zfs in use over the 0.6.3 that is running now.
>>>
>>> Thanks,
>>> bob
>>>
>>> _______________________________________________
>>> HPDD-discuss mailing list
>>> HPDD-discuss(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2:42 a.m.
New subject: Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and failing
Well, this does not seem as if it is going to work. I built the rpms
based upon the stock 2.6.32.504.16.2 kernel. Since the kernel built
with this method will not boot, I tried to instead install the Lustre
rpms on top of that stock kernel. The symbol tables are not compatible.
[root@umdist10 try2_Lustre2.7rpms]# yum localinstall
lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-mount-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
...
Installing :
lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 2/4
Installing :
lustre-osd-zfs-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 3/4
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dsl_prop_register
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_cursor_serialize
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_remove
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_hold_write
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dsl_prop_unregister
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zfs_attr_table
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_spill_rele
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_cursor_advance
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_prefetch
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_read
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_add_byte_array
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_cursor_init_serialized
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_assign
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_handle_get
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_size
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_cursor_fini
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zrl_add
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_free_long_range
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_add_uint64
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_next_nvpair
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_create
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_lookup
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_object_alloc
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_object_next
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_lookup_byte_array
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_object_free
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_own
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_unpack
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_buf_rele_array
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_create_flags
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_free
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol __cv_broadcast
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvpair_name
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_cursor_init
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_write
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_buf_rele
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_bulk_lookup
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_return_arcbuf
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_assign_arcbuf
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_object_size
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_callback_register
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_setup
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_update
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_disown
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_hold_sa_create
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_alloc
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_ds
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_lookup
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_spa
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol arc_remove_prune_callback
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol spa_get_dsl
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_userused_enabled
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_hold_zap
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zio_buf_free
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_lookup_uint64
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_get_txg
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_request_arcbuf
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_commit
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol txg_wait_callbacks
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_hold_sa
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dsl_pool_config_enter
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol spl_panic
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_remove_uint64
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dbuf_hold_impl
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_pool
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_add
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_add_int
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_free_range
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_size
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_buf_hold
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_replace_all_by_template
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_hold_bonus
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_increment_int
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zfs_prop_to_name
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol spa_freeze
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_remove
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zio_buf_alloc
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_cursor_retrieve
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol nvlist_pack
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zrl_remove
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_buf_rele
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol arc_add_prune_callback
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol txg_wait_synced
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_remove_int
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_object_info_from_db
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol zap_update
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dsl_pool_config_exit
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_abort
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_handle_destroy
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_objset_space
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_object_size_from_db
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dbuf_rele
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_object_set_blocksize
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_buf_hold_array_by_bonus
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol sa_bulk_update
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko needs
unknown symbol dmu_tx_hold_free
Installing : lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 4/4
So, I guess I'm back to square one. I'm also thinking that zfs on
linux is a big, big mistake.
bob
On 7/22/2015 10:41 PM, Scott Nolin wrote:
You can't just use *any* kernel, but yes lustre on zfs uses a
patchless kernel, since 2.4. I also found this not clear in any
documentation.
Scott
On 7/22/2015 4:12 PM, Bob Ball wrote:
> Hi, Ned,
>
> So, it seems that you are saying, for OSS that utilize zfs, not ldiskfs,
> then I can use a stock kernel and just add the Lustre rpms on top? That
> is not something that I had ever considered, but then, we've only gone
> to zfs since Lustre 2.7.0 came out. I will try to devise a test of
> this.
>
> It is frustrating though, to follow these Twiki procedures, and end up
> with a kernel that does not seem to understand the LVM disks underlying,
> even though it was built on that same machine. It just seems wrong.
>
> bob
>
> On 7/22/2015 1:10 PM, Ned Bass wrote:
>> Hi Bob,
>>
>>
>> On Wed, Jul 22, 2015 at 10:21:41AM -0400, Bob Ball wrote:
>>> Hi,
>>>
>>> I'm looking for someone who can give me advice on a problem I am
>>> having rebuilding the full set (server and client) of Lustre rpms,
>>> including zfs support on the server side. The 2.7.0 rpms as
>>> distributed were built with zfs 0.6.3, and do not work with 0.6.4.
>> It's not clear to me from your description how below the boot
>> failure is
>> related to Lustre or ZFS. But you may be able to avoid this whole issue
>> by using the stock kernel packaged by your distribution rather than
>> building your own. That _should_ be possible if you're not using
>> ldiskfs.
>>
>> Ned
>>
>>> So, I follow the directions here, fetching the current git source
>>> which is something like 2.7.56:
>>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>> I modified this as follows to include zfs:
>>>
>>> cd /usr/src/spl-0.6.4.2/
>>> ./configure --with-config=kernel
>>> make all
>>> cd ../zfs-0.6.4.2/
>>> ./configure --with-config=kernel
>>> make all
>>>
>>> Then, when I reach the stage of building the Lustre rpms, I prepare by
>>> ./configure
>>>
--with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.504.16.2.el6_lustre/
>>>
>>>
>>> --with-zfs=/usr/src/zfs-0.6.4.2 --with-spl=/usr/src/spl-0.6.4.2
>>>
>>> I install the kernel rpm, create the initramfs, and reboot.
>>>
>>> This is where it all goes horribly wrong. Our systems are built
>>> with LVM partitions, with the exception of the /boot partition on
>>> /dev/sda1. grub.conf is seen there in sda1, the kernel choice is
>>> made, it starts going. Then nothing. The console goes silent. It
>>> appears that the kernel cannot access the LVM partitions. From
>>> everything I can see, dracut knows about them, but I am at a loss on
>>> how to proceed.
>>>
>>> /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod
>>> --install 2.6.32.504.16.2.el6_lustre
>>>
>>> title Scientific Linux (2.6.32.504.16.2.el6_lustre)
>>> root (hd0,0)
>>> kernel /vmlinuz-2.6.32.504.16.2.el6_lustre ro
>>> root=/dev/mapper/vg0-lv_root rd_NO_LUKS rd_LVM_LV=vg0/lv_root
>>> LANG=en_US.UTF-8 rd_NO_MD rhgb quiet selinux=0 rd_LVM_LV=vg0/lv_swap
>>> SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc
>>> KEYTABLE=us rd_NO_DM rhgb quiet printk.time=1 console=tty0
>>> console=ttyS1,57600n8
>>> initrd /initramfs-2.6.32.504.16.2.el6_lustre.img
>>>
>>> Can anyone advise me on how to proceed with this? I'd really like
>>> to get the upgraded zfs in use over the 0.6.3 that is running now.
>>>
>>> Thanks,
>>> bob
>>>
>>> _______________________________________________
>>> HPDD-discuss mailing list
>>> HPDD-discuss(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/hpdd-discuss
>
> _______________________________________________
> HPDD-discuss mailing list
> HPDD-discuss(a)lists.01.org
> https://lists.01.org/mailman/listinfo/hpdd-discuss
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
7:16 a.m.
New subject: Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and failing
On Mon, Aug 17, 2015 at 12:42:02PM -0400, Bob Ball wrote:
Well, this does not seem as if it is going to work. I built the
rpms based upon the stock 2.6.32.504.16.2 kernel. Since the kernel
built with this method will not boot, I tried to instead install the
Lustre rpms on top of that stock kernel. The symbol tables are not
compatible.
[root@umdist10 try2_Lustre2.7rpms]# yum localinstall
lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-mount-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
...
Installing :
lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 2/4
Installing :
<snip>
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
needs unknown symbol dmu_buf_hold_array_by_bonus
WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
needs unknown symbol sa_bulk_update
WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
needs unknown symbol dmu_tx_hold_free
Installing : lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 4/4
So, I guess I'm back to square one. I'm also thinking that zfs on
linux is a big, big mistake.
Those warnings are unfortunate but I believe the installed osd_zfs.ko
module will still load and work correctly. It just isn't able to resolve
the symbols at RPM install time.
Ned
8:21 a.m.
New subject: Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and failing
Just a general comment as well:
Warnings like those at install time are often resolved by things you have yet to install,
even sometimes as part of the same package. They aren't a real issue unless you
cannot load the module and get the same type of messages then.
________________________________________
From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Ned Bass
[bass6(a)llnl.gov]
Sent: Monday, August 17, 2015 4:16 PM
To: Bob Ball
Cc: hpdd-discuss(a)ml01.01.org
Subject: Re: [HPDD-discuss] Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and
failing
On Mon, Aug 17, 2015 at 12:42:02PM -0400, Bob Ball wrote:
Well, this does not seem as if it is going to work. I built the
rpms based upon the stock 2.6.32.504.16.2 kernel. Since the kernel
built with this method will not boot, I tried to instead install the
Lustre rpms on top of that stock kernel. The symbol tables are not
compatible.
[root@umdist10 try2_Lustre2.7rpms]# yum localinstall
lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-mount-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
...
Installing :
lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 2/4
Installing :
<snip>
WARNING:
/lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
needs unknown symbol dmu_buf_hold_array_by_bonus
WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
needs unknown symbol sa_bulk_update
WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
needs unknown symbol dmu_tx_hold_free
Installing : lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 4/4
So, I guess I'm back to square one. I'm also thinking that zfs on
linux is a big, big mistake.
Those warnings are unfortunate but I believe the installed osd_zfs.ko
module will still load and work correctly. It just isn't able to resolve
the symbols at RPM install time.
Ned
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
11:15 p.m.
New subject: Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and failing
Thanks you, both. I will try again later today. I need to set up a
test bed.
bob
On 8/17/2015 6:21 PM, Patrick Farrell wrote:
Just a general comment as well:
Warnings like those at install time are often resolved by things you have yet to install,
even sometimes as part of the same package. They aren't a real issue unless you
cannot load the module and get the same type of messages then.
________________________________________
From: HPDD-discuss [hpdd-discuss-bounces(a)lists.01.org] on behalf of Ned Bass
[bass6(a)llnl.gov]
Sent: Monday, August 17, 2015 4:16 PM
To: Bob Ball
Cc: hpdd-discuss(a)ml01.01.org
Subject: Re: [HPDD-discuss] Trying to rebuild all Lustre rpms with zfs 0.6.4.2 -- and
failing
On Mon, Aug 17, 2015 at 12:42:02PM -0400, Bob Ball wrote:
> Well, this does not seem as if it is going to work. I built the
> rpms based upon the stock 2.6.32.504.16.2 kernel. Since the kernel
> built with this method will not boot, I tried to instead install the
> Lustre rpms on top of that stock kernel. The symbol tables are not
> compatible.
> [root@umdist10 try2_Lustre2.7rpms]# yum localinstall
> lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
> lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
> lustre-osd-zfs-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
lustre-osd-zfs-mount-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64.rpm
>
> ...
>
> Installing :
> lustre-modules-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 2/4
> Installing :
<snip>
> WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
> needs unknown symbol dmu_buf_hold_array_by_bonus
> WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
> needs unknown symbol sa_bulk_update
> WARNING: /lib/modules/2.6.32.504.16.2.el6_lustre/extra/kernel/fs/lustre/osd_zfs.ko
> needs unknown symbol dmu_tx_hold_free
> Installing : lustre-2.7.56-2.6.32.504.16.2.el6_lustre_gfe60e01.x86_64 4/4
>
> So, I guess I'm back to square one. I'm also thinking that zfs on
> linux is a big, big mistake.
Those warnings are unfortunate but I believe the installed osd_zfs.ko
module will still load and work correctly. It just isn't able to resolve
the symbols at RPM install time.
Ned
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
2588
days inactive
2615
days old
6 comments
4 participants
participants (4)
-
Bob Ball -
Ned Bass -
Patrick Farrell -
Scott Nolin