1:28 a.m.
ISSUE: I receive the following errors when attempting to mount ZFS:
KERNEL: Linux mds01 3.10.0-514.el7_lustre.x86_64 #1 SMP Wed Dec 7 15:50:02 PST 2016 x86_64
x86_64 x86_64 GNU/Linux
ERROR 1:
[root@mds01 ~]# mount -t lustre mds01-pool/mdsZFS /mnt/lustre/
mount.lustre: mount mds01-pool/mdsZFS at /mnt/lustre failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module 'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError: 56472:0:(genops.c:314:class_newdev()) OBD:
unknown type: osd-zfs
Dec 23 09:18:58 mds01 kernel: LustreError: 56472:0:(obd_config.c:366:class_attach())
Cannot create device mdsZFS-MDT0000-osd of type osd-zfs : -19
Dec 23 09:18:58 mds01 kernel: LustreError: 56472:0:(obd_mount.c:194:lustre_start_simple())
mdsZFS-MDT0000-osd attach error -19
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(obd_mount_server.c:1810:server_fill_super()) Unable to start osd on
mds01-pool/mdsZFS: -19
Dec 23 09:18:58 mds01 kernel: LustreError: 56472:0:(obd_mount.c:1449:lustre_fill_super())
Unable to mount (-19)
MODULES INSTALLED:
kernel-headers-3.10.0-514.el7_lustre.x86_64
kmod-lustre-osd-ldiskfs-2.9.0-1.el7.x86_64
lustre-osd-zfs-mount-2.9.0-1.el7.x86_64
kmod-lustre-tests-2.9.0-1.el7.x86_64
lustre-osd-ldiskfs-mount-2.9.0-1.el7.x86_64
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
kernel-devel-3.10.0-514.el7_lustre.x86_64
kmod-lustre-2.9.0-1.el7.x86_64
lustre-iokit-2.9.0-1.el7.x86_64
lustre-2.9.0-1.el7.x86_64
kernel-3.10.0-514.el7_lustre.x86_64
e2fsprogs-libs-1.42.13.wc5-7.el7.x86_64
e2fsprogs-1.42.13.wc5-7.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the 'kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64' module
installed, I had to use 'rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 -nodeps'.
Without -nodeps, I would receive dependency warnings about missing nearly two pages
references to KSYM.
Any ideas on how to overcome this issue?
Jeff Slapp | Director, Systems Engineering and Solution Architecture
DataCore Software Corporation
Corporate Park
6300 NW 5th Way
Ft. Lauderdale, FL 33309
http://www.datacore.com<http://www.datacore.com/>
THE DATA INFRASTRUCTURE SOFTWARE COMPANY
3:30 a.m.
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp(a)DataCore.com> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError: 56472:0:(genops.c:314:class_newdev()) OBD:
unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
4:18 a.m.
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp(a)DataCore.com> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
7:45 a.m.
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is compatible with
Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre kernel and
everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: hpdd-discuss(a)lists.01.org
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp(a)DataCore.com> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
1:31 a.m.
I just rebuilt the Lustre RPMs from source using rpmbuild. Interestingly, all RPMs were
created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: 'hpdd-discuss(a)lists.01.org' <hpdd-discuss(a)lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is compatible with
Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre kernel and
everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: hpdd-discuss(a)lists.01.org
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp(a)DataCore.com> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
7:20 a.m.
I'm having the same KSYM problem, CentOS 7.3, zfs-kmod repo enabled as
described at https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS
The missing dependencies from `yum -y install kmod-lustre-osd-zfs* lustre`
look like:
...
Error: Package: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 (lustre-server)
Requires: ksym(spa_get_dsl) = 0x6077e511
Installed: kmod-zfs-0.6.5.8-1.el7_3.centos.x86_64 (@zfs-kmod)
ksym(spa_get_dsl) = 0x62bac5c1
...
The 0.6.5.7 version of zfs-related RPMS from
http://yum.tamu.edu/zfsonlinux/epel/7/kmod/x86_64/ seems to match
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64,
but apparently not fully kernel-3.10.0-514.el7_lustre, with the end result
of: FATAL: Module zfs not found
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp <Jeff.Slapp(a)datacore.com> wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild.
Interestingly,
all RPMs were created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.
el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: 'hpdd-discuss(a)lists.01.org' <hpdd-discuss(a)lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is
compatible with Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre
kernel and everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: hpdd-discuss(a)lists.01.org
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-
release/el7.3.1611/server/RPMS/x86_64/
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
> On Dec 23, 2016, at 9:28 AM, Jeff Slapp <Jeff.Slapp(a)DataCore.com> wrote:
>
> ERROR 2 (in /var/log/messages):
>
> Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
>
> Dec 23 09:18:58 mds01 kernel: LustreError:
> 56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
>
<snip>
>
> MODULES INSTALLED:
>
<snip>
> kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
>
> zfs-release-1-3.el7.centos.noarch
>
> zfs-0.6.5.8-1.el7.centos.x86_64
>
> libzfs2-0.6.5.8-1.el7.centos.x86_64
>
> libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
>
> zfs-dkms-0.6.5.8-1.el7.centos.noarch
>
>
> NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’
module installed, I had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
–nodeps’. Without –nodeps, Idependency warnings about missing nearly two
pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the
version of ZFS you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
7:36 a.m.
My list of missing dependencies is much larger for some reason:
error: Failed dependencies:
zfs-kmod is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(__cv_broadcast) = 0x473952f6 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(arc_add_prune_callback) = 0x6fa427c8 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(arc_remove_prune_callback) = 0xd014a73b is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_assign_arcbuf) = 0xd4ce9943 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_buf_hold_array_by_bonus) = 0x330ef227 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_buf_rele) = 0x53e356d2 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_buf_rele_array) = 0x46d80b8c is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_free_long_range) = 0x389be439 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_free_range) = 0x90b66067 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_object_alloc) = 0x00b6fb3f is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_object_free) = 0x37337442 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_object_info_from_db) = 0x653f6c5a is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_object_next) = 0xe7e4b400 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_object_set_blocksize) = 0xdd72dc88 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_object_size_from_db) = 0x57b75eb4 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_disown) = 0x27d01e19 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_ds) = 0x25c947bd is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_own) = 0x72859ff6 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_pool) = 0xa8cb0bd0 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_spa) = 0xd0d4ba5d is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_space) = 0x0a5a5f8f is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_objset_userused_enabled) = 0x7f63a396 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_prefetch) = 0x7947c677 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_read) = 0x4f521623 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_request_arcbuf) = 0x12428829 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_return_arcbuf) = 0xbb5c175c is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_abort) = 0xa6835da4 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_assign) = 0xa7d102cc is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_callback_register) = 0xc10ae9f5 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_commit) = 0x9a88ba1e is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_create) = 0x5766b01d is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_get_txg) = 0xc2e7eba9 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_bonus) = 0x15ec1afe is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_free) = 0x513d69e5 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_sa) = 0xe2257879 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_sa_create) = 0x867b9689 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_spill) = 0x6a0cfcb3 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_write) = 0x08d8c736 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_tx_hold_zap) = 0xf617c183 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dmu_write) = 0xf6270318 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dsl_pool_config_enter) = 0x904c054e is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dsl_pool_config_exit) = 0xe8b19073 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dsl_prop_register) = 0xa6f021e0 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(dsl_prop_unregister) = 0x8611702e is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_add_byte_array) = 0x4ea06afa is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_alloc) = 0x75a78768 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_free) = 0x9d9f80c4 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_lookup_byte_array) = 0xcb59902f is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_next_nvpair) = 0x29c88b11 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_pack) = 0x424ac2e1 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_remove) = 0xcf24ceff is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_size) = 0x730e1886 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvlist_unpack) = 0x1cd81596 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(nvpair_name) = 0xd42a96fa is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_buf_hold) = 0x13e5ec7c is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_buf_rele) = 0xfb208743 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_bulk_lookup) = 0x29be42be is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_bulk_update) = 0x65ff4436 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_handle_destroy) = 0x20393eae is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_handle_get) = 0x9b0b81fd is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_lookup) = 0x4349aa0a is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_object_size) = 0xa7dee94a is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_replace_all_by_template) = 0x3e2fdfbc is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_setup) = 0x48da4662 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_size) = 0xef3e6514 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_spill_rele) = 0x3ae1b688 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(sa_update) = 0xa341f532 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(spa_freeze) = 0x58c6b5b2 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(spa_get_dsl) = 0x6077e511 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(spa_maxblocksize) = 0x588473de is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(spa_writeable) = 0x7ee456c6 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(spl_panic) = 0xbc32eee7 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(txg_wait_callbacks) = 0x90f50ab1 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(txg_wait_synced) = 0x769ff6de is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_add) = 0xefd09112 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_add_int) = 0xf266dd2c is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_add_uint64) = 0x8589c04d is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_create_flags) = 0xf357660a is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_cursor_advance) = 0x6bb73541 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_cursor_fini) = 0xa5814140 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_cursor_init) = 0x730b862a is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_cursor_init_serialized) = 0x662953e3 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_cursor_retrieve) = 0xd87504b1 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_cursor_serialize) = 0x3f455060 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_increment_int) = 0xa1b5abd7 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_lookup) = 0xeec0741f is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_lookup_uint64) = 0x5e3ba879 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_remove) = 0x3481f17f is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_remove_int) = 0xb0b9e33a is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_remove_uint64) = 0xe69d082e is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zap_update) = 0xf2bde2fb is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zfs_attr_table) = 0x7ae721f2 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zfs_prop_to_name) = 0xa483a8c3 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zio_buf_alloc) = 0x0c211976 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zio_buf_free) = 0x6e7a7166 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zrl_add) = 0xeac8a9a9 is needed by kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
ksym(zrl_remove) = 0x05836522 is needed by
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
From: Marcin Dulak [mailto:marcin.dulak@gmail.com]
Sent: Monday, December 26, 2016 3:20 PM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: Mohr Jr, Richard Frank (Rick Mohr) <rmohr(a)utk.edu>; hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I'm having the same KSYM problem, CentOS 7.3, zfs-kmod repo enabled as described at
https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS
The missing dependencies from `yum -y install kmod-lustre-osd-zfs* lustre` look like:
...
Error: Package: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 (lustre-server)
Requires: ksym(spa_get_dsl) = 0x6077e511
Installed: kmod-zfs-0.6.5.8-1.el7_3.centos.x86_64 (@zfs-kmod)
ksym(spa_get_dsl) = 0x62bac5c1
...
The 0.6.5.7 version of zfs-related RPMS from
http://yum.tamu.edu/zfsonlinux/epel/7/kmod/x86_64/ seems to match
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64,
but apparently not fully kernel-3.10.0-514.el7_lustre, with the end result of: FATAL:
Module zfs not found
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp
<Jeff.Slapp@datacore.com<mailto:Jeff.Slapp@datacore.com>> wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild. Interestingly, all RPMs were
created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: 'hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>'
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is compatible with
Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre kernel and
everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr)
[mailto:rmohr@utk.edu<mailto:rmohr@utk.edu>]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
9:25 a.m.
One has to specify --with zfs in order to build zfs support, moreover the
BuildRequires are missing in the lustre.spec file, so one has to guess them.
The following seems to build after installing lustre kernel and booting
into it. Have not tried if the build is functional.
# yum -y install
http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
# yum-config-manager --disable zfs
# yum-config-manager --enable zfs-kmod
# yum -y install kmod-zfs-devel libzfs2-devel
# yum -y install wget
# wget
https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7/server/SR...
# yum -y install rpm-build
# yum -y install libselinux-devel libtool
# uname -r
3.10.0-514.el7_lustre.x86_64
# rpm -qa | grep kernel | grep `uname -r`
kernel-devel-3.10.0-514.el7_lustre.x86_64
kernel-headers-3.10.0-514.el7_lustre.x86_64
kernel-3.10.0-514.el7_lustre.x86_64
# rpmbuild --rebuild --with zfs lustre-2.9.0-1.src.rpm
# yum -y install createrepo
# rpmbuild=`readlink -f ~/rpmbuild/RPMS`
# createrepo ${rpmbuild}
# cat <<END > /etc/yum.repos.d/lustre.repo
[lustre]
name=CentOS-$releasever - Lustre locally built RPMS
baseurl=file://${rpmbuild}
enabled=1
gpgcheck=0
END
# yum -y install kmod-lustre-osd-zfs* lustre
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp <Jeff.Slapp(a)datacore.com> wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild.
Interestingly,
all RPMs were created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.
el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: 'hpdd-discuss(a)lists.01.org' <hpdd-discuss(a)lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is
compatible with Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre
kernel and everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: hpdd-discuss(a)lists.01.org
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-
release/el7.3.1611/server/RPMS/x86_64/
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
> On Dec 23, 2016, at 9:28 AM, Jeff Slapp <Jeff.Slapp(a)DataCore.com> wrote:
>
> ERROR 2 (in /var/log/messages):
>
> Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
>
> Dec 23 09:18:58 mds01 kernel: LustreError:
> 56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
>
<snip>
>
> MODULES INSTALLED:
>
<snip>
> kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
>
> zfs-release-1-3.el7.centos.noarch
>
> zfs-0.6.5.8-1.el7.centos.x86_64
>
> libzfs2-0.6.5.8-1.el7.centos.x86_64
>
> libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
>
> zfs-dkms-0.6.5.8-1.el7.centos.noarch
>
>
> NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’
module installed, I had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
–nodeps’. Without –nodeps, Idependency warnings about missing nearly two
pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the
version of ZFS you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
4:54 a.m.
Thanks for the details Marcin. I missed that “—with zfs” switch… however I still receive
the missing KSYM errors when attempting to install the kmod-lustre-osd-zfs module.
From: Marcin Dulak [mailto:marcin.dulak@gmail.com]
Sent: Monday, December 26, 2016 5:25 PM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: Mohr Jr, Richard Frank (Rick Mohr) <rmohr(a)utk.edu>; hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
One has to specify --with zfs in order to build zfs support, moreover the BuildRequires
are missing in the lustre.spec file, so one has to guess them.
The following seems to build after installing lustre kernel and booting into it. Have not
tried if the build is functional.
# yum -y install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
# yum-config-manager --disable zfs
# yum-config-manager --enable zfs-kmod
# yum -y install kmod-zfs-devel libzfs2-devel
# yum -y install wget
# wget
https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7/server/SR...
# yum -y install rpm-build
# yum -y install libselinux-devel libtool
# uname -r
3.10.0-514.el7_lustre.x86_64
# rpm -qa | grep kernel | grep `uname -r`
kernel-devel-3.10.0-514.el7_lustre.x86_64
kernel-headers-3.10.0-514.el7_lustre.x86_64
kernel-3.10.0-514.el7_lustre.x86_64
# rpmbuild --rebuild --with zfs lustre-2.9.0-1.src.rpm
# yum -y install createrepo
# rpmbuild=`readlink -f ~/rpmbuild/RPMS`
# createrepo ${rpmbuild}
# cat <<END > /etc/yum.repos.d/lustre.repo
[lustre]
name=CentOS-$releasever - Lustre locally built RPMS
baseurl=file://${rpmbuild}
enabled=1
gpgcheck=0
END
# yum -y install kmod-lustre-osd-zfs* lustre
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp
<Jeff.Slapp@datacore.com<mailto:Jeff.Slapp@datacore.com>> wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild. Interestingly, all RPMs were
created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: 'hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>'
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is compatible with
Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre kernel and
everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr)
[mailto:rmohr@utk.edu<mailto:rmohr@utk.edu>]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
10:14 a.m.
On Tue, Dec 27, 2016 at 6:54 PM, Jeff Slapp <Jeff.Slapp(a)datacore.com> wrote:
Thanks for the details Marcin. I missed that “—with zfs” switch…
however I
still receive the missing KSYM errors when attempting to install the
kmod-lustre-osd-zfs module.
Note a similar discussion at
http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2016-December...
Most likely you are on an unclean system - make sure you are installing the
newly built RPMS and not the cached ones from the lustre repo.
Try the following Vagrantfile (available at
https://gist.github.com/marcindulak/ff2683c78c841f2fa562ee4412dd0a3e):
vagrant up
Marcin
*From:* Marcin Dulak [mailto:marcin.dulak@gmail.com]
*Sent:* Monday, December 26, 2016 5:25 PM
*To:* Jeff Slapp <Jeff.Slapp(a)DataCore.com>
*Cc:* Mohr Jr, Richard Frank (Rick Mohr) <rmohr(a)utk.edu>;
hpdd-discuss(a)lists.01.org
*Subject:* Re: [HPDD-discuss] Error when attempting to mount ZFS with
Lustre
One has to specify --with zfs in order to build zfs support, moreover the
BuildRequires are missing in the lustre.spec file, so one has to guess them.
The following seems to build after installing lustre kernel and booting
into it. Have not tried if the build is functional.
# yum -y install http://download.zfsonlinux.org
/epel/zfs-release.el7_3.noarch.rpm
# yum-config-manager --disable zfs
# yum-config-manager --enable zfs-kmod
# yum -y install kmod-zfs-devel libzfs2-devel
# yum -y install wget
# wget https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/
el7/server/SRPMS/lustre-2.9.0-1.src.rpm
# yum -y install rpm-build
# yum -y install libselinux-devel libtool
# uname -r
3.10.0-514.el7_lustre.x86_64
# rpm -qa | grep kernel | grep `uname -r`
kernel-devel-3.10.0-514.el7_lustre.x86_64
kernel-headers-3.10.0-514.el7_lustre.x86_64
kernel-3.10.0-514.el7_lustre.x86_64
# rpmbuild --rebuild --with zfs lustre-2.9.0-1.src.rpm
# yum -y install createrepo
# rpmbuild=`readlink -f ~/rpmbuild/RPMS`
# createrepo ${rpmbuild}
# cat <<END > /etc/yum.repos.d/lustre.repo
[lustre]
name=CentOS-$releasever - Lustre locally built RPMS
baseurl=file://${rpmbuild}
enabled=1
gpgcheck=0
END
# yum -y install kmod-lustre-osd-zfs* lustre
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp <Jeff.Slapp(a)datacore.com>
wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild. Interestingly,
all RPMs were created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.el
7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: 'hpdd-discuss(a)lists.01.org' <hpdd-discuss(a)lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is
compatible with Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre
kernel and everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)' <rmohr(a)utk.edu>
Cc: hpdd-discuss(a)lists.01.org
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-releas
e/el7.3.1611/server/RPMS/x86_64/
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr) [mailto:rmohr@utk.edu]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp(a)DataCore.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
> On Dec 23, 2016, at 9:28 AM, Jeff Slapp <Jeff.Slapp(a)DataCore.com> wrote:
>
> ERROR 2 (in /var/log/messages):
>
> Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
>
> Dec 23 09:18:58 mds01 kernel: LustreError:
> 56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
>
<snip>
>
> MODULES INSTALLED:
>
<snip>
> kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
>
> zfs-release-1-3.el7.centos.noarch
>
> zfs-0.6.5.8-1.el7.centos.x86_64
>
> libzfs2-0.6.5.8-1.el7.centos.x86_64
>
> libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
>
> zfs-dkms-0.6.5.8-1.el7.centos.noarch
>
>
> NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’
module installed, I had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
–nodeps’. Without –nodeps, Idependency warnings about missing nearly two
pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the
version of ZFS you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
12:16 a.m.
Excellent Marcin. I'll let you know how it goes. Thank you.
On Dec 27, 2016, at 6:14 PM, Marcin Dulak
<marcin.dulak@gmail.com<mailto:marcin.dulak@gmail.com>> wrote:
On Tue, Dec 27, 2016 at 6:54 PM, Jeff Slapp
<Jeff.Slapp@datacore.com<mailto:Jeff.Slapp@datacore.com>> wrote:
Thanks for the details Marcin. I missed that “—with zfs” switch… however I still receive
the missing KSYM errors when attempting to install the kmod-lustre-osd-zfs module.
Note a similar discussion at
http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2016-December...
Most likely you are on an unclean system - make sure you are installing the newly built
RPMS and not the cached ones from the lustre repo.
Try the following Vagrantfile (available at
https://gist.github.com/marcindulak/ff2683c78c841f2fa562ee4412dd0a3e):
vagrant up
Marcin
From: Marcin Dulak [mailto:marcin.dulak@gmail.com<mailto:marcin.dulak@gmail.com>]
Sent: Monday, December 26, 2016 5:25 PM
To: Jeff Slapp <Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>>
Cc: Mohr Jr, Richard Frank (Rick Mohr) <rmohr@utk.edu<mailto:rmohr@utk.edu>>;
hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
One has to specify --with zfs in order to build zfs support, moreover the BuildRequires
are missing in the lustre.spec file, so one has to guess them.
The following seems to build after installing lustre kernel and booting into it. Have not
tried if the build is functional.
# yum -y install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
# yum-config-manager --disable zfs
# yum-config-manager --enable zfs-kmod
# yum -y install kmod-zfs-devel libzfs2-devel
# yum -y install wget
# wget
https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7/server/SR...
# yum -y install rpm-build
# yum -y install libselinux-devel libtool
# uname -r
3.10.0-514.el7_lustre.x86_64
# rpm -qa | grep kernel | grep `uname -r`
kernel-devel-3.10.0-514.el7_lustre.x86_64
kernel-headers-3.10.0-514.el7_lustre.x86_64
kernel-3.10.0-514.el7_lustre.x86_64
# rpmbuild --rebuild --with zfs lustre-2.9.0-1.src.rpm
# yum -y install createrepo
# rpmbuild=`readlink -f ~/rpmbuild/RPMS`
# createrepo ${rpmbuild}
# cat <<END > /etc/yum.repos.d/lustre.repo
[lustre]
name=CentOS-$releasever - Lustre locally built RPMS
baseurl=file://${rpmbuild}
enabled=1
gpgcheck=0
END
# yum -y install kmod-lustre-osd-zfs* lustre
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp
<Jeff.Slapp@datacore.com<mailto:Jeff.Slapp@datacore.com>> wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild. Interestingly, all RPMs were
created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: 'hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>'
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is compatible with
Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre kernel and
everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr)
[mailto:rmohr@utk.edu<mailto:rmohr@utk.edu>]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the ‘kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64’ module installed, I
had to use ‘rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 –nodeps’. Without –nodeps,
Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
8:11 a.m.
Thank you very much Marcin. I was able to get it working with your instructions. I am so
glad I asked because if not, I would have never of figured out the precise order of
operations to make this work. I am not a Linux expert by trade, I know enough to get
around within the OS, but either this is a fairly complicated install process or I am
severely lacking in my Linux understanding. ;-)
In either case, it appears to be working. Below are the exact steps I followed to get the
system to where I needed it to be:
Using CentOS 7.3.1611 with the following roles enabled:
File and Storage Server
Guest Agents (if in a VM)
Large System Performance
Network File System Client
Performance Tools
Compatibility Libraries
Development Tools
hostname {YOUR SERVER NAME}
yum -y install
http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
yum clean all
vi /etc/yum.repos.d/lustre_server.repo
[lustre-server]
name=CentOS-$releasever - Lustre server
baseurl=https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7.3...
gpgcheck=0
kernel_version=`yum list --showduplicates kernel | grep lustre-server | awk '{print
$2}'`
kernel_firmware_version=`yum list --showduplicates kernel-firmware | grep lustre-server |
awk '{print $2}'`
yum -y install --nogpgcheck --setopt=protected_multilib=false kernel-${kernel_version}
kernel-firmware-${kernel_firmware_version} kernel-devel-${kernel_version}
kernel-headers-${kernel_version}
yum clean all
yum -y install yum-plugin-versionlock
yum versionlock add kernel
yum versionlock add kernel-firmware
yum versionlock add kernel-devel
yum versionlock add kernel-headers
yum clean all
yum-config-manager --disable lustre-server
yum -y install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
yum clean all
yum-config-manager --disable zfs
yum-config-manager --enable zfs-kmod
reboot
yum -y install wget
yum -y install rpm-build
yum -y install kmod-zfs-devel libzfs2-devel
yum -y install libselinux-devel libtool
rm -f lustre-2.9.0-1.src.rpm&& wget -q
https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7/server/SR...
rm -rf ~rpmbuild&& rpmbuild --rebuild --with zfs lustre-2.9.0-1.src.rpm
cd ~/rpmbuild/RPMS/`uname -m`/&& yum -y install kmod-lustre-osd-zfs-2.9.0*
kmod-lustre-2.9.0* lustre-osd-zfs-mount-2.9.0* lustre-2.9.0* lustre-iokit*
modprobe zfs
echo "options lnet networks=tcp0(eth0)" > /etc/modprobe.d/lustre.conf
zpool create -f mgs01-pool mirror /dev/sd[b-c]
mkfs.lustre --mdt --mgs --backfstype=zfs --fsname=mgsZFS --index=0 --mgsnode=[MGS IP
ADDRESS]@tcp0 mgs01-pool/mgsZFS
mkdir /mnt/mgsZFS
lctl dl
mount -t lustre mgs01-pool/mgsZFS /mnt/mgsZFS
lctl dl
Your instructions along with a very concise Lustre config guide
(http://www.pinedalab.org/wp-content/uploads/2015/10/Lustre-Workshop.pdf), and I am up and
running.
Hopefully this will help guide anyone else attempting to configure these two technologies
together.
Cheers.
From: Jeff Slapp
Sent: Wednesday, December 28, 2016 8:16 AM
To: Marcin Dulak <marcin.dulak(a)gmail.com>
Cc: hpdd-discuss(a)lists.01.org
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
Excellent Marcin. I'll let you know how it goes. Thank you.
On Dec 27, 2016, at 6:14 PM, Marcin Dulak
<marcin.dulak@gmail.com<mailto:marcin.dulak@gmail.com>> wrote:
On Tue, Dec 27, 2016 at 6:54 PM, Jeff Slapp
<Jeff.Slapp@datacore.com<mailto:Jeff.Slapp@datacore.com>> wrote:
Thanks for the details Marcin. I missed that "-with zfs" switch... however I
still receive the missing KSYM errors when attempting to install the kmod-lustre-osd-zfs
module.
Note a similar discussion at
http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2016-December...
Most likely you are on an unclean system - make sure you are installing the newly built
RPMS and not the cached ones from the lustre repo.
Try the following Vagrantfile (available at
https://gist.github.com/marcindulak/ff2683c78c841f2fa562ee4412dd0a3e):
vagrant up
Marcin
From: Marcin Dulak [mailto:marcin.dulak@gmail.com<mailto:marcin.dulak@gmail.com>]
Sent: Monday, December 26, 2016 5:25 PM
To: Jeff Slapp <Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>>
Cc: Mohr Jr, Richard Frank (Rick Mohr) <rmohr@utk.edu<mailto:rmohr@utk.edu>>;
hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
One has to specify --with zfs in order to build zfs support, moreover the BuildRequires
are missing in the lustre.spec file, so one has to guess them.
The following seems to build after installing lustre kernel and booting into it. Have not
tried if the build is functional.
# yum -y install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
# yum-config-manager --disable zfs
# yum-config-manager --enable zfs-kmod
# yum -y install kmod-zfs-devel libzfs2-devel
# yum -y install wget
# wget
https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7/server/SR...
# yum -y install rpm-build
# yum -y install libselinux-devel libtool
# uname -r
3.10.0-514.el7_lustre.x86_64
# rpm -qa | grep kernel | grep `uname -r`
kernel-devel-3.10.0-514.el7_lustre.x86_64
kernel-headers-3.10.0-514.el7_lustre.x86_64
kernel-3.10.0-514.el7_lustre.x86_64
# rpmbuild --rebuild --with zfs lustre-2.9.0-1.src.rpm
# yum -y install createrepo
# rpmbuild=`readlink -f ~/rpmbuild/RPMS`
# createrepo ${rpmbuild}
# cat <<END > /etc/yum.repos.d/lustre.repo
[lustre]
name=CentOS-$releasever - Lustre locally built RPMS
baseurl=file://${rpmbuild}
enabled=1
gpgcheck=0
END
# yum -y install kmod-lustre-osd-zfs* lustre
Marcin
On Mon, Dec 26, 2016 at 3:31 PM, Jeff Slapp
<Jeff.Slapp@datacore.com<mailto:Jeff.Slapp@datacore.com>> wrote:
I just rebuilt the Lustre RPMs from source using rpmbuild. Interestingly, all RPMs were
created except the one I need: kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm.
-----Original Message-----
From: Jeff Slapp
Sent: Sunday, December 25, 2016 3:46 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: 'hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>'
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
I suppose I would need someone from Intel to tell me if ZFS 0.6.5.8-1 is compatible with
Lustre 2.9.0-1.
I was able to build ZFS from source (SPL and ZFS modules) under the Lustre kernel and
everything went perfect right up to this step:
rpm -ivh kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64.rpm. Same KSYM errors.
Please advise whether I should use a previous version of Lustre (i.e. 2.8).
-----Original Message-----
From: Jeff Slapp
Sent: Friday, December 23, 2016 12:18 PM
To: 'Mohr Jr, Richard Frank (Rick Mohr)'
<rmohr@utk.edu<mailto:rmohr@utk.edu>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: RE: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
The pre-built Lustre RPMs were obtained here:
https://downloads.hpdd.intel.com/public/lustre/latest-release/el7.3.1611/...
I will investigate your suggestion. Thank you.
-----Original Message-----
From: Mohr Jr, Richard Frank (Rick Mohr)
[mailto:rmohr@utk.edu<mailto:rmohr@utk.edu>]
Sent: Friday, December 23, 2016 11:30 AM
To: Jeff Slapp <Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>>
Cc: hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>
Subject: Re: [HPDD-discuss] Error when attempting to mount ZFS with Lustre
On Dec 23, 2016, at 9:28 AM, Jeff Slapp
<Jeff.Slapp@DataCore.com<mailto:Jeff.Slapp@DataCore.com>> wrote:
ERROR 2 (in /var/log/messages):
Dec 23 09:18:58 mds01 kernel: LustreError: 158-c: Can't load module
'osd-zfs'
Dec 23 09:18:58 mds01 kernel: LustreError:
56472:0:(genops.c:314:class_newdev()) OBD: unknown type: osd-zfs
<snip>
MODULES INSTALLED:
<snip>
kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64
zfs-release-1-3.el7.centos.noarch
zfs-0.6.5.8-1.el7.centos.x86_64
libzfs2-0.6.5.8-1.el7.centos.x86_64
libzfs2-devel-0.6.5.8-1.el7.centos.x86_64
zfs-dkms-0.6.5.8-1.el7.centos.noarch
NOTE: In order to get the 'kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64' module
installed, I had to use 'rpm -ihv kmod-lustre-osd-zfs-2.9.0-1.el7.x86_64 -nodeps'.
Without -nodeps, Idependency warnings about missing nearly two pages references to KSYM.
Are you using the pre-built Lustre RPMs from OpenSFS? If so, see if the version of ZFS
you have installed matches what the RPMs were built against.
--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences http://www.nics.tennessee.edu
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss@lists.01.org<mailto:HPDD-discuss@lists.01.org>
https://lists.01.org/mailman/listinfo/hpdd-discuss
2090
days inactive
2095
days old
11 comments
3 participants
participants (3)
-
Jeff Slapp -
Marcin Dulak -
Mohr Jr, Richard Frank (Rick Mohr)