Hi all, I'm a little new to Lustre; my site manages a fair bit of data ... around 10 PB ... mostly now in NFS and we'd like to investigate using Lustre in our environment as a longer-term strategy for improved performance, availability and so on. However, our cluster runs entirely on Ubuntu Linux as a standard ... we're using 12.04 LTS right now in production with a house-build 3.4.61 kernel, however, I'm doing my Lustre experiments on Ubuntu 14 with the stock 3.13 kernel, source from Ubuntu's archives, just trying to follow the directions in Section 30 of the Lustre manual. Now, it's just not clear to me ... it looks like most of the packages and directions out there ... are for Red Hat ... I've been looking through the bug tracker and I see people discussing building Lustre on Debian or Ubuntu ... I don't know if this is just the client they are trying to build; I see a few tickets i.e. LU-5189 ... that imply that Lustre will run on Debian and derived platforms like Ubuntu ... but I don't have a good source anywhere that really walks through the build process ... However if I go and look at /home/build/lustre-release/lustre/kernel_patches or any of the files there under i.e. the "series" subdirectory ... I see nothing for Ubuntu, or Debian ... and a lot of what is there is for old 2.6.x RHEL and SLES kernels, or much lower revisions on the 3.x kernel train for same... So I suppose my questions are ... 1. Is it possible to build the server-side components of Lustre on Ubuntu 14? b. If so, does anyone have a source on any snippets of documentation, or a patch set, or information on how I can generate one of these patch series for the 3.13 kernel in Ubuntu 14 LTS? 2. I assume it's at least possible to run Ubuntu 14 LTS as a Lustre client? b. If so, is there any source for any direction on same? I suppose I would be willing to build the Lustre MGS/MDS and OSS machines with Fedora if I had to but it's pretty critical to have client support at least for Ubuntu machines ...that's possible, right? I see some cursory description of a Debian style Lustre build process in ticket LU-1706 but I don't see how it relates to the kernel ... should I just ... I haven't tried to just blindly follow that process; is that basically what I'm looking to do? There's support enough for 3.13 that ./configure will figure it out, so long as all the proper development packages are installed? So far I've got my Ubuntu kernel source in place; I've grabbed Lustre from Git; I ran autogen.sh per the documentation; now I'm at the stage where I need to patch and build the kernel and it's just not clear to me from the Lustre documentation or anything on the Ubuntu site, or just cursorily Googling around, what the next step is here to get the patches in and the build done... I have a "/lib/modules/3.13.0-53-generic/kernel/drivers/staging/lustre/lustre/llite/lustre.ko module sitting around on my basically stock Ubuntu 14 machine but searching for Lustre in apt doesn't give me much back ... it looks like there might be some nominal Lustre support in Ubuntu 14 but unclear what version or to what degree ... hopefully someone here can help me make sense of it all? TIA, Sean _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss

Ah, okay, thanks, Patrick, for helping me to get it all straight in my mind ... so basically, no chance on Ubuntu on the server-side; I think we can live with that if we have to... we can run Fedora or CentOS on the server-side (and there - Fedora Core 18 is the most recent supported?) and since the systems will be doing completely file service and there will be no interactive logon to the systems, it's not a huge deal that the userland lines up too much with our existing Ubuntu deployment on our cluster. I'll reload the pilot server machines with Fedora or CentOS and try the build process again there and try to carry that through to completion per the documentation; hopefully it will go more or less "by the book"; thanks for the guidance there. But for the clients, there's got to be some chance that client support is at least possible ... for Ubuntu 14? Even if I have to build from source and do a packaging ... that's okay ... the procedure would be basically as detailed in LU-1706? It's not clear to me if the stock 3.13 kernel is recent enough to have client support just there by default, and ... is there a strict dependency that the servers must be running the same Lustre revision as the clients do? Thanks, Sean On Tue, Jun 2, 2015 at 3:08 PM, Patrick Farrell <paf(a)cray.com&gt; wrote: > A short answer is that we have *client* support for very recent upstream > kernels, but server support is limited to those kernels for which you > observed patch series, primarily RHEL/CentOS 6 & 7 and some SLES versions. > Supported versions are listed in the lustre/kernel/which_patch file (and > vary slightly by Lustre version), but the newest supported anywhere is 3.10 > as it exists in RHEL 7. > > Trying to build the server for any other platform would be pretty tough > (since you'd have to port the patch series for the kernel itself & > ldiskfs), and it's not tested/supported, so there might be unknown issue as > well. I'd strongly caution against it for a production environment. > > > On 06/02/2015 01:51 PM, Sean Caron wrote: > > Hi all, > > I'm a little new to Lustre; my site manages a fair bit of data ... > around 10 PB ... mostly now in NFS and we'd like to investigate using > Lustre in our environment as a longer-term strategy for improved > performance, availability and so on. > > However, our cluster runs entirely on Ubuntu Linux as a standard ... > we're using 12.04 LTS right now in production with a house-build 3.4.61 > kernel, however, I'm doing my Lustre experiments on Ubuntu 14 with the > stock 3.13 kernel, source from Ubuntu's archives, just trying to follow the > directions in Section 30 of the Lustre manual. > > Now, it's just not clear to me ... it looks like most of the packages > and directions out there ... are for Red Hat ... I've been looking through > the bug tracker and I see people discussing building Lustre on Debian or > Ubuntu ... I don't know if this is just the client they are trying to > build; I see a few tickets i.e. LU-5189 ... that imply that Lustre will run > on Debian and derived platforms like Ubuntu ... but I don't have a good > source anywhere that really walks through the build process ... > > However if I go and look > at /home/build/lustre-release/lustre/kernel_patches or any of the files > there under i.e. the "series" subdirectory ... I see nothing for Ubuntu, or > Debian ... and a lot of what is there is for old 2.6.x RHEL and SLES > kernels, or much lower revisions on the 3.x kernel train for same... > > So I suppose my questions are ... > > 1. Is it possible to build the server-side components of Lustre on > Ubuntu 14? > > b. If so, does anyone have a source on any snippets of documentation, > or a patch set, or information on how I can generate one of these patch > series for the 3.13 kernel in Ubuntu 14 LTS? > > 2. I assume it's at least possible to run Ubuntu 14 LTS as a Lustre > client? > > b. If so, is there any source for any direction on same? > > I suppose I would be willing to build the Lustre MGS/MDS and OSS > machines with Fedora if I had to but it's pretty critical to have client > support at least for Ubuntu machines ...that's possible, right? > > I see some cursory description of a Debian style Lustre build process > in ticket LU-1706 but I don't see how it relates to the kernel ... should I > just ... I haven't tried to just blindly follow that process; is that > basically what I'm looking to do? There's support enough for 3.13 that > ./configure will figure it out, so long as all the proper development > packages are installed? > > So far I've got my Ubuntu kernel source in place; I've grabbed Lustre > from Git; I ran autogen.sh per the documentation; now I'm at the stage > where I need to patch and build the kernel and it's just not clear to me > from the Lustre documentation or anything on the Ubuntu site, or just > cursorily Googling around, what the next step is here to get the patches in > and the build done... > > I have a > "/lib/modules/3.13.0-53-generic/kernel/drivers/staging/lustre/lustre/llite/lustre.ko > module sitting around on my basically stock Ubuntu 14 machine but searching > for Lustre in apt doesn't give me much back ... it looks like there might > be some nominal Lustre support in Ubuntu 14 but unclear what version or to > what degree ... hopefully someone here can help me make sense of it all? > > TIA, > > Sean > > > > _______________________________________________ > HPDD-discuss mailing listHPDD-discuss@lists.01.orghttps://lists.01.org/mailman/listinfo/hpdd-discuss > > > > _______________________________________________ > HPDD-discuss mailing list > HPDD-discuss(a)lists.01.org > https://lists.01.org/mailman/listinfo/hpdd-discuss > >

Unless I missed something, Fedora is *not* a supported server OS. It's just CentOS/RHEL and SLES. Back before client support was moved to "should just work", it was mentioned as a supported *client* OS in the which_patch file. So if your source is old enough, that might appear there. For clients, yes, I think you should be able to build clients on Ubuntu 14. I can't really speak to the process as I've never tried it, though it's been discussed on this list in the past. LU-1706 is quite old and refers to b2_1 for Ubuntu 10. You could start there, but I'd encourage you to look at the list archives for a newer process. As far as kernel/OS to use on the servers, Lustre version is a concern: only the current development branch (which will someday be Lustre 2.8) has CentOS 7 support... But using the development branch in production could be unpleasant. It's certainly going to be less reliable than one of the release versions. So with Lustre 2.7 and older, you are limited to CentOS 6. Which doesn't help with your MD RAID concerns. So: Old kernel, or development version of Lustre. Sorry I don't have a better choice for you. - Patrick On 06/02/2015 02:36 PM, Sean Caron wrote: And, any suggestion, if it would make more sense, to use Centos 7, versus Centos 6 ... I have hardware support and software stability concerns to go too far back on the kernels, particularly back to 2.6.x, because we plan to use MD RAID on JBODs for the underlying datastore and I remember many problems that could have been triggered in i.e. MD in a high-throughput environment in the older 2.6.x and 3.0, 3.2, 3.4 kernels ... Or Fedora ... on the server-side? Looking for the solution that will give me the best prospects for long-term support, best support on modern hardware, most bug fixes, but also to have it be fully compatible with Lustre and to be able to successfully build all components of the latest release without too much hacking ... any advice would be much appreciated. Thanks, Sean On Tue, Jun 2, 2015 at 3:19 PM, Sean Caron <scaron(a)umich.edu&gt; wrote: > Ah, okay, thanks, Patrick, for helping me to get it all straight in my > mind ... so basically, no chance on Ubuntu on the server-side; I think we > can live with that if we have to... we can run Fedora or CentOS on the > server-side (and there - Fedora Core 18 is the most recent supported?) and > since the systems will be doing completely file service and there will be > no interactive logon to the systems, it's not a huge deal that the userland > lines up too much with our existing Ubuntu deployment on our cluster. > > I'll reload the pilot server machines with Fedora or CentOS and try the > build process again there and try to carry that through to completion per > the documentation; hopefully it will go more or less "by the book"; thanks > for the guidance there. > > But for the clients, there's got to be some chance that client support > is at least possible ... for Ubuntu 14? Even if I have to build from source > and do a packaging ... that's okay ... the procedure would be basically as > detailed in LU-1706? It's not clear to me if the stock 3.13 kernel is > recent enough to have client support just there by default, and ... is > there a strict dependency that the servers must be running the same Lustre > revision as the clients do? > > Thanks, > > Sean > > > > On Tue, Jun 2, 2015 at 3:08 PM, Patrick Farrell <paf(a)cray.com&gt; wrote: > >> A short answer is that we have *client* support for very recent >> upstream kernels, but server support is limited to those kernels for which >> you observed patch series, primarily RHEL/CentOS 6 & 7 and some SLES >> versions. Supported versions are listed in the lustre/kernel/which_patch >> file (and vary slightly by Lustre version), but the newest supported >> anywhere is 3.10 as it exists in RHEL 7. >> >> Trying to build the server for any other platform would be pretty tough >> (since you'd have to port the patch series for the kernel itself & >> ldiskfs), and it's not tested/supported, so there might be unknown issue as >> well. I'd strongly caution against it for a production environment. >> >> >> On 06/02/2015 01:51 PM, Sean Caron wrote: >> >> Hi all, >> >> I'm a little new to Lustre; my site manages a fair bit of data ... >> around 10 PB ... mostly now in NFS and we'd like to investigate using >> Lustre in our environment as a longer-term strategy for improved >> performance, availability and so on. >> >> However, our cluster runs entirely on Ubuntu Linux as a standard ... >> we're using 12.04 LTS right now in production with a house-build 3.4.61 >> kernel, however, I'm doing my Lustre experiments on Ubuntu 14 with the >> stock 3.13 kernel, source from Ubuntu's archives, just trying to follow the >> directions in Section 30 of the Lustre manual. >> >> Now, it's just not clear to me ... it looks like most of the packages >> and directions out there ... are for Red Hat ... I've been looking through >> the bug tracker and I see people discussing building Lustre on Debian or >> Ubuntu ... I don't know if this is just the client they are trying to >> build; I see a few tickets i.e. LU-5189 ... that imply that Lustre will run >> on Debian and derived platforms like Ubuntu ... but I don't have a good >> source anywhere that really walks through the build process ... >> >> However if I go and look >> at /home/build/lustre-release/lustre/kernel_patches or any of the files >> there under i.e. the "series" subdirectory ... I see nothing for Ubuntu, or >> Debian ... and a lot of what is there is for old 2.6.x RHEL and SLES >> kernels, or much lower revisions on the 3.x kernel train for same... >> >> So I suppose my questions are ... >> >> 1. Is it possible to build the server-side components of Lustre on >> Ubuntu 14? >> >> b. If so, does anyone have a source on any snippets of documentation, >> or a patch set, or information on how I can generate one of these patch >> series for the 3.13 kernel in Ubuntu 14 LTS? >> >> 2. I assume it's at least possible to run Ubuntu 14 LTS as a Lustre >> client? >> >> b. If so, is there any source for any direction on same? >> >> I suppose I would be willing to build the Lustre MGS/MDS and OSS >> machines with Fedora if I had to but it's pretty critical to have client >> support at least for Ubuntu machines ...that's possible, right? >> >> I see some cursory description of a Debian style Lustre build process >> in ticket LU-1706 but I don't see how it relates to the kernel ... should I >> just ... I haven't tried to just blindly follow that process; is that >> basically what I'm looking to do? There's support enough for 3.13 that >> ./configure will figure it out, so long as all the proper development >> packages are installed? >> >> So far I've got my Ubuntu kernel source in place; I've grabbed Lustre >> from Git; I ran autogen.sh per the documentation; now I'm at the stage >> where I need to patch and build the kernel and it's just not clear to me >> from the Lustre documentation or anything on the Ubuntu site, or just >> cursorily Googling around, what the next step is here to get the patches in >> and the build done... >> >> I have a >> "/lib/modules/3.13.0-53-generic/kernel/drivers/staging/lustre/lustre/llite/lustre.ko >> module sitting around on my basically stock Ubuntu 14 machine but searching >> for Lustre in apt doesn't give me much back ... it looks like there might >> be some nominal Lustre support in Ubuntu 14 but unclear what version or to >> what degree ... hopefully someone here can help me make sense of it all? >> >> TIA, >> >> Sean >> >> >> >> _______________________________________________ >> HPDD-discuss mailing listHPDD-discuss@lists.01.orghttps://lists.01.org/mailman/listinfo/hpdd-discuss >> >> >> >> _______________________________________________ >> HPDD-discuss mailing list >> HPDD-discuss(a)lists.01.org >> https://lists.01.org/mailman/listinfo/hpdd-discuss >> >> >