Re: [HPDD-discuss] [Lustre-discuss] Lustre and OFED
>>>> "Anthony" == Anthony Alba
<ascanio.alba7(a)gmail.com> writes:
Anthony> Slightly off-topic: for lustre/Mellanox/MPI shops do your
Anthony> MPI apps (OpenMPI?) see tangible benefits of fca, hcoll,
Anthony> knem from the MLNX OFED on a mixture of FDR10 and
Anthony> FDR). These are the some reasons given for the MLNX OFED
Anthony> stack.
Yes there are definitely significant advantages from using fca and
hcoll. I've seen a 12% speedup with these two running an OpenFOAM benchmark
on a 4(!!!)-node FDR system. Would have never thought, it will help
that much already with so few nodes. The good news is, that you don't need
the Mellanox stack for this, just the fca and hcoll libs. I've run this
using Qlustar with OpenMPI 1.8.2 and stock OFED 3.12 / kernel 3.12.
No experience with knem yet.
--
Roland
-------
http://www.q-leap.com / http://qlustar.com
--- HPC / Storage / Cloud Linux Cluster OS ---
> >I see all the questions and ask myself:
>
> >Is the difference between OFED and the infiniband drivers in the
> >CentOS
> repos tangible?
>
> It depends. I found for most cases it is not worth the extra
> effort to use the OFED/Mellanox stack.
>
> Now if you have FDR Mellanox cards then yes using the
> Mellanox/OFED 3.12 stack makes a difference.
>
> We did have one cluster that was having problems with the RHEL
> infiniband stack and going to the
>
> Mellanox stack helped in that case.
> >We had the OFED problems and just switched to the CentOS
> >drivers, since
> all the clients are running with the Debian/kernel.org drivers it
> didn't seem to make a big difference anyhow...
>
> >>On Fri, Oct 3, 2014 at 9:40 AM, Indivar Nair
> >><indivar.nair(a)techterra.in>
> wrote:
>
> >>Hi ...,
>
> >>I can see quite a few mails on Lustre and OFED.
> >>
>
> >>Are there any known issues with CentOS6.5+Lustre 2.5.3+Mellanox
> >>OFED
> 2.3 combination?