7:39 a.m.
I see all the questions and ask myself:
Is the difference between OFED and the infiniband drivers in the CentOS repos tangible?
It depends. I found for most cases it is not worth the extra effort to use the
OFED/Mellanox stack.
Now if you have FDR Mellanox cards then yes using the Mellanox/OFED 3.12 stack makes a
difference.
We did have one cluster that was having problems with the RHEL infiniband stack and going
to the
Mellanox stack helped in that case.
We had the OFED problems and just switched to the CentOS drivers,
since all the clients are running with the Debian/kernel.org<http://kernel.org>
drivers it didn't seem to make a big difference anyhow...
>On Fri, Oct 3, 2014 at 9:40 AM, Indivar Nair
<indivar.nair@techterra.in<mailto:indivar.nair@techterra.in>> wrote:
>Hi ...,
>I can see quite a few mails on Lustre and OFED.
>
>Are there any known issues with CentOS6.5+Lustre 2.5.3+Mellanox OFED 2.3 combination?
10:45 a.m.
New subject: [Lustre-discuss] Lustre and OFED
Slightly off-topic: for lustre/Mellanox/MPI shops do your MPI apps
(OpenMPI?) see tangible benefits of fca, hcoll, knem from the MLNX OFED on
a mixture of FDR10 and FDR). These are the some reasons given for the MLNX
OFED stack.
On 7 Oct, 2014 4:39 am, "Simmons, James A." <simmonsja(a)ornl.gov> wrote:
>I see all the questions and ask myself:
>Is the difference between OFED and the infiniband drivers in the CentOS
repos tangible?
It depends. I found for most cases it is not worth the extra effort to use
the OFED/Mellanox stack.
Now if you have FDR Mellanox cards then yes using the Mellanox/OFED 3.12
stack makes a difference.
We did have one cluster that was having problems with the RHEL infiniband
stack and going to the
Mellanox stack helped in that case.
>We had the OFED problems and just switched to the CentOS drivers, since
all the clients are running with the Debian/kernel.org drivers it didn't
seem to make a big difference anyhow...
>
>>On Fri, Oct 3, 2014 at 9:40 AM, Indivar Nair <indivar.nair(a)techterra.in>
wrote:
>>Hi ...,
>>I can see quite a few mails on Lustre and OFED.
>>
>>Are there any known issues with CentOS6.5+Lustre 2.5.3+Mellanox OFED
2.3 combination?
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss
7:45 p.m.
New subject: [Lustre-discuss] Lustre and OFED
>>>> "Anthony" == Anthony Alba
<ascanio.alba7(a)gmail.com> writes:
Anthony> Slightly off-topic: for lustre/Mellanox/MPI shops do your
Anthony> MPI apps (OpenMPI?) see tangible benefits of fca, hcoll,
Anthony> knem from the MLNX OFED on a mixture of FDR10 and
Anthony> FDR). These are the some reasons given for the MLNX OFED
Anthony> stack.
Yes there are definitely significant advantages from using fca and
hcoll. I've seen a 12% speedup with these two running an OpenFOAM benchmark
on a 4(!!!)-node FDR system. Would have never thought, it will help
that much already with so few nodes. The good news is, that you don't need
the Mellanox stack for this, just the fca and hcoll libs. I've run this
using Qlustar with OpenMPI 1.8.2 and stock OFED 3.12 / kernel 3.12.
No experience with knem yet.
--
Roland
-------
http://www.q-leap.com / http://qlustar.com
--- HPC / Storage / Cloud Linux Cluster OS ---
> >I see all the questions and ask myself:
>
> >Is the difference between OFED and the infiniband drivers in the
> >CentOS
> repos tangible?
>
> It depends. I found for most cases it is not worth the extra
> effort to use the OFED/Mellanox stack.
>
> Now if you have FDR Mellanox cards then yes using the
> Mellanox/OFED 3.12 stack makes a difference.
>
> We did have one cluster that was having problems with the RHEL
> infiniband stack and going to the
>
> Mellanox stack helped in that case.
> >We had the OFED problems and just switched to the CentOS
> >drivers, since
> all the clients are running with the Debian/kernel.org drivers it
> didn't seem to make a big difference anyhow...
>
> >>On Fri, Oct 3, 2014 at 9:40 AM, Indivar Nair
> >><indivar.nair(a)techterra.in>
> wrote:
>
> >>Hi ...,
>
> >>I can see quite a few mails on Lustre and OFED.
> >>
>
> >>Are there any known issues with CentOS6.5+Lustre 2.5.3+Mellanox
> >>OFED
> 2.3 combination?
2902
days inactive
2904
days old
2 comments
3 participants
participants (3)
-
Anthony Alba -
rf＠q-leap.de -
Simmons, James A.