4:25 a.m.
Just to understand it correctly.
If I have 1 MDS, 2 OSS with 6 OST each(created through LVM) and 4 lustreclients.
So, as per your statement, its equivalent to 1 NameNode(=>
LustreClient1) and 3 DataNode(=>lustreclient2, lustreclient3,
lustreclient3),while MDS and OSS/OST will be undisturbed and will
neither be namenode/datanode.. am I right?
All I dint get this point .."I would also keep the same total number
of OSTs and total number of disks on all datanodes." Can you please
clarify.
Regarding LUG, I will try to see if I can attend it.Thanks for sharing it.
On 3/20/13, Diep, Minh <minh.diep(a)intel.com> wrote:
Hi,
There isn't a simple or trivial comparison between Hadoop+HDFS and
Hadoop+Lustre.
A typical approach (IMHO) is keeping the same number of Lustre client with
Hadoop datanode.
I would also keep the same total number of OSTs and total number of disks
on all datanodes.
Please let me know what you find. My finding was that since Lustre is best
for large file sequential IO, benchmark such as TestDFSIO show Lustre
perform better while terasort does not.
There will be a talk by Intel at LUG this year about how Intel will
improve Hadoop running on Lustre. Please attend if you have a chance.
Thanks
-Minh
On 3/19/13 10:25 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>Hi,
>
>All I have been reading
>http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where
>it talks about how lustre can be more efficient for BigMapOutput kind
>of application.
>I just thought to try my hands comparing HDFS Vs Lustre.
>
>To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My
>Plan is to install Hadoop on 2 Lustre Clients.
>I am going to run wordcount example with this setup
>
>Versus
>
>I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup.
>
>I wonder how am I going to compare HDFS and Lustre through what number
>of nodes to actually equilibrium the setup?
>
>If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4
>DataNode. Will this setup be balanced comparison.
>Please suggest.
>_______________________________________________
>HPDD-discuss mailing list
>HPDD-discuss(a)lists.01.org
>https://lists.01.org/mailman/listinfo/hpdd-discuss