Re: [HPDD-discuss] Comparing lustre and hadoop distributed filesystem..
Hi,
There isn't a simple or trivial comparison between Hadoop+HDFS and
Hadoop+Lustre.
A typical approach (IMHO) is keeping the same number of Lustre client with
Hadoop datanode.
I would also keep the same total number of OSTs and total number of disks
on all datanodes.
Please let me know what you find. My finding was that since Lustre is best
for large file sequential IO, benchmark such as TestDFSIO show Lustre
perform better while terasort does not.
There will be a talk by Intel at LUG this year about how Intel will
improve Hadoop running on Lustre. Please attend if you have a chance.
Thanks
-Minh
On 3/19/13 10:25 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
Hi,
All I have been reading
http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where
it talks about how lustre can be more efficient for BigMapOutput kind
of application.
I just thought to try my hands comparing HDFS Vs Lustre.
To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My
Plan is to install Hadoop on 2 Lustre Clients.
I am going to run wordcount example with this setup
Versus
I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup.
I wonder how am I going to compare HDFS and Lustre through what number
of nodes to actually equilibrium the setup?
If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4
DataNode. Will this setup be balanced comparison.
Please suggest.
_______________________________________________
HPDD-discuss mailing list
HPDD-discuss(a)lists.01.org
https://lists.01.org/mailman/listinfo/hpdd-discuss