Yes, you are correct. I wrote lustreclient3 twice, instead of lustreclient4. I got your point on this. >while MDS and OSS/OST will be undisturbed and will >neither be namenode/datanode.. am I right? It can be namenode but should not be datanode since we are not recommending mount lustre client on servers. You said it can be namenode. How is it possible? Say, if I take MDS as namenode. In case of namenode, we usually take mount point as /mnt/lustre. But there is no such mount point here in MDS. Same for OSS / OST.

One more doubt is: You expressed .."If you have 2 oss with 6 OST each, resulting total of 12 disks, then you might use 3 disks on each of 4 datanode (ie. Total 12 disks.) However, you are using LVM. That's different." I dint understand why are we concerned regarding 3 disk on each of 4 namenode? Are you talking about Hadoop + HDFS here.

Though I will go ahead and test the environment, and then come back with more results meanwhile. Thanks for all the suggestion. Its really great to see such an active mailinglist.

On 3/20/13, Diep, Minh <minh.diep(a)intel.com&gt; wrote: > > > On 3/20/13 10:21 AM, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: > >>Just to understand it correctly. >>If I have 1 MDS, 2 OSS with 6 OST each(created through LVM) and 4 >>lustreclients. >>So, as per your statement, its equivalent to 1 NameNode(=> >>LustreClient1) and 3 DataNode(=>lustreclient2, lustreclient3, >>lustreclient3), > Should be 4 datanode + 1 namenode. You had lustreclient3 twice? > >>while MDS and OSS/OST will be undisturbed and will >>neither be namenode/datanode.. am I right? > It can be namenode but should not be datanode since we are not > recommending mount lustre client on servers >> >>All I dint get this point .."I would also keep the same total number >>of OSTs and total number of disks on all datanodes." Can you please >>clarify. > If you have 2 oss with 6 OST each, resulting total of 12 disks, then you > might use 3 disks on each of 4 datanode (ie. Total 12 disks.) > However, you are using LVM. That's different. > > You can start out with what you have to see how the perf numbers turn >out, > but it's difficult to draw any conclusion if we are not comparing > apple-to-apple. > > Thanks > -Minh > >> >>Regarding LUG, I will try to see if I can attend it.Thanks for sharing >>it. >> >>On 3/20/13, Diep, Minh <minh.diep(a)intel.com&gt; wrote: >>> Hi, >>> >>> There isn't a simple or trivial comparison between Hadoop+HDFS and >>> Hadoop+Lustre. >>> A typical approach (IMHO) is keeping the same number of Lustre client >>>with >>> Hadoop datanode. >>> I would also keep the same total number of OSTs and total number of >>>disks >>> on all datanodes. >>> >>> Please let me know what you find. My finding was that since Lustre is >>>best >>> for large file sequential IO, benchmark such as TestDFSIO show Lustre >>> perform better while terasort does not. >>> >>> There will be a talk by Intel at LUG this year about how Intel will >>> improve Hadoop running on Lustre. Please attend if you have a chance. >>> >>> Thanks >>> -Minh >>> >>> On 3/19/13 10:25 AM, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: >>> >>>>Hi, >>>> >>>>All I have been reading >>>>http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdfFile link where >>>>it talks about how lustre can be more efficient for BigMapOutput kind >>>>of application. >>>>I just thought to try my hands comparing HDFS Vs Lustre. >>>> >>>>To Test it, I have 1 MDS, 2 OSS/OST and 2 Lustre Client running. My >>>>Plan is to install Hadoop on 2 Lustre Clients. >>>>I am going to run wordcount example with this setup >>>> >>>>Versus >>>> >>>>I have 1 Namenode and 4 DataNode and general Hadoop + HDFS setup. >>>> >>>>I wonder how am I going to compare HDFS and Lustre through what number >>>>of nodes to actually equilibrium the setup? >>>> >>>>If I take 1 MDS, 2 OSS/OST and 2 Lustre Client vs 1 Namenode and 4 >>>>DataNode. Will this setup be balanced comparison. >>>>Please suggest. >>>>_______________________________________________ >>>>HPDD-discuss mailing list &gt;&gt;&gt;&gt;HPDD-discuss(a)lists.01.org >>>>https://lists.01.org/mailman/listinfo/hpdd-discuss >>> >>> > > _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss