Hello, I am in verse to setup Hadoop over lustre(replacing HDFS). I have 1 MDS, 2 OSS/OST and 2 Lustre Client. My MDS shows: [code] [root@MDS ~]# lctl list_nids 10.84.214.185@tcp [/code] Lustre Client shows: [code] [root@lustreclient1 ~]# lfs df -h UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6% /mnt/lustre[MDT:0] lustre-OST0000_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:0] lustre-OST0001_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:1] lustre-OST0002_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:2] lustre-OST0003_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:3] lustre-OST0004_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:4] lustre-OST0005_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:5] lustre-OST0006_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:6] lustre-OST0007_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:7] lustre-OST0008_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:8] lustre-OST0009_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:9] lustre-OST000a_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:10] lustre-OST000b_UUID 5.9G 276.1M 5.3G 5% /mnt/lustre[OST:11] filesystem summary: 70.9G 3.2G 64.0G 5% /mnt/lustre [/code] Now I installed Hadoop on two Lustre Client(untouching MDS and OSS). My core-site.xml shows: [code] <property> <name>fs.default.name</name> <value>file:///mnt/lustre</value> </property> <property> <name>mapred.system.dir</name> <value>${fs.default.name}/hadoop_tmp/mapred/system</value> <description>The shared directory where MapReduce stores control files. </description> </property> [/code] My conf/masters shows [code] lustreclient1 [/code] My conf/slaves shows: [code] lustreclient1 lustreclient2 I have no idea if I need any further configuration file changes. Do I need just the above configuration. What about hdfs-site.xml and mapred-site.xml? _______________________________________________ HPDD-discuss mailing list HPDD-discuss(a)lists.01.org https://lists.01.org/mailman/listinfo/hpdd-discuss

Hi, Thanks for the quick response. All I understand is: Master Node (NameNode) ===================== File: conf/core-site.xml <property> <name>fs.default.name</name> <value>file:///</value> </property> <property> <name>fs.file.impl</name> <value>org.apache.hadoop.fs.LocalFileSystem</value> <name>hadoop.tmp.dir</name> <value>/mnt/lustre/hadoop_tmp/lustrecient1</value> File: mapred-site.xml <name>mapred.job.tracker</name> <value>lustreclient1:9101</value> Slave Nodes(DataNodes) ====================== File: conf/core-site.xml <property> <name>fs.default.name</name> <value>file:///</value> </property> <property> <name>fs.file.impl</name> <value>org.apache.hadoop.fs.LocalFileSystem</value> <name>hadoop.tmp.dir</name> <value>/mnt/lustre/hadoop_tmp/lustrecient2</value> File:mapred-site.xml In mapred-site.xml: <name>mapred.job.tracker</name> <value>lustreclient1:9101</value> <== Is it correct? Please confirm if the entry is correct? On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com&gt; wrote: > Hi, > > I would suggest you set this instead. > > > <name>fs.default.name</name> > <value>file:///</value> > > > <name>fs.file.impl</name> > <value>org.apache.hadoop.fs.LocalFileSystem</value> > > > We set different paths to hadoop.tmp.dir on every node since they are > sharing the same space. > On master > <name>hadoop.tmp.dir</name> > <value>/mnt/lustre/hadoop_tmp/lustrecient1</value> > > > > On slave > <value>/mnt/lustre/hadoop_tmp/lustrecient2</value> > > > In mapred-site.xml: > <name>mapred.job.tracker</name> > <value>client1:9101</value> > > > On master, don't start hdfs since you are using lustre. Start mapred >only. > > > HTH > -Minh > > On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: > >>Hello, >> >>I am in verse to setup Hadoop over lustre(replacing HDFS). >>I have 1 MDS, 2 OSS/OST and 2 Lustre Client. >>My MDS shows: >> >>[code] >>[root@MDS ~]# lctl list_nids >>10.84.214.185@tcp >>[/code] >> >> >>Lustre Client shows: >>[code] >>[root@lustreclient1 ~]# lfs df -h >>UUID bytes Used Available Use% Mounted on >>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6% >>/mnt/lustre[MDT:0] >>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:0] >>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:1] >>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:2] >>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:3] >>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:4] >>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:5] >>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:6] >>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:7] >>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:8] >>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:9] >>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:10] >>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5% >>/mnt/lustre[OST:11] >> >>filesystem summary: 70.9G 3.2G 64.0G 5% >>/mnt/lustre >>[/code] >> >>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS). >> >>My core-site.xml shows: >> >>[code] >><property> >><name>fs.default.name</name> >><value>file:///mnt/lustre</value> >></property> >><property> >><name>mapred.system.dir</name> >><value>${fs.default.name}/hadoop_tmp/mapred/system</value> >><description>The shared directory where MapReduce stores control >>files. >></description> >></property> >>[/code] >> >>My conf/masters shows >>[code] >>lustreclient1 >>[/code] >> >>My conf/slaves shows: >> >>[code] >>lustreclient1 >>lustreclient2 >> >>I have no idea if I need any further configuration file changes. >> >>Do I need just the above configuration. >>What about hdfs-site.xml and mapred-site.xml? >>_______________________________________________ >>HPDD-discuss mailing list &gt;&gt;HPDD-discuss(a)lists.01.org >>https://lists.01.org/mailman/listinfo/hpdd-discuss >

Thanks. One more query- I can see that when we run wordcount example in case of HDFS, we simply used to run command as: $bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/ebooks /user/hduser/ebooks-output Before which we used to copy Local data into HDFS as shown: bin/hadoop dfs -copyFromLocal /tmp/ebooks /user/hduser/ebooks. If I want to run wordcount example in case of lustre, what would be the right approach? Please suggest. On Tue, Mar 19, 2013 at 9:40 PM, Diep, Minh <minh.diep(a)intel.com&gt; wrote: > Yes, that is correct. > > On 3/19/13 9:05 AM, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: > >>Hi, >> >>Thanks for the quick response. >>All I understand is: >> >> >>Master Node (NameNode) >>===================== >> >>File: conf/core-site.xml >> >><property> >><name>fs.default.name</name> >><value>file:///</value> >></property> >><property> >> >><name>fs.file.impl</name> >><value>org.apache.hadoop.fs.LocalFileSystem</value> >> >><name>hadoop.tmp.dir</name> >><value>/mnt/lustre/hadoop_tmp/lustrecient1</value> >> >> >> >>File: mapred-site.xml >> >> >><name>mapred.job.tracker</name> >><value>lustreclient1:9101</value> >> >> >> >>Slave Nodes(DataNodes) >>====================== >> >>File: conf/core-site.xml >> >><property> >><name>fs.default.name</name> >><value>file:///</value> >></property> >><property> >> >><name>fs.file.impl</name> >><value>org.apache.hadoop.fs.LocalFileSystem</value> >> >><name>hadoop.tmp.dir</name> >><value>/mnt/lustre/hadoop_tmp/lustrecient2</value> >> >>File:mapred-site.xml >> >>In mapred-site.xml: >><name>mapred.job.tracker</name> >><value>lustreclient1:9101</value> <== Is it correct? >> >>Please confirm if the entry is correct? >> >> >> >> >> >> >>On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com&gt; wrote: >>> Hi, >>> >>> I would suggest you set this instead. >>> >>> >>> <name>fs.default.name</name> >>> <value>file:///</value> >>> >>> >>> <name>fs.file.impl</name> >>> <value>org.apache.hadoop.fs.LocalFileSystem</value> >>> >>> >>> We set different paths to hadoop.tmp.dir on every node since they are >>> sharing the same space. >>> On master >>> <name>hadoop.tmp.dir</name> >>> <value>/mnt/lustre/hadoop_tmp/lustrecient1</value> >>> >>> >>> >>> On slave >>> <value>/mnt/lustre/hadoop_tmp/lustrecient2</value> >>> >>> >>> In mapred-site.xml: >>> <name>mapred.job.tracker</name> >>> <value>client1:9101</value> >>> >>> >>> On master, don't start hdfs since you are using lustre. Start mapred >>>only. >>> >>> >>> HTH >>> -Minh >>> >>> On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote: >>> >>>>Hello, >>>> >>>>I am in verse to setup Hadoop over lustre(replacing HDFS). >>>>I have 1 MDS, 2 OSS/OST and 2 Lustre Client. >>>>My MDS shows: >>>> >>>>[code] >>>>[root@MDS ~]# lctl list_nids >>>>10.84.214.185@tcp >>>>[/code] >>>> >>>> >>>>Lustre Client shows: >>>>[code] >>>>[root@lustreclient1 ~]# lfs df -h >>>>UUID bytes Used Available Use% Mounted >>>>on >>>>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6% >>>>/mnt/lustre[MDT:0] >>>>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:0] >>>>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:1] >>>>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:2] >>>>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:3] >>>>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:4] >>>>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:5] >>>>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:6] >>>>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:7] >>>>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:8] >>>>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:9] >>>>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:10] >>>>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5% >>>>/mnt/lustre[OST:11] >>>> >>>>filesystem summary: 70.9G 3.2G 64.0G 5% >>>>/mnt/lustre >>>>[/code] >>>> >>>>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS). >>>> >>>>My core-site.xml shows: >>>> >>>>[code] >>>><property> >>>><name>fs.default.name</name> >>>><value>file:///mnt/lustre</value> >>>></property> >>>><property> >>>><name>mapred.system.dir</name> >>>><value>${fs.default.name}/hadoop_tmp/mapred/system</value> >>>><description>The shared directory where MapReduce stores control >>>>files. >>>></description> >>>></property> >>>>[/code] >>>> >>>>My conf/masters shows >>>>[code] >>>>lustreclient1 >>>>[/code] >>>> >>>>My conf/slaves shows: >>>> >>>>[code] >>>>lustreclient1 >>>>lustreclient2 >>>> >>>>I have no idea if I need any further configuration file changes. >>>> >>>>Do I need just the above configuration. >>>>What about hdfs-site.xml and mapred-site.xml? >>>>_______________________________________________ >>>>HPDD-discuss mailing list &gt;&gt;&gt;&gt;HPDD-discuss(a)lists.01.org >>>>https://lists.01.org/mailman/listinfo/hpdd-discuss >>> >