2:07 a.m.
It would be /mnt/lustre/<some data dir>
On 3/19/13 9:48 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
Thanks.
One more query- I can see that when we run wordcount example in case
of HDFS, we simply used to run command as:
$bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/ebooks
/user/hduser/ebooks-output
Before which we used to copy Local data into HDFS as shown:
bin/hadoop dfs -copyFromLocal /tmp/ebooks /user/hduser/ebooks.
If I want to run wordcount example in case of lustre, what would be
the right approach?
Please suggest.
On Tue, Mar 19, 2013 at 9:40 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
> Yes, that is correct.
>
> On 3/19/13 9:05 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>
>>Hi,
>>
>>Thanks for the quick response.
>>All I understand is:
>>
>>
>>Master Node (NameNode)
>>=====================
>>
>>File: conf/core-site.xml
>>
>><property>
>><name>fs.default.name</name>
>><value>file:///</value>
>></property>
>><property>
>>
>><name>fs.file.impl</name>
>><value>org.apache.hadoop.fs.LocalFileSystem</value>
>>
>><name>hadoop.tmp.dir</name>
>><value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>>
>>
>>
>>File: mapred-site.xml
>>
>>
>><name>mapred.job.tracker</name>
>><value>lustreclient1:9101</value>
>>
>>
>>
>>Slave Nodes(DataNodes)
>>======================
>>
>>File: conf/core-site.xml
>>
>><property>
>><name>fs.default.name</name>
>><value>file:///</value>
>></property>
>><property>
>>
>><name>fs.file.impl</name>
>><value>org.apache.hadoop.fs.LocalFileSystem</value>
>>
>><name>hadoop.tmp.dir</name>
>><value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>>
>>File:mapred-site.xml
>>
>>In mapred-site.xml:
>><name>mapred.job.tracker</name>
>><value>lustreclient1:9101</value> <== Is it correct?
>>
>>Please confirm if the entry is correct?
>>
>>
>>
>>
>>
>>
>>On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
>>> Hi,
>>>
>>> I would suggest you set this instead.
>>>
>>>
>>> <name>fs.default.name</name>
>>> <value>file:///</value>
>>>
>>>
>>> <name>fs.file.impl</name>
>>> <value>org.apache.hadoop.fs.LocalFileSystem</value>
>>>
>>>
>>> We set different paths to hadoop.tmp.dir on every node since they are
>>> sharing the same space.
>>> On master
>>> <name>hadoop.tmp.dir</name>
>>> <value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>>>
>>>
>>>
>>> On slave
>>> <value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>>>
>>>
>>> In mapred-site.xml:
>>> <name>mapred.job.tracker</name>
>>> <value>client1:9101</value>
>>>
>>>
>>> On master, don't start hdfs since you are using lustre. Start mapred
>>>only.
>>>
>>>
>>> HTH
>>> -Minh
>>>
>>> On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com>
wrote:
>>>
>>>>Hello,
>>>>
>>>>I am in verse to setup Hadoop over lustre(replacing HDFS).
>>>>I have 1 MDS, 2 OSS/OST and 2 Lustre Client.
>>>>My MDS shows:
>>>>
>>>>[code]
>>>>[root@MDS ~]# lctl list_nids
>>>>10.84.214.185@tcp
>>>>[/code]
>>>>
>>>>
>>>>Lustre Client shows:
>>>>[code]
>>>>[root@lustreclient1 ~]# lfs df -h
>>>>UUID bytes Used Available Use% Mounted
>>>>on
>>>>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6%
>>>>/mnt/lustre[MDT:0]
>>>>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:0]
>>>>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:1]
>>>>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:2]
>>>>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:3]
>>>>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:4]
>>>>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:5]
>>>>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:6]
>>>>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:7]
>>>>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:8]
>>>>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:9]
>>>>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:10]
>>>>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5%
>>>>/mnt/lustre[OST:11]
>>>>
>>>>filesystem summary: 70.9G 3.2G 64.0G 5%
>>>>/mnt/lustre
>>>>[/code]
>>>>
>>>>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS).
>>>>
>>>>My core-site.xml shows:
>>>>
>>>>[code]
>>>><property>
>>>><name>fs.default.name</name>
>>>><value>file:///mnt/lustre</value>
>>>></property>
>>>><property>
>>>><name>mapred.system.dir</name>
>>>><value>${fs.default.name}/hadoop_tmp/mapred/system</value>
>>>><description>The shared directory where MapReduce stores control
>>>>files.
>>>></description>
>>>></property>
>>>>[/code]
>>>>
>>>>My conf/masters shows
>>>>[code]
>>>>lustreclient1
>>>>[/code]
>>>>
>>>>My conf/slaves shows:
>>>>
>>>>[code]
>>>>lustreclient1
>>>>lustreclient2
>>>>
>>>>I have no idea if I need any further configuration file changes.
>>>>
>>>>Do I need just the above configuration.
>>>>What about hdfs-site.xml and mapred-site.xml?
>>>>_______________________________________________
>>>>HPDD-discuss mailing list
>>>>HPDD-discuss(a)lists.01.org
>>>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>>>
>