3:48 a.m.
Thanks.
One more query- I can see that when we run wordcount example in case
of HDFS, we simply used to run command as:
$bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/ebooks
/user/hduser/ebooks-output
Before which we used to copy Local data into HDFS as shown:
bin/hadoop dfs -copyFromLocal /tmp/ebooks /user/hduser/ebooks.
If I want to run wordcount example in case of lustre, what would be
the right approach?
Please suggest.
On Tue, Mar 19, 2013 at 9:40 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
Yes, that is correct.
On 3/19/13 9:05 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>Hi,
>
>Thanks for the quick response.
>All I understand is:
>
>
>Master Node (NameNode)
>=====================
>
>File: conf/core-site.xml
>
><property>
><name>fs.default.name</name>
><value>file:///</value>
></property>
><property>
>
><name>fs.file.impl</name>
><value>org.apache.hadoop.fs.LocalFileSystem</value>
>
><name>hadoop.tmp.dir</name>
><value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>
>
>
>File: mapred-site.xml
>
>
><name>mapred.job.tracker</name>
><value>lustreclient1:9101</value>
>
>
>
>Slave Nodes(DataNodes)
>======================
>
>File: conf/core-site.xml
>
><property>
><name>fs.default.name</name>
><value>file:///</value>
></property>
><property>
>
><name>fs.file.impl</name>
><value>org.apache.hadoop.fs.LocalFileSystem</value>
>
><name>hadoop.tmp.dir</name>
><value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>
>File:mapred-site.xml
>
>In mapred-site.xml:
><name>mapred.job.tracker</name>
><value>lustreclient1:9101</value> <== Is it correct?
>
>Please confirm if the entry is correct?
>
>
>
>
>
>
>On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
>> Hi,
>>
>> I would suggest you set this instead.
>>
>>
>> <name>fs.default.name</name>
>> <value>file:///</value>
>>
>>
>> <name>fs.file.impl</name>
>> <value>org.apache.hadoop.fs.LocalFileSystem</value>
>>
>>
>> We set different paths to hadoop.tmp.dir on every node since they are
>> sharing the same space.
>> On master
>> <name>hadoop.tmp.dir</name>
>> <value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>>
>>
>>
>> On slave
>> <value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>>
>>
>> In mapred-site.xml:
>> <name>mapred.job.tracker</name>
>> <value>client1:9101</value>
>>
>>
>> On master, don't start hdfs since you are using lustre. Start mapred
>>only.
>>
>>
>> HTH
>> -Minh
>>
>> On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com>
wrote:
>>
>>>Hello,
>>>
>>>I am in verse to setup Hadoop over lustre(replacing HDFS).
>>>I have 1 MDS, 2 OSS/OST and 2 Lustre Client.
>>>My MDS shows:
>>>
>>>[code]
>>>[root@MDS ~]# lctl list_nids
>>>10.84.214.185@tcp
>>>[/code]
>>>
>>>
>>>Lustre Client shows:
>>>[code]
>>>[root@lustreclient1 ~]# lfs df -h
>>>UUID bytes Used Available Use% Mounted on
>>>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6%
>>>/mnt/lustre[MDT:0]
>>>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:0]
>>>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:1]
>>>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:2]
>>>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:3]
>>>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:4]
>>>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:5]
>>>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:6]
>>>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:7]
>>>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:8]
>>>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:9]
>>>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:10]
>>>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5%
>>>/mnt/lustre[OST:11]
>>>
>>>filesystem summary: 70.9G 3.2G 64.0G 5%
>>>/mnt/lustre
>>>[/code]
>>>
>>>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS).
>>>
>>>My core-site.xml shows:
>>>
>>>[code]
>>><property>
>>><name>fs.default.name</name>
>>><value>file:///mnt/lustre</value>
>>></property>
>>><property>
>>><name>mapred.system.dir</name>
>>><value>${fs.default.name}/hadoop_tmp/mapred/system</value>
>>><description>The shared directory where MapReduce stores control
>>>files.
>>></description>
>>></property>
>>>[/code]
>>>
>>>My conf/masters shows
>>>[code]
>>>lustreclient1
>>>[/code]
>>>
>>>My conf/slaves shows:
>>>
>>>[code]
>>>lustreclient1
>>>lustreclient2
>>>
>>>I have no idea if I need any further configuration file changes.
>>>
>>>Do I need just the above configuration.
>>>What about hdfs-site.xml and mapred-site.xml?
>>>_______________________________________________
>>>HPDD-discuss mailing list
>>>HPDD-discuss(a)lists.01.org
>>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>>