3:10 a.m.
Yes, that is correct.
On 3/19/13 9:05 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
Hi,
Thanks for the quick response.
All I understand is:
Master Node (NameNode)
=====================
File: conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>file:///</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
<name>hadoop.tmp.dir</name>
<value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
File: mapred-site.xml
<name>mapred.job.tracker</name>
<value>lustreclient1:9101</value>
Slave Nodes(DataNodes)
======================
File: conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>file:///</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
<name>hadoop.tmp.dir</name>
<value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
File:mapred-site.xml
In mapred-site.xml:
<name>mapred.job.tracker</name>
<value>lustreclient1:9101</value> <== Is it correct?
Please confirm if the entry is correct?
On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
> Hi,
>
> I would suggest you set this instead.
>
>
> <name>fs.default.name</name>
> <value>file:///</value>
>
>
> <name>fs.file.impl</name>
> <value>org.apache.hadoop.fs.LocalFileSystem</value>
>
>
> We set different paths to hadoop.tmp.dir on every node since they are
> sharing the same space.
> On master
> <name>hadoop.tmp.dir</name>
> <value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
>
>
>
> On slave
> <value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
>
>
> In mapred-site.xml:
> <name>mapred.job.tracker</name>
> <value>client1:9101</value>
>
>
> On master, don't start hdfs since you are using lustre. Start mapred
>only.
>
>
> HTH
> -Minh
>
> On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>
>>Hello,
>>
>>I am in verse to setup Hadoop over lustre(replacing HDFS).
>>I have 1 MDS, 2 OSS/OST and 2 Lustre Client.
>>My MDS shows:
>>
>>[code]
>>[root@MDS ~]# lctl list_nids
>>10.84.214.185@tcp
>>[/code]
>>
>>
>>Lustre Client shows:
>>[code]
>>[root@lustreclient1 ~]# lfs df -h
>>UUID bytes Used Available Use% Mounted on
>>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6%
>>/mnt/lustre[MDT:0]
>>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:0]
>>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:1]
>>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:2]
>>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:3]
>>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:4]
>>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:5]
>>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:6]
>>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:7]
>>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:8]
>>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:9]
>>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:10]
>>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5%
>>/mnt/lustre[OST:11]
>>
>>filesystem summary: 70.9G 3.2G 64.0G 5%
>>/mnt/lustre
>>[/code]
>>
>>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS).
>>
>>My core-site.xml shows:
>>
>>[code]
>><property>
>><name>fs.default.name</name>
>><value>file:///mnt/lustre</value>
>></property>
>><property>
>><name>mapred.system.dir</name>
>><value>${fs.default.name}/hadoop_tmp/mapred/system</value>
>><description>The shared directory where MapReduce stores control
>>files.
>></description>
>></property>
>>[/code]
>>
>>My conf/masters shows
>>[code]
>>lustreclient1
>>[/code]
>>
>>My conf/slaves shows:
>>
>>[code]
>>lustreclient1
>>lustreclient2
>>
>>I have no idea if I need any further configuration file changes.
>>
>>Do I need just the above configuration.
>>What about hdfs-site.xml and mapred-site.xml?
>>_______________________________________________
>>HPDD-discuss mailing list
>>HPDD-discuss(a)lists.01.org
>>https://lists.01.org/mailman/listinfo/hpdd-discuss
>