3:05 a.m.
Hi,
Thanks for the quick response.
All I understand is:
Master Node (NameNode)
=====================
File: conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>file:///</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
<name>hadoop.tmp.dir</name>
<value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
File: mapred-site.xml
<name>mapred.job.tracker</name>
<value>lustreclient1:9101</value>
Slave Nodes(DataNodes)
======================
File: conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>file:///</value>
</property>
<property>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
<name>hadoop.tmp.dir</name>
<value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
File:mapred-site.xml
In mapred-site.xml:
<name>mapred.job.tracker</name>
<value>lustreclient1:9101</value> <== Is it correct?
Please confirm if the entry is correct?
On Tue, Mar 19, 2013 at 8:30 PM, Diep, Minh <minh.diep(a)intel.com> wrote:
Hi,
I would suggest you set this instead.
<name>fs.default.name</name>
<value>file:///</value>
<name>fs.file.impl</name>
<value>org.apache.hadoop.fs.LocalFileSystem</value>
We set different paths to hadoop.tmp.dir on every node since they are
sharing the same space.
On master
<name>hadoop.tmp.dir</name>
<value>/mnt/lustre/hadoop_tmp/lustrecient1</value>
On slave
<value>/mnt/lustre/hadoop_tmp/lustrecient2</value>
In mapred-site.xml:
<name>mapred.job.tracker</name>
<value>client1:9101</value>
On master, don't start hdfs since you are using lustre. Start mapred only.
HTH
-Minh
On 3/19/13 4:48 AM, "linux freaker" <linuxfreaker(a)gmail.com> wrote:
>Hello,
>
>I am in verse to setup Hadoop over lustre(replacing HDFS).
>I have 1 MDS, 2 OSS/OST and 2 Lustre Client.
>My MDS shows:
>
>[code]
>[root@MDS ~]# lctl list_nids
>10.84.214.185@tcp
>[/code]
>
>
>Lustre Client shows:
>[code]
>[root@lustreclient1 ~]# lfs df -h
>UUID bytes Used Available Use% Mounted on
>lustre-MDT0000_UUID 4.5G 274.3M 3.9G 6%
>/mnt/lustre[MDT:0]
>lustre-OST0000_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:0]
>lustre-OST0001_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:1]
>lustre-OST0002_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:2]
>lustre-OST0003_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:3]
>lustre-OST0004_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:4]
>lustre-OST0005_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:5]
>lustre-OST0006_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:6]
>lustre-OST0007_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:7]
>lustre-OST0008_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:8]
>lustre-OST0009_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:9]
>lustre-OST000a_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:10]
>lustre-OST000b_UUID 5.9G 276.1M 5.3G 5%
>/mnt/lustre[OST:11]
>
>filesystem summary: 70.9G 3.2G 64.0G 5% /mnt/lustre
>[/code]
>
>Now I installed Hadoop on two Lustre Client(untouching MDS and OSS).
>
>My core-site.xml shows:
>
>[code]
><property>
><name>fs.default.name</name>
><value>file:///mnt/lustre</value>
></property>
><property>
><name>mapred.system.dir</name>
><value>${fs.default.name}/hadoop_tmp/mapred/system</value>
><description>The shared directory where MapReduce stores control
>files.
></description>
></property>
>[/code]
>
>My conf/masters shows
>[code]
>lustreclient1
>[/code]
>
>My conf/slaves shows:
>
>[code]
>lustreclient1
>lustreclient2
>
>I have no idea if I need any further configuration file changes.
>
>Do I need just the above configuration.
>What about hdfs-site.xml and mapred-site.xml?
>_______________________________________________
>HPDD-discuss mailing list
>HPDD-discuss(a)lists.01.org
>https://lists.01.org/mailman/listinfo/hpdd-discuss