Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?

Saturday, 11 May 2013

I can see the error regarding org.apache.fs. am I missing anything?
On 10 May 2013 16:09, "linux freaker" <linuxfreaker(a)gmail.com&gt; wrote:

...
 I tried running this way too but no luck:

 [root@lustreclient1 hadoop]# bin/hadoop jar hadoop-test-1.1.1.jar
 TestDFSIO -write -nrFiles 10 -fileSize 1000 -resFile
 /mnt/lustre/benchmarks/TestDFSIO/io_control
 TestDFSIO.0.0.4
 13/05/10 16:05:29 INFO fs.TestDFSIO: nrFiles = 10
 13/05/10 16:05:29 INFO fs.TestDFSIO: fileSize (MB) = 1000
 13/05/10 16:05:29 INFO fs.TestDFSIO: bufferSize = 1000000
 13/05/10 16:05:30 INFO fs.TestDFSIO: creating control file: 1000 mega
 bytes, 10 files
 13/05/10 16:05:30 INFO util.NativeCodeLoader: Loaded the native-hadoop
 library
 13/05/10 16:05:30 INFO fs.TestDFSIO: created control files for: 10 files
 13/05/10 16:05:30 INFO mapred.FileInputFormat: Total input paths to
 process : 10
 13/05/10 16:05:30 INFO mapred.JobClient: Running job: job_201305101207_0007
 13/05/10 16:05:31 INFO mapred.JobClient:  map 0% reduce 0%
 13/05/10 16:06:01 INFO mapred.JobClient:  map 20% reduce 0%
 13/05/10 16:06:07 INFO mapred.JobClient:  map 40% reduce 0%
 13/05/10 16:06:08 INFO mapred.JobClient:  map 40% reduce 13%
 13/05/10 16:06:12 INFO mapred.JobClient:  map 50% reduce 13%
 13/05/10 16:06:13 INFO mapred.JobClient:  map 60% reduce 13%
 13/05/10 16:06:18 INFO mapred.JobClient:  map 60% reduce 16%
 13/05/10 16:06:24 INFO mapred.JobClient:  map 60% reduce 20%
 13/05/10 16:06:33 INFO mapred.JobClient: Task Id :
 attempt_201305101207_0007_m_000004_0, Status : FAILED
 java.io.FileNotFoundException: File
 file:/benchmarks/TestDFSIO/io_control/in_file_test_io_6 does not exist.
         at
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
         at
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
         at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
         at
 org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
         at
 org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
         at

org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
         at

org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
         at
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:416)
         at
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
         at org.apache.hadoop.mapred.Child.main(Child.java:249)

 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task

outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000004_0&filter=stdout
 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task

outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000004_0&filter=stderr
 13/05/10 16:06:33 INFO mapred.JobClient: Task Id :
 attempt_201305101207_0007_m_000005_0, Status : FAILED
 java.io.FileNotFoundException: File
 file:/benchmarks/TestDFSIO/io_control/in_file_test_io_9 does not exist.
         at
 org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
         at
 org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
         at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
         at
 org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
         at
 org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
         at

org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
         at

org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
         at
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:416)
         at
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
         at org.apache.hadoop.mapred.Child.main(Child.java:249)

 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task

outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000005_0&filter=stdout
 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task

outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000005_0&filter=stderr
 13/05/10 16:06:37 INFO mapred.JobClient:  map 80% reduce 20%

 On Fri, May 10, 2013 at 3:47 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote:

> I tried creating a directory /benchmarks/TestDFSIO under / directory and
> linked it as:
> ln -s /benchmarks /mnt/lustre.
> But I am getting this error:
>
>
> # bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -read -nrFiles 10
> -filesize 1000
> TestDFSIO.0.0.4
> 13/05/10 15:42:38 INFO fs.TestDFSIO: nrFiles = 10
> 13/05/10 15:42:38 INFO fs.TestDFSIO: fileSize (MB) = 1
> 13/05/10 15:42:38 INFO fs.TestDFSIO: bufferSize = 1000000
> 13/05/10 15:42:38 INFO fs.TestDFSIO: creating control file: 1 mega bytes,
> 10 files
> 13/05/10 15:42:38 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/05/10 15:42:38 INFO fs.TestDFSIO: created control files for: 10 files
> 13/05/10 15:42:39 INFO mapred.FileInputFormat: Total input paths to
> process : 10
> 13/05/10 15:42:39 INFO mapred.JobClient: Running job:
> job_201305101207_0005
> 13/05/10 15:42:40 INFO mapred.JobClient:  map 0% reduce 0%
> 13/05/10 15:43:12 INFO mapred.JobClient: Task Id :
> attempt_201305101207_0005_m_000000_0, Status : FAILED
> java.io.FileNotFoundException: File
> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_5 does not exist.
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>         at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
>         at
>
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>         at
>
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
>         at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> 13/05/10 15:43:12 WARN mapred.JobClient: Error reading task
>
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0005_m_000000_0&filter=stdout
> 13/05/10 15:43:12 WARN mapred.JobClient: Error reading task
>
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0005_m_000000_0&filter=stderr
> 13/05/10 15:43:12 INFO mapred.JobClient: Task Id :
> attempt_201305101207_0005_m_000001_0, Status : FAILED
> java.io.FileNotFoundException: File
> file:/benchmarks/TestDFSIO/io_control/in_file_test_io_0 does not exist.
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>         at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
>         at
>
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
>         at
>
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
>         at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
>
>
>
> On Thu, May 9, 2013 at 8:32 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote:
>
>> Do I need to create benchmark folder under /mnt/lustre?
>>
>>
>> On Thu, May 9, 2013 at 8:24 PM, Diep, Minh <minh.diep(a)intel.com&gt; wrote:
>>
>>>  You should be able to use the exact same command as in hdfs but make
>>> sure you create a symlink from /benchmark or /benchmarks (don't remember
>>> with the s or not) to lustre FS.
>>>
>>>  HTH
>>> -Minh
>>>
>>>   From: linux freaker <linuxfreaker(a)gmail.com&gt;
>>> Date: Thursday, May 9, 2013 1:16 AM
>>> To: &quot;hpdd-discuss(a)lists.01.org&quot; <hpdd-discuss(a)lists.01.org&gt;
>>> Subject: [HPDD-discuss] How to run TestDfsio on hadoop running on
>>> lustre?
>>>
>>>   Any suggestion?
>>>
>>
>>
>

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?