2:35 a.m.
I believe you tried –read before did a –write.
From: linux freaker <linuxfreaker@gmail.com<mailto:linuxfreaker@gmail.com>>
Date: Friday, May 10, 2013 9:33 AM
To: Minh Diep <minh.diep@intel.com<mailto:minh.diep@intel.com>>
Cc: "hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?
I have already tried it in first stage but it didn't work.
On 10 May 2013 21:37, "Diep, Minh"
<minh.diep@intel.com<mailto:minh.diep@intel.com>> wrote:
Try this
Rm –f /benchmarks/*
Bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO –write –nrFiles 10 –fileSize 1000
From: linux freaker <linuxfreaker@gmail.com<mailto:linuxfreaker@gmail.com>>
Date: Friday, May 10, 2013 8:51 AM
To: Minh Diep <minh.diep@intel.com<mailto:minh.diep@intel.com>>,
"hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?
I can see the error regarding org.apache.fs. am I missing anything?
On 10 May 2013 16:09, "linux freaker"
<linuxfreaker@gmail.com<mailto:linuxfreaker@gmail.com>> wrote:
I tried running this way too but no luck:
[root@lustreclient1 hadoop]# bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -write
-nrFiles 10 -fileSize 1000 -resFile /mnt/lustre/benchmarks/TestDFSIO/io_control
TestDFSIO.0.0.4
13/05/10 16:05:29 INFO fs.TestDFSIO: nrFiles = 10
13/05/10 16:05:29 INFO fs.TestDFSIO: fileSize (MB) = 1000
13/05/10 16:05:29 INFO fs.TestDFSIO: bufferSize = 1000000
13/05/10 16:05:30 INFO fs.TestDFSIO: creating control file: 1000 mega bytes, 10 files
13/05/10 16:05:30 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/05/10 16:05:30 INFO fs.TestDFSIO: created control files for: 10 files
13/05/10 16:05:30 INFO mapred.FileInputFormat: Total input paths to process : 10
13/05/10 16:05:30 INFO mapred.JobClient: Running job: job_201305101207_0007
13/05/10 16:05:31 INFO mapred.JobClient: map 0% reduce 0%
13/05/10 16:06:01 INFO mapred.JobClient: map 20% reduce 0%
13/05/10 16:06:07 INFO mapred.JobClient: map 40% reduce 0%
13/05/10 16:06:08 INFO mapred.JobClient: map 40% reduce 13%
13/05/10 16:06:12 INFO mapred.JobClient: map 50% reduce 13%
13/05/10 16:06:13 INFO mapred.JobClient: map 60% reduce 13%
13/05/10 16:06:18 INFO mapred.JobClient: map 60% reduce 16%
13/05/10 16:06:24 INFO mapred.JobClient: map 60% reduce 20%
13/05/10 16:06:33 INFO mapred.JobClient: Task Id : attempt_201305101207_0007_m_000004_0,
Status : FAILED
java.io.FileNotFoundException: File
file:/benchmarks/TestDFSIO/io_control/in_file_test_io_6 does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
13/05/10 16:06:33 WARN mapred.JobClient: Error reading task
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000004_0&filter=stdout
13/05/10 16:06:33 WARN mapred.JobClient: Error reading task
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000004_0&filter=stderr
13/05/10 16:06:33 INFO mapred.JobClient: Task Id : attempt_201305101207_0007_m_000005_0,
Status : FAILED
java.io.FileNotFoundException: File
file:/benchmarks/TestDFSIO/io_control/in_file_test_io_9 does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
13/05/10 16:06:33 WARN mapred.JobClient: Error reading task
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000005_0&filter=stdout
13/05/10 16:06:33 WARN mapred.JobClient: Error reading task
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000005_0&filter=stderr
13/05/10 16:06:37 INFO mapred.JobClient: map 80% reduce 20%
On Fri, May 10, 2013 at 3:47 PM, linux freaker
<linuxfreaker@gmail.com<mailto:linuxfreaker@gmail.com>> wrote:
I tried creating a directory /benchmarks/TestDFSIO under / directory and linked it as:
ln -s /benchmarks /mnt/lustre.
But I am getting this error:
# bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -read -nrFiles 10 -filesize 1000
TestDFSIO.0.0.4
13/05/10 15:42:38 INFO fs.TestDFSIO: nrFiles = 10
13/05/10 15:42:38 INFO fs.TestDFSIO: fileSize (MB) = 1
13/05/10 15:42:38 INFO fs.TestDFSIO: bufferSize = 1000000
13/05/10 15:42:38 INFO fs.TestDFSIO: creating control file: 1 mega bytes, 10 files
13/05/10 15:42:38 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/05/10 15:42:38 INFO fs.TestDFSIO: created control files for: 10 files
13/05/10 15:42:39 INFO mapred.FileInputFormat: Total input paths to process : 10
13/05/10 15:42:39 INFO mapred.JobClient: Running job: job_201305101207_0005
13/05/10 15:42:40 INFO mapred.JobClient: map 0% reduce 0%
13/05/10 15:43:12 INFO mapred.JobClient: Task Id : attempt_201305101207_0005_m_000000_0,
Status : FAILED
java.io.FileNotFoundException: File
file:/benchmarks/TestDFSIO/io_control/in_file_test_io_5 does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
13/05/10 15:43:12 WARN mapred.JobClient: Error reading task
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0005_m_000000_0&filter=stdout
13/05/10 15:43:12 WARN mapred.JobClient: Error reading task
outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0005_m_000000_0&filter=stderr
13/05/10 15:43:12 INFO mapred.JobClient: Task Id : attempt_201305101207_0005_m_000001_0,
Status : FAILED
java.io.FileNotFoundException: File
file:/benchmarks/TestDFSIO/io_control/in_file_test_io_0 does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
On Thu, May 9, 2013 at 8:32 PM, linux freaker
<linuxfreaker@gmail.com<mailto:linuxfreaker@gmail.com>> wrote:
Do I need to create benchmark folder under /mnt/lustre?
On Thu, May 9, 2013 at 8:24 PM, Diep, Minh
<minh.diep@intel.com<mailto:minh.diep@intel.com>> wrote:
You should be able to use the exact same command as in hdfs but make sure you create a
symlink from /benchmark or /benchmarks (don't remember with the s or not) to lustre
FS.
HTH
-Minh
From: linux freaker <linuxfreaker@gmail.com<mailto:linuxfreaker@gmail.com>>
Date: Thursday, May 9, 2013 1:16 AM
To: "hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>"
<hpdd-discuss@lists.01.org<mailto:hpdd-discuss@lists.01.org>>
Subject: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?
Any suggestion?