Re: [HPDD-discuss] How to run TestDfsio on hadoop running on lustre?

I tried running this way too but no luck: [root@lustreclient1 hadoop]# bin/hadoop jar hadoop-test-1.1.1.jar TestDFSIO -write -nrFiles 10 -fileSize 1000 -resFile /mnt/lustre/benchmarks/TestDFSIO/io_control TestDFSIO.0.0.4 13/05/10 16:05:29 INFO fs.TestDFSIO: nrFiles = 10 13/05/10 16:05:29 INFO fs.TestDFSIO: fileSize (MB) = 1000 13/05/10 16:05:29 INFO fs.TestDFSIO: bufferSize = 1000000 13/05/10 16:05:30 INFO fs.TestDFSIO: creating control file: 1000 mega bytes, 10 files 13/05/10 16:05:30 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/05/10 16:05:30 INFO fs.TestDFSIO: created control files for: 10 files 13/05/10 16:05:30 INFO mapred.FileInputFormat: Total input paths to process : 10 13/05/10 16:05:30 INFO mapred.JobClient: Running job: job_201305101207_0007 13/05/10 16:05:31 INFO mapred.JobClient: map 0% reduce 0% 13/05/10 16:06:01 INFO mapred.JobClient: map 20% reduce 0% 13/05/10 16:06:07 INFO mapred.JobClient: map 40% reduce 0% 13/05/10 16:06:08 INFO mapred.JobClient: map 40% reduce 13% 13/05/10 16:06:12 INFO mapred.JobClient: map 50% reduce 13% 13/05/10 16:06:13 INFO mapred.JobClient: map 60% reduce 13% 13/05/10 16:06:18 INFO mapred.JobClient: map 60% reduce 16% 13/05/10 16:06:24 INFO mapred.JobClient: map 60% reduce 20% 13/05/10 16:06:33 INFO mapred.JobClient: Task Id : attempt_201305101207_0007_m_000004_0, Status : FAILED java.io.FileNotFoundException: File file:/benchmarks/TestDFSIO/io_control/in_file_test_io_6 does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000004_0&filter=stdout 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000004_0&filter=stderr 13/05/10 16:06:33 INFO mapred.JobClient: Task Id : attempt_201305101207_0007_m_000005_0, Status : FAILED java.io.FileNotFoundException: File file:/benchmarks/TestDFSIO/io_control/in_file_test_io_9 does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474) at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43) at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000005_0&filter=stdout 13/05/10 16:06:33 WARN mapred.JobClient: Error reading task outputhttp://lustreclient2:50060/tasklog?plaintext=true&attemptid=attempt_201305101207_0007_m_000005_0&filter=stderr 13/05/10 16:06:37 INFO mapred.JobClient: map 80% reduce 20% On Fri, May 10, 2013 at 3:47 PM, linux freaker <linuxfreaker(a)gmail.com&gt;wrote: