Hadoop TestDFSIO

@VM

 

[bigdata@bigdata hadoop]$ hadoop jar hadoop-test-1.0.4.jar TestDFSIO -write -nrFiles 10 -fileSize 20

Warning: $HADOOP_HOME is deprecated.

 

TestDFSIO.0.0.4

13/04/21 12:54:47 INFO fs.TestDFSIO: nrFiles = 10

13/04/21 12:54:47 INFO fs.TestDFSIO: fileSize (MB) = 20

13/04/21 12:54:47 INFO fs.TestDFSIO: bufferSize = 1000000

13/04/21 12:54:48 INFO fs.TestDFSIO: creating control file: 20 mega bytes, 10 files

13/04/21 12:54:48 INFO fs.TestDFSIO: created control files for: 10 files

13/04/21 12:54:48 INFO mapred.FileInputFormat: Total input paths to process : 10

13/04/21 12:54:48 INFO mapred.JobClient: Running job: job_201304060732_0008

13/04/21 12:54:49 INFO mapred.JobClient:  map 0% reduce 0%

13/04/21 12:55:02 INFO mapred.JobClient:  map 20% reduce 0%

13/04/21 12:55:09 INFO mapred.JobClient:  map 40% reduce 0%

13/04/21 12:55:15 INFO mapred.JobClient:  map 60% reduce 0%

13/04/21 12:55:18 INFO mapred.JobClient:  map 60% reduce 13%

13/04/21 12:55:21 INFO mapred.JobClient:  map 80% reduce 20%

13/04/21 12:55:27 INFO mapred.JobClient:  map 100% reduce 26%

13/04/21 12:55:41 INFO mapred.JobClient:  map 100% reduce 100%

13/04/21 12:55:46 INFO mapred.JobClient: Job complete: job_201304060732_0008

13/04/21 12:55:46 INFO mapred.JobClient: Counters: 30

13/04/21 12:55:46 INFO mapred.JobClient:   Job Counters

13/04/21 12:55:46 INFO mapred.JobClient:     Launched reduce tasks=1

13/04/21 12:55:46 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=45841

13/04/21 12:55:46 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0

13/04/21 12:55:46 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0

13/04/21 12:55:46 INFO mapred.JobClient:     Launched map tasks=10

13/04/21 12:55:46 INFO mapred.JobClient:     Data-local map tasks=10

13/04/21 12:55:46 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=36159

13/04/21 12:55:46 INFO mapred.JobClient:   File Input Format Counters

13/04/21 12:55:46 INFO mapred.JobClient:     Bytes Read=1120

13/04/21 12:55:46 INFO mapred.JobClient:   File Output Format Counters

13/04/21 12:55:46 INFO mapred.JobClient:     Bytes Written=78

13/04/21 12:55:46 INFO mapred.JobClient:   FileSystemCounters

13/04/21 12:55:46 INFO mapred.JobClient:     FILE_BYTES_READ=833

13/04/21 12:55:46 INFO mapred.JobClient:     HDFS_BYTES_READ=2340

13/04/21 12:55:46 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=239157

13/04/21 12:55:46 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=209715278

13/04/21 12:55:46 INFO mapred.JobClient:   Map-Reduce Framework

13/04/21 12:55:46 INFO mapred.JobClient:     Map output materialized bytes=887

13/04/21 12:55:46 INFO mapred.JobClient:     Map input records=10

13/04/21 12:55:46 INFO mapred.JobClient:     Reduce shuffle bytes=799

13/04/21 12:55:46 INFO mapred.JobClient:     Spilled Records=100

13/04/21 12:55:46 INFO mapred.JobClient:     Map output bytes=727

13/04/21 12:55:46 INFO mapred.JobClient:     Total committed heap usage (bytes)=2764374016

13/04/21 12:55:46 INFO mapred.JobClient:     CPU time spent (ms)=11070

13/04/21 12:55:46 INFO mapred.JobClient:     Map input bytes=260

13/04/21 12:55:46 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1220

13/04/21 12:55:46 INFO mapred.JobClient:     Combine input records=0

13/04/21 12:55:46 INFO mapred.JobClient:     Reduce input records=50

13/04/21 12:55:46 INFO mapred.JobClient:     Reduce input groups=5

13/04/21 12:55:46 INFO mapred.JobClient:     Combine output records=0

13/04/21 12:55:46 INFO mapred.JobClient:     Physical memory (bytes) snapshot=2931482624

13/04/21 12:55:46 INFO mapred.JobClient:     Reduce output records=5

13/04/21 12:55:46 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=14189375488

13/04/21 12:55:46 INFO mapred.JobClient:     Map output records=50

13/04/21 12:55:46 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write

13/04/21 12:55:46 INFO fs.TestDFSIO:            Date & time: Sun Apr 21 12:55:46 CST 2013

13/04/21 12:55:46 INFO fs.TestDFSIO:        Number of files: 10

13/04/21 12:55:46 INFO fs.TestDFSIO: Total MBytes processed: 200

13/04/21 12:55:46 INFO fs.TestDFSIO:      Throughput mb/sec: 27.359781121751027

13/04/21 12:55:46 INFO fs.TestDFSIO: Average IO rate mb/sec: 31.685556411743164

13/04/21 12:55:46 INFO fs.TestDFSIO:  IO rate std deviation: 10.648457663926225

13/04/21 12:55:46 INFO fs.TestDFSIO:     Test exec time sec: 58.006

13/04/21 12:55:46 INFO fs.TestDFSIO:

你可能感兴趣的:(hadoop)