WARN io.ReadaheadPool: Failed readahead on ifile EBADF: Bad file descriptor 失败案例vs成功案例
终于正确配置好Hadoop了，我用的是centos7 伪分布式安装，做个记录，留个纪念，也为和我遇到同样问题的道友“排忧解难”一下
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hadoop jar ./share/hadoop/mapreduce//hadoop-mapreduce-examples-3.1.3.jar grep /opt/module/hadoop-3.1.3/input /opt/module/hadoop-3.1.3/output1 'dfs[a-z]+'
2022-05-11 17:36:48,699 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2022-05-11 17:36:48,764 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-05-11 17:36:48,765 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2022-05-11 17:36:49,178 INFO input.FileInputFormat: Total input files to process : 9
2022-05-11 17:36:49,229 INFO mapreduce.JobSubmitter: number of splits:9
2022-05-11 17:36:49,558 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local334235216_0001
2022-05-11 17:36:49,560 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 17:36:49,692 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 17:36:49,693 INFO mapreduce.Job: Running job: job_local334235216_0001
2022-05-11 17:36:49,694 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 17:36:49,720 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:49,720 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:49,721 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 17:36:49,799 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 17:36:49,799 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000000_0
2022-05-11 17:36:49,841 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:49,842 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:49,953 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:49,957 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hadoop-policy.xml:0+11392
2022-05-11 17:36:50,167 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:50,167 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:50,167 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:50,167 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:50,167 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:50,176 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:50,304 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 17:36:50,714 INFO mapreduce.Job: Job job_local334235216_0001 running in uber mode : false
2022-05-11 17:36:50,732 INFO mapreduce.Job: map 0% reduce 0%
2022-05-11 17:36:50,779 INFO mapred.LocalJobRunner:
2022-05-11 17:36:50,782 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:50,782 INFO mapred.MapTask: Spilling map output
2022-05-11 17:36:50,782 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2022-05-11 17:36:50,782 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2022-05-11 17:36:50,808 INFO mapred.MapTask: Finished spill 0
2022-05-11 17:36:50,821 INFO mapred.Task: Task:attempt_local334235216_0001_m_000000_0 is done. And is in the process of committing
2022-05-11 17:36:50,826 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:50,826 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000000_0' done.
2022-05-11 17:36:50,834 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000000_0: Counters: 23
File System Counters
FILE: Number of bytes read=317748
FILE: Number of bytes written=822141
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=11392
HDFS: Number of bytes written=0
HDFS: Number of read operations=5
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=265
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=134
Combine input records=1
Combine output records=1
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=266338304
File Input Format Counters
Bytes Read=11392
2022-05-11 17:36:50,834 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000000_0
2022-05-11 17:36:50,835 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000001_0
2022-05-11 17:36:50,836 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:50,836 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:50,837 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:50,838 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/capacity-scheduler.xml:0+8260
2022-05-11 17:36:51,029 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,030 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,030 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,030 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,030 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,031 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,041 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,041 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,046 INFO mapred.Task: Task:attempt_local334235216_0001_m_000001_0 is done. And is in the process of committing
2022-05-11 17:36:51,051 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,052 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000001_0' done.
2022-05-11 17:36:51,052 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000001_0: Counters: 23
File System Counters
FILE: Number of bytes read=318960
FILE: Number of bytes written=822179
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=19652
HDFS: Number of bytes written=0
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=220
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=139
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=371720192
File Input Format Counters
Bytes Read=8260
2022-05-11 17:36:51,052 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000001_0
2022-05-11 17:36:51,052 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000002_0
2022-05-11 17:36:51,053 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,053 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,054 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,054 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-acls.xml:0+3518
2022-05-11 17:36:51,084 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,084 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,084 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,084 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,085 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,085 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,092 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,092 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,095 INFO mapred.Task: Task:attempt_local334235216_0001_m_000002_0 is done. And is in the process of committing
2022-05-11 17:36:51,099 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,099 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000002_0' done.
2022-05-11 17:36:51,099 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000002_0: Counters: 23
File System Counters
FILE: Number of bytes read=320172
FILE: Number of bytes written=822217
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=23170
HDFS: Number of bytes written=0
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=135
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=129
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=10
Total committed heap usage (bytes)=458227712
File Input Format Counters
Bytes Read=3518
2022-05-11 17:36:51,099 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000002_0
2022-05-11 17:36:51,100 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000003_0
2022-05-11 17:36:51,100 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,100 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,101 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,101 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hdfs-site.xml:0+1093
2022-05-11 17:36:51,123 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,123 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,123 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,123 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,123 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,124 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,129 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,129 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,132 INFO mapred.Task: Task:attempt_local334235216_0001_m_000003_0 is done. And is in the process of committing
2022-05-11 17:36:51,135 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,136 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000003_0' done.
2022-05-11 17:36:51,136 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000003_0: Counters: 23
File System Counters
FILE: Number of bytes read=321384
FILE: Number of bytes written=822255
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=24263
HDFS: Number of bytes written=0
HDFS: Number of read operations=11
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=32
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=130
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=9
Total committed heap usage (bytes)=470810624
File Input Format Counters
Bytes Read=1093
2022-05-11 17:36:51,136 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000003_0
2022-05-11 17:36:51,136 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000004_0
2022-05-11 17:36:51,137 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,137 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,137 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,154 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/core-site.xml:0+1041
2022-05-11 17:36:51,177 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,177 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,177 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,177 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,177 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,179 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,185 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,185 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,188 INFO mapred.Task: Task:attempt_local334235216_0001_m_000004_0 is done. And is in the process of committing
2022-05-11 17:36:51,190 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,190 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000004_0' done.
2022-05-11 17:36:51,191 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000004_0: Counters: 23
File System Counters
FILE: Number of bytes read=322084
FILE: Number of bytes written=822293
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=25304
HDFS: Number of bytes written=0
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=28
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=130
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=5
Total committed heap usage (bytes)=463994880
File Input Format Counters
Bytes Read=1041
2022-05-11 17:36:51,191 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000004_0
2022-05-11 17:36:51,191 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000005_0
2022-05-11 17:36:51,192 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,192 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,192 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,193 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/mapred-site.xml:0+758
2022-05-11 17:36:51,211 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,211 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,211 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,211 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,211 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,212 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,221 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,221 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,223 INFO mapred.Task: Task:attempt_local334235216_0001_m_000005_0 is done. And is in the process of committing
2022-05-11 17:36:51,225 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,225 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000005_0' done.
2022-05-11 17:36:51,226 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000005_0: Counters: 23
File System Counters
FILE: Number of bytes read=322784
FILE: Number of bytes written=822331
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=26062
HDFS: Number of bytes written=0
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=21
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=132
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=758
2022-05-11 17:36:51,226 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000005_0
2022-05-11 17:36:51,226 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000006_0
2022-05-11 17:36:51,227 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,227 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,227 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,228 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/yarn-site.xml:0+690
2022-05-11 17:36:51,245 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,245 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,245 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,245 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,245 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,246 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,252 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,252 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,254 INFO mapred.Task: Task:attempt_local334235216_0001_m_000006_0 is done. And is in the process of committing
2022-05-11 17:36:51,258 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,258 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000006_0' done.
2022-05-11 17:36:51,259 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000006_0: Counters: 23
File System Counters
FILE: Number of bytes read=323484
FILE: Number of bytes written=822369
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=26752
HDFS: Number of bytes written=0
HDFS: Number of read operations=17
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=19
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=130
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=690
2022-05-11 17:36:51,259 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000006_0
2022-05-11 17:36:51,259 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000007_0
2022-05-11 17:36:51,260 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,260 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,260 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,261 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-site.xml:0+682
2022-05-11 17:36:51,277 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,277 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,277 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,277 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,277 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,278 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,283 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,283 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,285 INFO mapred.Task: Task:attempt_local334235216_0001_m_000007_0 is done. And is in the process of committing
2022-05-11 17:36:51,288 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,288 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000007_0' done.
2022-05-11 17:36:51,289 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000007_0: Counters: 23
File System Counters
FILE: Number of bytes read=324184
FILE: Number of bytes written=822407
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=27434
HDFS: Number of bytes written=0
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=20
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=129
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=682
2022-05-11 17:36:51,289 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000007_0
2022-05-11 17:36:51,289 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000008_0
2022-05-11 17:36:51,291 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,291 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,291 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,292 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/httpfs-site.xml:0+620
2022-05-11 17:36:51,308 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,308 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,308 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,308 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,308 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,308 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,313 INFO mapred.LocalJobRunner:
2022-05-11 17:36:51,313 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,316 INFO mapred.Task: Task:attempt_local334235216_0001_m_000008_0 is done. And is in the process of committing
2022-05-11 17:36:51,319 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,319 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000008_0' done.
2022-05-11 17:36:51,320 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000008_0: Counters: 23
File System Counters
FILE: Number of bytes read=324372
FILE: Number of bytes written=822445
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28054
HDFS: Number of bytes written=0
HDFS: Number of read operations=21
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=17
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=132
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=620
2022-05-11 17:36:51,320 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000008_0
2022-05-11 17:36:51,320 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 17:36:51,381 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 17:36:51,381 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_r_000000_0
2022-05-11 17:36:51,388 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,388 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,388 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,391 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3f9f840d
2022-05-11 17:36:51,392 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 17:36:51,426 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=329200416, maxSingleShuffleLimit=82300104, mergeThreshold=217272288, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 17:36:51,429 INFO reduce.EventFetcher: attempt_local334235216_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 17:36:51,469 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,475 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000002_0
2022-05-11 17:36:51,481 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
2022-05-11 17:36:51,484 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,484 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000008_0
2022-05-11 17:36:51,485 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
2022-05-11 17:36:51,485 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,486 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000005_0
2022-05-11 17:36:51,486 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
2022-05-11 17:36:51,488 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,488 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000004_0
2022-05-11 17:36:51,488 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
2022-05-11 17:36:51,489 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,490 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000001_0
2022-05-11 17:36:51,490 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
2022-05-11 17:36:51,491 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,491 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000007_0
2022-05-11 17:36:51,491 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
2022-05-11 17:36:51,492 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2022-05-11 17:36:51,493 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local334235216_0001_m_000000_0
2022-05-11 17:36:51,493 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->33
2022-05-11 17:36:51,494 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,494 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000006_0
2022-05-11 17:36:51,494 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
2022-05-11 17:36:51,495 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,495 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000003_0
2022-05-11 17:36:51,495 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 35, usedMemory ->37
2022-05-11 17:36:51,496 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 17:36:51,496 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 17:36:51,497 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,507 INFO mapred.Merger: Merging 9 sorted segments
2022-05-11 17:36:51,546 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,546 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2022-05-11 17:36:51,546 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,547 INFO reduce.MergeManagerImpl: Merged 9 segments, 37 bytes to disk to satisfy reduce memory limit
2022-05-11 17:36:51,546 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,547 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2022-05-11 17:36:51,548 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 17:36:51,548 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 17:36:51,548 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2022-05-11 17:36:51,549 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 17:36:51,641 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2022-05-11 17:36:51,735 INFO mapreduce.Job: map 100% reduce 0%
2022-05-11 17:36:51,859 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 17:36:51,922 INFO mapred.Task: Task:attempt_local334235216_0001_r_000000_0 is done. And is in the process of committing
2022-05-11 17:36:51,924 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 17:36:51,924 INFO mapred.Task: Task attempt_local334235216_0001_r_000000_0 is allowed to commit now
2022-05-11 17:36:51,939 INFO output.FileOutputCommitter: Saved output of task 'attempt_local334235216_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/grep-temp-482164475
2022-05-11 17:36:51,940 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 17:36:51,940 INFO mapred.Task: Task 'attempt_local334235216_0001_r_000000_0' done.
2022-05-11 17:36:51,941 INFO mapred.Task: Final Counters for attempt_local334235216_0001_r_000000_0: Counters: 29
File System Counters
FILE: Number of bytes read=324758
FILE: Number of bytes written=822470
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28054
HDFS: Number of bytes written=111
HDFS: Number of read operations=26
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=73
Reduce input records=1
Reduce output records=1
Spilled Records=1
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=0
Total committed heap usage (bytes)=470286336
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=111
2022-05-11 17:36:51,941 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_r_000000_0
2022-05-11 17:36:51,941 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 17:36:52,736 INFO mapreduce.Job: map 100% reduce 100%
2022-05-11 17:36:52,736 INFO mapreduce.Job: Job job_local334235216_0001 completed successfully
2022-05-11 17:36:52,752 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=3219930
FILE: Number of bytes written=8223107
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=240137
HDFS: Number of bytes written=111
HDFS: Number of read operations=143
HDFS: Number of large read operations=0
HDFS: Number of write operations=12
Map-Reduce Framework
Map input records=757
Map output records=1
Map output bytes=17
Map output materialized bytes=73
Input split bytes=1185
Combine input records=1
Combine output records=1
Reduce input groups=1
Reduce shuffle bytes=73
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=36
Total committed heap usage (bytes)=4382523392
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=28054
File Output Format Counters
Bytes Written=111
2022-05-11 17:36:52,769 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 17:36:52,922 INFO input.FileInputFormat: Total input files to process : 1
2022-05-11 17:36:52,924 INFO mapreduce.JobSubmitter: number of splits:1
2022-05-11 17:36:52,937 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1614699978_0002
2022-05-11 17:36:52,937 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 17:36:52,992 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 17:36:52,993 INFO mapreduce.Job: Running job: job_local1614699978_0002
2022-05-11 17:36:52,994 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 17:36:52,994 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:52,994 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:52,994 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 17:36:53,001 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 17:36:53,001 INFO mapred.LocalJobRunner: Starting task: attempt_local1614699978_0002_m_000000_0
2022-05-11 17:36:53,003 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:53,003 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:53,003 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:53,004 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/grep-temp-482164475/part-r-00000:0+111
2022-05-11 17:36:53,021 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:53,021 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:53,021 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:53,021 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:53,021 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:53,023 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:53,042 INFO mapred.LocalJobRunner:
2022-05-11 17:36:53,042 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:53,042 INFO mapred.MapTask: Spilling map output
2022-05-11 17:36:53,042 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2022-05-11 17:36:53,042 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2022-05-11 17:36:53,043 INFO mapred.MapTask: Finished spill 0
2022-05-11 17:36:53,045 INFO mapred.Task: Task:attempt_local1614699978_0002_m_000000_0 is done. And is in the process of committing
2022-05-11 17:36:53,047 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:53,047 INFO mapred.Task: Task 'attempt_local1614699978_0002_m_000000_0' done.
2022-05-11 17:36:53,048 INFO mapred.Task: Final Counters for attempt_local1614699978_0002_m_000000_0: Counters: 22
File System Counters
FILE: Number of bytes read=641323
FILE: Number of bytes written=1644014
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28165
HDFS: Number of bytes written=111
HDFS: Number of read operations=32
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Map-Reduce Framework
Map input records=1
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=131
Combine input records=0
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=469237760
File Input Format Counters
Bytes Read=111
2022-05-11 17:36:53,048 INFO mapred.LocalJobRunner: Finishing task: attempt_local1614699978_0002_m_000000_0
2022-05-11 17:36:53,048 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 17:36:53,049 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 17:36:53,049 INFO mapred.LocalJobRunner: Starting task: attempt_local1614699978_0002_r_000000_0
2022-05-11 17:36:53,051 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:53,051 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:53,051 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:53,051 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4f5e74fe
2022-05-11 17:36:53,051 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 17:36:53,052 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=328466432, maxSingleShuffleLimit=82116608, mergeThreshold=216787856, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 17:36:53,053 INFO reduce.EventFetcher: attempt_local1614699978_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 17:36:53,054 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1614699978_0002_m_000000_0 decomp: 21 len: 25 to MEMORY
2022-05-11 17:36:53,055 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1614699978_0002_m_000000_0
2022-05-11 17:36:53,055 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
2022-05-11 17:36:53,055 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 17:36:53,056 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 17:36:53,056 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 17:36:53,057 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 17:36:53,057 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2022-05-11 17:36:53,058 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit
2022-05-11 17:36:53,058 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2022-05-11 17:36:53,058 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 17:36:53,058 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 17:36:53,059 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2022-05-11 17:36:53,059 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 17:36:53,067 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 17:36:53,076 INFO mapred.Task: Task:attempt_local1614699978_0002_r_000000_0 is done. And is in the process of committing
2022-05-11 17:36:53,078 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 17:36:53,078 INFO mapred.Task: Task attempt_local1614699978_0002_r_000000_0 is allowed to commit now
2022-05-11 17:36:53,084 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1614699978_0002_r_000000_0' to hdfs://localhost:9000/opt/module/hadoop-3.1.3/output1
2022-05-11 17:36:53,085 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 17:36:53,085 INFO mapred.Task: Task 'attempt_local1614699978_0002_r_000000_0' done.
2022-05-11 17:36:53,085 INFO mapred.Task: Final Counters for attempt_local1614699978_0002_r_000000_0: Counters: 29
File System Counters
FILE: Number of bytes read=641405
FILE: Number of bytes written=1644039
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28165
HDFS: Number of bytes written=122
HDFS: Number of read operations=37
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=25
Reduce input records=1
Reduce output records=1
Spilled Records=1
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=469237760
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=11
2022-05-11 17:36:53,085 INFO mapred.LocalJobRunner: Finishing task: attempt_local1614699978_0002_r_000000_0
2022-05-11 17:36:53,085 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 17:36:53,995 INFO mapreduce.Job: Job job_local1614699978_0002 running in uber mode : false
2022-05-11 17:36:53,995 INFO mapreduce.Job: map 100% reduce 100%
2022-05-11 17:36:53,995 INFO mapreduce.Job: Job job_local1614699978_0002 completed successfully
2022-05-11 17:36:54,004 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=1282728
FILE: Number of bytes written=3288053
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=56330
HDFS: Number of bytes written=233
HDFS: Number of read operations=69
HDFS: Number of large read operations=0
HDFS: Number of write operations=14
Map-Reduce Framework
Map input records=1
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=131
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=25
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=938475520
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=111
File Output Format Counters
Bytes Written=11
[hadoop@hadoop2 hadoop-3.1.3]$ ll -al
总用量 180
drwxr-xr-x. 15 hadoop hadoop 232 5月 11 17:05 .
drwxr-xr-x. 4 hadoop hadoop 46 5月 2 08:21 ..
drwxr-xr-x. 2 hadoop hadoop 183 9月 12 2019 bin
drwxr-xr-x. 3 hadoop hadoop 20 9月 12 2019 etc
drwxr-xr-x. 2 hadoop hadoop 106 9月 12 2019 include
drwxr-xr-x. 3 hadoop hadoop 20 9月 12 2019 lib
drwxr-xr-x. 4 hadoop hadoop 288 9月 12 2019 libexec
-rw-rw-r--. 1 hadoop hadoop 147145 9月 4 2019 LICENSE.txt
drwxrwxr-x. 2 hadoop hadoop 4096 5月 11 17:05 logs
-rw-rw-r--. 1 hadoop hadoop 21867 9月 4 2019 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop 1366 9月 4 2019 README.txt
drwxr-xr-x. 3 hadoop hadoop 4096 9月 12 2019 sbin
drwxr-xr-x. 4 hadoop hadoop 31 9月 12 2019 share
drwxrwxr-x. 3 hadoop hadoop 17 5月 11 09:04 tmp
drwxrwxr-x. 4 hadoop hadoop 31 5月 11 17:36 tmpp
drwxrwxr-x. 2 hadoop hadoop 22 5月 2 22:53 wcinput
drwxr-xr-x. 2 hadoop hadoop 88 5月 2 22:46 wcoutput
drwxr-xr-x. 2 hadoop hadoop 88 5月 2 22:54 wcoutput2
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hdfs dfs -cat /opt/module/hadoop-3.1.3/output1/*
2022-05-11 18:01:37,074 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1 dfsadmin
成功案例：
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hadoop jar ./share/hadoop/mapreduce//hadoop-mapreduce-examples-3.1.3.jar grep /opt/module/hadoop-3.1.3/input ./output1 'dfs[a-z.]+' 2022-05-11 18:04:56,846 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2022-05-11 18:04:56,940 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-05-11 18:04:56,940 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2022-05-11 18:04:57,369 INFO input.FileInputFormat: Total input files to process : 9
2022-05-11 18:04:57,393 INFO mapreduce.JobSubmitter: number of splits:9
2022-05-11 18:04:57,535 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1922828463_0001
2022-05-11 18:04:57,536 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 18:04:57,637 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 18:04:57,637 INFO mapreduce.Job: Running job: job_local1922828463_0001
2022-05-11 18:04:57,638 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 18:04:57,645 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:57,645 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:57,646 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 18:04:57,691 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 18:04:57,692 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000000_0
2022-05-11 18:04:57,716 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:57,716 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:57,734 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:57,739 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hadoop-policy.xml:0+11392
2022-05-11 18:04:57,815 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:57,815 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:57,815 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:57,815 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:57,815 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:57,824 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:57,850 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 18:04:57,939 INFO mapred.LocalJobRunner:
2022-05-11 18:04:57,942 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:57,942 INFO mapred.MapTask: Spilling map output
2022-05-11 18:04:57,942 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2022-05-11 18:04:57,942 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2022-05-11 18:04:57,956 INFO mapred.MapTask: Finished spill 0
2022-05-11 18:04:57,967 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000000_0 is done. And is in the process of committing
2022-05-11 18:04:57,971 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:57,971 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000000_0' done.
2022-05-11 18:04:57,981 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000000_0: Counters: 23
File System Counters
FILE: Number of bytes read=317748
FILE: Number of bytes written=824593
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=11392
HDFS: Number of bytes written=0
HDFS: Number of read operations=5
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=265
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=134
Combine input records=1
Combine output records=1
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=267386880
File Input Format Counters
Bytes Read=11392
2022-05-11 18:04:57,982 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000000_0
2022-05-11 18:04:57,982 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000001_0
2022-05-11 18:04:57,989 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:57,989 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:57,990 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:57,991 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/capacity-scheduler.xml:0+8260
2022-05-11 18:04:58,319 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,319 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,319 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,319 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,319 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,321 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,329 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,329 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,332 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000001_0 is done. And is in the process of committing
2022-05-11 18:04:58,336 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,336 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000001_0' done.
2022-05-11 18:04:58,337 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000001_0: Counters: 23
File System Counters
FILE: Number of bytes read=318960
FILE: Number of bytes written=824631
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=19652
HDFS: Number of bytes written=0
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=220
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=139
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=372768768
File Input Format Counters
Bytes Read=8260
2022-05-11 18:04:58,337 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000001_0
2022-05-11 18:04:58,337 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000002_0
2022-05-11 18:04:58,338 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,338 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,339 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,340 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-acls.xml:0+3518
2022-05-11 18:04:58,524 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,524 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,524 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,524 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,524 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,525 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,531 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,531 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,534 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000002_0 is done. And is in the process of committing
2022-05-11 18:04:58,538 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,538 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000002_0' done.
2022-05-11 18:04:58,538 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000002_0: Counters: 23
File System Counters
FILE: Number of bytes read=320172
FILE: Number of bytes written=824669
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=23170
HDFS: Number of bytes written=0
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=135
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=129
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=9
Total committed heap usage (bytes)=458227712
File Input Format Counters
Bytes Read=3518
2022-05-11 18:04:58,539 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000002_0
2022-05-11 18:04:58,539 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000003_0
2022-05-11 18:04:58,539 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,539 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,540 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,541 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hdfs-site.xml:0+1093
2022-05-11 18:04:58,563 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,563 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,563 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,563 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,563 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,564 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,572 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,572 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,572 INFO mapred.MapTask: Spilling map output
2022-05-11 18:04:58,572 INFO mapred.MapTask: bufstart = 0; bufend = 84; bufvoid = 104857600
2022-05-11 18:04:58,572 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2022-05-11 18:04:58,574 INFO mapred.MapTask: Finished spill 0
2022-05-11 18:04:58,575 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000003_0 is done. And is in the process of committing
2022-05-11 18:04:58,578 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,578 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000003_0' done.
2022-05-11 18:04:58,579 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000003_0: Counters: 23
File System Counters
FILE: Number of bytes read=321384
FILE: Number of bytes written=824797
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=24263
HDFS: Number of bytes written=0
HDFS: Number of read operations=11
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=32
Map output records=3
Map output bytes=84
Map output materialized bytes=96
Input split bytes=130
Combine input records=3
Combine output records=3
Spilled Records=3
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=6
Total committed heap usage (bytes)=470810624
File Input Format Counters
Bytes Read=1093
2022-05-11 18:04:58,579 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000003_0
2022-05-11 18:04:58,579 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000004_0
2022-05-11 18:04:58,580 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,580 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,580 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,582 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/core-site.xml:0+1041
2022-05-11 18:04:58,606 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,606 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,606 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,606 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,606 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,607 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,613 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,613 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,615 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000004_0 is done. And is in the process of committing
2022-05-11 18:04:58,618 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,618 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000004_0' done.
2022-05-11 18:04:58,619 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000004_0: Counters: 23
File System Counters
FILE: Number of bytes read=322084
FILE: Number of bytes written=824835
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=25304
HDFS: Number of bytes written=0
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=28
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=130
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=7
Total committed heap usage (bytes)=463994880
File Input Format Counters
Bytes Read=1041
2022-05-11 18:04:58,619 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000004_0
2022-05-11 18:04:58,619 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000005_0
2022-05-11 18:04:58,620 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,620 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,621 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,622 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/mapred-site.xml:0+758
2022-05-11 18:04:58,642 INFO mapreduce.Job: Job job_local1922828463_0001 running in uber mode : false
2022-05-11 18:04:58,648 INFO mapreduce.Job: map 100% reduce 0%
2022-05-11 18:04:58,649 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,649 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,649 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,649 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,649 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,651 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,658 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,658 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,660 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000005_0 is done. And is in the process of committing
2022-05-11 18:04:58,663 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,663 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000005_0' done.
2022-05-11 18:04:58,664 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000005_0: Counters: 23
File System Counters
FILE: Number of bytes read=322784
FILE: Number of bytes written=824873
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=26062
HDFS: Number of bytes written=0
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=21
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=132
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=4
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=758
2022-05-11 18:04:58,664 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000005_0
2022-05-11 18:04:58,664 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000006_0
2022-05-11 18:04:58,664 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,665 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,665 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,666 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/yarn-site.xml:0+690
2022-05-11 18:04:58,686 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,686 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,686 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,686 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,686 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,687 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,692 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,692 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,694 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000006_0 is done. And is in the process of committing
2022-05-11 18:04:58,698 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,698 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000006_0' done.
2022-05-11 18:04:58,699 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000006_0: Counters: 23
File System Counters
FILE: Number of bytes read=323484
FILE: Number of bytes written=824911
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=26752
HDFS: Number of bytes written=0
HDFS: Number of read operations=17
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=19
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=130
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=5
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=690
2022-05-11 18:04:58,699 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000006_0
2022-05-11 18:04:58,699 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000007_0
2022-05-11 18:04:58,700 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,700 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,700 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,701 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-site.xml:0+682
2022-05-11 18:04:58,719 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,720 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,720 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,720 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,720 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,721 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,725 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,725 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,728 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000007_0 is done. And is in the process of committing
2022-05-11 18:04:58,730 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,730 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000007_0' done.
2022-05-11 18:04:58,731 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000007_0: Counters: 23
File System Counters
FILE: Number of bytes read=324184
FILE: Number of bytes written=824949
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=27434
HDFS: Number of bytes written=0
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=20
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=129
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=682
2022-05-11 18:04:58,731 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000007_0
2022-05-11 18:04:58,731 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000008_0
2022-05-11 18:04:58,732 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,732 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,732 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,733 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/httpfs-site.xml:0+620
2022-05-11 18:04:58,752 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,753 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,753 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,753 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,753 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,753 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,758 INFO mapred.LocalJobRunner:
2022-05-11 18:04:58,758 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,760 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000008_0 is done. And is in the process of committing
2022-05-11 18:04:58,762 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,762 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000008_0' done.
2022-05-11 18:04:58,763 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000008_0: Counters: 23
File System Counters
FILE: Number of bytes read=324372
FILE: Number of bytes written=824987
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28054
HDFS: Number of bytes written=0
HDFS: Number of read operations=21
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
Map-Reduce Framework
Map input records=17
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=132
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=3
Total committed heap usage (bytes)=470286336
File Input Format Counters
Bytes Read=620
2022-05-11 18:04:58,763 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000008_0
2022-05-11 18:04:58,763 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 18:04:58,766 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 18:04:58,766 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_r_000000_0
2022-05-11 18:04:58,773 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,773 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,773 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,776 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@323c3357
2022-05-11 18:04:58,777 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 18:04:58,795 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=329200416, maxSingleShuffleLimit=82300104, mergeThreshold=217272288, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 18:04:58,799 INFO reduce.EventFetcher: attempt_local1922828463_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 18:04:58,831 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,835 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000007_0
2022-05-11 18:04:58,836 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
2022-05-11 18:04:58,838 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,839 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000001_0
2022-05-11 18:04:58,839 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
2022-05-11 18:04:58,840 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,841 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000004_0
2022-05-11 18:04:58,841 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
2022-05-11 18:04:58,842 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000003_0 decomp: 92 len: 96 to MEMORY
2022-05-11 18:04:58,843 INFO reduce.InMemoryMapOutput: Read 92 bytes from map-output for attempt_local1922828463_0001_m_000003_0
2022-05-11 18:04:58,843 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 92, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->98
2022-05-11 18:04:58,843 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-05-11 18:04:58,844 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,852 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000006_0
2022-05-11 18:04:58,852 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 98, usedMemory ->100
2022-05-11 18:04:58,853 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2022-05-11 18:04:58,853 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1922828463_0001_m_000000_0
2022-05-11 18:04:58,853 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 6, commitMemory -> 100, usedMemory ->121
2022-05-11 18:04:58,855 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,855 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000002_0
2022-05-11 18:04:58,855 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 121, usedMemory ->123
2022-05-11 18:04:58,856 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,856 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000005_0
2022-05-11 18:04:58,857 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 123, usedMemory ->125
2022-05-11 18:04:58,858 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,858 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000008_0
2022-05-11 18:04:58,858 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 125, usedMemory ->127
2022-05-11 18:04:58,858 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 18:04:58,859 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 18:04:58,859 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 18:04:58,865 INFO mapred.Merger: Merging 9 sorted segments
2022-05-11 18:04:58,865 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 78 bytes
2022-05-11 18:04:58,866 INFO reduce.MergeManagerImpl: Merged 9 segments, 127 bytes to disk to satisfy reduce memory limit
2022-05-11 18:04:58,866 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
2022-05-11 18:04:58,867 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 18:04:58,867 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 18:04:58,867 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 87 bytes
2022-05-11 18:04:58,868 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 18:04:58,914 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2022-05-11 18:04:58,945 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 18:04:59,378 INFO mapred.Task: Task:attempt_local1922828463_0001_r_000000_0 is done. And is in the process of committing
2022-05-11 18:04:59,380 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 18:04:59,380 INFO mapred.Task: Task attempt_local1922828463_0001_r_000000_0 is allowed to commit now
2022-05-11 18:04:59,394 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1922828463_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/grep-temp-1352764713
2022-05-11 18:04:59,394 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 18:04:59,394 INFO mapred.Task: Task 'attempt_local1922828463_0001_r_000000_0' done.
2022-05-11 18:04:59,395 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_r_000000_0: Counters: 29
File System Counters
FILE: Number of bytes read=324938
FILE: Number of bytes written=825102
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28054
HDFS: Number of bytes written=219
HDFS: Number of read operations=26
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=4
Reduce shuffle bytes=163
Reduce input records=4
Reduce output records=4
Spilled Records=4
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=0
Total committed heap usage (bytes)=470286336
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=219
2022-05-11 18:04:59,395 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_r_000000_0
2022-05-11 18:04:59,395 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 18:04:59,653 INFO mapreduce.Job: map 100% reduce 100%
2022-05-11 18:04:59,654 INFO mapreduce.Job: Job job_local1922828463_0001 completed successfully
2022-05-11 18:04:59,664 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=3220110
FILE: Number of bytes written=8248347
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=240137
HDFS: Number of bytes written=219
HDFS: Number of read operations=143
HDFS: Number of large read operations=0
HDFS: Number of write operations=12
Map-Reduce Framework
Map input records=757
Map output records=4
Map output bytes=101
Map output materialized bytes=163
Input split bytes=1185
Combine input records=4
Combine output records=4
Reduce input groups=4
Reduce shuffle bytes=163
Reduce input records=4
Reduce output records=4
Spilled Records=8
Shuffled Maps =9
Failed Shuffles=0
Merged Map outputs=9
GC time elapsed (ms)=37
Total committed heap usage (bytes)=4384620544
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=28054
File Output Format Counters
Bytes Written=219
2022-05-11 18:04:59,679 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 18:04:59,697 INFO input.FileInputFormat: Total input files to process : 1
2022-05-11 18:04:59,699 INFO mapreduce.JobSubmitter: number of splits:1
2022-05-11 18:04:59,723 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local324626594_0002
2022-05-11 18:04:59,723 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 18:04:59,776 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 18:04:59,777 INFO mapreduce.Job: Running job: job_local324626594_0002
2022-05-11 18:04:59,778 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 18:04:59,778 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:59,778 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:59,779 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 18:04:59,785 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 18:04:59,785 INFO mapred.LocalJobRunner: Starting task: attempt_local324626594_0002_m_000000_0
2022-05-11 18:04:59,787 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:59,787 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:59,787 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:59,789 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/grep-temp-1352764713/part-r-00000:0+219
2022-05-11 18:04:59,805 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:59,805 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:59,805 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:59,805 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:59,805 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:59,806 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:59,821 INFO mapred.LocalJobRunner:
2022-05-11 18:04:59,821 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:59,821 INFO mapred.MapTask: Spilling map output
2022-05-11 18:04:59,821 INFO mapred.MapTask: bufstart = 0; bufend = 101; bufvoid = 104857600
2022-05-11 18:04:59,821 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
2022-05-11 18:04:59,823 INFO mapred.MapTask: Finished spill 0
2022-05-11 18:04:59,824 INFO mapred.Task: Task:attempt_local324626594_0002_m_000000_0 is done. And is in the process of committing
2022-05-11 18:04:59,828 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:59,828 INFO mapred.Task: Task 'attempt_local324626594_0002_m_000000_0' done.
2022-05-11 18:04:59,828 INFO mapred.Task: Final Counters for attempt_local324626594_0002_m_000000_0: Counters: 22
File System Counters
FILE: Number of bytes read=641505
FILE: Number of bytes written=1644282
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28273
HDFS: Number of bytes written=219
HDFS: Number of read operations=32
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Map-Reduce Framework
Map input records=4
Map output records=4
Map output bytes=101
Map output materialized bytes=115
Input split bytes=132
Combine input records=0
Spilled Records=4
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=0
Total committed heap usage (bytes)=469237760
File Input Format Counters
Bytes Read=219
2022-05-11 18:04:59,828 INFO mapred.LocalJobRunner: Finishing task: attempt_local324626594_0002_m_000000_0
2022-05-11 18:04:59,828 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 18:04:59,829 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 18:04:59,830 INFO mapred.LocalJobRunner: Starting task: attempt_local324626594_0002_r_000000_0
2022-05-11 18:04:59,832 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:59,832 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:59,832 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:59,832 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@23ea483f
2022-05-11 18:04:59,833 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 18:04:59,833 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=328466432, maxSingleShuffleLimit=82116608, mergeThreshold=216787856, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 18:04:59,834 INFO reduce.EventFetcher: attempt_local324626594_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 18:04:59,836 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local324626594_0002_m_000000_0 decomp: 111 len: 115 to MEMORY
2022-05-11 18:04:59,837 INFO reduce.InMemoryMapOutput: Read 111 bytes from map-output for attempt_local324626594_0002_m_000000_0
2022-05-11 18:04:59,837 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 111, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->111
2022-05-11 18:04:59,838 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 18:04:59,841 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 18:04:59,841 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 18:04:59,842 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 18:04:59,842 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 101 bytes
2022-05-11 18:04:59,842 INFO reduce.MergeManagerImpl: Merged 1 segments, 111 bytes to disk to satisfy reduce memory limit
2022-05-11 18:04:59,843 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
2022-05-11 18:04:59,843 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 18:04:59,843 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 18:04:59,843 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 101 bytes
2022-05-11 18:04:59,843 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 18:04:59,858 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 18:04:59,871 INFO mapred.Task: Task:attempt_local324626594_0002_r_000000_0 is done. And is in the process of committing
2022-05-11 18:04:59,873 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 18:04:59,873 INFO mapred.Task: Task attempt_local324626594_0002_r_000000_0 is allowed to commit now
2022-05-11 18:04:59,885 INFO output.FileOutputCommitter: Saved output of task 'attempt_local324626594_0002_r_000000_0' to hdfs://localhost:9000/user/hadoop/output1
2022-05-11 18:04:59,888 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 18:04:59,888 INFO mapred.Task: Task 'attempt_local324626594_0002_r_000000_0' done.
2022-05-11 18:04:59,888 INFO mapred.Task: Final Counters for attempt_local324626594_0002_r_000000_0: Counters: 29
File System Counters
FILE: Number of bytes read=641767
FILE: Number of bytes written=1644397
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28273
HDFS: Number of bytes written=296
HDFS: Number of read operations=37
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=115
Reduce input records=4
Reduce output records=4
Spilled Records=4
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=469237760
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=77
2022-05-11 18:04:59,888 INFO mapred.LocalJobRunner: Finishing task: attempt_local324626594_0002_r_000000_0
2022-05-11 18:04:59,889 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 18:05:00,778 INFO mapreduce.Job: Job job_local324626594_0002 running in uber mode : false
2022-05-11 18:05:00,778 INFO mapreduce.Job: map 100% reduce 100%
2022-05-11 18:05:00,779 INFO mapreduce.Job: Job job_local324626594_0002 completed successfully
2022-05-11 18:05:00,789 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=1283272
FILE: Number of bytes written=3288679
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=56546
HDFS: Number of bytes written=515
HDFS: Number of read operations=69
HDFS: Number of large read operations=0
HDFS: Number of write operations=14
Map-Reduce Framework
Map input records=4
Map output records=4
Map output bytes=101
Map output materialized bytes=115
Input split bytes=132
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=115
Reduce input records=4
Reduce output records=4
Spilled Records=8
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=938475520
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=219
File Output Format Counters
Bytes Written=77
查看目录下文件（没有显示出output1）：
[hadoop@hadoop2 hadoop-3.1.3]$ ll
总用量 180
drwxr-xr-x. 2 hadoop hadoop 183 9月 12 2019 bin
drwxr-xr-x. 3 hadoop hadoop 20 9月 12 2019 etc
drwxr-xr-x. 2 hadoop hadoop 106 9月 12 2019 include
drwxr-xr-x. 3 hadoop hadoop 20 9月 12 2019 lib
drwxr-xr-x. 4 hadoop hadoop 288 9月 12 2019 libexec
-rw-rw-r--. 1 hadoop hadoop 147145 9月 4 2019 LICENSE.txt
drwxrwxr-x. 2 hadoop hadoop 4096 5月 11 17:05 logs
-rw-rw-r--. 1 hadoop hadoop 21867 9月 4 2019 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop 1366 9月 4 2019 README.txt
drwxr-xr-x. 3 hadoop hadoop 4096 9月 12 2019 sbin
drwxr-xr-x. 4 hadoop hadoop 31 9月 12 2019 share
drwxrwxr-x. 3 hadoop hadoop 17 5月 11 09:04 tmp
drwxrwxr-x. 4 hadoop hadoop 31 5月 11 17:36 tmpp
drwxrwxr-x. 2 hadoop hadoop 22 5月 2 22:53 wcinput
drwxr-xr-x. 2 hadoop hadoop 88 5月 2 22:46 wcoutput
drwxr-xr-x. 2 hadoop hadoop 88 5月 2 22:54 wcoutput2
查看运行结果的命令（查看的是位于 HDFS 中的输出结果）：
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hdfs dfs -cat output1/*
2022-05-11 18:06:36,892 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1 dfsadmin
1 dfs.replication
1 dfs.namenode.name.dir
1 dfs.datanode.data.dir
数据是存储在hdfs中的，我们也可以把数据取回到本地：
//如果本地有同名文件，要先删去，这里浅试一下，做个复习
[hadoop@hadoop2 hadoop-3.1.3]$ rm -r ./output1
rm: 无法删除"./output1": 没有那个文件或目录//说明没有
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hdfs dfs -get output1 ./output1
2022-05-11 18:16:14,963 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
[hadoop@hadoop2 hadoop-3.1.3]$ cat ./output1/*
1 dfsadmin
1 dfs.replication
1 dfs.namenode.name.dir
1 dfs.datanode.data.dir
Hadoop 运行程序时，输出目录不能存在，否则会提示错误 “org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/hadoop/output already exists” ，因此若要再次执行，需要执行如下命令删除 output 文件夹:
./bin/hdfs dfs -rm -r output # 删除 output 文件夹
关闭hadoop：
[hadoop@hadoop2 hadoop-3.1.3]$ ./sbin/stop-dfs.sh
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [hadoop2]
开启：
[hadoop@hadoop2 hadoop-3.1.3]$ jps
7360 DataNode
7713 Jps
7570 SecondaryNameNode
7192 NameNode
有关此方面的问题，欢迎来问，因为刚刚结束所以有很大的热情呢~