WARN io.ReadaheadPool: Failed readahead on ifile EBADF: Bad file descriptor 失败案例vs成功案例

终于正确配置好Hadoop了,我用的是centos7 伪分布式安装,做个记录,留个纪念,也为和我遇到同样问题的道友“排忧解难”一下

[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hadoop jar ./share/hadoop/mapreduce//hadoop-mapreduce-examples-3.1.3.jar  grep /opt/module/hadoop-3.1.3/input /opt/module/hadoop-3.1.3/output1 'dfs[a-z]+' 
2022-05-11 17:36:48,699 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2022-05-11 17:36:48,764 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-05-11 17:36:48,765 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2022-05-11 17:36:49,178 INFO input.FileInputFormat: Total input files to process : 9
2022-05-11 17:36:49,229 INFO mapreduce.JobSubmitter: number of splits:9
2022-05-11 17:36:49,558 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local334235216_0001
2022-05-11 17:36:49,560 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 17:36:49,692 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 17:36:49,693 INFO mapreduce.Job: Running job: job_local334235216_0001
2022-05-11 17:36:49,694 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 17:36:49,720 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:49,720 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:49,721 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 17:36:49,799 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 17:36:49,799 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000000_0
2022-05-11 17:36:49,841 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:49,842 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:49,953 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:49,957 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hadoop-policy.xml:0+11392
2022-05-11 17:36:50,167 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:50,167 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:50,167 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:50,167 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:50,167 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:50,176 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:50,304 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 17:36:50,714 INFO mapreduce.Job: Job job_local334235216_0001 running in uber mode : false
2022-05-11 17:36:50,732 INFO mapreduce.Job:  map 0% reduce 0%
2022-05-11 17:36:50,779 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:50,782 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:50,782 INFO mapred.MapTask: Spilling map output
2022-05-11 17:36:50,782 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2022-05-11 17:36:50,782 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2022-05-11 17:36:50,808 INFO mapred.MapTask: Finished spill 0
2022-05-11 17:36:50,821 INFO mapred.Task: Task:attempt_local334235216_0001_m_000000_0 is done. And is in the process of committing
2022-05-11 17:36:50,826 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:50,826 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000000_0' done.
2022-05-11 17:36:50,834 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000000_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=317748
		FILE: Number of bytes written=822141
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=11392
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=5
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=265
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=134
		Combine input records=1
		Combine output records=1
		Spilled Records=1
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=266338304
	File Input Format Counters 
		Bytes Read=11392
2022-05-11 17:36:50,834 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000000_0
2022-05-11 17:36:50,835 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000001_0
2022-05-11 17:36:50,836 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:50,836 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:50,837 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:50,838 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/capacity-scheduler.xml:0+8260
2022-05-11 17:36:51,029 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,030 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,030 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,030 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,030 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,031 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,041 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,041 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,046 INFO mapred.Task: Task:attempt_local334235216_0001_m_000001_0 is done. And is in the process of committing
2022-05-11 17:36:51,051 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,052 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000001_0' done.
2022-05-11 17:36:51,052 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000001_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=318960
		FILE: Number of bytes written=822179
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=19652
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=7
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=220
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=139
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=371720192
	File Input Format Counters 
		Bytes Read=8260
2022-05-11 17:36:51,052 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000001_0
2022-05-11 17:36:51,052 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000002_0
2022-05-11 17:36:51,053 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,053 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,054 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,054 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-acls.xml:0+3518
2022-05-11 17:36:51,084 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,084 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,084 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,084 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,085 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,085 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,092 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,092 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,095 INFO mapred.Task: Task:attempt_local334235216_0001_m_000002_0 is done. And is in the process of committing
2022-05-11 17:36:51,099 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,099 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000002_0' done.
2022-05-11 17:36:51,099 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000002_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=320172
		FILE: Number of bytes written=822217
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=23170
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=9
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=135
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=129
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=10
		Total committed heap usage (bytes)=458227712
	File Input Format Counters 
		Bytes Read=3518
2022-05-11 17:36:51,099 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000002_0
2022-05-11 17:36:51,100 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000003_0
2022-05-11 17:36:51,100 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,100 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,101 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,101 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hdfs-site.xml:0+1093
2022-05-11 17:36:51,123 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,123 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,123 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,123 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,123 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,124 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,129 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,129 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,132 INFO mapred.Task: Task:attempt_local334235216_0001_m_000003_0 is done. And is in the process of committing
2022-05-11 17:36:51,135 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,136 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000003_0' done.
2022-05-11 17:36:51,136 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000003_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=321384
		FILE: Number of bytes written=822255
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=24263
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=11
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=32
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=130
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=9
		Total committed heap usage (bytes)=470810624
	File Input Format Counters 
		Bytes Read=1093
2022-05-11 17:36:51,136 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000003_0
2022-05-11 17:36:51,136 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000004_0
2022-05-11 17:36:51,137 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,137 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,137 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,154 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/core-site.xml:0+1041
2022-05-11 17:36:51,177 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,177 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,177 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,177 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,177 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,179 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,185 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,185 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,188 INFO mapred.Task: Task:attempt_local334235216_0001_m_000004_0 is done. And is in the process of committing
2022-05-11 17:36:51,190 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,190 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000004_0' done.
2022-05-11 17:36:51,191 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000004_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=322084
		FILE: Number of bytes written=822293
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=25304
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=13
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=28
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=130
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=5
		Total committed heap usage (bytes)=463994880
	File Input Format Counters 
		Bytes Read=1041
2022-05-11 17:36:51,191 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000004_0
2022-05-11 17:36:51,191 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000005_0
2022-05-11 17:36:51,192 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,192 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,192 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,193 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/mapred-site.xml:0+758
2022-05-11 17:36:51,211 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,211 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,211 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,211 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,211 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,212 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,221 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,221 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,223 INFO mapred.Task: Task:attempt_local334235216_0001_m_000005_0 is done. And is in the process of committing
2022-05-11 17:36:51,225 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,225 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000005_0' done.
2022-05-11 17:36:51,226 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000005_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=322784
		FILE: Number of bytes written=822331
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=26062
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=15
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=21
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=132
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=3
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=758
2022-05-11 17:36:51,226 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000005_0
2022-05-11 17:36:51,226 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000006_0
2022-05-11 17:36:51,227 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,227 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,227 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,228 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/yarn-site.xml:0+690
2022-05-11 17:36:51,245 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,245 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,245 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,245 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,245 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,246 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,252 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,252 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,254 INFO mapred.Task: Task:attempt_local334235216_0001_m_000006_0 is done. And is in the process of committing
2022-05-11 17:36:51,258 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,258 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000006_0' done.
2022-05-11 17:36:51,259 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000006_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=323484
		FILE: Number of bytes written=822369
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=26752
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=17
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=19
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=130
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=3
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=690
2022-05-11 17:36:51,259 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000006_0
2022-05-11 17:36:51,259 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000007_0
2022-05-11 17:36:51,260 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,260 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,260 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,261 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-site.xml:0+682
2022-05-11 17:36:51,277 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,277 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,277 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,277 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,277 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,278 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,283 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,283 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,285 INFO mapred.Task: Task:attempt_local334235216_0001_m_000007_0 is done. And is in the process of committing
2022-05-11 17:36:51,288 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,288 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000007_0' done.
2022-05-11 17:36:51,289 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000007_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=324184
		FILE: Number of bytes written=822407
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=27434
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=19
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=20
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=129
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=3
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=682
2022-05-11 17:36:51,289 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000007_0
2022-05-11 17:36:51,289 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_m_000008_0
2022-05-11 17:36:51,291 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,291 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,291 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,292 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/httpfs-site.xml:0+620
2022-05-11 17:36:51,308 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:51,308 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:51,308 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:51,308 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:51,308 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:51,308 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:51,313 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:51,313 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:51,316 INFO mapred.Task: Task:attempt_local334235216_0001_m_000008_0 is done. And is in the process of committing
2022-05-11 17:36:51,319 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:51,319 INFO mapred.Task: Task 'attempt_local334235216_0001_m_000008_0' done.
2022-05-11 17:36:51,320 INFO mapred.Task: Final Counters for attempt_local334235216_0001_m_000008_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=324372
		FILE: Number of bytes written=822445
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28054
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=21
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=17
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=132
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=3
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=620
2022-05-11 17:36:51,320 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_m_000008_0
2022-05-11 17:36:51,320 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 17:36:51,381 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 17:36:51,381 INFO mapred.LocalJobRunner: Starting task: attempt_local334235216_0001_r_000000_0
2022-05-11 17:36:51,388 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:51,388 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:51,388 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:51,391 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3f9f840d
2022-05-11 17:36:51,392 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 17:36:51,426 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=329200416, maxSingleShuffleLimit=82300104, mergeThreshold=217272288, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 17:36:51,429 INFO reduce.EventFetcher: attempt_local334235216_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 17:36:51,469 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,475 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000002_0
2022-05-11 17:36:51,481 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
2022-05-11 17:36:51,484 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,484 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000008_0
2022-05-11 17:36:51,485 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
2022-05-11 17:36:51,485 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,486 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000005_0
2022-05-11 17:36:51,486 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
2022-05-11 17:36:51,488 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,488 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000004_0
2022-05-11 17:36:51,488 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
2022-05-11 17:36:51,489 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,490 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000001_0
2022-05-11 17:36:51,490 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
2022-05-11 17:36:51,491 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,491 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000007_0
2022-05-11 17:36:51,491 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
2022-05-11 17:36:51,492 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2022-05-11 17:36:51,493 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local334235216_0001_m_000000_0
2022-05-11 17:36:51,493 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->33
2022-05-11 17:36:51,494 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,494 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000006_0
2022-05-11 17:36:51,494 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 33, usedMemory ->35
2022-05-11 17:36:51,495 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local334235216_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
2022-05-11 17:36:51,495 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local334235216_0001_m_000003_0
2022-05-11 17:36:51,495 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 35, usedMemory ->37
2022-05-11 17:36:51,496 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 17:36:51,496 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 17:36:51,497 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,507 INFO mapred.Merger: Merging 9 sorted segments
2022-05-11 17:36:51,546 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,546 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2022-05-11 17:36:51,546 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,503 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,547 INFO reduce.MergeManagerImpl: Merged 9 segments, 37 bytes to disk to satisfy reduce memory limit
2022-05-11 17:36:51,546 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 17:36:51,547 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2022-05-11 17:36:51,548 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 17:36:51,548 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 17:36:51,548 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10 bytes
2022-05-11 17:36:51,549 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 17:36:51,641 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2022-05-11 17:36:51,735 INFO mapreduce.Job:  map 100% reduce 0%
2022-05-11 17:36:51,859 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 17:36:51,922 INFO mapred.Task: Task:attempt_local334235216_0001_r_000000_0 is done. And is in the process of committing
2022-05-11 17:36:51,924 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 17:36:51,924 INFO mapred.Task: Task attempt_local334235216_0001_r_000000_0 is allowed to commit now
2022-05-11 17:36:51,939 INFO output.FileOutputCommitter: Saved output of task 'attempt_local334235216_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/grep-temp-482164475
2022-05-11 17:36:51,940 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 17:36:51,940 INFO mapred.Task: Task 'attempt_local334235216_0001_r_000000_0' done.
2022-05-11 17:36:51,941 INFO mapred.Task: Final Counters for attempt_local334235216_0001_r_000000_0: Counters: 29
	File System Counters
		FILE: Number of bytes read=324758
		FILE: Number of bytes written=822470
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28054
		HDFS: Number of bytes written=111
		HDFS: Number of read operations=26
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=3
	Map-Reduce Framework
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=73
		Reduce input records=1
		Reduce output records=1
		Spilled Records=1
		Shuffled Maps =9
		Failed Shuffles=0
		Merged Map outputs=9
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=470286336
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Output Format Counters 
		Bytes Written=111
2022-05-11 17:36:51,941 INFO mapred.LocalJobRunner: Finishing task: attempt_local334235216_0001_r_000000_0
2022-05-11 17:36:51,941 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 17:36:52,736 INFO mapreduce.Job:  map 100% reduce 100%
2022-05-11 17:36:52,736 INFO mapreduce.Job: Job job_local334235216_0001 completed successfully
2022-05-11 17:36:52,752 INFO mapreduce.Job: Counters: 35
	File System Counters
		FILE: Number of bytes read=3219930
		FILE: Number of bytes written=8223107
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=240137
		HDFS: Number of bytes written=111
		HDFS: Number of read operations=143
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=12
	Map-Reduce Framework
		Map input records=757
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=73
		Input split bytes=1185
		Combine input records=1
		Combine output records=1
		Reduce input groups=1
		Reduce shuffle bytes=73
		Reduce input records=1
		Reduce output records=1
		Spilled Records=2
		Shuffled Maps =9
		Failed Shuffles=0
		Merged Map outputs=9
		GC time elapsed (ms)=36
		Total committed heap usage (bytes)=4382523392
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=28054
	File Output Format Counters 
		Bytes Written=111
2022-05-11 17:36:52,769 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 17:36:52,922 INFO input.FileInputFormat: Total input files to process : 1
2022-05-11 17:36:52,924 INFO mapreduce.JobSubmitter: number of splits:1
2022-05-11 17:36:52,937 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1614699978_0002
2022-05-11 17:36:52,937 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 17:36:52,992 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 17:36:52,993 INFO mapreduce.Job: Running job: job_local1614699978_0002
2022-05-11 17:36:52,994 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 17:36:52,994 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:52,994 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:52,994 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 17:36:53,001 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 17:36:53,001 INFO mapred.LocalJobRunner: Starting task: attempt_local1614699978_0002_m_000000_0
2022-05-11 17:36:53,003 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:53,003 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:53,003 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:53,004 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/grep-temp-482164475/part-r-00000:0+111
2022-05-11 17:36:53,021 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 17:36:53,021 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 17:36:53,021 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 17:36:53,021 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 17:36:53,021 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 17:36:53,023 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 17:36:53,042 INFO mapred.LocalJobRunner: 
2022-05-11 17:36:53,042 INFO mapred.MapTask: Starting flush of map output
2022-05-11 17:36:53,042 INFO mapred.MapTask: Spilling map output
2022-05-11 17:36:53,042 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2022-05-11 17:36:53,042 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2022-05-11 17:36:53,043 INFO mapred.MapTask: Finished spill 0
2022-05-11 17:36:53,045 INFO mapred.Task: Task:attempt_local1614699978_0002_m_000000_0 is done. And is in the process of committing
2022-05-11 17:36:53,047 INFO mapred.LocalJobRunner: map
2022-05-11 17:36:53,047 INFO mapred.Task: Task 'attempt_local1614699978_0002_m_000000_0' done.
2022-05-11 17:36:53,048 INFO mapred.Task: Final Counters for attempt_local1614699978_0002_m_000000_0: Counters: 22
	File System Counters
		FILE: Number of bytes read=641323
		FILE: Number of bytes written=1644014
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28165
		HDFS: Number of bytes written=111
		HDFS: Number of read operations=32
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=6
	Map-Reduce Framework
		Map input records=1
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=131
		Combine input records=0
		Spilled Records=1
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=469237760
	File Input Format Counters 
		Bytes Read=111
2022-05-11 17:36:53,048 INFO mapred.LocalJobRunner: Finishing task: attempt_local1614699978_0002_m_000000_0
2022-05-11 17:36:53,048 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 17:36:53,049 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 17:36:53,049 INFO mapred.LocalJobRunner: Starting task: attempt_local1614699978_0002_r_000000_0
2022-05-11 17:36:53,051 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 17:36:53,051 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 17:36:53,051 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 17:36:53,051 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4f5e74fe
2022-05-11 17:36:53,051 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 17:36:53,052 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=328466432, maxSingleShuffleLimit=82116608, mergeThreshold=216787856, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 17:36:53,053 INFO reduce.EventFetcher: attempt_local1614699978_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 17:36:53,054 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1614699978_0002_m_000000_0 decomp: 21 len: 25 to MEMORY
2022-05-11 17:36:53,055 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1614699978_0002_m_000000_0
2022-05-11 17:36:53,055 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->21
2022-05-11 17:36:53,055 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 17:36:53,056 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 17:36:53,056 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 17:36:53,057 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 17:36:53,057 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2022-05-11 17:36:53,058 INFO reduce.MergeManagerImpl: Merged 1 segments, 21 bytes to disk to satisfy reduce memory limit
2022-05-11 17:36:53,058 INFO reduce.MergeManagerImpl: Merging 1 files, 25 bytes from disk
2022-05-11 17:36:53,058 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 17:36:53,058 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 17:36:53,059 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11 bytes
2022-05-11 17:36:53,059 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 17:36:53,067 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 17:36:53,076 INFO mapred.Task: Task:attempt_local1614699978_0002_r_000000_0 is done. And is in the process of committing
2022-05-11 17:36:53,078 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 17:36:53,078 INFO mapred.Task: Task attempt_local1614699978_0002_r_000000_0 is allowed to commit now
2022-05-11 17:36:53,084 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1614699978_0002_r_000000_0' to hdfs://localhost:9000/opt/module/hadoop-3.1.3/output1
2022-05-11 17:36:53,085 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 17:36:53,085 INFO mapred.Task: Task 'attempt_local1614699978_0002_r_000000_0' done.
2022-05-11 17:36:53,085 INFO mapred.Task: Final Counters for attempt_local1614699978_0002_r_000000_0: Counters: 29
	File System Counters
		FILE: Number of bytes read=641405
		FILE: Number of bytes written=1644039
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28165
		HDFS: Number of bytes written=122
		HDFS: Number of read operations=37
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=8
	Map-Reduce Framework
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=25
		Reduce input records=1
		Reduce output records=1
		Spilled Records=1
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=469237760
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Output Format Counters 
		Bytes Written=11
2022-05-11 17:36:53,085 INFO mapred.LocalJobRunner: Finishing task: attempt_local1614699978_0002_r_000000_0
2022-05-11 17:36:53,085 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 17:36:53,995 INFO mapreduce.Job: Job job_local1614699978_0002 running in uber mode : false
2022-05-11 17:36:53,995 INFO mapreduce.Job:  map 100% reduce 100%
2022-05-11 17:36:53,995 INFO mapreduce.Job: Job job_local1614699978_0002 completed successfully
2022-05-11 17:36:54,004 INFO mapreduce.Job: Counters: 35
	File System Counters
		FILE: Number of bytes read=1282728
		FILE: Number of bytes written=3288053
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=56330
		HDFS: Number of bytes written=233
		HDFS: Number of read operations=69
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=14
	Map-Reduce Framework
		Map input records=1
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=131
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=25
		Reduce input records=1
		Reduce output records=1
		Spilled Records=2
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=938475520
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=111
	File Output Format Counters 
		Bytes Written=11

[hadoop@hadoop2 hadoop-3.1.3]$ ll -al
总用量 180
drwxr-xr-x. 15 hadoop hadoop    232 511 17:05 .
drwxr-xr-x.  4 hadoop hadoop     46 52 08:21 ..
drwxr-xr-x.  2 hadoop hadoop    183 912 2019 bin
drwxr-xr-x.  3 hadoop hadoop     20 912 2019 etc
drwxr-xr-x.  2 hadoop hadoop    106 912 2019 include
drwxr-xr-x.  3 hadoop hadoop     20 912 2019 lib
drwxr-xr-x.  4 hadoop hadoop    288 912 2019 libexec
-rw-rw-r--.  1 hadoop hadoop 147145 94 2019 LICENSE.txt
drwxrwxr-x.  2 hadoop hadoop   4096 511 17:05 logs
-rw-rw-r--.  1 hadoop hadoop  21867 94 2019 NOTICE.txt
-rw-rw-r--.  1 hadoop hadoop   1366 94 2019 README.txt
drwxr-xr-x.  3 hadoop hadoop   4096 912 2019 sbin
drwxr-xr-x.  4 hadoop hadoop     31 912 2019 share
drwxrwxr-x.  3 hadoop hadoop     17 511 09:04 tmp
drwxrwxr-x.  4 hadoop hadoop     31 511 17:36 tmpp
drwxrwxr-x.  2 hadoop hadoop     22 52 22:53 wcinput
drwxr-xr-x.  2 hadoop hadoop     88 52 22:46 wcoutput
drwxr-xr-x.  2 hadoop hadoop     88 52 22:54 wcoutput2
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hdfs dfs -cat /opt/module/hadoop-3.1.3/output1/*
2022-05-11 18:01:37,074 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1	dfsadmin

成功案例:

[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hadoop jar ./share/hadoop/mapreduce//hadoop-mapreduce-examples-3.1.3.jar  grep /opt/module/hadoop-3.1.3/input ./output1 'dfs[a-z.]+' 2022-05-11 18:04:56,846 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2022-05-11 18:04:56,940 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-05-11 18:04:56,940 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2022-05-11 18:04:57,369 INFO input.FileInputFormat: Total input files to process : 9
2022-05-11 18:04:57,393 INFO mapreduce.JobSubmitter: number of splits:9
2022-05-11 18:04:57,535 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1922828463_0001
2022-05-11 18:04:57,536 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 18:04:57,637 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 18:04:57,637 INFO mapreduce.Job: Running job: job_local1922828463_0001
2022-05-11 18:04:57,638 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 18:04:57,645 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:57,645 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:57,646 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 18:04:57,691 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 18:04:57,692 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000000_0
2022-05-11 18:04:57,716 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:57,716 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:57,734 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:57,739 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hadoop-policy.xml:0+11392
2022-05-11 18:04:57,815 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:57,815 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:57,815 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:57,815 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:57,815 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:57,824 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:57,850 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 18:04:57,939 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:57,942 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:57,942 INFO mapred.MapTask: Spilling map output
2022-05-11 18:04:57,942 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2022-05-11 18:04:57,942 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2022-05-11 18:04:57,956 INFO mapred.MapTask: Finished spill 0
2022-05-11 18:04:57,967 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000000_0 is done. And is in the process of committing
2022-05-11 18:04:57,971 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:57,971 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000000_0' done.
2022-05-11 18:04:57,981 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000000_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=317748
		FILE: Number of bytes written=824593
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=11392
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=5
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=265
		Map output records=1
		Map output bytes=17
		Map output materialized bytes=25
		Input split bytes=134
		Combine input records=1
		Combine output records=1
		Spilled Records=1
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=267386880
	File Input Format Counters 
		Bytes Read=11392
2022-05-11 18:04:57,982 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000000_0
2022-05-11 18:04:57,982 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000001_0
2022-05-11 18:04:57,989 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:57,989 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:57,990 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:57,991 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/capacity-scheduler.xml:0+8260
2022-05-11 18:04:58,319 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,319 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,319 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,319 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,319 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,321 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,329 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,329 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,332 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000001_0 is done. And is in the process of committing
2022-05-11 18:04:58,336 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,336 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000001_0' done.
2022-05-11 18:04:58,337 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000001_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=318960
		FILE: Number of bytes written=824631
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=19652
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=7
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=220
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=139
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=372768768
	File Input Format Counters 
		Bytes Read=8260
2022-05-11 18:04:58,337 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000001_0
2022-05-11 18:04:58,337 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000002_0
2022-05-11 18:04:58,338 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,338 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,339 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,340 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-acls.xml:0+3518
2022-05-11 18:04:58,524 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,524 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,524 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,524 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,524 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,525 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,531 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,531 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,534 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000002_0 is done. And is in the process of committing
2022-05-11 18:04:58,538 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,538 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000002_0' done.
2022-05-11 18:04:58,538 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000002_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=320172
		FILE: Number of bytes written=824669
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=23170
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=9
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=135
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=129
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=9
		Total committed heap usage (bytes)=458227712
	File Input Format Counters 
		Bytes Read=3518
2022-05-11 18:04:58,539 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000002_0
2022-05-11 18:04:58,539 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000003_0
2022-05-11 18:04:58,539 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,539 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,540 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,541 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/hdfs-site.xml:0+1093
2022-05-11 18:04:58,563 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,563 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,563 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,563 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,563 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,564 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,572 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,572 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,572 INFO mapred.MapTask: Spilling map output
2022-05-11 18:04:58,572 INFO mapred.MapTask: bufstart = 0; bufend = 84; bufvoid = 104857600
2022-05-11 18:04:58,572 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2022-05-11 18:04:58,574 INFO mapred.MapTask: Finished spill 0
2022-05-11 18:04:58,575 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000003_0 is done. And is in the process of committing
2022-05-11 18:04:58,578 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,578 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000003_0' done.
2022-05-11 18:04:58,579 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000003_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=321384
		FILE: Number of bytes written=824797
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=24263
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=11
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=32
		Map output records=3
		Map output bytes=84
		Map output materialized bytes=96
		Input split bytes=130
		Combine input records=3
		Combine output records=3
		Spilled Records=3
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=6
		Total committed heap usage (bytes)=470810624
	File Input Format Counters 
		Bytes Read=1093
2022-05-11 18:04:58,579 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000003_0
2022-05-11 18:04:58,579 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000004_0
2022-05-11 18:04:58,580 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,580 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,580 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,582 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/core-site.xml:0+1041
2022-05-11 18:04:58,606 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,606 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,606 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,606 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,606 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,607 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,613 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,613 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,615 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000004_0 is done. And is in the process of committing
2022-05-11 18:04:58,618 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,618 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000004_0' done.
2022-05-11 18:04:58,619 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000004_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=322084
		FILE: Number of bytes written=824835
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=25304
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=13
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=28
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=130
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=7
		Total committed heap usage (bytes)=463994880
	File Input Format Counters 
		Bytes Read=1041
2022-05-11 18:04:58,619 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000004_0
2022-05-11 18:04:58,619 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000005_0
2022-05-11 18:04:58,620 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,620 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,621 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,622 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/mapred-site.xml:0+758
2022-05-11 18:04:58,642 INFO mapreduce.Job: Job job_local1922828463_0001 running in uber mode : false
2022-05-11 18:04:58,648 INFO mapreduce.Job:  map 100% reduce 0%
2022-05-11 18:04:58,649 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,649 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,649 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,649 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,649 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,651 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,658 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,658 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,660 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000005_0 is done. And is in the process of committing
2022-05-11 18:04:58,663 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,663 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000005_0' done.
2022-05-11 18:04:58,664 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000005_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=322784
		FILE: Number of bytes written=824873
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=26062
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=15
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=21
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=132
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=4
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=758
2022-05-11 18:04:58,664 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000005_0
2022-05-11 18:04:58,664 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000006_0
2022-05-11 18:04:58,664 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,665 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,665 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,666 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/yarn-site.xml:0+690
2022-05-11 18:04:58,686 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,686 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,686 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,686 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,686 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,687 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,692 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,692 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,694 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000006_0 is done. And is in the process of committing
2022-05-11 18:04:58,698 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,698 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000006_0' done.
2022-05-11 18:04:58,699 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000006_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=323484
		FILE: Number of bytes written=824911
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=26752
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=17
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=19
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=130
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=5
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=690
2022-05-11 18:04:58,699 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000006_0
2022-05-11 18:04:58,699 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000007_0
2022-05-11 18:04:58,700 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,700 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,700 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,701 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/kms-site.xml:0+682
2022-05-11 18:04:58,719 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,720 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,720 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,720 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,720 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,721 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,725 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,725 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,728 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000007_0 is done. And is in the process of committing
2022-05-11 18:04:58,730 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,730 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000007_0' done.
2022-05-11 18:04:58,731 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000007_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=324184
		FILE: Number of bytes written=824949
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=27434
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=19
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=20
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=129
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=3
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=682
2022-05-11 18:04:58,731 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000007_0
2022-05-11 18:04:58,731 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_m_000008_0
2022-05-11 18:04:58,732 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,732 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,732 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,733 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/opt/module/hadoop-3.1.3/input/httpfs-site.xml:0+620
2022-05-11 18:04:58,752 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:58,753 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:58,753 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:58,753 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:58,753 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:58,753 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:58,758 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:58,758 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:58,760 INFO mapred.Task: Task:attempt_local1922828463_0001_m_000008_0 is done. And is in the process of committing
2022-05-11 18:04:58,762 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:58,762 INFO mapred.Task: Task 'attempt_local1922828463_0001_m_000008_0' done.
2022-05-11 18:04:58,763 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_m_000008_0: Counters: 23
	File System Counters
		FILE: Number of bytes read=324372
		FILE: Number of bytes written=824987
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28054
		HDFS: Number of bytes written=0
		HDFS: Number of read operations=21
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=1
	Map-Reduce Framework
		Map input records=17
		Map output records=0
		Map output bytes=0
		Map output materialized bytes=6
		Input split bytes=132
		Combine input records=0
		Combine output records=0
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=3
		Total committed heap usage (bytes)=470286336
	File Input Format Counters 
		Bytes Read=620
2022-05-11 18:04:58,763 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_m_000008_0
2022-05-11 18:04:58,763 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 18:04:58,766 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 18:04:58,766 INFO mapred.LocalJobRunner: Starting task: attempt_local1922828463_0001_r_000000_0
2022-05-11 18:04:58,773 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:58,773 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:58,773 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:58,776 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@323c3357
2022-05-11 18:04:58,777 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 18:04:58,795 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=329200416, maxSingleShuffleLimit=82300104, mergeThreshold=217272288, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 18:04:58,799 INFO reduce.EventFetcher: attempt_local1922828463_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 18:04:58,831 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,835 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000007_0
2022-05-11 18:04:58,836 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
2022-05-11 18:04:58,838 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,839 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000001_0
2022-05-11 18:04:58,839 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
2022-05-11 18:04:58,840 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000004_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,841 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000004_0
2022-05-11 18:04:58,841 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
2022-05-11 18:04:58,842 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000003_0 decomp: 92 len: 96 to MEMORY
2022-05-11 18:04:58,843 INFO reduce.InMemoryMapOutput: Read 92 bytes from map-output for attempt_local1922828463_0001_m_000003_0
2022-05-11 18:04:58,843 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 92, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->98
2022-05-11 18:04:58,843 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2022-05-11 18:04:58,844 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,852 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000006_0
2022-05-11 18:04:58,852 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 98, usedMemory ->100
2022-05-11 18:04:58,853 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2022-05-11 18:04:58,853 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local1922828463_0001_m_000000_0
2022-05-11 18:04:58,853 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 6, commitMemory -> 100, usedMemory ->121
2022-05-11 18:04:58,855 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,855 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000002_0
2022-05-11 18:04:58,855 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 121, usedMemory ->123
2022-05-11 18:04:58,856 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,856 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000005_0
2022-05-11 18:04:58,857 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 123, usedMemory ->125
2022-05-11 18:04:58,858 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1922828463_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2022-05-11 18:04:58,858 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1922828463_0001_m_000008_0
2022-05-11 18:04:58,858 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 125, usedMemory ->127
2022-05-11 18:04:58,858 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 18:04:58,859 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 18:04:58,859 INFO reduce.MergeManagerImpl: finalMerge called with 9 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 18:04:58,865 INFO mapred.Merger: Merging 9 sorted segments
2022-05-11 18:04:58,865 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 78 bytes
2022-05-11 18:04:58,866 INFO reduce.MergeManagerImpl: Merged 9 segments, 127 bytes to disk to satisfy reduce memory limit
2022-05-11 18:04:58,866 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
2022-05-11 18:04:58,867 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 18:04:58,867 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 18:04:58,867 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 87 bytes
2022-05-11 18:04:58,868 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 18:04:58,914 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2022-05-11 18:04:58,945 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 18:04:59,378 INFO mapred.Task: Task:attempt_local1922828463_0001_r_000000_0 is done. And is in the process of committing
2022-05-11 18:04:59,380 INFO mapred.LocalJobRunner: 9 / 9 copied.
2022-05-11 18:04:59,380 INFO mapred.Task: Task attempt_local1922828463_0001_r_000000_0 is allowed to commit now
2022-05-11 18:04:59,394 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1922828463_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/grep-temp-1352764713
2022-05-11 18:04:59,394 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 18:04:59,394 INFO mapred.Task: Task 'attempt_local1922828463_0001_r_000000_0' done.
2022-05-11 18:04:59,395 INFO mapred.Task: Final Counters for attempt_local1922828463_0001_r_000000_0: Counters: 29
	File System Counters
		FILE: Number of bytes read=324938
		FILE: Number of bytes written=825102
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28054
		HDFS: Number of bytes written=219
		HDFS: Number of read operations=26
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=3
	Map-Reduce Framework
		Combine input records=0
		Combine output records=0
		Reduce input groups=4
		Reduce shuffle bytes=163
		Reduce input records=4
		Reduce output records=4
		Spilled Records=4
		Shuffled Maps =9
		Failed Shuffles=0
		Merged Map outputs=9
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=470286336
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Output Format Counters 
		Bytes Written=219
2022-05-11 18:04:59,395 INFO mapred.LocalJobRunner: Finishing task: attempt_local1922828463_0001_r_000000_0
2022-05-11 18:04:59,395 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 18:04:59,653 INFO mapreduce.Job:  map 100% reduce 100%
2022-05-11 18:04:59,654 INFO mapreduce.Job: Job job_local1922828463_0001 completed successfully
2022-05-11 18:04:59,664 INFO mapreduce.Job: Counters: 35
	File System Counters
		FILE: Number of bytes read=3220110
		FILE: Number of bytes written=8248347
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=240137
		HDFS: Number of bytes written=219
		HDFS: Number of read operations=143
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=12
	Map-Reduce Framework
		Map input records=757
		Map output records=4
		Map output bytes=101
		Map output materialized bytes=163
		Input split bytes=1185
		Combine input records=4
		Combine output records=4
		Reduce input groups=4
		Reduce shuffle bytes=163
		Reduce input records=4
		Reduce output records=4
		Spilled Records=8
		Shuffled Maps =9
		Failed Shuffles=0
		Merged Map outputs=9
		GC time elapsed (ms)=37
		Total committed heap usage (bytes)=4384620544
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=28054
	File Output Format Counters 
		Bytes Written=219
2022-05-11 18:04:59,679 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 18:04:59,697 INFO input.FileInputFormat: Total input files to process : 1
2022-05-11 18:04:59,699 INFO mapreduce.JobSubmitter: number of splits:1
2022-05-11 18:04:59,723 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local324626594_0002
2022-05-11 18:04:59,723 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-11 18:04:59,776 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-05-11 18:04:59,777 INFO mapreduce.Job: Running job: job_local324626594_0002
2022-05-11 18:04:59,778 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-05-11 18:04:59,778 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:59,778 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:59,779 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-05-11 18:04:59,785 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-05-11 18:04:59,785 INFO mapred.LocalJobRunner: Starting task: attempt_local324626594_0002_m_000000_0
2022-05-11 18:04:59,787 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:59,787 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:59,787 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:59,789 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/grep-temp-1352764713/part-r-00000:0+219
2022-05-11 18:04:59,805 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2022-05-11 18:04:59,805 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2022-05-11 18:04:59,805 INFO mapred.MapTask: soft limit at 83886080
2022-05-11 18:04:59,805 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2022-05-11 18:04:59,805 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2022-05-11 18:04:59,806 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2022-05-11 18:04:59,821 INFO mapred.LocalJobRunner: 
2022-05-11 18:04:59,821 INFO mapred.MapTask: Starting flush of map output
2022-05-11 18:04:59,821 INFO mapred.MapTask: Spilling map output
2022-05-11 18:04:59,821 INFO mapred.MapTask: bufstart = 0; bufend = 101; bufvoid = 104857600
2022-05-11 18:04:59,821 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
2022-05-11 18:04:59,823 INFO mapred.MapTask: Finished spill 0
2022-05-11 18:04:59,824 INFO mapred.Task: Task:attempt_local324626594_0002_m_000000_0 is done. And is in the process of committing
2022-05-11 18:04:59,828 INFO mapred.LocalJobRunner: map
2022-05-11 18:04:59,828 INFO mapred.Task: Task 'attempt_local324626594_0002_m_000000_0' done.
2022-05-11 18:04:59,828 INFO mapred.Task: Final Counters for attempt_local324626594_0002_m_000000_0: Counters: 22
	File System Counters
		FILE: Number of bytes read=641505
		FILE: Number of bytes written=1644282
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28273
		HDFS: Number of bytes written=219
		HDFS: Number of read operations=32
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=6
	Map-Reduce Framework
		Map input records=4
		Map output records=4
		Map output bytes=101
		Map output materialized bytes=115
		Input split bytes=132
		Combine input records=0
		Spilled Records=4
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=469237760
	File Input Format Counters 
		Bytes Read=219
2022-05-11 18:04:59,828 INFO mapred.LocalJobRunner: Finishing task: attempt_local324626594_0002_m_000000_0
2022-05-11 18:04:59,828 INFO mapred.LocalJobRunner: map task executor complete.
2022-05-11 18:04:59,829 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2022-05-11 18:04:59,830 INFO mapred.LocalJobRunner: Starting task: attempt_local324626594_0002_r_000000_0
2022-05-11 18:04:59,832 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-05-11 18:04:59,832 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-05-11 18:04:59,832 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-05-11 18:04:59,832 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@23ea483f
2022-05-11 18:04:59,833 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2022-05-11 18:04:59,833 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=328466432, maxSingleShuffleLimit=82116608, mergeThreshold=216787856, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2022-05-11 18:04:59,834 INFO reduce.EventFetcher: attempt_local324626594_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2022-05-11 18:04:59,836 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local324626594_0002_m_000000_0 decomp: 111 len: 115 to MEMORY
2022-05-11 18:04:59,837 INFO reduce.InMemoryMapOutput: Read 111 bytes from map-output for attempt_local324626594_0002_m_000000_0
2022-05-11 18:04:59,837 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 111, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->111
2022-05-11 18:04:59,838 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2022-05-11 18:04:59,841 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 18:04:59,841 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2022-05-11 18:04:59,842 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 18:04:59,842 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 101 bytes
2022-05-11 18:04:59,842 INFO reduce.MergeManagerImpl: Merged 1 segments, 111 bytes to disk to satisfy reduce memory limit
2022-05-11 18:04:59,843 INFO reduce.MergeManagerImpl: Merging 1 files, 115 bytes from disk
2022-05-11 18:04:59,843 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2022-05-11 18:04:59,843 INFO mapred.Merger: Merging 1 sorted segments
2022-05-11 18:04:59,843 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 101 bytes
2022-05-11 18:04:59,843 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 18:04:59,858 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-05-11 18:04:59,871 INFO mapred.Task: Task:attempt_local324626594_0002_r_000000_0 is done. And is in the process of committing
2022-05-11 18:04:59,873 INFO mapred.LocalJobRunner: 1 / 1 copied.
2022-05-11 18:04:59,873 INFO mapred.Task: Task attempt_local324626594_0002_r_000000_0 is allowed to commit now
2022-05-11 18:04:59,885 INFO output.FileOutputCommitter: Saved output of task 'attempt_local324626594_0002_r_000000_0' to hdfs://localhost:9000/user/hadoop/output1
2022-05-11 18:04:59,888 INFO mapred.LocalJobRunner: reduce > reduce
2022-05-11 18:04:59,888 INFO mapred.Task: Task 'attempt_local324626594_0002_r_000000_0' done.
2022-05-11 18:04:59,888 INFO mapred.Task: Final Counters for attempt_local324626594_0002_r_000000_0: Counters: 29
	File System Counters
		FILE: Number of bytes read=641767
		FILE: Number of bytes written=1644397
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=28273
		HDFS: Number of bytes written=296
		HDFS: Number of read operations=37
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=8
	Map-Reduce Framework
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=115
		Reduce input records=4
		Reduce output records=4
		Spilled Records=4
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=469237760
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Output Format Counters 
		Bytes Written=77
2022-05-11 18:04:59,888 INFO mapred.LocalJobRunner: Finishing task: attempt_local324626594_0002_r_000000_0
2022-05-11 18:04:59,889 INFO mapred.LocalJobRunner: reduce task executor complete.
2022-05-11 18:05:00,778 INFO mapreduce.Job: Job job_local324626594_0002 running in uber mode : false
2022-05-11 18:05:00,778 INFO mapreduce.Job:  map 100% reduce 100%
2022-05-11 18:05:00,779 INFO mapreduce.Job: Job job_local324626594_0002 completed successfully
2022-05-11 18:05:00,789 INFO mapreduce.Job: Counters: 35
	File System Counters
		FILE: Number of bytes read=1283272
		FILE: Number of bytes written=3288679
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=56546
		HDFS: Number of bytes written=515
		HDFS: Number of read operations=69
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=14
	Map-Reduce Framework
		Map input records=4
		Map output records=4
		Map output bytes=101
		Map output materialized bytes=115
		Input split bytes=132
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=115
		Reduce input records=4
		Reduce output records=4
		Spilled Records=8
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=0
		Total committed heap usage (bytes)=938475520
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=219
	File Output Format Counters 
		Bytes Written=77

查看目录下文件(没有显示出output1):

[hadoop@hadoop2 hadoop-3.1.3]$ ll
总用量 180
drwxr-xr-x. 2 hadoop hadoop    183 912 2019 bin
drwxr-xr-x. 3 hadoop hadoop     20 912 2019 etc
drwxr-xr-x. 2 hadoop hadoop    106 912 2019 include
drwxr-xr-x. 3 hadoop hadoop     20 912 2019 lib
drwxr-xr-x. 4 hadoop hadoop    288 912 2019 libexec
-rw-rw-r--. 1 hadoop hadoop 147145 94 2019 LICENSE.txt
drwxrwxr-x. 2 hadoop hadoop   4096 511 17:05 logs
-rw-rw-r--. 1 hadoop hadoop  21867 94 2019 NOTICE.txt
-rw-rw-r--. 1 hadoop hadoop   1366 94 2019 README.txt
drwxr-xr-x. 3 hadoop hadoop   4096 912 2019 sbin
drwxr-xr-x. 4 hadoop hadoop     31 912 2019 share
drwxrwxr-x. 3 hadoop hadoop     17 511 09:04 tmp
drwxrwxr-x. 4 hadoop hadoop     31 511 17:36 tmpp
drwxrwxr-x. 2 hadoop hadoop     22 52 22:53 wcinput
drwxr-xr-x. 2 hadoop hadoop     88 52 22:46 wcoutput
drwxr-xr-x. 2 hadoop hadoop     88 52 22:54 wcoutput2

查看运行结果的命令(查看的是位于 HDFS 中的输出结果):

[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hdfs dfs -cat output1/*
2022-05-11 18:06:36,892 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1	dfsadmin
1	dfs.replication
1	dfs.namenode.name.dir
1	dfs.datanode.data.dir

数据是存储在hdfs中的,我们也可以把数据取回到本地:

//如果本地有同名文件,要先删去,这里浅试一下,做个复习
[hadoop@hadoop2 hadoop-3.1.3]$ rm -r ./output1
rm: 无法删除"./output1": 没有那个文件或目录//说明没有
[hadoop@hadoop2 hadoop-3.1.3]$ ./bin/hdfs dfs -get output1 ./output1
2022-05-11 18:16:14,963 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
[hadoop@hadoop2 hadoop-3.1.3]$ cat ./output1/*
1	dfsadmin
1	dfs.replication
1	dfs.namenode.name.dir
1	dfs.datanode.data.dir

Hadoop 运行程序时,输出目录不能存在,否则会提示错误 “org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:9000/user/hadoop/output already exists” ,因此若要再次执行,需要执行如下命令删除 output 文件夹:

./bin/hdfs dfs -rm -r output    # 删除 output 文件夹

关闭hadoop:

[hadoop@hadoop2 hadoop-3.1.3]$ ./sbin/stop-dfs.sh
Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [hadoop2]

开启:

[hadoop@hadoop2 hadoop-3.1.3]$ jps
7360 DataNode
7713 Jps
7570 SecondaryNameNode
7192 NameNode

有关此方面的问题,欢迎来问,因为刚刚结束所以有很大的热情呢~

你可能感兴趣的:(笔记,实践经验,hadoop,mapreduce,大数据,hdfs,hbase)