2.mr对wordcount性能测试50g数据

1.对4.1中生成50g单词进行统计

基于mr磁盘测试

开始:19/12/27 16:23:45

结束:19/12/27 16:26:22

 

 

 

19/12/27 16:26:15 INFO mapreduce.Job: map 100% reduce 75%

19/12/27 16:26:16 INFO mapreduce.Job: map 100% reduce 82%

19/12/27 16:26:17 INFO mapreduce.Job: map 100% reduce 87%

19/12/27 16:26:18 INFO mapreduce.Job: map 100% reduce 92%

19/12/27 16:26:19 INFO mapreduce.Job: map 100% reduce 95%

19/12/27 16:26:20 INFO mapreduce.Job: map 100% reduce 99%

19/12/27 16:26:21 INFO mapreduce.Job: map 100% reduce 100%

19/12/27 16:26:22 INFO mapreduce.Job: Job job_1577432091531_0001 completed successfully

19/12/27 16:26:22 INFO mapreduce.Job: Counters: 55

File System Counters

FILE: Number of bytes read=5603673641

FILE: Number of bytes written=3363374125

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=54698304302

HDFS: Number of bytes written=535602831

HDFS: Number of read operations=1850

HDFS: Number of large read operations=0

HDFS: Number of write operations=200

HDFS: Number of bytes read erasure-coded=0

Job Counters

Launched map tasks=450

Launched reduce tasks=100

Data-local map tasks=438

Rack-local map tasks=12

Total time spent by all maps in occupied slots (ms)=19917612

Total time spent by all reduces in occupied slots (ms)=3050464

Total time spent by all map tasks (ms)=19917612

Total time spent by all reduce tasks (ms)=3050464

Total vcore-milliseconds taken by all map tasks=19917612

Total vcore-milliseconds taken by all reduce tasks=3050464

Total megabyte-milliseconds taken by all map tasks=20395634688

Total megabyte-milliseconds taken by all reduce tasks=3123675136

Map-Reduce Framework

Map input records=1373955

Map output records=5042161744

Map output bytes=75172225085

Map output materialized bytes=1352230193

Input split bytes=63450

Combine input records=5042161744

Combine output records=117362307

Reduce input groups=23909852

Reduce shuffle bytes=1352230193

Reduce input records=117362307

Reduce output records=23909852

Spilled Records=349748802

Shuffled Maps =45000

Failed Shuffles=0

Merged Map outputs=45000

GC time elapsed (ms)=349516

CPU time spent (ms)=16941720

Physical memory (bytes) snapshot=358828400640

Virtual memory (bytes) snapshot=1499865440256

Total committed heap usage (bytes)=413318774784

Peak Map Physical memory (bytes)=720048128

Peak Map Virtual memory (bytes)=2736115712

Peak Reduce Physical memory (bytes)=562401280

Peak Reduce Virtual memory (bytes)=2844385280

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=54698240852

File Output Format Counters

Bytes Written=535602831

 

 

 

 

 

 

 

9/12/27 16:26:22 INFO mapreduce。工作：计数器：55

文件系统计数器

FILE：读取的字节数= 5603673641

FILE：写入的字节数= 3363374125

FILE：读取操作数= 0

FILE：大读取操作数= 0

FILE：写操作数= 0

HDFS：读取的字节数= 54698304302

HDFS：写入的字节数= 535602831

HDFS：读取操作数= 1850

HDFS：大型读取操作数= 0

HDFS：写入操作数= 200

HDFS：读取的字节数擦除编码= 0

工作柜台

启动的map任务= 450

启动减少任务= 100

数据局部map任务= 438

机架本地map任务= 12

所有map在占用的插槽中花费的总时间（ms）= 19917612

所有人花费的总时间减少了占用的时隙（ms）= 3050464

所有map任务花费的总时间（ms）= 19917612

所有缩减任务花费的总时间（毫秒）= 3050464

所有map任务花费的总vcore-毫秒数= 19917612

所有reduce任务花费的总vcore-毫秒数= 3050464

所有map任务花费的总兆字节毫秒数= 20395634688

所有缩减任务花费的总兆字节-毫秒= 3123675136

Map-Reduce框架

map输入记录= 1373955

map输出记录= 5042161744

映射输出字节= 75172225085

映射输出实例化字节= 1352230193

输入拆分字节= 63450

合并输入记录= 5042161744

合并输出记录= 117362307

减少输入组= 23909852

减少随机播放字节= 1352230193

减少输入记录= 117362307

减少输出记录= 23909852

溢出记录= 349748802

随机map= 45000

随机洗牌失败= 0

合并map输出= 45000

已用GC时间（ms）= 349516

花的CPU时间（毫秒）= 16941720

物理内存（字节）快照= 358828400640

虚拟内存（字节）快照= 1499865440256

已提交的总堆使用量（字节）= 413318774784

峰图物理内存（字节）= 720048128

峰图虚拟内存（字节）= 2736115712

峰值减少物理内存（字节）= 562401280

峰值减少虚拟内存（字节）= 2844385280

随机错误

BAD_ID = 0

连接= 0

IO_ERROR = 0

WRONG_LENGTH = 0

WRONG_MAP = 0

WRONG_REDUCE = 0

文件输入格式计数器

读取的字节数= 54698240852

文件输出格式计数器

写入的字节数= 535602831