pig on tez测试
测试环境
- pig-0.14.0
- hadoop-2.5.2 ()1+2)
hive on tez 测试后,很好奇,pig是否可以在tez上运行呢?从官网上可以看到pig on tez的描述,就想应该是可以pig on tez的.
pig安装过程略过…
准备数据集
[hadoop@mymaster ~]$ wget http://hortonassets.s3.amazonaws.com/pig/lahman591-csv.zip
[hadoop@mymaster ~]$ unzip lahman591-csv.zip
[hadoop@mymaster ~]$ hadoop fs mkdir /test
[hadoop@mymaster ~]$ hadoop fs -put lahman591-csv/Batting.csv /test // 将Batting.csv上传到hdfs编写pig测试脚本
[hadoop@mymaster ~]$ mkdir pig
[hadoop@mymaster ~]$ vim pig/test.pig
batting = LOAD '/test/Batting.csv' USING PigStorage(',');
raw_runs = FILTER batting BY $1>0;
runs = FOREACH raw_runs GENERATE $0 AS playerID, $1 AS year, $8 AS runs;
grp_data = GROUP runs BY (year);
max_runs = FOREACH grp_data GENERATE group as grp, MAX(runs.runs) AS max_runs;
join_max_runs = JOIN max_runs BY ($0, max_runs), runs BY (year, runs);
join_data = FOREACH join_max_runs GENERATE $0 AS year, $2 AS playerID, $1 AS runs;
DUMP join_data;mr方式运行 test.pig
[hadoop@mymaster ~]$ /usr/local/pig-0.14.0/bin/pig -x mr pig/test.pig // 片段 Input(s): Successfully read 95195 records (6399268 bytes) from: "/test/Batting.csv" Output(s): Successfully stored 151 records (4507 bytes) in: "hdfs://10.128.17.21:9000/tmp/temp1552838877/tmp1249909816" Counters: Total records written : 151 Total bytes written : 4507 Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0 ------- (1988,boggswa01,128.0) (1989,boggswa01,113.0) (1990,henderi01,119.0) (1991,molitpa01,133.0) (1992,phillto02,114.0) (1993,dykstle01,143.0) (1994,thomafr04,106.0) (1995,biggicr01,123.0) (1996,burksel01,142.0) (1997,biggicr01,146.0) (1998,sosasa01,134.0) (1999,bagweje01,143.0) (2000,bagweje01,152.0) (2001,sosasa01,146.0) (2002,soriaal01,128.0) (2003,pujolal01,137.0) (2004,pujolal01,133.0) (2005,pujolal01,129.0) (2006,sizemgr01,134.0) (2007,rodrial01,143.0) (2008,ramirha01,125.0) (2009,pujolal01,124.0) (2010,pujolal01,115.0) (2011,grandcu01,136.0) 2015-06-02 13:49:39,574 [main] INFO org.apache.pig.Main - Pig script completed in 1 minute, 10 seconds and 20 milliseconds (70020 ms)运行情况:
输入:95195 records
输出:151 records
耗时:70s
tez 方式运行test.pig
[hadoop@mymaster pig]$ /usr/local/pig-0.14.0/bin/pig -x tez pig/test.pig HadoopVersion: 2.5.2 PigVersion: 0.14.0 TezVersion: 0.5.2 UserId: hadoop FileName: 1.pig StartedAt: 2015-06-02 13:50:03 FinishedAt: 2015-06-02 13:50:34 Features: HASH_JOIN,GROUP_BY,FILTER Success! DAG PigLatin:1.pig-0_scope-0: ApplicationId: job_1432693876849_0008 TotalLaunchedTasks: 3 FileBytesRead: 3494886 FileBytesWritten: 5509316 HdfsBytesRead: 6398886 HdfsBytesWritten: 4507 Input(s): Successfully read 95195 records (6398886 bytes) from: "/test/lahman591-csv/Batting.csv" Output(s): Successfully stored 151 records (4507 bytes) in: "hdfs://10.128.17.21:9000/tmp/temp-1130777030/tmp93164502" (1994,thomafr04,106.0) (1995,biggicr01,123.0) (1996,burksel01,142.0) (1997,biggicr01,146.0) (1998,sosasa01,134.0) (1999,bagweje01,143.0) (2000,bagweje01,152.0) (2001,sosasa01,146.0) (2002,soriaal01,128.0) (2003,pujolal01,137.0) (2004,pujolal01,133.0) (2005,pujolal01,129.0) (2006,sizemgr01,134.0) (2007,rodrial01,143.0) (2008,ramirha01,125.0) (2009,pujolal01,124.0) (2010,pujolal01,115.0) (2011,grandcu01,136.0) 2015-06-02 13:50:34,623 [main] INFO org.apache.pig.Main - Pig script completed in 34 seconds and 350 milliseconds (34350 ms) 2015-06-02 13:50:34,634 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down thread pool 运行情况:
输入:95195 records
输出:151 records
耗时:34s
测试结果 tez比yarn快2倍多
根据本轮测试效果差异不大,要根据mr的串联数和数据大小 进行严格的测试 才能达到理想的官网测试性能指标,但是可以肯定的是,mr任务串联越多,tez的性能越显著.
hive on tez 的配置稍嫌麻烦,相对来说pig on tez的测试环境相当容易
参考:http://zh.hortonworks.com/hadoop-tutorial/faster-pig-tez/