hive Error: GC overhead limit exceeded
+ hive -e 'use trainschain;insert overwrite table dwd_acct_settlecore_clear_mer_detail_h_d partition (dp,end_date) select id,mer_clear_no,merchant_no,account_evidence_no,product,transaction_no,payment_no,pay_seq,order_id,transaction_type,ori_transaction_no,ori_order_id,country,currency,transaction_amount,customer_id,card_no,store_code,transaction_created_date,transaction_success_date,payment_method,payment_method2,promotion_amount,extra_attribute,clearing_flag,clear_status,settle_type,rate_condition_type,merchant_fee_amount,refundable_mer_fee_amount,refund_merchant_fee_amount,algorithm_type,algorithm_value,mer_batch_no,mer_clear_date,mer_set_date,set_type,mer_pay_date,trade_settled,fee_setttled,merchant_refund_fee,created_date,modified_date,effect,is_calculate_vat,vat_rate,vat_fee,refunded_vat_fee,is_refund_vat,wht_rate,wht_fee,refunded_wht_fee,is_refund_wht,is_wht_set,settle_amount,merchant_fee_no,rate_no,service_code,set_no,attach,trade_date,is_recon,account_type,refund_type,start_date,dp,end_date from trainschain.dwd_s_acct_settlecore_clear_mer_detail_i_d_his_tmp'
Logging initialized using configuration in jar:file:/export/servers/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/jars/hive-common-1.1.0-cdh5.14.4.jar!/hive-log4j.properties
OK
Time taken: 1.177 seconds
setfacl: Permission denied. user=lixianwei6 is not the owner of inode=.hive-staging_hive_2020-03-26_09-41-50_930_2885942923653611875-1
Query ID = lixianwei6_20200326094141_b47b8591-ae35-41e1-b847-0754a3b4b8d5
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1574761440025_893687, Tracking URL = http://THA-223-35-30.h.chinabank.com.cn:8088/proxy/application_1574761440025_893687/
Kill Command = /export/servers/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/bin/hadoop job -kill job_1574761440025_893687
Hadoop job information for Stage-1: number of mappers: 16; number of reducers: 0
2020-03-26 09:42:01,021 Stage-1 map = 0%, reduce = 0%
2020-03-26 09:42:09,360 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 9.81 sec
2020-03-26 09:42:10,398 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 14.69 sec
2020-03-26 09:42:11,439 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 20.69 sec
2020-03-26 09:42:18,709 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 318.91 sec
2020-03-26 09:42:23,913 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 385.65 sec
2020-03-26 09:42:24,956 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 692.36 sec
2020-03-26 09:42:30,168 Stage-1 map = 72%, reduce = 0%, Cumulative CPU 858.85 sec
2020-03-26 09:42:31,217 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 1370.6 sec
2020-03-26 09:42:43,712 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 2722.88 sec
2020-03-26 09:42:48,888 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 3144.38 sec
2020-03-26 09:42:49,924 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 3409.25 sec
2020-03-26 09:42:56,148 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 4110.04 sec
2020-03-26 09:43:10,680 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 3764.08 sec
2020-03-26 09:43:16,894 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 3056.9 sec
2020-03-26 09:43:34,474 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 4026.54 sec
2020-03-26 09:43:40,668 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 4194.77 sec
2020-03-26 09:44:22,946 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 6258.44 sec
2020-03-26 09:44:43,571 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 5407.69 sec
2020-03-26 09:44:45,629 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 5462.79 sec
2020-03-26 09:45:06,212 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 5996.83 sec
2020-03-26 09:45:34,021 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 6072.8 sec
2020-03-26 09:45:56,689 Stage-1 map = 94%, reduce = 0%, Cumulative CPU 6323.27 sec
2020-03-26 09:46:40,997 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 5382.81 sec
MapReduce Total cumulative CPU time: 0 days 1 hours 29 minutes 42 seconds 810 msec
Ended Job = job_1574761440025_893687 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1574761440025_893687_m_000015 (and more) from job job_1574761440025_893687
Examining task ID: task_1574761440025_893687_m_000005 (and more) from job job_1574761440025_893687
Examining task ID: task_1574761440025_893687_m_000003 (and more) from job job_1574761440025_893687
Task with the most failures(7):
-----
Task ID:
task_1574761440025_893687_m_000003
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1574761440025_893687&tipid=task_1574761440025_893687_m_000003
-----
Diagnostic Messages for this Task:
Error: GC overhead limit exceeded
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
设置参数：
hive-site.xml 的 Hive 服务高级配置代码段（安全阀）
hive-site.xml 的 Hive 复制高级配置代码段（安全阀）
hive-site.xml 的 Hive 客户端高级配置代码段（安全阀）
mapreduce.reduce.memory.mb=5120
mapreduce.map.memory.mb=5120
mapreduce.map.java.opts=-Xmx4096m -XX:+UseConcMarkSweepGC
hive.groupby.skewindata=true
hive.optimize.skewjoin=true