map和reduce 个数的设定 （Hive优化）经典

一、<wbr><wbr><wbr>控制hive任务中的map数:</wbr></wbr></wbr><wbr></wbr>

1.<wbr><wbr><wbr>通常情况下，作业会通过input的目录产生一个或者多个map任务。</wbr></wbr></wbr><wbr><br> 主要的决定因素有： input的文件总个数，input的文件大小，集群设置的文件块大小(目前为128M, 可在hive中通过setdfs.block.size;命令查看到，该参数不能自定义修改)；<br><br><span style="color:#ff0ff">2.<wbr><wbr><wbr>举例：</wbr></wbr></wbr></span><wbr><br> a)<wbr><wbr><wbr>假设input目录下有1个文件a,大小为780M,那么hadoop会将该文件a分隔成7个块（6个128m的块和1个12m的块），从而产生7个map数<br> b)<wbr><wbr><wbr>假设input目录下有3个文件a,b,c,大小分别为10m，20m，130m，那么hadoop会分隔成4个块（10m,20m,128m,2m）,从而产生4个map数<br> 即，如果文件大于块大小(128m),那么会拆分，如果小于块大小，则把该文件当成一个块。<br><br><span style="color:#ff0ff">3.<wbr><wbr><wbr>是不是map数越多越好？</wbr></wbr></wbr></span><wbr><br> 答案是否定的。如果一个任务有很多小文件（远远小于块大小128m）,则每个小文件也会被当做一个块，用一个map任务来完成，<br> 而一个map任务启动和初始化的时间远远大于逻辑处理的时间，就会造成很大的资源浪费。<br> 而且，同时可执行的map数是受限的。</wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr>

<wbr></wbr>

4.<wbr><wbr><wbr>是不是保证每个map处理接近128m的文件块，就高枕无忧了？</wbr></wbr></wbr><wbr><br> 答案也是不一定。比如有一个127m的文件，正常会用一个map去完成，但这个文件只有一个或者两个小字段，却有几千万的记录，<br> 如果map处理的逻辑比较复杂，用一个map任务去做，肯定也比较耗时。<br><br> 针对上面的问题3和4，我们需要采取两种方式来解决：即减少map数和增加map数；<br><br><strong><span style="color:#ff00">如何合并小文件，减少map数？</span><wbr></wbr></strong><br><wbr><wbr><wbr>假设一个SQL任务：<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>Select count(1) from popt_tbaccountcopy_mes where pt =‘2012-07-04’;<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>该任务的inputdir<wbr>/group/p_sdo_data/p_sdo_data_etl/pt/popt_tbaccountcopy_mes/pt=2012-07-04<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>共有194个文件，其中很多是远远小于128m的小文件，总大小9G，正常执行会用194个map任务。<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>Map总共消耗的计算资源： SLOTS_MILLIS_MAPS= 623,020<br><br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>我通过以下方法来在map执行前合并小文件，减少map数：<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>set mapred.max.split.size=100000000;<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr> setmapred.min.split.size.per.node=100000000;<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr> setmapred.min.split.size.per.rack=100000000;<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr> sethive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>再执行上面的语句，用了74个map任务，map消耗的计算资源：SLOTS_MILLIS_MAPS=333,500<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>对于这个简单SQL任务，执行时间上可能差不多，但节省了一半的计算资源。<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>大概解释一下，100000000表示100M, sethive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;这个参数表示执行前进行小文件合并，<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>前面三个参数确定合并文件块的大小，大于文件块大小128m的，按照128m来分隔，小于128m,大于100m的，按照100m来分隔，把那些小于100m的（包括小文件和分隔大文件剩下的），<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>进行合并,最终生成了74个块。<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><br><strong><span style="color:#ff00">如何适当的增加map数？</span><wbr></wbr></strong><br><br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>当input的文件都很大，任务逻辑复杂，map执行非常慢的时候，可以考虑增加Map数，来使得每个map处理的数据量减少，从而提高任务的执行效率。<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>假设有这样一个任务：<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>Select data_desc,<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>count(1),<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr>count(distinct id),<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr> sum(casewhen …),<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr> sum(casewhen ...),<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr><wbr> sum(…)<br><wbr><wbr><wbr><wbr><wbr><wbr><wbr>from a group by data_desc<br><wbr><wbr><wbr><wbr><wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr></wbr>