[原]Hadoop/Yarn/MapReduce内存分配（配置）方案

以horntonworks给出推荐配置为蓝本，给出一种常见的Hadoop集群上各组件的内存分配方案。方案最右侧一栏是一个8G VM的分配方案，方案预留1-2G的内存给操作系统，分配4G给Yarn/MapReduce，当然也包括了HIVE，剩余的2-3G是在需要使用HBase时预留给HBase的。

Configuration File	Configuration Setting	Value Calculation	8G VM (4G For MR)
yarn-site.xml	yarn.nodemanager.resource.memory-mb	= containers * RAM-per-container	4096
yarn-site.xml	yarn.scheduler.minimum-allocation-mb	= RAM-per-container	1024
yarn-site.xml	yarn.scheduler.maximum-allocation-mb	= containers * RAM-per-container	4096
mapred-site.xml	mapreduce.map.memory.mb	= RAM-per-container	1024
mapred-site.xml	mapreduce.reduce.memory.mb	= 2 * RAM-per-container	2048
mapred-site.xml	mapreduce.map.java.opts	= 0.8 * RAM-per-container	819
mapred-site.xml	mapreduce.reduce.java.opts	= 0.8 * 2 * RAM-per-container	1638
yarn-site.xml (check)	yarn.app.mapreduce.am.resource.mb	= 2 * RAM-per-container	2048
yarn-site.xml (check)	yarn.app.mapreduce.am.command-opts	= 0.8 * 2 * RAM-per-container	1638
tez-site.xml	tez.am.resource.memory.mb	= RAM-per-container	1024
tez-site.xml	tez.am.java.opts	= 0.8 * RAM-per-container	819
tez-site.xml	hive.tez.container.size	= RAM-per-container	1024
tez-site.xml	hive.tez.java.opts	= 0.8 * RAM-per-container	819

 

作者：bluishglc 发表于2015-1-6 8:30:20 原文链接

阅读：218 评论：0 查看评论