hadoop中
和balance一样 机架感应对于集群来说 也是一个非常重要的工具。
默认情况下是不开启的 所有的机器都用 一个默认的机架 "default/rack"
配置机架感应的过程如下:
1.编写shell脚本:
vi $HADOOP_HOME/bin/rackaware.sh
HADOOP_CONF=$HADOOP_HOME/conf
while [ $# -gt 0 ] ; do
nodeArg=$1
exec< ${HADOOP_CONF}/topology.data
result=""
while read line ; do
ar=( $line )
if [ "${ar[0]}" = "$nodeArg" ] ; then
result="${ar[1]}"
fi
done
shift
if [ -z "$result" ] ; then
echo -n "/default/rack "
else
echo -n "$result "
fi
done
2.配置机器与机架的对应关系文件
配置好所有datanode 机器与机架的对应关系
vi $HADOOP_HOME/conf/topology.data
192-168-103-43 /dc1/rack1
192-168-103-44 /dc1/rack1
192-168-103-45 /dc1/rack2
192.168.103.43 /dc1/rack1
192.168.103.44 /dc1/rack1
192.168.103.45 /dc1/rack2
3.修改master的配置文件 core-site.xml
<property>
<name>topology.script.file.name</name>
<value>/opt/hadoop-0.20.2-cdh3u6/bin/rackaware.sh</value>
</property>
<property>
<name>topology.script.number.args</name>
<value>3</value>
</property>
添加以上配置
4.重启hadoop集群
log里面会看到如下信息:
2014-07-29 18:11:59,116 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /dc1/rack2/192-168-103-45
2014-07-29 18:11:59,131 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /dc1/rack1/192-168-103-44
2014-07-29 18:11:16,326 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /dc1/rack1/192.168.103.44:50010
2014-07-29 18:11:16,350 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /dc1/rack2/192.168.103.45:50010