自己搭建的hadoop集群一般默认不具备机架感知能力,为了提高hadoop集群的网络性能,我们常常需要配置机架感知。下面是以下步骤:
一、修改配置文件core-site.xml,添加配置属性:
<property> <name>topology.script.file.name</name> <value>/home/r203/hadoop-0.20.2/rackaware.py</value> </property>
二、编写自己的脚本rackaware.py:
#!/usr/bin/python #-*-coding:UTF-8 -*- import sys rack = { "14a":"rack1","14b":"rack1","14c":"rack1","14d":"rack1","14e":"rack1","14f":"rack1","14g":"rack1","14h":"rack1", "14i":"rack1","14j":"rack1","14k":"rack1","14l":"rack1","14m":"rack1","14n":"rack1","14o":"rack1","14p":"rack1", "14q":"rack1","14r":"rack1","14s":"rack1","14t":"rack1","14u":"rack1","14v":"rack1","14w":"rack1","14x":"rack1", "15a":"rack2","15b":"rack2","15c":"rack2","15d":"rack2","15e":"rack2","15f":"rack2","15g":"rack2","15h":"rack2", "15i":"rack2","15j":"rack2","15k":"rack2","15l":"rack2","15m":"rack2","15n":"rack2","15o":"rack2","15p":"rack2", "15q":"rack2","15r":"rack2","15s":"rack2","15t":"rack2","15u":"rack2","15v":"rack2","15w":"rack2","15x":"rack2", "16a":"rack3","16b":"rack3","16c":"rack3","16d":"rack3","16e":"rack3","16f":"rack3","16g":"rack3","16h":"rack3", "16i":"rack3","16j":"rack3","16k":"rack3","16l":"rack3","16m":"rack3","16n":"rack3","16o":"rack3","16p":"rack3", "16q":"rack3","16r":"rack3","16s":"rack3","16t":"rack3","16u":"rack3","16v":"rack3","16w":"rack3","16x":"rack3", } if __name__=="__main__": print "/" + rack.get(sys.argv[1],"rack0")
注:脚本中可以写主机名也可以写ip地址。
三、重启namenode。