1. core-site.xml文件
<configuration xmlns:xi="http://www.w3.org/2001/XInclude"> <xi:includehref="mountTable.xml" /> <property> <name>fs.defaultFS</name> <value>viewfs://mycluster</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/home/hihadoop/hadoop/jns</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/hadoop/tmp</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>hadoopa.highgo.com:2181</value> </property> </configuration>
2. mountTable.xml文件
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.viewfs.mounttable.mycluster.homedir</name> <value>/home</value> </property> <property> <name>fs.viewfs.mounttable.mycluster.link./home</name> <value>hdfs://ns1/home</value> </property> <property> <name>fs.viewfs.mounttable.mycluster.link./tmp</name> <value>hdfs://ns1/tmp</value> </property> <property> <name>fs.viewfs.mounttable.mycluster.link./projects/foo</name> <value>hdfs://ns2/projects/foo</value> </property> <property> <name>fs.viewfs.mounttable.mycluster.link./projects/bar</name> <value>hdfs://ns2/projects/bar</value> </property> </configuration>
3. hdfs-site.xml文件(使用ViewFs的话,相关的配置,上边的配置已经完成了,下面是一个两个高可用的集群联邦的配置)
<configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/hadoop/dfs/data/data1</value> </property> <!-- <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoopb.highgo.com:50090</value> </property> <property> <name>dfs.namenode.secondary.https-address</name> <value>hadoopb.highgo.com:50091</value> </property> --> <property> <name>dfs.nameservices</name> <value>ns1,ns2</value> </property> <property> <name>dfs.ha.namenodes.ns1</name> <value>nna,nnb</value> </property> <property> <name>dfs.ha.namenodes.ns2</name> <value>nng,nnh</value> </property> <property> <name>dfs.namenode.rpc-address.ns1.nna</name> <value>hadoopa.highgo.com:8020</value> </property> <property> <name>dfs.namenode.rpc-address.ns1.nnb</name> <value>hadoopb.highgo.com:8020</value> </property> <property> <name>dfs.namenode.http-address.ns1.nna</name> <value>hadoopa.highgo.com:50070</value> </property> <property> <name>dfs.namenode.http-address.ns1.nnb</name> <value>hadoopb.highgo.com:50070</value> </property> <property> <name>dfs.namenode.shared.edits.dir.ns1</name> <value>qjournal://hadoopa.highgo.com:8485;hadoopb.highgo.com:8485;hadoopc.highgo.com:8485/ns1</value> </property> <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.namenode.rpc-address.ns2.nng</name> <value>hadooph.highgo.com:8020</value> </property> <property> <name>dfs.namenode.rpc-address.ns2.nnh</name> <value>hadoopg.highgo.com:8020</value> </property> <property> <name>dfs.namenode.http-address.ns2.nng</name> <value>hadooph.highgo.com:50070</value> </property> <property> <name>dfs.namenode.http-address.ns2.nnh</name> <value>hadoopg.highgo.com:50070</value> </property> <!-- <property> <name>dfs.namenode.shared.edits.dir.ns2</name> <value>qjournal://hadoopf.highgo.com:8485;hadoopg.highgo.com:8485;hadooph.highgo.com:8485/ns2</value> </property>--> <property> <name>dfs.client.failover.proxy.provider.ns2</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hihadoop/.ssh/id_rsa</value> </property> <property> <name>dfs.ha.automatic-failover.enabled.ns1</name> <value>true</value> </property> <property> <name>dfs.ha.automatic-failover.enabled.ns2</name> <value>true</value> </property> </configuration>
4. 格式化集群。需要在两个NameNode集群上分别格式化。启动整个JNs,在每一个集群上,在其中一个NameNode执行
hdfs namenode -format -clusterId mycluster
然后启动
hadoop-daemon.sh start namenode
在另一个NameNode上执行
hdfs namenode –bootstrapStandby
然后启动
hadoop-daemon.sh start namenode
在两个NameNode集群的任意NameNode上执行
hdfs zkfc –formatZK
在所有节点执行
hadoop-daemon.sh start zkfc
启动所有的DataNode
5. 此时可以用Hadoop shell进行一些操作了,操作的目录是所用的NameNode就是挂载表中的配置。