Hadoop Usage

hadoop.tmp.dir

f you are using Hadoop on ubuntu, don't accept the default value for hadoop.tmp.dir. The default value is /tmp/hadoop-${user.name}.

 

With /tmp/hadoop-${user.name} as hadoop.tmp.dir, I need to format namenode whenenver I restarted ubuntu. So use another value for this Hadoop configuration.

fs.default.name

Make sure that /rise.cn.abc.com is DNS name accessible by all the hadoop nodes.

 

<property>
    <name>fs.default.name</name>
  <value>hdfs://rise.cn.abc.com:9000/</value>
</property>

 

If rise.cn.abc.com is bound to 127.0.0.1 in /etc/hosts, only datanodes which exists on the same machine as the namenode can connect to the namenode.

 

Namenode cannot accept connection from datanode is a detailed description of this problem.

 

HADOOP_CONF_DIR

 

If unset, HADOOP_CONF_DIR is $HADOOP_HOME/conf. It can be also specified on hadoop comamnd. For example,

 

hadoop --config /home/jing/dev/hadoop/conf/xincl-conf/  fs -ls /

 

--config expects a directory. There is another option -conf for adding additional configurations. -conf expects a file. And it is specified after hadoop subcommand. For exmaple.

 

hadoop fs -conf /home/jing/dev/hadoop/conf/single/hadoop.xml  -ls /