（转载）Hadoop2.7.1配置
http://my.oschina.net/laigous/blog/478438
hadoop伪分布式搭建
其实官网有比较详实的说明，英语好的可以直接看官网，地址
1、安装JDK1.7
这个省略，官网显示1.6可以，但是我用openjdk1.6出了异常，JDK1.6没试，直接用了JDK1.7
配置好环境变量
|
1
|
vi
/etc/profile
|
|
1
2
3
|
export
JAVA_HOME=
/usr/local/jdk1
.7.0_79
export
CLASSPATH=.:$JAVE_HOME
/lib
.tools.jar
export
PATH=$PATH:$JAVA_HOME
/bin
|
添加完后执行命令使配置生效
|
1
|
source
/etc/profile
|
2、安装ssh、rsync,（以ubuntu为例）
|
1
2
|
$
sudo
apt-get
install
ssh
$
sudo
apt-get
install
rsync
|
3、下载hadoop编译包，镜像地址（国内镜像竟然比US的还慢，不能忍，注意2.7的编译包为64位）
查是32还是64位的办法
|
1
2
|
cd
hadoop-2.7.0
/lib/native
file
libhadoop.so.1.0.0
|
|
1
|
hadoop-2.7.0
/lib/native/libhadoop
.so.1.0.0: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped
|
4、配置环境变量
hadoop配置文件指定java路径
etc/hadoop/hadoop-env.sh
|
1
|
export
JAVA_HOME=
/usr/local/jdk1
.7.0_79
|
系统环境变量
|
1
2
3
4
|
export
HADOOP_HOME=
/usr/local/hadoop-2
.7.0
export
PATH=$PATH:$HADOOP_HOME
/bin
export
HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME
/lib/native
export
HADOOP_OPTS=
"-Djava.library.path=$HADOOP_HOME/lib"
|
后两条不加会出现
You have loaded library /usr/hadoop/hadoop-2.7.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
添加完后执行命令使配置生效
|
1
|
source
/etc/profile
|
执行命令查看是否成功
|
1
|
hadoop version
|
5、修改hadoop配置文件
etc/hadoop/core-site.xml:
|
1
2
3
4
5
6
|
<
configuration
>
<
property
>
<
name
>fs.defaultFS</
name
>
<
value
>hdfs://localhost:9000</
value
>
</
property
>
</
configuration
>
|
etc/hadoop/hdfs-site.xml:
|
1
2
3
4
5
6
|
<
configuration
>
<
property
>
<
name
>dfs.replication</
name
>
<
value
>1</
value
>
</
property
>
</
configuration
>
|
6、设置hadoop ssh无密码访问
|
1
2
3
|
$
ssh
-keygen -t dsa -P
''
-f ~/.
ssh
/id_dsa
$
cat
~/.
ssh
/id_dsa
.pub >> ~/.
ssh
/authorized_keys
$
export
HADOOP\_PREFIX=
/usr/local/hadoop-2
.7.0
|
7、格式化节点和启动hadoop
|
1
2
|
$ bin
/hdfs
namenode -
format
$ sbin
/start-dfs
.sh
|
打开浏览器 http://localhost:50070/看是否成功
hdfs配置：username最好和当前用户名相同，不然会可能出现权限问题
|
1
2
|
$ bin
/hdfs
dfs -
mkdir
/user
$ bin
/hdfs
dfs -
mkdir
/user/
<username>
|
8、yarn配置
etc/hadoop/mapred-site.xml:
|
1
2
3
4
5
6
|
<
configuration
>
<
property
>
<
name
>mapreduce.framework.name</
name
>
<
value
>yarn</
value
>
</
property
>
</
configuration
>
|
etc/hadoop/yarn-site.xml:
|
1
2
3
4
5
6
|
<
configuration
>
<
property
>
<
name
>yarn.nodemanager.aux-services</
name
>
<
value
>mapreduce_shuffle</
value
>
</
property
>
</
configuration
>
|
启动yarn
|
1
|
$ sbin
/start-yarn
.sh
|
http://localhost:8088/查看是否成功
至此hadoop单节点伪分布式安装配置完成