标签(空格分隔): hadoop
VMware Workstation 10.0.1(本课程学习实验全部在虚拟机里面完成)
系统:Centos6.5 2.6.32-358.el6.x86_64
内存:最低2GB 处理器:1 网卡:NAT模式(自动获取IP)
1,修改主机名
# vi /etc/sysconfig/network
HOSTNAME=hadoop-yarn.beifeng.com
2,关闭selinux和防火墙
# chkconfig iptables off
# vi /etc/sysconfig/selinux
SELINUX=disabled
3,添加DNS
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DNS1=8.8.8.8
DNS2=8.8.4.4
4,新建几个目录
[root@localhost /]# mkdir -p /opt/softwares
[root@localhost /]# mkdir -p /opt/tools
[root@localhost /]# mkdir -p /opt/modules
5,解压jdk包到指定目录下
[root@hadoop-yarn softwares]# tar -zxf jdk-7u67-linux-x64.tar.gz -C /opt/modules/
6,删除系统自带的jdk
# rpm -qa|grep java
# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
# rpm -e --nodeps tzdata-java-2012j-1.el6.noarch
7,修改环境变量 /etc/profile
vi /etc/profile
export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile
# java -version //java命令验证一下版本
8,安装依赖包
# yum -y install autoconf automake libtool cmake
# yum -y install ncurses-devel
# yum -y install openssl-devel
# yum -y install lzo-devel zlib-devel gcc gcc-c++
9,安装protobuf
[root@hadoop-yarn softwares]# tar -zxf protobuf-2.5.0.tar.gz -C /opt/modules/
[root@hadoop-yarn modules]# cd protobuf-2.5.0/
# ./configure
# make
# make install
10,安装findbugs
下载解压apache-maven-3.0.5-bin.tar.gz
[root@hadoop-yarn softwares]# tar -zxf findbugs-1.3.9.tar.gz -C /opt/modules/
vi /etc/profile //增加环境变量 ------------------------------------
export FINDBUGS_HOME=/opt/modules/findbugs-1.3.9 export PATH=$PATH:$FINDBUGS_HOME/bin
------------------------------------ # source /etc/profile # findbugs -version //验证
11,配置maven环境
下载解压apache-maven-3.0.5-bin.tar.gz
[root@hadoop-yarn softwares]# tar -zxf apache-maven-3.0.5-bin.tar.gz -C /opt/modules/
vi /etc/profile //增加环境变量 ------------------------------------
export MAVEN_HOME=/opt/modules/apache-maven-3.0.5 export PATH=$PATH:$MAVEN_HOME/bin
------------------------------------ # source /etc/profile
[root@hadoop-yarn /]# mkdir ~/.m2/ //home目录下
[root@hadoop-yarn /]# cp /opt/modules/apache-maven-3.0.5/conf/settings.xml /root/.m2/
上传部分包(为节省时间我已经部分包打包成mvn-resp.tar.gz)
# tar -zxf mvn-resp.tar.gz //解压以后在当前目录会有一个.m2的隐藏目录
# cd .m2
# cp -r repository /root/.m2/
12,编译安装hadoop
hadoop 下载网址(所有的):
http://archive.apache.org/dist/hadoop/common/
源码包管理地址:
https://github.com/apache/hadoop
# tar -zxf hadoop-2.5.0-src.tar.gz -C /opt/modules/
使用maven命令编译hadoop
# cd hadoop-2.5.0-src/
# mvn package -Pdist,native -DskipTests -Dtar
13.移动目录
# cd /彩电opt/modules/hadoop-2.5.0-src/hadoop-dist/target
# mv hadoop-2.5.0.tar.gz /opt/softwares/hadoop-2.5.0-bf.tar.gz
移出我们编译好的hadoop目录
# cd /opt/modules/hadoop-2.5.0-src/hadoop-dist/target
# mv hadoop-2.5.0/ /opt/modules/
(/opt/modules/hadoop-2.5.0/这就是我们hadoop的编译好的目录)
Hadoop默认情况下是不支持snappy压缩的,如果需要支持snappy的话,需要自行编译hadoop2.x源码完成:
1) 安装sanppy
2) 编译haodop 2.x源码
mvn package -Pdist,native -DskipTests -Dtar -Drequire.snappy
编译完成以后会生成新的/opt/modules/hadoop-2.5.0-src/target/hadoop-2.5.0/lib/native,然后替换原来的即可。
编译需要较长时间。
**报错Out Of Memory,由于内存原因导致,加export MAVEN_OPTS=”-Xms256m -Xmx512m”
**如果出现类似下面这样的错误,多尝试几次”mvn package -Pdist,native -DskipTests –Dtar”命令,这是因为网络不稳定造成的。
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution module-javadocs of goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar failed: Plugin org.apache.maven.plugins:maven-javadoc-plugin:2.8.1 or one of its dependencies could not be resolved: Could not transfer artifact com.thoughtworks.qdox:qdox:jar:1.12 from/to central (http://repo.maven.apache.org/maven2): Connection to http://repo.maven.apache.org refused: Connection refused -> [Help 1]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-project: Execution module-javadocs of goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar failed: Plugin org.apache.maven.plugins:maven-javadoc-plugin:2.8.1 or one of its dependencies could not be resolved: Could not transfer artifact com.thoughtworks.qdox:qdox:jar:1.12 from/to central (http://repo.maven.apache.org/maven2): Connection to http://repo.maven.apache.org refused: Connection refused -> [Help 1]