cdh hadoop 集群搭建

CDH5 Hadoop集群完全离线安装说明

安装准备工具：

1. 操作系统centos7

2. Jdk环境：jdk包

3. Jdbc连接包：jdbc连接包mysql-connector-java-5.1.39-bin.jar

下载地址:http://dev.mysql.com/downloads/connector/j/

4. cdh 安装相关包

（1）cloudera manager包 下载地址：

http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz

（2）CDH-5.7.2-1.cdh5.7.2.p0.11-el7.parcel

（3）CDH-5.7.2-1.cdh5.7.2.p0.11-el7.parcel.sha1

（4）manifest.json

（2）~（4）的文件下载路径：http://archive.cloudera.com/cdh5/parcels/5.7.2/

5.linux系统的mysql数据库

以上准备工具如图所示：

安装前准备工作介绍；

1. Linux版本 必须支持cloudera manager（其中python版本必须一直，否则后期出现问题特别难解决）

2. Linux系统安装之后卸载虚拟网卡：ifconfig

会提示网络问题，提示网络速度问题。如果linux系统自带virbr0,需要卸载虚拟网络。命令如何：yum -y remove libvirt*   然后 reboot重启。

3

开始安装：

1. 安装jdk（每台机器都要） ，首先卸载原有的openJDK：具体操作指令如下

卸载自带jd k:]

杀进成

rm -f /var/run/yum.pid

[root@cdh1~]$ java -version

[root@cdh1~]$ rpm -qa | grep jdk

java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64

java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64

[root@cdh1~]# yum -y remove java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64

[root@cdh1~]# yum -y remove java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64

[root@cdh1~]# java -version

bash: /usr/bin/java: No such file or directory

安装jdk:

方法一：使用rpm命令安装：

[root@one-centos 下载] # rpm -ivh jdk-8u111-linux-x64.rpm

设置环境变量：
[root@one-centos 下载]# vim /etc/profile
在打开的profile文件中添加如下内容：
#set java environment
JAVA_HOME=/usr/java/jdk1.8.0_111
JRE_HOME=/usr/java/jdk1.8.0_111/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

修改生效

[root@one-centos 下载]# source /etc/profile

验证JDK有效性：

[root@one-centos yt]# java -version

[root@one-centos yt]# java

[root@one-centos yt]# javac

[root@one-centos yt]# echo $JAVA_HOME

/usr/java/jdk1.8.0_111

方法二：用tar.gz包安装JDK

在/usr/目录下创建java目录：

[root@localhost ~]# mkdir/usr/java

[root@localhost ~]# cd /usr/java

下载JDK，然后解压：

[root@localhost java]# curl -O http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.tar.gz

[root@localhost java]# tar -zxvf jdk-8u111-linux-x64.tar.gz

设置环境变量：（同上）

[root@one-centos 下载]# vim /etc/profile

在打开的profile文件中添加如下内容：

#set java environment

JAVA_HOME=/usr/java/jdk1.8.0_111

JRE_HOME=/usr/java/jdk1.8.0_111/jre

CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

export JAVA_HOME JRE_HOME CLASS_PATH PATH

让修改生效：（同上）

[root@one-centos 下载]# source /etc/profile

验证JDK有效性：（同上）

[root@one-centos yt]# java -version

[root@one-centos yt]# java

[root@one-centos yt]# javac

[root@one-centos yt]# echo $JAVA_HOME

/usr/java/jdk1.8.0_111

2. 修改每台节点服务器的有关配置hostname、selinux关闭，防火墙关闭;hostname修改：分别对三台都进行更改，并且注意每台名称和ip，每台都要配上hosts。下面以一台机器为例：

Hostname:

[root@cdh1~]# vi /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=cdh1

 

[root@cdh1~]# vi /etc/hosts127.0.0.1 localhost.cdh1192.168.42.128  cdh1192.168.42.129  cdh2192.168.42.130  cdh3

Selinux: 机器重启后生效；

[root@cdh1~]# vi /etc/sysconfig/selinux

SELINUX=disabled

[root@cdh1~]#sestatus -v

SELinux status: disabled

表示已经关闭了

防火墙：关闭防火墙

[root@cdh1~]# systemctl stop firewalld

[root@cdh1~]# systemctl disable firewalld

rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'

rm '/etc/systemd/system/basic.target.wants/firewalld.service'

[root@cdh1~]# systemctl status firewalld

firewalld.service - firewalld - dynamic firewall daemon

   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)

   Active: inactive (dead)

NTP服务器配置(用于3个节点间实现时间同步)

 

[root@cdh1~]#yum -y install ntp

更改master的节点

[root@cdh1~]## vi /etc/ntp.conf

注释掉所有server *.*.*的指向，新添加一条可连接的ntp服务器(我选的本公司的ntp测试服务器)

server 172.30.0.19 iburst

在其他节点上把ntp指向master服务器地址即可(/etc/ntp.conf下)

server 192.168.42.128 iburst

[root@cdh1~]## systemctl start ntpd  //启动ntp服务

[root@cdh1~]## systemctl status ntpd //查看ntp服务状态

 

SSH无密码登录配置，各个节点都需要设置免登录密码

下面以192.168.42.128到192.168.42.129的免密登录设置举例

[root@cdh1 /]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):/root/.ssh/id_rsa already exists.

Overwrite (y/n)? y

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

1d:e9:b4:ed:1d:e5:c6:a7:f3:23:ac:02:2b:8c:fc:ca root@cdh1

The key's randomart image is:

+--[ RSA 2048]----+

|                 |

|           .     |

|          +     .|

|         + +   + |

|        S + . . =|

|       .   . . +.|

|  . o   o   o +  |

|  .o o . .   o + |

|   Eo..   ... . o|

+-----------------+

[root@cdh1 /]# ssh-copy-id 192.168.42.129

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@192.168.42.129's password:

Number of key(s) added: 1

 

Now try logging into the machine, with:   "ssh '192.168.42.129'"

and check to make sure that only the key(s) you wanted were added.

 

安装mysql 
centos7自带的是mariadb，需要先卸载掉

 

[root@cdh1 /]# rpm -qa | grep mariadb

mariadb-libs-5.5.41-2.el7_0.x86_64

[root@cdh1 /]# rpm -e --nodeps mariadb-libs-5.5.41-2.el7_0.x86_64

[root@cdh1 /]# tar -xvf  MySQL-5.6.24-1.linux_glibc2.5.x86_64.rpm-bundle.tar   //mysql rpm包拷贝到服务器上然后解压

[root@cdh1 /]# rpm -ivh MySQL-*.rpm  //安装释出的全部rpm

[root@cdh1 /]# cp /usr/share/mysql/my-default.cnf /etc/my.cnf

[root@cdh1 /]# vi /etc/my.cnf    //在配置文件中增加以下配置并保存

[mysqld]

default-storage-engine = innodb

innodb_file_per_table

collation-server = utf8_general_ci

init-connect = 'SET NAMES utf8'

character-set-server = utf8

 

[root@cdh1 /]# yum install -y perl-Module-Install.noarch

[root@cdh1 /]# /usr/bin/mysql_install_db   //初始化mysql

[root@cdh1 /]# service mysql restart       //启动mysql

 ERROR! MySQL server PID file could not be found!

Starting MySQL... SUCCESS!

[root@cdh1 /]#  cat /root/.mysql_secret    //查看mysql root初始化密码

# The random password set for the root user at Fri Sep 22 11:13:25 2017 (local time): 9mp7uYFmgt6drdq3

[root@cdh1 /]#  mysql -u root -p          //登录进行去更改密码

mysql> SET PASSWORD=PASSWORD('123456');

mysql> update user set host='%' where user='root' and host='localhost';   //允许mysql远程访问

Query OK, 1 row affected (0.05 sec)

Rows matched: 1  Changed: 1  Warnings: 0

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

 

[root@cdh1 /]#  chkconfig mysql on   //配置开机启动

 

[root@cdh1 /]# tar -zcvf mysql-connector-java-5.1.44.tar.gz // 解压mysql-connector-java-5.1.44.tar.gz得到mysql-connector-java-5.1.44-bin.jar

[root@cdh1 /]#  mkdir /usr/share/java                          // 在各节点创建java文件夹

[root@cdh1 /]#  cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar       //将mysql-connector-java-5.1.44-bin.jar拷贝到/usr/share/java路径下并重命名为mysql-connector-java.jar

创建数据库

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.00 sec)

create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.00 sec)

create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.00 sec)

 create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.00 sec)

create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Query OK, 1 row affected (0.00 sec)

grant all on *.* to root@"%" Identified by "123456";

安装Cloudera-Manager

//解压cm tar包到指定目录所有服务器都要(或者在主节点解压好，然后通过scp到各个节点同一目录下)

[root@cdh1 ~]#mkdir /opt/cloudera-manager

[root@cdh1 ~]# tar -axvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager  

//创建cloudera-scm用户（所有节点）

[root@cdh1 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm  

//在主节点创建cloudera-manager-server的本地元数据保存目录

[root@cdh1 ~]# mkdir /var/cloudera-scm-server

[root@cdh1 ~]# chown cloudera-scm:cloudera-scm /var/cloudera-scm-server

[root@cdh1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera-manager

//配置从节点cloudera-manger-agent指向主节点服务器

[root@cdh1 ~]# vi /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini

将server_host改为CMS所在的主机名即cdh1

//主节点中创建parcel-repo仓库目录

[root@cdh1 ~]# mkdir -p /opt/cloudera/parcel-repo

[root@cdh1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

[root@cdh1 ~]# cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo

注意：其中CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 后缀要把1去掉

//所有节点创建parcels目录

[root@cdh1 ~]# mkdir -p /opt/cloudera/parcels

[root@cdh1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

解释：Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/opt/cloudera/parcels目录中

//初始脚本配置数据库scm_prepare_database.sh(在主节点上)

[root@cdh1 ~]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -p123456 --scm-host cdh1 scmdbn scmdbu scmdbp

说明：这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指：

mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。

-cdh1：数据库建立在cdh1主机上面,也就是主节点上面。

-uroot：root身份运行mysql。-123456：mysql的root密码是***。

--scm-host cdh1：CMS的主机，一般是和mysql安装的主机是在同一个主机上,最后三个参数是：数据库名，数据库用户名，数据库密码。

 

如果报错：

ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8'

java.sql.SQLException: Access denied for user 'root'@'cdh1' (using password: YES)

则参考 http://forum.spring.io/forum/spring-projects/web/57254-java-sql-sqlexception-access-denied-for-user-root-localhost-using-password-yes

运行如下命令：

update user set PASSWORD=PASSWORD('123456') where user='root';

GRANT ALL PRIVILEGES ON *.* TO 'root'@'cdh1' IDENTIFIED BY '123456' WITH GRANT OPTION;

FLUSH PRIVILEGES;

//启动主节点

[root@cdh1 ~]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server

[root@cdh1 ~]# chkconfig cloudera-scm-server on

[root@cdh1 ~]# vi /etc/init.d/cloudera-scm-server

CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default

[root@cdh1 ~]# service cloudera-scm-server start

//同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-server restart

//启动cloudera-scm-agent所有节点

[root@cdhX ~]# mkdir /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-agent

[root@cdhX ~]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent

[root@cdhX ~]# chkconfig cloudera-scm-agent on

[root@cdhX ~]# vi /etc/init.d/cloudera-scm-agent

CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default

[root@cdhX ~]# service cloudera-scm-agent start

//同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-agent restart

问题解决

 

参考：https://www.cnblogs.com/zhangleisanshi/p/7575579.html

副本块不足问题参考：http://blog.csdn.net/kissmelove01/article/details/44680255

2、CDH环境检测2处警告修复

2.1、检查主机正确性时出现 “Clouder% 建议将 /proc/sys/vm/sw%ppiness 设

置为 0。当前设置为 30。” 的警告，进⾏行行如下设定

# vi /etc/sysctl.conf

vm.sw%ppiness = 0

# sysctl –p

2.2、检查主机正确性时出现 “已启⽤用“透明⼤大⻚页⾯面”，它可能会导致重⼤大的性能

问题。” 的警告，进⾏行行如下设定

echo never > /sys/kernel/mm/tr%nsp%rent_hugep%ge/en%bled

echo never > /sys/kernel/mm/tr%nsp%rent_hugep%ge/defr%g

# vi /etc/rc.loc%l

echo never > /sys/kernel/mm/tr%nsp%rent_hugep%ge/en%bled

echo never > /sys/kernel/mm/tr%nsp%rent_hugep%ge/defr%g

3.网络问题，提示网络速度问题。如果linux系统自带virbr0,需要卸载虚拟网络。命令如何：yum -y remove libvirt*   然后 reboot重启。