cdh离线安装部署

cdh安装准备环境的博文请参考： https://mp.csdn.net/console/editor/html/105581435

 

CM离线部署:

  server agent

[root@mycluster001 cdh5.16.2]# mv cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz /tmp/

[root@mycluster001 cdh5.16.2]# cd tmp

[root@mycluster001 tmp]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz mycluster002:/tmp
root@mycluster002's password: 
cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz       100%  799MB 114.2MB/s   00:07    
[root@mycluster001 tmp]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz mycluster003:/tmp
root@mycluster003's password: 
cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz       100%  799MB 266.4MB/s   00:03   

 

以下3台都需要操作：

[root@mycluster001 tmp]#  mkdir /opt/cloudera-manager   

[root@mycluster001 tmp]# tar -zxvf cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz -C /opt/cloudera-manager

root@mycluster001 tmp]# cd /opt/cloudera-manager/
[root@mycluster001 cloudera-manager]# ll
total 8
drwxr-xr-x 4 1106 4001 4096 May 18  2019 cloudera
drwxr-xr-x 9 1106 4001 4096 May 18  2019 cm-5.16.2

[root@mycluster001 cloudera-scm-agent]# sed -i 's#server_host=localhost#server_host=mycluster001#g' /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-agent/config.ini 

[root@mycluster001 cloudera-scm-agent]# head /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-agent/config.ini
[General]
# Hostname of the CM server.

server_host=mycluster001

创建用户并赋予权限

[root@mycluster001 cloudera-scm-server]# useradd --system --home=/opt/cloudera-manager/cm-5.16.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "cloudera scm user" cloudera-scm

[root@mycluster001 cloudera-scm-server]# chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager/
[root@mycluster001 cloudera-scm-server]# ll /opt/cloudera-manager/
total 8
drwxr-xr-x 4 cloudera-scm cloudera-scm 4096 May 18  2019 cloudera
drwxr-xr-x 9 cloudera-scm cloudera-scm 4096 May 18  2019 cm-5.16.2
[root@mycluster001 cloudera-manager]# cd cm-5.16.2/
[root@mycluster001 cm-5.16.2]# ll
total 28
drwxr-xr-x 7 cloudera-scm cloudera-scm 4096 May 18  2019 etc
drwxr-xr-x 5 cloudera-scm cloudera-scm 4096 May 18  2019 lib
drwxr-xr-x 3 cloudera-scm cloudera-scm 4096 May 18  2019 lib64
drwxr-xr-x 5 cloudera-scm cloudera-scm 4096 May 18  2019 log
drwxr-xr-x 3 cloudera-scm cloudera-scm 4096 May 18  2019 run
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 May 18  2019 sbin
drwxr-xr-x 3 cloudera-scm cloudera-scm 4096 May 18  2019 share

 

以下是第一台机器操作：

[root@mycluster001 cloudera-scm-server]# cd /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-server

[root@mycluster001 cloudera-scm-server]# vi db.properties

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=mycluster001

# The database name
com.cloudera.cmf.db.name=cmf

# The database user
com.cloudera.cmf.db.user=cmf

# The database user's password
com.cloudera.cmf.db.password=000000

# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL

 

Parcel离线部署

解压parcel包：

[root@mycluster001 cdh5.16.2]#  tar -zxvf CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel

[root@mycluster001 cdh5.16.2]# ll
total 2082876
drwxr-xr-x 11 1106 4001       4096 Jun  3  2019 CDH-5.16.2-1.cdh5.16.2.p0.8
-rw-r--r--  1 root root 2132782197 Apr 17 19:14 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
-rw-r--r--  1 root root         41 Jun 18  2019 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1
-rw-r--r--  1 root root      66804 Jun 18  2019 manifest.json

[root@mycluster001 cdh5.16.2]# cd CDH-5.16.2-1.cdh5.16.2.p0.8
[root@mycluster001 CDH-5.16.2-1.cdh5.16.2.p0.8]# ll
total 96
drwxr-xr-x  2 1106 4001  4096 Jun  3  2019 bin
drwxr-xr-x 27 1106 4001  4096 Jun  3  2019 etc
drwxr-xr-x  5 1106 4001  4096 Jun  3  2019 include
drwxr-xr-x  2 1106 4001 65536 Jun  3  2019 jars
drwxr-xr-x 38 1106 4001  4096 Jun  3  2019 lib
drwxr-xr-x  3 1106 4001  4096 Jun  3  2019 lib64
drwxr-xr-x  3 1106 4001  4096 Jun  3  2019 libexec
drwxr-xr-x  2 1106 4001  4096 Jun  3  2019 meta
drwxr-xr-x  4 1106 4001  4096 Jun  3  2019 share

重命名：

[root@mycluster001 cdh5.16.2]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha

如果不去掉1，会在部署的过程中，cm会认为该parcel文件下载未完成，会持续下载

 

比对parcel包是否破损：

[root@mycluster001 cdh5.16.2]# cat CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
eec004b83305131a1982bb36f1ae2be54b6ef0d6
[root@mycluster001 cdh5.16.2]# sha1sum CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
eec004b83305131a1982bb36f1ae2be54b6ef0d6  CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel

 

[root@mycluster001 cdh5.16.2]#mkdir -p /opt/cloudera/parcel-repo

[root@mycluster001 cdh5.16.2]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel* /opt/cloudera/parcel-repo

[root@mycluster001 cdh5.16.2]# mv manifest.json /opt/cloudera/parcel-repo/

[root@mycluster001 cdh5.16.2]# cd /opt/cloudera/parcel-repo/
[root@mycluster001 parcel-repo]# ll
total 2082872
-rw-r--r-- 1 root root 2132782197 Apr 17 19:14 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
-rw-r--r-- 1 root root         41 Jun 18  2019 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
-rw-r--r-- 1 root root      66804 Jun 18  2019 manifest.json

[root@mycluster001 parcel-repo]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/

[root@mycluster001 parcel-repo]# ll /opt/cloudera/parcel-repo/
total 2082872
-rw-r--r-- 1 cloudera-scm cloudera-scm 2132782197 Apr 17 19:14 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
-rw-r--r-- 1 cloudera-scm cloudera-scm         41 Jun 18  2019 CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
-rw-r--r-- 1 cloudera-scm cloudera-scm      66804 Jun 18  2019 manifest.json

 

正式集群部署

1、创建所有节点的软件安装目录 用户和用户组的权限

多台操作：

[root@mycluster001 cloudera]# mkdir -p /opt/cloudera/parcels
[root@mycluster001 cloudera]# ll
total 8
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Apr 18 15:46 parcel-repo
drwxr-xr-x 2 root         root         4096 Apr 18 15:53 parcels
[root@mycluster001 cloudera]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/
[root@mycluster001 cloudera]# ll
total 8
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Apr 18 15:46 parcel-repo
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Apr 18 15:53 parcels

2、mycluser001机器启动cm server

[root@mycluster001 init.d]# cd /opt/cloudera-manager/cm-5.16.2/etc/init.d

[root@mycluster001 init.d]# ll
total 32
-rwxr-xr-x 1 cloudera-scm cloudera-scm 8871 May 18  2019 cloudera-scm-agent
-rwxr-xr-x 1 cloudera-scm cloudera-scm 8417 May 18  2019 cloudera-scm-server
-rwxr-xr-x 1 cloudera-scm cloudera-scm 4444 May 18  2019 cloudera-scm-server-db
[root@mycluster001 init.d]# ./cloudera-scm-server start 

这个时候新开一个远程户端，切到log路径：

[root@mycluster001 cloudera-scm-server]# tail -F /opt/cloudera-manager/cm-5.16.2/log/cloudera-scm-server/cloudera-scm-server.log 

[root@mycluster001 init.d]# netstat -lntup 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:7180            0.0.0.0:*               LISTEN      4320/java           
tcp        0      0 0.0.0.0:7182            0.0.0.0:*               LISTEN      4320/java           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2931/sshd           
tcp6       0      0 :::3306                 :::*                    LISTEN      4251/mysqld    

1）阿里云web防火墙放开 7180

 

2）等待1min   打开浏览器

http://120.55.92.62:7180/cmf/login

120.55.92.62为mycluster001的公网ip

3）定位log，找到error

[root@mycluster001 cloudera-scm-server]# tail -F /opt/cloudera-manager/cm-5.16.2/log/cloudera-scm-server/cloudera-scm-server.log 

 

3、启动所有节点agent（3台）

[root@mycluster001 init.d]# /opt/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-agent  start
Starting cloudera-scm-agent:                               [  OK  ]
[root@mycluster001 init.d]# /opt/cloudera-manager/cm-5.16.2/etc/init.d/cloudera-scm-agent  status
cloudera-scm-agent (pid  4485) is running...

 

4、cm web界面配置

选择free版本即可，生产环境是够用的，且免费版本是没有节点限制的。

根据这里的警告信息进行操作，3台都要操作

[root@mycluster001 init.d]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@mycluster001 init.d]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

然后再重新运行

至此cdh安装完成

 

5、web界面管理

因为我们的ntp  server只部署在001机器上，所以可以将其他机器抑制

现在已经没有报错信息了。

处理以下警告信息：

剩下的根据提示修改即可，结果见下图：

 

 

ok，至此cdh已安装ok,至于cdh的各种操作以及故障排查等操作，以后视情况看是否再出相关博客吧。各位辛苦了。

 

--------------------------

      用人品去感动别人，用行动去带动别人，用阳光去照耀别人，用坚持去赢得别人，要求自己每天都去做与目标有关的事情，哪怕每天只进步一点点，坚持下来你就是最优秀卓越的！欢迎大家加入大数据qq交流群：725967421  一起交流，一起进步！！
--------------------------