CDH6.2 集群离线部署

CDH6.2 集群离线部署

---

服务器集群节点角色规划

用户	主机名	内网IP	SSH端口	系统	角色
root	cm-server-master	192.168.2.131	22	CentOS Linux release 7.6.1810 (Core)	nginx、postgresql、cm-server、cm-agent
root	cm-node-02	192.168.2.132	22	CentOS Linux release 7.6.1810 (Core)	cm-agent
root	cm-node-03	192.168.2.133	22	CentOS Linux release 7.6.1810 (Core)	cm-agent

系统环境准备

• 安装一些个人常用的基础安装包

 yum  -y install  epel-release.noarch
 yum  -y install   psmisc gcc gcc-c++ texinfo wget unzip zip gcc libticonv-devel  libcurl-devel curl nmap iotop dstat tree mlocate ntpdate openssh-clients  net-tools vim  ntsysv nmap curl lrzsz  sysstat  libselinux-python   pcre pcre-devel zlib zlib-devel openssl openssl-devel readline-devel bzip2  httpd-devel python-devel python-pip python-setuptools   lsof  sqlite-devel  nscd bind-utils telnet rsync tcpdump expect  nc    ntp  lftp   bash-completion ipset ipvsadm 

• 更改服务hosts文件

  cat  >> /etc/hosts << EOF
  192.168.2.131 cm-server-master
  192.168.2.132 cm-node-02
  192.168.2.133 cm-node-03
  EOF
  

• 关闭防火墙

  systemctl stop firewalld
  systemctl stop iptables
  systemctl disable firewalld
  systemctl disable iptables
  
  

• 关闭selinux

  sed -i 's/SELINUX=enforcing/SELINUX=disabled/'  /etc/sysconfig/selinux
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/'   /etc/selinux/config
  setenforce 0
  

• 三台时间同步要一致

  systemctl start ntpd
  systemctl enable ntpd
  
  

• 关闭swap分区

  swapoff -a
  sed  -i  '/swap/d' /etc/fstab
  mount  -a
  

  • 然后修改/etc/fstab，把swap分区相关的配置注释掉

• 内核参数调整

  cat >> /etc/sysctl.conf <

• 配置资源限制

  sh -c " cat  >>/etc/security/limits.conf <> /etc/security/limits.d/20-nproc.conf << EOF
     *  soft    nproc     unlimited
     *  hard    nproc     unlimited
  EOF"
  

• 系统语言环境为 UTF-8

  echo 'LANG="zh_CN.UTF-8"' >/etc/locale.conf
  source  /etc/locale.conf
  

- 关闭透明大页
  ```
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
  echo never > /sys/kernel/mm/transparent_hugepage/defrag
  cat  >>/etc/rc.d/rc.local << EOF
  if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
  fi

  if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
    echo never > /sys/kernel/mm/transparent_hugepage/defrag
  fi
  EOF
  chmod +x /etc/rc.d/rc.local
  ```

cm-server 配置ssh 免密码登录认证

• 生成ssh秘钥对

  ssh-keygen -t rsa  
  

• 拷贝SSH公钥到目标服务器账号~/.ssh/authorized_keys(600权限)文件中

  ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
  ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
  ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
  
  

在cm-server服务器上下载CDH6.2相关的安装包

• 链接: https://pan.baidu.com/s/1qcb-SWB4itv1rN9-fBoVgQ 提取码: tqu2
• 云盘中的parcel-repo-6.2.0、cloudera-repos-6.2.0 文件夹下载到家目录下

安装nginx 做cm本地yum repo服务器

• 安装启动nginx

  yum install -y yum-utils
  tee -a /etc/yum.repos.d/nginx.repo <

• 配置Cloudera Manager包 yum源

  tee -a /etc/nginx/conf.d/cm6.conf << EOF
  server {
  listen 8090;
  server_name 0.0.0.0;
  root /data/nginx/cm6;
  location /cm6 {
       autoindex on;
       autoindex_exact_size off;
       autoindex_localtime on;
    }
  }
  mkdir -p /data/nginx/cm6
  cp ~/cloudera-repos-6.2.0/* /data/nginx/cm6/
  nginx -s reload
  yum install -y createrepo
  cd /data/nginx/cm6
  createrepo .
  tee -a /etc/yum.repos.d/cm6.repo << EOF
  [cloudera-manager]
  name=Cloudera Manager 6.2.0
  baseurl=http://192.168.2.131:8090/cm6
  gpgcheck=0
  enabled=1
  EOF
  

安装postgressql

• 使用docker安装postgres

  mkdir -p /data/postgres-data
  docker run -it -d --name postgres --restart always --privileged=true -v /data/postgres-data:/var/lib/postgresql/data -e POSTGRES_PASSWORD="password" -e POSTGRES_USER="root" -p 5432:5432 -d postgres:9.5.12
  
  

• 创建CDH需要使用的用户和数据库

  create user cdh password 'password';
  CREATE DATABASE cdhdb;
  CREATE DATABASE hivedb;
  CREATE DATABASE rmdb;
  CREATE DATABASE amdb;
  grant all  on database  cdhdb to  cdh;
  grant all  on database  hivedb to  cdh;
  grant all  on database  rmdb to  cdh;
  grant all  on database  amdb to  cdh;
  

安装和配置cm-server

• yum 安装cm-server

  yum install -y cloudera-manager-daemons cloudera-manager-server oracle-j2sdk1.8
  

• 配置Parcel存储库

  cp ~/parcel-repo-6.2.0/* /opt/cloudera/parcel-repo/   
  sha1sum /opt/cloudera/parcel-repo/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel | awk '{ print $1 }' > /opt/cloudera/parcel-repo/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha
  sha1sum /opt/cloudera/parcel-repo/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373-el7.parcel | awk '{ print $1 }' > /opt/cloudera/parcel-repo/GPLEXTRAS-6.2.0-1.gplextras6.2.0.p0.967373-el7.parcel.sha
  

• 初始化cm-server 的数据库

  sh /opt/cloudera/cm/schema/scm_prepare_database.sh -h 192.168.2.131 -P 5432 postgresql cdhdb cdh password
  

• 启动cm-server

  systemctl enable cloudera-scm-server
  systemctl start  cloudera-scm-server
  tail -f200 /var/log/cloudera-scm-server/cloudera-scm-server.log
  

配置CDH集群

1. 打开Cloudera Manager 管理界面 http://192.168.2.131:7180 默认账号:admin 密码：admin
2. 勾选接受许可
3. 选择企业试用版
4. 集群安装–>填写集群名字–>添加agent节点–> 设置本地存储仓库(自定义地址：http://192.168.2.131:8090/cm6)
   （如果设置完选择存储库之后，点击继续，如果无法进行下一步，需要手动拼接下一步 URL地址到浏览器
   参数： step=javaOptionsStep
   ）
5. 勾选安装jdk
6. 设置SSH凭证
7. 等待agent 安装
8. 检查网络和主机环境，所有状态通过检查才表示正常
9. 配置CDH服务，选择自定义服务,可以选择HDFS、Yarn、hive 等最基本的服务，后续其他服务可以添加。
10. 等待集群安装完成正常启动

CDH集群配置lzo 压缩

• 在 HDFS 的配置上添加 lzo 的压缩格式

    com.hadoop.compression.lzo.LzoCodec
    com.hadoop.compression.lzo.LzopCodec
  

• 在 YARN 的配置上添加 classpath

   /opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/*
  

• 检查lzo 配置： hadoop checknative -a

  21/09/03 16:20:38 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
  21/09/03 16:20:38 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
  Native library checking:
  hadoop:  true /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native/libhadoop.so.1.0.0
  zlib:    true /lib64/libz.so.1
  zstd  :  true /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native/libzstd.so.1
  snappy:  true /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native/libsnappy.so.1
  lz4:     true revision:10301
  bzip2:   true /lib64/libbz2.so.1
  openssl: true /lib64/libcrypto.so
  ISA-L:   true /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native/libisal.so.2
  

hive开启分割符配置

• 在CMF 管理界面指定 Hive 的 JAR 辅助目录，路径为：
  /opt/cloudera/parcels/CDH/lib/hive/contrib

如何更改集群节点中的hostname

1. 在节点上修改主机名
2. 修改/etc/hosts 文件 主机名与IP的映射关系
3. 停止服务cm-server、cm-agent
4. 修改cm数据库中的HOSTS表中的name字段

   select host_id, host_identifier, name, ip_address  from HOSTS;
   update HOSTS set name='new-hostname' where host_id=xxx;
   

5. 启动服务cm-server、cm-agent

重置admin用户的密码为admin

use cdhdb;
update USERS set PASSWORD_HASH='2518db3278f704558ece63f469af87a7c12dbad8ff0971d3b677e291d61b44fc', PASSWORD_SALT=-5792243850706636683 WHERE USER_ID = 1;

CDH集群删除重新部署

• server

  systemctl stop  cloudera-scm-server.service 
  rm /var/lib/cloudera-scm-server/* -rf
  数据库删除重新创建并初始化数据库
    
  

• agent

  systemctl stop  cloudera-scm-agent.service
  rm /var/lib/cloudera-scm-agent/* -rf
  
  

• 启动server 和agent

  systemctl start  cloudera-scm-server.service 
   systemctl start cloudera-scm-agent.service