在centos7 安装 CDH6.0.1

在centos7 安装 CDH6.0.1

官方安装教程
https://www.cloudera.com/documentation/enterprise/6/6.0/topics/installation.html

相关包下载

Cloudera Manager下载地址:
https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64
cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm

jdk
oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

以上安装包自己放在能找得到的位置,个人是放在/opt目录下。

CDH安装包下载地址:https://archive.cloudera.com/cdh6/6.0.1/parcels/,需要下载以下文件:
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256
manifest.json

以上三个文件放到/opt/cloudera/parcel-repo/目录下,如果没有此目录,可以自己手动创建。

网速好的话,也可以在线安装的。不需要提前下载这些安装包。

介绍

是在个人PC机上,用虚拟机来装CDH的。毕竟服务器那玩意儿暂时太贵,玩不起。何况也是自己拿来熟悉CDH平台的,也没有必要买云服务器。

安装前的准备

  1. 环境:Centos7 64位, CDH6.0.1;
  2. 至少准备三台虚拟机;

    192.168.237.145 cdh.145.sky Server+agent
    192.168.237.146 cdh.146.sky agent
    192.168.237.147 cdh.147.sky agent

  3. 建议:

    3台机器,内存最好不低于10G,硬盘空间最好大于50G。
    内存太低了的话,后面在启动集群过程中,可能会显示out of memory而导致组件启动失败。

开始安装

  1. 配置 hostname(完全限定的域名(FQDN))以及免密登陆(所有主机)

    sudo hostnamectl set-hostname cdh.145.sky

  2. 编辑 /etc/hosts 文件使用群集中每个主机的IP地址和完全限定的域名(FQDN)

    192.168.237.145 cdh.145.sky
    192.168.237.146 cdh.146.sky
    192.168.237.147 cdh.147.sky

  3. 配置免密登陆(所有主机)

    ssh-keygen
    ssh-copy-id [email protected]

  4. 禁用防火墙和selinux(所有主机)

    查看防火墙:systemctl status firewalld.service 绿的表示防火墙开启。
    执行关闭命令:systemctl stop firewalld.service
    执行开机禁用防火墙命令:systemctl disable firewalld.service

    关闭selinux:
    vim /etc/sysconfig/selinux
    SELINUX=enforcing 改为 SELINUX=disabled

  5. 安装Java(所有主机)

    centos7 自带OpenJdk,我们需要卸载并且安装Oracle的。
    在下载 Cloudera Manager 包的时候,里边包含Java的rpm包,直接用这个进行默认安装就可以。
    oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

    网速好的话,也可以在线安装
    yum install oracle-j2sdk1.8

    默认安装路径
    /usr/java/jdk1.8.0_141-cloudera

  6. 安装NTP(所有主机)

    1. yum -y install ntp
      编辑 /etc/ntp.conf 文件添加NTP服务器
      server 0.pool.ntp.org
      server 1.pool.ntp.org
      server 2.pool.ntp.org

    2. 启动ntp
      systemctl enable ntpd
      systemctl start ntpd

    3. 同步节点
      ntpdate -u 0.pool.ntp.org

    4. 同步系统时钟
      hwclock --systohc

  7. 安装cloudera manager包

    Master节点:
    yum -y install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server

    Node节点:
    yum -y install cloudera-manager-agent cloudera-manager-daemons

    本地安装:
    (网络好的话,可以忽略此步骤。)
    cloudera-manager-agent-6.0.1-610811.el7.x8664.rpm
    cloudera-manager-daemons-6.0.1-610811.el7.x8664.rpm
    cloudera-manager-server-6.0.1-610811.el7.x8664.rpm
    先安装 cloudera-manager-daemons-6.0.1-610811.el7.x8664.rpm 再安装其他的。

  8. 启用Auto-TLS以自动创建证书:(Master)

    JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services

  9. 安装mysql(Master)

    wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
    yum -y install mysql-community-release-el7-5.noarch.rpm
    yum -y update
    yum -y install mysql-server

    *.Move old InnoDB log files /var/lib/mysql/iblogfile0 and /var/lib/mysql/iblogfile1 out of /var/lib/mysql/ to a backup location.
    rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1

    1. 启动mysql
      systemctl enable mysqld
      systemctl start mysqld

    2. 初始化Mysql
      /usr/bin/mysql_secure_installation

    [...]
    Enter current password for root (enter for none):
    OK, successfully used password, moving on...
    [...]
    Set root password? [Y/n] Y
    New password:root
    Re-enter new password:root
    Remove anonymous users? [Y/n] Y
    [...]
    Disallow root login remotely? [Y/n] N
    [...]
    Remove test database and access to it [Y/n] Y
    [...]
    Reload privilege tables now? [Y/n] Y
    All done!

  10. 安装MySQL JDBC驱动程序

    wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

    tar zxvf mysql-connector-java-5.1.46.tar.gz

    cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

  11. 必须配置数据库以支持MySQL UTF8 字符集编码。

    mysql -u root -p root

    create database scm default character set utf8 default collate utf8_general_ci;
    grant all on scm.* to 'scm'@'%' identified by 'root';
    create database amon default character set utf8 default collate utf8_general_ci;
    grant all on amon.* to 'amon'@'%' identified by 'root';
    create database rman default character set utf8 default collate utf8_general_ci;
    grant all on rman.* to 'rman'@'%' identified by 'root';
    create database hue default character set utf8 default collate utf8_general_ci;
    grant all on hue.* to 'hue'@'%' identified by 'root';
    create database metastore default character set utf8 default collate utf8_general_ci;
    grant all on metastore.* to 'hive'@'%' identified by 'root';
    create database sentry default character set utf8 default collate utf8_general_ci;
    grant all on sentry.* to 'sentry'@'%' identified by 'root';
    create database nav default character set utf8 default collate utf8_general_ci;
    grant all on nav.* to 'nav'@'%' identified by 'root';
    create database oozie default character set utf8 default collate utf8_general_ci;
    grant all on oozie.* to 'oozie'@'%' identified by 'root';
    flush privileges;

  12. 禁用透明大页面压缩

    echo never > /sys/kernel/mm/transparent_hugepage/defrag
    echo never > /sys/kernel/mm/transparent_hugepage/enabled

    并将上面的两条命令写入开机自启动
    vim /etc/rc.local

  13. 优化交换分区

    vim /etc/sysctl.conf

    vm.swappiness = 10

    sysctl -p /etc/sysctl.conf

  14. 设置Cloudera Manager数据库

    mysql在本地时执行:
    rm -rf /etc/cloudera-scm-server/db.mgmt.properties
    /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

  15. 启动Cloudera Manager Server

    设置Cloudera Manager数据库后,启动Cloudera Manager Server,然后登录Cloudera Manager Admin Console:

    启动Cloudera Manager Server:
    systemctl start cloudera-scm-server

    可以查看启动日志
    tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

    看到此条信息,说明启动完成
    INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

    将CHD6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中,如果没有此目录,可以自己创建。(如果是在线安装,那么此步骤可以忽略。)

    注意:
    最后将CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256,重命名为CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha,这点必须注意,否则系统会重新下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel。

  16. 登录

    在Web浏览器中,转到 http://cdh.145.sky:7180,其中域名是运行Cloudera Manager Server的主机的FQDN或IP地址。

    如果有启用自动TLS,则会重定向到https:cdh.145.sky:7183,并显示安全警告。你可能需要表明你信任该证书,或单击以继续执行Cloudera Manager Server主机。

    登录Cloudera Manager Admin Console。默认凭据为:
    Username: admin
    Password: admin

  17. 在CM管理界面,按照操作一步一步进行操作即可。

    安装完成界面:
    在centos7 安装 CDH6.0.1_第1张图片

  18. 如果部署失败,可以卸载删除Cloudera Manger环境,重新部署。

    systemctl stop cloudera-scm-server
    systemctl stop cloudera-scm-agent

    yum -y remove 'cloudera-manager-*'

    yum clean all

    umount cm_processes
    umount /var/run/cloudera-scm-agent/process

    rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*
    rm -rf /tmp/.scmpreparenode.lock
    rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
    rm -Rf datadrivepath/dfs datadrivepath/mapred datadrivepath/yarn

    systemctl stop mysqld

    yum -y remove mysql
    rm -rf /var/lib/mysql
    rm -rf /var/log/mysqld.log
    rm -rf /var/lib/mysql/mysql
    rm -rf /usr/lib64/mysql
    rm -rf /usr/share/mysql
    rm -rf /opt/cloudera

    rpm -qa | grep -i mysql
    卸载MySQL相关的文件。

写在结尾

文章若有不对之处,敬请谅解,欢迎指出,共同学习进步。

你可能感兴趣的:(在centos7 安装 CDH6.0.1)