VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2

安装虚拟机CDH版本,五台虚拟机

一、前期准备配置操作 (root 用户操作)

    1、设置 用户名 配置IP 

        设置虚拟机为静态IP:编辑--> 虚拟网络编辑器 

        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第1张图片

 

        配置 [root@localhost Desktop]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736  的时候对应关系如下

         VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第2张图片

        用户名:[root@localhost Desktop]# vi /etc/sysconfig/network 编辑  HOSTNAME=cdh1

        

        配置 IP 映射:[root@localhost Desktop]# vi /etc/hosts  编辑IP 和 对应的用户名

        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第3张图片

        配置IP:[root@localhost Desktop]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736 

        修改 BOOTPROTO=static

        添加

        GATEWAY=192.168.108.2
        IPADDR=172.20.235.101 
        NETMASK=255.255.255.0

        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第4张图片

        重启network:[root@localhost Desktop]# systemctl restart network

        其他四台做同样的操作 (注**  虚拟机连不上外网请看: https://mp.csdn.net/postlist

    2、禁用IPV6 (所有机器)

        修改配置文件,然后重启:

        vi /etc/default/grub 添加 禁用ipv6:ipv6.disable=1

        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第5张图片

        重新生成配置:/boot/grub2/grub.cfg

        grub2-mkconfig -o /boot/grub2/grub.cfg

        重启机器:reboot 

        查看ipv6是否禁用成功:lsmod|grep ipv6

    3、SSH免密码登录

        3.1、创建一对公钥文件(~/.ssh/id_rsa.pub)和私钥文件 (~/.ssh/id_rsa

         [root@cdh1 anni]# cd ~/.ssh/

         [root@cdh1 .ssh]# ssh-keygen -t rsa  创建公钥和私钥

        3.2、在其他从节点上执行ssh (ssh 用户名 如:ssh cdh2) 创建 ~/.ssh 文件夹

        3.3、复制公钥到其它节点上(所有节点都要执行

 

                VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第6张图片

                scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/

 

                scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/
                scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/

 

                scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/

        3.4、生成文件 authorized_keys (在其他从节点上)

            cat id_rsa.pub >> authorized_keys

        3.5、然后把文件 authorized_keys 复制到主节点上

            scp /root/.ssh/authorized_keys [email protected]:/root/.ssh/

        3.6、修改主机配置文件 /etc/ssh/sshd_config

            VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第7张图片

             启动ssh服务: /bin/systemctl start sshd.service

            测试:ssh cdh5   退出“exit”

            [root@cdh3 .ssh]# service sshd restart 重启sshd 服务

        关于CRT连接虚拟机的时候出错:参考链接:https://blog.csdn.net/pansanday/article/details/75265951

        SecureCRT A public key file has not been specified by this session...   

 

 

 之前连接linux服务器时, 都是输入用户名和密码进行登录的. 后因安全问题, 改用了Agent keys的方式进行登录. 管理员提供了一个 *.pub的文件, 说是导入之后不用输入密码即可进行连接.

 

 

SecureCRT的Tools -> Manage Agent Keys -> 点击右侧的Add... -> 选择提供的 *.pub文件, 再次登录即可.

 

 

但问题来了, 每次关闭了SecureCRT, 再次连接这个服务器时, 又要求导入, 具体信息如下:

 

 

先是弹出: All available GSSAPI mechanisms failed.

 

 

 

 

确定之后, 报出了: A public key file has not been specified for this session. Would you like to specify one now?

 

 

 

 

选择是之后, 就又要求导入key了...

 

 

后在网上搜索, 网上给的解决方案是: 修改/etc/ssh/sshd_config, 将PasswordAuthentication项改为yes,重启sshd服务(service sshd restart)即可

 

 

但是因为安全原因, 是不可能改为yes的, 后来经过大神指点, 修改了secureCRT的配置, 解决了每次都需要选择key的问题:

 

 

解决方案:

 

 

右击连击 -> Properties -> 在左侧的Category中, 依次选择Connection / SSH2 , 在右侧的Authentication中, 取消勾选 Password, Keyboard Interactive, GSSAPI -> OK即可

 

        xshell不能通过密码连接虚拟机:

                VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第8张图片

修改配置:/etc/ssh/sshd_config

# Authentication:
#LoginGraceTime 2m
PermitRootLogin yes
StrictModes no
#MaxAuthTries 6

#MaxSessions 10

注释掉:PermitRootLogin yes
             StrictModes no

 

        服务器连不上解决办法:Restarting network (via systemctl):  Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details.

        参考:centos7 无法启动网络(service network restart)错误解决办法  

         链接:https://blog.csdn.net/zkja595470467/article/details/53007915

    3.7、关闭防火墙和selinux (所有节点)

        参考:https://blog.csdn.net/spacefortress/article/details/77141208

 

CentOS7关闭防火墙和SELinux

关闭防火墙

临时关闭防火墙

systemctl stop firewalld

临时打开防火墙

systemctl start firewalld

防火墙开机关闭

systemctl disable firewalld

防火墙开机启动

systemctl enable firewalld

查看防火墙状态

systemctl status firewalld

关闭selinux

临时关闭SELinux

setenforce 0

临时打开SELinux

setenforce 1

开机关闭SELinux

vi /etc/selinux/config
修改"SELINUX=disabled"为"SELINUX=disabled"

查看SELinux状态

getenforce

 

    3.8、安装jdk并设置java_home(所有节点)

 

            使用CentOS7卸载自带jdk安装自己的JDK1.8

            参考链接:https://blog.csdn.net/magicianjun/article/details/78530129

            检查系统自带的安装包:

            VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第9张图片

            卸载自带的java:

                rpm -e --nodeps java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64
                rpm -e --nodeps java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64
                rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64
                rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64

            安装下载好的java:

 

                mkdir -p /usr/java

                上传tar包并解压:

            

            配置环境变量:

                    echo 'export JAVA_HOME=/usr/java/jdk1.8.0_172/' >> /etc/profile
                    echo 'export JRE_HOME=${JAVA_HOME}/jre' >> /etc/profile
                    echo 'export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib' >> /etc/profile
                    echo 'export PATH=${JAVA_HOME}/bin:$PATH' >> /etc/profile
                    生效配置文件:source /etc/profile
                    查看Java版本:java -version

    3.9、系统时间同步(所有节点)

            参考链接:https://www.cnblogs.com/rwxwsblog/p/5799764.html

                             https://blog.csdn.net/yuanfang_way/article/details/53959591

            安装ntp:yum install ntp -y

            设置中国时间:timedatectl set-timezone Asia/Shanghai

            将ntp设置为缺省启动:systemctl enable ntpd

            修改启动参数,增加-g -x参数,允许ntp服务在系统时间误差较大时也能正常工作:

                    echo 'OPTIONS="-x"' >> /etc/sysconfig/ntpd

            启动ntp服务:systemctl restart ntpd

            配置服务节点:vi /etc/ntp.conf

                             server 节点添加:127.127.1.0 代表自身

                            

                            client 节点添加:192.168.140.101 服务节点的IP

                            

            重启ntpd服务器:systemctl restart ntpd

            在client节点同步server节点的时间:ntpdate 192.168.140.101

            client节点启动ntpd服务:systemctl start ntpd

                                                    systemctl enable ntpd

            所有节点启动时间同步:timedatectl set-ntp yes

     4.0、设置用户文件打开数目、最大进程数、内存(所有节点)

                参考:https://blog.csdn.net/wangkai_123456/article/details/78481698

                修改nr_open的值为:10240000

                        添加参数:echo "fs.nr_open=10240000" >> /etc/sysctl.conf

                        生效配置文件:sysctl -p

                 修改/etc/systemd/system.conf,主要用于systemd service的资源限制

 

                        添加参数:echo "DefaultLimitCORE=infinity" >> /etc/systemd/system.conf

 

                                         echo "DefaultLimitNOFILE=10240000" >> /etc/systemd/system.conf
                                         echo "DefaultLimitNPROC=10240000" >> /etc/systemd/system.conf

 

                修改/etc/security/limits.conf文件:

                    echo "root             soft    fsize           unlimited" >> /etc/security/limits.conf
                    echo "root             hard    fsize           unlimited" >> /etc/security/limits.conf
                    echo "root             soft    cpu           unlimited" >> /etc/security/limits.conf
                    echo "root             hard    cpu           unlimited" >> /etc/security/limits.conf
                    echo "root             soft    as           unlimited" >> /etc/security/limits.conf
                    echo "root             hard    as           unlimited" >> /etc/security/limits.conf
                    echo "root             soft    nofile           10240000" >> /etc/security/limits.conf
                    echo "root             hard    nofile           10240000" >> /etc/security/limits.conf
                    echo "root             soft    nproc           10240000" >> /etc/security/limits.conf
                    echo "root             hard    nproc           10240000" >> /etc/security/limits.conf

            修改/etc/security/limits.d/20-nproc.conf文件:

                    修改
                    *          soft    nproc     4096
                    为
                    *          soft    nproc     10240000

    4.1、安装MySQL (查了好多文档终于搞定了 哈哈)

            https://blog.csdn.net/kwame211/article/details/75351394

            https://blog.csdn.net/bao19901210/article/details/51917641

            

          CentOS7下安装Mysql失败经历--CentOS7使用yum安装和卸载Mysql过程

            https://www.cnblogs.com/Lenbrother/articles/6203620.html

            

        CentOS 7下彻底卸载MySQL数据库

            https://zhangzifan.com/centos-7-remove-mysql.html

            [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it

        https://blog.csdn.net/jd_bai2016/article/details/68070076

            

mysql cluster (mysql 集群)安装配置方案   :https://www.cnblogs.com/jedy/p/4422547.html

            /usr/local/mysql/bin/mysqld -f /data/mysql/mysql-cluster/config.ini

            /usr/local/mysql/bin/ndbd --initial --ndb-connectstring=192.168.140.201

/usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data

            安装步骤: 参考连接:https://blog.csdn.net/yabingshi_tech/article/details/47952135

            机器五台:

            192.168.140.201    管理节点   sql节点
            192.168.140.202    数据节点
            192.168.140.203    数据节点
            192.168.140.204    sql节点

            192.168.140.205    sql节点

            1.1 准备好下载的mysql5.7 : mysql-cluster-gpl-7.5.10-linux-glibc2.12-x86_64.tar.gz

            1.2  所有节点执行

                    mkdir -p /data/mysql/mysql-cluster/ 

                    上传 mysql-cluster-gpl-7.5.10-linux-glibc2.12-x86_64.tar.gz 到目录 /data/mysql/mysql-cluster/ 并解压

                    mv /data/mysql/mysql-cluster/mysql-cluster-gpl-7.5.10-linux-glibc2.12-x86_64 /usr/local/mysql

                   

  1. #新建mysql用户及用户组,授权  
  2. #groupadd mysql   
  3. #useradd -g mysql -s /usr/sbin/nologin mysql  
  4. #chown -R mysql:mysql /usr/local/mysql 

                    chown -R mysql:mysql /usr/local/mysql/*

             1.3 管理节点配置和启动:192.168.140.201

                   全局配置:

                    vi /data/mysql/mysql-cluster/config.ini     配置信息如下

                        [ndbd default]
                        NoOfReplicas=2
                        
                        DataMemory=128M
                        IndexMemory=18M
                        
                        [ndb_mgmd]
                        NodeId=11
                        hostname=192.168.140.201
                        datadir=/data/mysql/mysql-cluster
                        
                        [ndbd]
                        NodeId=21
                        hostname=192.168.140.202
                        datadir=/usr/local/mysql/data
                        
                        [ndbd]
                        NodeId=22
                        hostname=192.168.140.203
                        datadir=/usr/local/mysql/data
                        
                        [mysqld]
                        NodeId=31
                        hostname=192.168.140.201
                        
                        [mysqld]
                        NodeId=32
                        hostname=192.168.140.204
                        
                        [mysqld]
                        NodeId=33
                        hostname=192.168.140.205
                        
                        [mysqld]

                    注:NoOfReplicas表示数据节点的冗余数量。

如果是2 表有两份数据,如果是1表示只有一份数据. 当NoOfReplicas=2时, 你可以试验一下如果一个data node 停掉了,cluster 可以照常工作; 如果NoOfReplicas=1, 一个data node 停掉了,cluster 就会自动停掉.

                    启动MGM node 刷新配置 :/data/mysql/mysql-cluster/config.ini 

                    --/usr/local/mysql/bin/mysqld -f /data/mysql/mysql-cluster/config.ini

                    1./usr/local/mysql/bin/ndb_mgmd -f /data/mysql/mysql-cluster/config.ini --initial

2./usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data

                     执行如下图的步骤

                           注:1报的错 跟用户有关 ,mysql为了安全起见不希望使用root启动,最好的方式还是创建一个管理MySQL的用户启动。

                                    2检查进程是否启动

                                    3查看管理节点的全局配置是否起作用

                        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第10张图片

                         错误信息:

                              

                        解决方法:修改配置/etc/my.cnf  把 sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES 注释掉。

                1.4 数据节点配置启动

                      先把/etc/my.cnf文件先删除,然后再创建的。

                      编辑信息如下:

                        [mysqld]
                        ndbcluster
                        ndb-connectstring=192.168.140.201
                        [mysql_cluster]
                        ndb-connectstring=192.168.140.201

                    初始化启动数据节点:mkdir -p /usr/local/mysql/data

                                        /usr/local/mysql/bin/ndbd --initial --ndb-connectstring=192.168.140.201

                        

                    通过管理节点查看状态:

                        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第11张图片

                    1.5 sql节点配置

                          先把/etc/my.cnf文件先删除,然后再创建的。

                                编辑信息如下:

                                    主节点:

[client]
socket=/usr/local/mysql/mysql.sock
[mysql_cluster]
ndb-connectstring=192.168.140.101
[mysqld]
socket=/usr/local/mysql/mysql.sock
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
ndbcluster
ndb-connectstring=192.168.140.101
user=mysql
skip-name-resolve
#skip-grant-tables
# Recommended in standard MySQL setup
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appr
opriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
server-id=1
# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES

                                    从节点

                                    [client]
                                    socket=/usr/local/mysql/mysql.sock
                                    [mysqld]
                                    basedir=/usr/local/mysql
                                    datadir=/usr/local/mysql/data
                                    ndbcluster
                                    ndb-connectstring=192.168.140.201
                                    [mysql_cluster]

                                    ndb-connectstring=192.168.140.201

                        复制:cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysql 方便mysql启动

                         创建目录并赋权:mkdir -p /usr/local/mysql/data , chown -R mysql:mysql /usr/local/mysql/*

                                                     mkdir -p /var/lib/mysql/, chown -R mysql:mysql /var/lib/mysql/

                    初始化sql节点:  /usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data

                          启动mysql:service mysql restart

                                

                               -- update user set authentication_string=password('123456') where user='root';

                            --SET PASSWORD = PASSWORD('123456'); 

                                --grant all privileges on *.* to root@'%' identified by '***831';

                    回到管理节点查看状态:

                                VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第12张图片 

 

                               初始化数据库:mysql_secure_installation

        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第13张图片

                            

                           准备附加数据库:

                            

1)附加数据库列表

Role Database User Password
Activity Monitor amon amon amon_password
Reports Manager rman rman rman_password
Hive Metastore Server metastore hive hive_password
Sentry Server sentry sentry sentry_password
Cloudera Navigator Audit Server nav nav nav_password

Cloudera Navigator Metadata Server

navms navms navms_password

2)创建数据库并配置管理账号密码

mysql -uroot -p
create database amon default character set utf8;
grant all privileges on amon.* to 'amon'@'%' identified by 'amon_password';

create database rman default character set utf8;
grant all privileges on rman.* to 'rman'@'%' identified by 'rman_password';

create database metastore default character set utf8;
grant all privileges on metastore.* to 'hive'@'%' identified by 'hive_password';

create database sentry default character set utf8;
grant all privileges on sentry.* to 'sentry'@'%' identified by 'sentry_password';

create database nav default character set utf8;
grant all privileges on nav.* to 'nav'@'%' identified by 'nav_password';

create database navms default character set utf8;
grant all privileges on navms.* to 'navms'@'%' identified by 'navms_password';

flush privileges;

 

create database hue default character set utf8;
grant all privileges on hue.* to 'hue'@'%' identified by 'huee_password';

 

flush privileges;

create database hive default character set utf8;
grant all privileges on hive.* to 'hive'@'%' identified by 'hive_password';
flush privileges;

                        错误信息:

                                

                        解决办法:

 

这个其实与validate_password_policy的值有关。

validate_password_policy有以下取值:

Policy Tests Performed
0 or LOW Length
1 or MEDIUM Length; numeric, lowercase/uppercase, and special characters
2 or STRONG Length; numeric, lowercase/uppercase, and special characters; dictionary file

默认是1,即MEDIUM,所以刚开始设置的密码必须符合长度,且必须含有数字,小写或大写字母,特殊字符。

有时候,只是为了自己测试,不想密码设置得那么复杂,譬如说,我只想设置root的密码为123456。

必须修改两个全局参数:

首先,修改validate_password_policy参数的值

mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)

这样,判断密码的标准就基于密码的长度了。这个由validate_password_length参数来决定。

 

                        配置oozie库

 

                            VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第14张图片

create database oozie default character set utf8;
grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie_password';
grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie_password';
flush privileges;

                                配置oozie软连接

                             mkdir -p /opt/cloudera/parcels/CDH/lib/oozie/lib/

                             cd /opt/cloudera/parcels/CDH/lib/oozie/lib/

              ln -s /usr/share/java/mysql-connector-java.jar mysql-connector-java.jar

 

                        1.6 配置path

 

                                在各节点配置PATH

                                vi /root/.bash_profile

                                在PATH这一行后面,添加/usr/local/mysql/bin

                                如:PATH=$PATH:$HOME/bin:/usr/local/mysql/bin

                                        source /root/.bash_profile

                        1.7 重置密码解决MySQL for Linux错误 ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

                            https://www.cnblogs.com/gumuzi/p/5711495.html

                4.2、安装Cloudera Manager

                            参考链接链接:http://blog.51cto.com/cmdschool/1916322(集群安装)

                        添加yum源:(所有节点)

curl https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo > /etc/yum.repos.d/cloudera-manager.repo

           查看cdh配置信息: /etc/yum.repos.d/cloudera-manager.repo

                        添加配置:vim /etc/yum.repos.d/mysql-community.repo (所有节点)

[mysql56-community]
name=MySQL 5.6 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.6-community/el/7/$basearch/
enabled=1 --开启
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
 
[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/
enabled=0 --关闭
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

                        安装Cloudera Manager包 (主节点)

                                    yum install -y cloudera-manager-daemons cloudera-manager-server

                        安装mysql (主节点)

                                    mkdir -p /etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

                yum install -y mysql-community-server mysql-community-devel mysql-community-client mysql-community-libs mysql-community-common mysql-community-libs-compat

                        安装Cloudera Manager Agent包 (所有节点)

                                    yum install -y cloudera-manager-agent cloudera-manager-daemons

          检查线程配置(所有节点)

vim /etc/security/limits.d/cloudera-scm.conf

#
# (c) Copyright 2014 Cloudera, Inc.
#
cloudera-scm    soft  nofile  32768
cloudera-scm    soft  nproc   65536
cloudera-scm    hard  nofile  1048576
cloudera-scm    hard  nproc   unlimited
cloudera-scm    hard  memlock unlimited
cloudera-scm    soft  memlock unlimited

                            创建压缩包存放目录(所有节点)         

 

                mkdir -p /opt/cloudera/parcels

                chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

 

                           配置无密码登陆sudo访问(所有节点) visudo

 

                                增加如下配置:

                                            %cloudera-scm        ALL=(ALL)       NOPASSWD: ALL

                                         确认包含:Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

            配置su的limit (所有节点)

                   echo 'session         required        pam_limits.so' >> /etc/pam.d/su

                            安装MySQL JDBC Driver (所有节点)

                                    cd /usr/share/java/

                                    wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.39.tar.gz

                                    tar -xzvf mysql-connector-java-5.1.39.tar.gz

                                    cp mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar mysql-connector-java.jar

                            

                            准备scm库

                                VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第15张图片

create database scm default character set utf8;
grant all privileges on *.* to scm@'cdh1' identified by 'scm_password';
flush privileges;

                                配置scm库连接参数:

                cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.properties.default

                                    vi db.properties 添加如下配置(主节点):

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cdh1
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=scm_password

com.cloudera.cmf.db.setupType=EXTERNAL

                            启动服务: /etc/init.d/cloudera-scm-server start

            开机自启:chkconfig cloudera-scm-server on

                            故障排除:tail -f /var/log/cloudera-scm-server/cloudera-scm-server.out                        

                                                1.1 错误分析:java_home找不到  

                                                        解决方法:在/etc/init.d/cloudera-scm-server最开始加入java_home的环境变量

                     1.2 错误分析:找不到mysql的jdbc驱动

                         解决方法:建立软连接或者复制MySQL的jdbc驱动到目录:/usr/share/cmf/lib  权限给777

问题:java.sql.SQLException: Access denied for user 'scm'@'192.168.140.101' (using password: YES)

VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第16张图片

参考连接:https://blog.csdn.net/lxf20054658/article/details/73695469/

解决办法:重新设置scm用户密码并保存

 

use mysql;

desc user;

drop user 'scm'@'%';

CREATE USER 'scm'@'%' IDENTIFIED BY 'scm_password';

GRANT ALL PRIVILEGES ON *.* TO scm@"%" IDENTIFIED BY "scm_password";

update user set authentication_string = password('scm_password') where User='scm';

select Host,User,authentication_string  from user where User='scm'; 

 

flush privileges;
quit;

 

              启动服务ok:

                    

              指定管理服务器和指定包存放的位置(所有节点):vim /etc/cloudera-scm-agent/config.ini

                   确认添加以下配置:

 

                    server_host=cdh1

                    server_port=7182

                    parcel_dir=/opt/cloudera/parcels

            启动服务并配置开机自启(所有节点):

                    

                    /etc/init.d/cloudera-scm-agent start

                    chkconfig cloudera-scm-agent on

            故障排除:tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.out

 

 

            CDH安装时,无法纳管全部的节点的一个bug

 

                https://blog.csdn.net/eyoulc123/article/details/52214190

修改cdh5集群中主机节点IP或hostName 变成本地的情况:

第一步,关闭服务

  1. 关闭集群服务,及 Cloudera Management Service,
  2. 关闭cm服务:( cm安装节点 )

    命令 : service cloudera-scm-server stop

         service cloudera-scm-agent stop( 没有可忽略 )

 第二步:登录MySQL数据库修改 HOSTS表中的数据

  mysql -u root -p *****

 进入scm库 找到hosts表修改数据:

VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第17张图片

update HOSTS set NAME='cdh5',IP_ADDRESS='192.168.140.105' where HOST_ID=5;

update HOSTS set NAME='cdh2',IP_ADDRESS='192.168.140.102' where HOST_ID=2;

commit;退出

CDH安装步骤:

参考:https://blog.csdn.net/qq_37667074/article/details/78731529

 

        4.3、登录  哈哈

            VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第18张图片

            根据提示到如下页面:选择主机cdh1

            VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第19张图片

            VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第20张图片

              VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第21张图片

                VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第22张图片

 

有一个问题:

运行

echo 10 > /proc/sys/vm/swappiness

要永久改变

vim   /etc/sysctl.conf

vm.swappiness=10

 

echo never>/sys/kernel/mm/transparent_hugepage/defrag

 

echo never>/sys/kernel/mm/transparent_hugepage/enabled

 

 

 

接下来点击继续然后选择安装的服务,及数据库配置就行了。

    VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第23张图片

    VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第24张图片

        VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第25张图片

VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第26张图片

VMware-workstation14 PRO 安装配置Centos7 安装CDH5.14.2_第27张图片

 

 

 

 

 

你可能感兴趣的:(#,Hadoop安装)