CDH搭建大数据集群环境准备-阿里云主机
首先声明，这篇博文相对比较长，大约需要10 min
1、阿里云官网登录，购买ecs云服务器 www.aliyun.com/
如果你是新用户，需要注册，之后充值100
购买ecs服务器
我们可以看到公网和内网IP，将公网IP配置笔记本上的hosts文件，地址自己去找吧，这里不说了，我配置后的结果：
这里顺便说下停止ecs服务器
2、配置远程连接
我这里使用的远程连接工具是xshell,你们也可以使用crt或者MobaXterm工具
输入用户root和自己设置的密码就OK了
然后以同样的方式进行后两台的连接
至此，3台机器已能够实现远程连接。
3、安装包准备
1）JDK
2）MySQL
3）MySQL jdbc jar包
4）cm tar包
下载地址： http://archive.cloudera.com/cm5/cm/5/
5）parcel文件 3个
下载地址：http://archive.cloudera.com/cdh5/parcels/5.16.2/
cm和parcel官网下载会非常慢，可以考虑使用迅雷
最终结果:
上传安装包到/root/cdh5.16.2，上传前增加滴1台机器的带宽
然后上传，之后再将带宽修改为1M即可。
4、集群节点初始化
1）hosts文件准备（3台都需要配置）
[root@mycluster001 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.170.107 mycluster001
172.16.170.106 mycluster002
172.16.170.105 mycluster0032）防火墙关闭
自己的机房或者IDC托管的机器需要关闭，如果是云主机可以忽略关闭操作，但是云主机有web防火墙
systemctl stop firewalld
systemctl disable firewalld
iptables -F3）关闭selinux,云主机是关闭状态的，如果不是disabled,可以使用下列命令进行关闭
sed -i 's#SELINUX=enable#SELINUX=disabled#g' /etc/selinux/config4）时区/时钟同步(如果机器为云主机，下面操作可忽略)
[root@mycluster001 ~]# timedatectl
Local time: Sat 2020-04-18 11:22:06 CST
Universal time: Sat 2020-04-18 03:22:06 UTC
RTC time: Sat 2020-04-18 11:22:06
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: yes
DST active: n/antp选择第一台节点作为ntp的主节点，其他节点为从节点
yum -y install ntp（3台都需要操作）
vi /etc/ntp.conf（只在第一台机器进行配置）
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不可用，可以使用当前本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些网段的机器来同步时间
restrict 172.16.170.0 mask 255.255.255.0 nomodify notrap
除了001机器，其他都关闭
systemctl stop ntpd
systemctl disable ntpd
[root@mycluster002 ~]# ntpdate mycluster001
18 Apr 11:54:58 ntpdate[2394]: adjust time server 172.16.170.107 offset -0.004965 sec
[root@mycluster003 ~]# ntpdate mycluster001
18 Apr 11:55:23 ntpdate[2383]: adjust time server 172.16.170.107 offset 0.002791 sec在002和003机器配置定时任务
[root@mycluster002 ~]# crontab -e
no crontab for root - using an empty one
00 00 * * * ntpdate mycluster0015）jdk tar包解压部署
（3台均需操作） mkdir /usr/java/ tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java/ cd /usr/java/ [root@mycluster001 java]# ll total 4 drwxr-xr-x 8 10 143 4096 Apr 11 2015 jdk1.8.0_45 [root@mycluster001 java]# chown -R root:root jdk1.8.0_45/ [root@mycluster001 java]# ll total 4 drwxr-xr-x 8 root root 4096 Apr 11 2015 jdk1.8.0_45坑： 上面解压后的jdk属主和属组改为root
vi /etc/profile
source /etc/profile
java -version #查看是否安装成功
6）离线部署MySQL
1.解压及创建目录
[root@mycluster001 cdh5.16.2]# mv mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz.zip.001 /usr/local/
[root@mycluster001 local]# tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
[root@mycluster001 local]# mv mysql-5.7.11-linux-glibc2.5-x86_64 mysql
[root@mycluster001 local]# mkdir mysql/arch mysql/data mysql/tmp
2.创建my.cnf
[root@mycluster001 local]# vi /etc/my.cnf
[client]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4
[mysqld]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
skip-slave-start
skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M
table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600
# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32
#isolation level and default engine
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED
server-id = 1739
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
pid-file = /usr/local/mysql/data/hostname.pid
#open performance schema
log-warnings
sysdate-is-now
binlog_format = ROW
log_bin_trust_function_creators=1
log-error = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7
innodb_write_io_threads=16
relay-log = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info
log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF
# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
#other logs
#general_log =1
#general_log_file = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err
#for replication slave
sync_binlog = 500
#for innodb options
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend
innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M
#根据生产需要，调整pool size
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp
innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2
innodb_locks_unsafe_for_binlog=1
#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on
#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1
[mysqldump]
quick
max_allowed_packet = 128M
[mysql]
no-auto-rehash
default-character-set=utf8mb4
[mysqlhotcopy]
interactive-timeout
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
3.创建用户组及用户
[root@mycluster001 local]# groupadd -g 101 dba
[root@mycluster001 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
根据命令操作结果的提示：
4.copy 环境变量配置文件至mysqladmin用户的home目录中,为了以下步骤配置个人环境变量
[root@mycluster001 local]# cp /etc/skel/.* /usr/local/mysql ###important
5.配置环境变量
[root@mycluster001 local]# vi mysql/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH
unset USERNAME
#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1
## end
6.赋权限和用户组，切换用户mysqladmin，安装
[root@mycluster001 local]# chown mysqladmin:dba /etc/my.cnf
[root@mycluster001 local]# chmod 640 /etc/my.cnf
[root@mycluster001 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@mycluster001 local]# chmod -R 755 /usr/local/mysql
7.配置服务及开机自启动
[root@mycluster001 local]# cd /usr/local/mysql
[root@mycluster001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql
[root@mycluster001 mysql]# chmod +x /etc/rc.d/init.d/mysql
[root@mycluster001 mysql]# chkconfig --add mysql
[root@mycluster001 mysql]# chkconfig --level 345 mysql on
8.安装libaio及安装mysql的初始db
[root@mycluster001 mysql]# yum -y install libaio
[root@mycluster001 mysql]# sudo su - mysqladmin
Last login: Sat Apr 18 12:51:42 CST 2020 on pts/0
mycluster001:mysqladmin:/usr/local/mysql:>bin/mysqld \
> --defaults-file=/etc/my.cnf \
> --user=mysqladmin \
> --basedir=/usr/local/mysql/ \
> --datadir=/usr/local/mysql/data/ \
> --initialize
在初始化时如果加上 –initial-insecure，则会创建空密码的 root@localhost 账号，否则会创建带密码的 root@localhost 账号，密码直接写在 log-error 日志文件中
（在5.6版本中是放在 ~/.mysql_secret 文件里，更加隐蔽，不熟悉的话可能会无所适从）
9.查看临时密码
mycluster001:mysqladmin:/usr/local/mysql:>cd data/
mycluster001:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password
2020-04-18T05:01:49.024813Z 1 [Note] A temporary password is generated for root@localhost: (Oz5geZ31RaX
10.启动
mycluster001:mysqladmin:/usr/local/mysql/data:>/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &
11.登录及修改用户密码
mycluster001:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p'Oz5geZ31RaX'
mysql> alter user root@localhost identified by '000000';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '000000' ;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
12.重启
mycluster001:mysqladmin:/usr/local/mysql/data:>service mysql restart
Shutting down MySQL..2020-04-18T05:08:53.985098Z mysqld_safe mysqld from pid file /usr/local/mysql/data/hostname.pid ended
[ OK ]
Starting MySQL.[ OK ]
[1]+ Done /usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf
mycluster001:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p000000
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
7）添加database并授权
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
mysql> create database cmf default character set utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all privileges on cmf.* to cmf@'%' identified by '000000';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>
mysql> create database amon default character set utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all privileges on amon.* to amon@'%' identified by '000000';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>
mysql> flush privileges
-> ;
Query OK, 0 rows affected (0.00 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| amon |
| cmf |
| mysql |
| performance_schema |
| sys |
+--------------------+
6 rows in set (0.00 sec)8)移动并重命名jdbc连接包
[root@mycluster001 cdh5.16.2]# ls
CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel manifest.json
CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 mysql-connector-java-5.1.47.jar
cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz
[root@mycluster001 cdh5.16.2]# mkdir -p /usr/share/java
[root@mycluster001 cdh5.16.2]# mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
[root@mycluster001 cdh5.16.2]# ll /usr/share/java/ |grep mysql-con
-rw-r--r-- 1 root root 1007502 Mar 2 21:13 mysql-connector-java.jar
至此，cdh安装环境已经准备好。离线安装部署请阅读：https://mp.csdn.net/console/editor/html/105597346
--------------------------
用人品去感动别人，用行动去带动别人，用阳光去照耀别人，用坚持去赢得别人，要求自己每天都去做与目标有关的事情，哪怕每天只进步一点点，坚持下来你就是最优秀卓越的！欢迎大家加入大数据qq交流群：725967421 一起交流，一起进步！！
--------------------------