Cloudera Manager 6.2 构建CDH 集群

 

                                           强大自己是唯一获得幸福的途径，这是长远的，而非当下的玩乐！

---

 

Cloudera Manager	6.2
OS	Centos 7.4
Jdk	1.8

Cloudera Manager	192.168.253.230
cdh-host1	192.168.253.231
cdh-host2	192.168.253.232

 

修改hosts文件

安装工具及关闭防火墙、selinux

yum install -y bash-completion
systemctl stop firewalld
systemctl disable firewalld
setenforce 0

安装ntp

yum install ntp
systemctl start ntpd
systemctl enable ntpd

请确认本机python版本为2.7

配置server免密访问agent

 

配置存储库

wget https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/cloudera-manager.repo -O /etc/yum.repos.d/cloudera-manager.repo
rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera

安装JDK

yum install oracle-j2sdk1.8

安装Cloudera Manager Server

yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
sed -i "s#CMF_JAVA_OPTS=\"-Xmx2G#CMF_JAVA_OPTS=\"-Xmx4G#g" /etc/default/cloudera-scm-server

安装mariadb及配置

yum -y install mariadb-server
echo "更改配置"
cat > /etc/my.cnf << SUCESS
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED

#symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

log_bin=/var/lib/mysql/mysql_binary_log

server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
SUCESS

systemctl start mariadb
systemctl enable mariadb
echo "可参照下图"
/usr/bin/mysql_secure_installation
echo "登入数据库"
mysql -u root -p

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

grant all privileges on *.* to 'root'@'%' identified by '密码' with grant option;
flush privileges;

安装驱动程序

cd /usr/local/src/
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

 

设置Cloudera Manager数据库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root

 确认出现如下字段

Enter SCM password:
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

安装CDH和其他软件

systemctl start cloudera-scm-server.service
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

等待完全启动，出现如下字段

浏览器访问 http://:7180

欢迎页、Accept License同意后、点击继续

Select Edition页根据自己需求选择对应功能版本后继续

 

跳到欢迎页  继续

Cluster Basics 页为集群取名称或默认后   继续

Specify Hosts 页填写hosts文件的主机名，支持[1-9]等简单正则点击搜索，搜索出节点后继续

 

选择存储库 页默认配置    继续

JDK 安装选项选择安装jdk后   继续

 

提供 SSH 登录凭据页 且一定确认server可以保证通过此处后的server可以登录到agent上执行所需操作(root)

        根据自身配置填写私钥或密码等信息后   继续

 

 

Install Agents页 会自动安装agent

 

Install Parcels页 自动安装 无需操作  安装完后 继续

 

Inspect Cluster页 依次点击两次网络和主机检查

有警告，点击检查结果，根据提示操作后，点击重新运行，无报错后  继续

 

 

跳转群集设置

 

Select Services页  根据自身需求选择要安装的服务后(后续还可以加装)  继续

自定义角色分配页  根据自身的需求选择对应服务安装的节点，注(大部分服务都应为奇数个)

可参考：https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_ig_host_allocations.html#host_role_assignments

 

数据库设置页  根据上面设置的数据库信息进行填写，填写完成后，点击测试连接，通过后  继续

审核更改页 默认继续

等待安装完成后 完成