------------------------------------------------------------------------
pkgname | n1 | n2 | n3 | pkgtype
------------------------------------------------------------------------
#### common ####
net-tools (ALL)
gcc (ALL)
perl (ALL)
jdk-8u152-linux-x64.tar.gz (ALL)
zookeeper-server Y Y Y yum
cyrus-sasl ? ? ? yum
cyrus-sasl-devel ? ? ? yum
kafka_2.11-1.0.0.tgz Y Y Y
#### mysql ####
perl Y yum
mysql-community-common-5.7.20-1.el7.x86_64.rpm
Y rpm
mysql-community-libs-5.7.20-1.el7.x86_64.rpm
Y rpm
mysql-community-client-5.7.20-1.el7.x86_64.rpm
Y rpm
mysql-community-server-5.7.20-1.el7.x86_64.rpm
Y rpm
#### hadoop ####
hadoop-hdfs-namenode Y Y N yum
hadoop-hdfs-secondarynamenode* X X X yum (不要安装)
hadoop-hdfs-datanode Y Y Y + yum
hadoop-hdfs-journalnode Y Y Y yum (奇数: 3,5,7,..., 资源消耗低)
hadoop-mapreduce Y Y Y yum
hadoop-hdfs-zkfc Y Y N yum
hadoop-yarn-resourcemanager Y Y N yum (最好单独一台)
hadoop-yarn-nodemanager Y Y Y + yum
hadoop-lzo Y Y Y + yum
hadoop-mapreduce-historyserver Y N N yum
hadoop-libhdfs Y Y N yum
#### hbase ####
hbase-master Y Y N yum
hbase-regionserver Y Y Y yum
#### hive ####
hive-metastore Y N N yum
hive-server2 Y Y Y yum
hive-hbase Y Y Y yum
hive-hcatalog Y Y Y yum
hive-webhcat Y Y Y yum
hive-webhcat-server Y N N yum
#### spark ####
spark-master Y Y N yum
spark-worker Y Y Y yum
spark-history-server Y N N yum
------------------------------------------------------------------------
如上表,对pkgtype为yum的执行下面的命令:
yum install -y pkgname
例如, 对于n1, n2执行(Y - 安装, N - 不安装):
yum install -y hadoop-hdfs-namenode hadoop-hdfs-datanode
yum install -y hadoop-hdfs-datanode
hive remote模式:
hive-metastore 安装在n1上, 需要在n3上安装mysql-server,下面是安装mysql的步骤:
wget ftp://pub:[email protected]/rpm/mysql-community-common-5.7.20-1.el7.x86_64.rpm
wget ftp://pub:[email protected]/rpm/mysql-community-libs-5.7.20-1.el7.x86_64.rpm
wget ftp://pub:[email protected]/rpm/mysql-community-client-5.7.20-1.el7.x86_64.rpm
wget ftp://pub:[email protected]/rpm/mysql-community-server-5.7.20-1.el7.x86_64.rpm
a.首先卸载mariadb
# rpm -qa |grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
# rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
b. 安装mysql-server
# rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm
# rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm
# rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm
# yum install -y perl
# rpm -ivh mysql-community-server-5.7.20-1.el7.x86_64.rpm
c. 启动 mysql
# service mysqld start
或
# systemctl start mysqld
d. 查看安装过程的临时密码:
# sudo grep 'temporary password' /var/log/mysqld.log
YVo)b8Er=g
d. 更改临时密码
# sudo mysql_secure_installation
New password:
Abc1234!
e. 更改mysql数据库文件路径
查看配置文件路径
# mysql --help|grep my.cnf
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf
查看默认数据文件路径:
# mysqladmin -u root -p'Abc1234!' variables | grep datadir
| datadir | /var/lib/mysql/
修改到:/data/mysql
# systemctl stop mysqld.service
# mv /var/lib/mysql /data/
[client]
port = 3306
socket = /data/mysql/mysql.sock #### (必须)
[mysqld]
port = 3306
datadir=/data/mysql
socket=/data/mysql/mysql.sock
再启动mysqld:
# systemctl start mysqld
再次查看数据文件路径:
# mysqladmin -u root -p'Abc1234!' variables | grep datadir
这样就把数据库文件改到/data/mysql下面了。
最后设置mysql开机启动(待测试):
# systemctl enable mysqld.service
在需要访问mysql服务的机器上安装mysql客户端, 这里是在n1上安装:
首先卸载mariadb
# rpm -qa |grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
# rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
安装mysql-client
# rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm
# rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm
# rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm
# mysql -u root -pAbc1234!
mysql: [Warning] Using a password on the command line interface can be insecure.需要在n3上设置mysql可以远程访问, 回到n3上,执行:
登录到mysql:
# mysql -u root -pAbc1234!
设置在n1上可以访问mysql:
mysql > grant all privileges on *.* to 'root'@'cent7-n1.pepstack.com' identified by 'Abc1234!'
mysql > flush privileges;
为hive 创建mysql数据库 hive_metastore (hive-schema-1.1.0.mysql.sql):
在n3 (mysql所在的机器上),执行:
# cd /usr/lib/hive/scripts/metastore/upgrade/mysql/
# mysql -u root -pAbc1234!
为hive创建表:
mysql> create database hive_metastore;
mysql> use hive_metastore;
mysql> source ./hive-schema-1.1.0.mysql.sql;
mysql> quit;
# mysql -u root -pAbc1234!
mysql> create user 'hive'@'cent7-n1.pepstack.com' identified by 'Abc1234!';
mysql> revoke all privileges,grant option from 'hive'@'cent7-n1.pepstack.com';
mysql> grant all privileges on hive_metastore.* to 'hive'@'cent7-n1.pepstack.com';
mysql> flush privileges;
mysql> quit;
# mysql -u hive -pAbc1234! -h cent7-n3.pepstack.com