Docker 构建 Hive 1.2.2 单节点

1. 借鉴

Hadoop集群添加新节点
为什么Hive里，要用mysql?
centos7下安装mysql5.7（rpm）
error: Failed dependencies: perl(Data::Dumper) is needed by MySQL-server-5.6.46-1.el7.x8
如何在DOCKER容器中运行MySQL服务
rpm 完全卸载mysql
Tez 0.9安装部署+hive on tez配置 + Tez-UI

2. 开始

我们的节点规划如下：

hive01[172.173.16.22]

镜像准备

hive是依托于hadoop的，所以hive机器上需要hadoop环境

1. docker hub 下载

  docker pull caiserkaiser/hadoop:2.7.2

2. 构建
   caiser/hadoop:2.7.2 镜像

创建自定义网络

Docker 网络操作

docker network create -d bridge --subnet "172.173.16.0/24" --gateway "172.173.16.1"  datastore_net

启动容器

docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hadoop:2.7.2

下载并配置hive

1. 下载hive

2. 拷贝到容器内

   docker cp ~/Downloads/apache-hive-1.2.2-bin.tar.gz e2d21b78fdb0:/opt/envs
   

3. 解压并重命名

   mkdir hive-1.2.2 && tar -zxvf apache-hive-1.2.2-bin.tar.gz -C hive-1.2.2 --strip-components 1
   

配置

hive-env.sh

• a. 备份

  cp /opt/envs/hive-1.2.2/conf/hive-env.sh.template /opt/envs/hive-1.2.2/conf/hive-env.sh
  

• b. 编辑hive-env.sh

   vi /opt/envs/hive-1.2.2/conf/hive-env.sh
  

• c. 配置

  ①. 配置 HADOOP_HOME

   export HADOOP_HOME=/opt/envs/hadoop-2.7.2
  

  ②. 配置 HIVE_CONF_DIR

  export HIVE_CONF_DIR=/opt/envs/hive-1.2.2/conf
  

安装which

不安装which，启动datanode时会报找不到文件的错误

yum install which

安装mysql

为啥需要mysql

[借鉴引用]
你在哪路径下，执行hive指令，就在哪路径下生成metastore_db建一套数据库文件，这样是极其不合适的，公司里每个人若不一样，则会显得非常混杂。导致员工之间无法公用交流，也不便于管理。

• a . 重要如果安装了mysql，先删除，再重新安装

   rpm -qa | grep -i mysql
   rpm -ev --nodeps xxx
  
   find / -name mysql
   find / -name my.cnf
  
   rm -rf  xxx/mysql
   rm -rf /etc/my.cnf
  

• b. 下载mysql

   MySQL-client-5.6.48-1.el7.x86_64.rpm
   MySQL-server-5.6.48-1.el7.x86_64.rpm
  

• c. 拷贝到容器

   docker cp ~/Downloads/MySQL-client-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs
   docker cp ~/Downloads/MySQL-server-5.6.48-1.el7.x86_64.rpm e2d21b78fdb0:/opt/envs
  

• d. 安装mysql服务端

  ① 安装服务端

  rpm -ivh MySQL-server-5.6.48-1.el7.x86_64.rpm
  

  ② 出现问题:

  /usr/bin/perl is needed by MySQL-server-5.6.48-1.el7.x86_64
  libaio.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
  libaio.so.1(LIBAIO_0.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
  libaio.so.1(LIBAIO_0.4)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
  libnuma.so.1()(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
  libnuma.so.1(libnuma_1.1)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
  libnuma.so.1(libnuma_1.2)(64bit) is needed by MySQL-server-5.6.48-1.el7.x86_64
  net-tools is needed by MySQL-server-5.6.48-1.el7.x86_64
  perl(Data::Dumper) is needed by MySQL-server-5.6.48-1.el7.x86_64
  

  ③ 安装以下依赖

   yum install -y perl perl-devel autoconf libaio numactl net-tools iproute
  

  ④ server安装完成[看到以下内容证明完成了]

  A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
  You will find that password in '/root/.mysql_secret'.
  
  You must change that password on your first connect,
  no other statement but 'SET PASSWORD' will be accepted.
  See the manual for the semantics of the 'password expired' flag.
  
  Also, the account for the anonymous user has been removed.
  
  In addition, you can run:
  
  /usr/bin/mysql_secure_installation
  
  which will also give you the option of removing the test database.
  This is strongly recommended for production servers.
  
  See the manual for more instructions.
  
  Please report any problems at http://bugs.mysql.com/
  
  The latest information about MySQL is available on the web at
  
    http://www.mysql.com
  
  Support MySQL by buying support/licenses at http://shop.mysql.com
  
  New default config file was created as /usr/my.cnf and
  will be used by default by the server when you start it.
  You may edit this file to change server settings
  

• e. 查看随机生成的密码

  cat /root/.mysql_secret
  

• f. 启动mysql

  ① 启动

  /etc/init.d/mysql start
  或 service mysql start
  

  ② 启动成功日志如下：

  Starting MySQL.Logging to '/var/lib/mysql/e2d21b78fdb0.err'.
  SUCCESS!
  

• g. 安装mysql客户端

   rpm -ivh MySQL-client-5.6.48-1.el7.x86_64.rpm
  

• h. 连接mysql，使用root用户

  mysql -uroot -p
  

• i. 修改mysql root用户密码

  set password=password('密码');
  quit;
  

• j. 修改用户表，将host表的host列修改为任意ip可访问，并删除其他host（否则在后面启动时会报没有权限）

  use mysql;
  
  update user set host = '%' where host = 'localhost';
  
  delete from user where host = 'e2d21b78fdb0';
  delete from user where Host='127.0.0.1';
  delete from user where Host='::1';
  
  flush privileges;
  

• k. 安装mysql-connector-java.jar

  ① 下载mysql-connector-java.jar

  ② 将jar包拷贝到 hive-1.2.2/lib 下

配置Metastore到Mysql

1. 在hive-1.2.2/conf下创建hive-site.xml

   vi hive-site.xml
   

2. 配置如下:

   
   
   
      
        javax.jdo.option.ConnectionURL
        jdbc:mysql://hive01:3306/metastore?createDatabaseIfNotExist=true
        JDBC connect string for a JDBC metastore
      
      
        javax.jdo.option.ConnectionDriverName
        com.mysql.jdbc.Driver
        Driver class name for a JDBC metastore
      
      
        javax.jdo.option.ConnectionUserName
        root
        username to use against metastore database
      
      
        javax.jdo.option.ConnectionPassword
        密码
        password to use against metastore database
      
      
      
        hive.cli.print.header
        true
      
      
        hive.cli.print.current.db
        true
      
   
   

配置运行日志便于查询

1. 备份

   cp /opt/envs/hive-1.2.2/conf/hive-log4j.properties.template /opt/envs/hive-1.2.2/conf/hive-log4j.properties
   

2. 修改

   vi /opt/envs/hive-1.2.2/conf/hive-log4j.properties
   

3. 修改hive.log.dir属性

   hive.log.dir=/opt/envs/hive-1.2.2/logs
   

保存为镜像并移除容器

docker commit e2d21b78fdb0 caiser/hive:1.2.2
docker rm e2d21b78fdb0

启动容器

docker run -it -d --network datastore_net --ip 172.173.16.22 --name hive01 caiser/hive:1.2.2 bin/bash

配置ssh免密登录

1. 进入容器

   docker exec -it hive01 /bin/bash
   

2. 到~/.ssh目录下生成秘钥

   ssh-keygen -t rsa
   

3. 拷贝秘钥到hadoop01，hadoop02和hadoop03

   a.[如果没开启]开启ssh服务[ps -ef | grep ssh]

   /usr/sbin/sshd -D &
   

   b. 拷贝秘钥到hadoop01，hadoop02，hadoop03

   ssh-copy-id hadoop01
   ssh-copy-id hadoop02
   ssh-copy-id hadoop03
   

4. hadoop01,hadoop02和hadoop03依次执行

   ssh-copy-id hive01
   

向hadoop集群中添加节点

• 动态
  ① 在hive01机器的hadoop/sbin目录下启动datanode：

  hadoop-daemon.sh start datanode
  

  ② 在hive01机器的hadoop/sbin目录下启动nodemanager

  yarn-daemon.sh start nodemanager
  

  ③ jps查看datanode和nodemanager是否已启动
  ④ 回到namenode节点打印集群信息，或网页登录50070端口查看节点数量

  hdfs dfsadmin -report
  

• 静态

  ① 停止hadoop集群
  ② 保证各节点hadoop的配置文件内容一致
  ③ 分别配置4个节点hadoop的salves文件

   hadoop01
   hadoop02
   hadoop03
   hive01
  

  ④ 重启hadoop集群

启动hive

1. 启动

   bin/hive
   

2. 看到这种就是成了

   Logging initialized using configuration in file:/opt/envs/hive-1.2.2/conf/hive-log4j.properties
   hive (default)> 
   

3. 大功告成