hive3.1.1集群搭建

hive作为一个数据仓库建立在hadoop的基础上，提供了一套类似mysql的语法，用于做报表统计、数据分析等。在hive2.x之前，hive是不支持联机事务处理的，也就是说使用hive操作sql，会有很大的时间延迟。而hive3.x版本则有了改进，hive引擎支持tez和spark,在查询速度上有了显著的提升。接下来我们来安装搭建hive3.1.1:

##准备环境

• 虚拟机4台（hadoop1,hadoop2,hadoop3,hadoop4）
• 搭建hadoop集群
  hadoop3.1.1搭建教程

##mysql安装步骤

1. 首先需要安装mysql在机器上，点击获取安装包：mysql5.7

这里，我们也可以选择使用命令下载。

cd /usr/local.  //切换到下载目录
wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.24-linux-glibc2.12-x86_64.tar.gz

2. 创建安装目录

   mkdir -p /usr/local/mysql
   

3. 解压mysql安装包

   tar -zxvf mysql-5.7.24-linux-glibc2.12-x86_64.tar.gz -C /usr/local/mysql
   

4. 创建mysql用户和用户组

   groupadd mysql
   useradd -r -g mysql mysql
   

5. 创建my.cnf文件

   vi /etc/my.cnf
   

   拷贝以下内容

   [mysqld]
   basedir = /usr/local/mysql
   datadir = /usr/local/mysql/data
   port = 3306
   socket = /tmp/mysql.sock
   
   sql_mode=NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER
   explicit_defaults_for_timestamp=true
   
   [mysqld_safe]
   pid-file = /tmp/mysqld/mysqld.pid
   log-error = /var/log/mysqld.log
   

6. 创建my.cnf上配置的文件，mysql.sock, mysqld.pid, mysqld.log

   mkdir -p /tmp/mysqld
   cd /tmp
   touch mysql.sock
   chown -R mysql:mysql mysql.sock
   chmod 755 mysql.sock
   
   cd mysqld
   touch mysqld.pid
   chmod 755 mysqld.pid
   chown -R mysql:mysql mysqld.pid
   
   cd /var/log
   touch mysqld.log
   chmod 755 mysqld.log
   

7. 配置mysql环境变量

   vi /etc/profile
   

   添加如下内容：

   export MYSQL_HOME=/usr/local/mysql
   export PATH=$PATH:$MYSQL_HOME/bin
   

   让环境变量立即生效

   source /etc/profile
   

8. 初始化mysql

   mysqld --initialize --user=mysql
   

    注：执行上一段命令可以会报libaio缺失的错误，则执行下面命令：
    
    centos版本：
    yum install -y libaio-dev
    
    ubuntu版本：
    apt-get install libaio-dev
   

   如果还是下载不到libaio依赖包，该考虑换一个新的源了。
   步骤：

   1.备份源表

   cp /etc/apt/sources.list /etc/apt/sources.list.bak
   

   2.网上找到新的源,删除原来的源，添加新的源，这里提供一些源

   deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
   deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
   deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
   deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
   deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
   deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
   deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
   deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
   deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
   deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
   
   deb http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
   deb http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
   deb http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
   deb http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
   deb http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse
   deb-src http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse
   deb-src http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse
   deb-src http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse
   deb-src http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse
   deb-src http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse	
   
   

   3.更新源的操作

   apt-get update
   

   4.再次下载libaio依赖包

   apt-get install libaio-dev
   

9. 如果上述步骤没有报错，则留意初始化之后的密码

   ...'root'@localhost: XXXXX //保留密码
   

   如果上述步骤报错，则需要删除mysql数据，再次执行初始化操作,留意初始化密码

   rm -rf /usr/local/mysql/data
   

10. 更改mysql数据的所属用户及用户组

    chown -R mysql:mysql /usr/local/mysql/data
    

11. 配置mysql启动服务

    cp /usr/local/mysqlsupport-files/mysql.server /etc/init.d/mysqld
    

12. 启动mysql服务

    /etc/init.d/mysqld start
    

13. 启动mysql客户端

    mysql -u root -p
    

    输入初始化密码

14. 更改初始化密码

    alter user 'root'@'localhost' identified by '123456'; 
    flush privileges;
    

hive安装步骤

1. 切换到下载目录下，执行下载命令

   wget http://mirror.bit.edu.cn/apache/hive/hive-3.1.1/apache-hive-3.1.1-bin.tar.gz
   

2. 创建hive的安装目录

   mkdir -p /usr/local/hive
   

3. 解压hive安装包

   tar -zxvf apache-hive-3.1.1-bin.tar.gz -C /usr/local
   

4. 更改hive安装名称

   cd /usr/local
   mv apache-hive-3.1.1-bin hive
   

5. 创建hive链接

   ln -s apache-hive-3.1.1-bin hive
   

6. 配置hive的环境变量

   vi /etc/profile
   

   拷贝如下内容：

   export HIVE_HOME=/usr/local/hive
   export PATH=$PATH:$HIVE_HOME/bin
   

   让环境变量立即生效

   source /etc/profile
   

7. 测试hive执行是否成功

   hive --version
   

8. 创建hive的hive-site.xml文件

   cd /usr/local/hive/conf
   cp hive-default.xml.template hive-site.xml
   

   替换如下内容：

   
   	
   		javax.jdo.option.ConnectionURL
   		jdbc:mysql://localhost:3306/hive?useSSL=false
   	
   	
   		javax.jdo.option.ConnectionDriverName
   		com.mysql.jdbc.Driver
   	
   	
   		javax.jdo.option.ConnectionUserName
   		root
   	
   	
   		javax.jdo.option.ConnectionPassword
   		123456
   	
   	
   		hive.metastore.schema.verification
   		false
   	
   
   
   

   注意修改连接mysql的用户名和密码

9. 打开mysql客户端，创建hive数据库

   create database hive;
   

10. 进入hive的lib目录，下载mysql连接包

    wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.47/mysql-connector-java-5.1.47.jar
    

11. 启动hadoop,hive需要hadoop来存储数据

    start-dfs.sh 		//hadoop1节点启动
    start-yarn.sh 	//hadoop2节点启动
    

12. 初始化hive

    schematool -dbType mysql -initSchema
    

13. 进入hive的bin目录，启动hive

    cd /usr/local/hive/bin
    hive	//启动hive
    

14. 进入hive命令界面，创建一个数据库测试hive.

    create database test;
    

15. 打开hadoop的hdfs管理界面
    

hive集群搭建完成～