Apache Hive安装部署

Apache Hive安装部署

Hive元数据

• 描述数据的数据，主要描述数据属性信息，用来支持如指示存储位置，历史数据，资源查找，文件记录等功能。
• 存储在关系型数据库中。如hive内置Derby，或第三方MySql
• Metastore元数据服务。作用管理元数据，对外暴露服务。

metastore服务配置三种模式：

• 内嵌模式
• 本地模式
• 远程模式

区分配置方式方法：

• metastore是否需要单独配置，单独启动
• metadaa存储在内置derby中，还是第三方RDBMS

	内嵌	本地	远程
meta单独配置，启动	否	否	是
metadata存储介质	Derby	Mysql	Mysql

部署实战

安装前准备：

• Hive基于Hadoop，保证服务器基础环境正常
  • 集群时间同步，防火墙关闭，Host映射，免密登录，JDK安装
• Hadoop集群健康可用
  • 等待HDFS安全模式关闭后再运行Hive

️安装MySQL

• MySQL只需一台机器安装并需要授权远程访问

• 卸载CentOS7自带mariadb

  [root@node1 ~]# rpm -qa|grep mariadb
  mariadb-libs-5.5.60-1.el7_5.x86_64
  [root@node1 ~]# rpm -e mariadb-libs-5.5.60-1.el7_5.x86_64 --nodeps
  
  [root@node1 ~]# rpm -qa|grep mariadb
  
  

• 安装mysql

  [root@node1 ~]# mkdir /export/software/mysql
  #上传mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar 到上述文件夹下  解压
  [root@node1 ~]# ls /export/software/mysql/
  mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar
  [root@node1 ~]# cd /export/software/mysql/
  [root@node1 mysql]# tar xvf mysql-5.7.29-1.el7.x86_64.rpm-bundle.tar
  mysql-community-embedded-devel-5.7.29-1.el7.x86_64.rpm
  mysql-community-test-5.7.29-1.el7.x86_64.rpm
  mysql-community-embedded-5.7.29-1.el7.x86_64.rpm
  mysql-community-embedded-compat-5.7.29-1.el7.x86_64.rpm
  mysql-community-libs-5.7.29-1.el7.x86_64.rpm
  mysql-community-client-5.7.29-1.el7.x86_64.rpm
  mysql-community-server-5.7.29-1.el7.x86_64.rpm
  mysql-community-devel-5.7.29-1.el7.x86_64.rpm
  mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm
  mysql-community-common-5.7.29-1.el7.x86_64.rpm
  #安装依赖
  [root@node1 mysql]# yum -y install libaio
  [root@node1 mysql]# yum install -y net-tools
  
  #开始安装
  [root@node1 mysql]# rpm -ivh mysql-community-common-5.7.29-1.el7.x86_64.rpm mysql-community-libs-5.7.29-1.el7.x86_64.rpm mysql-community-client-5.7.29-1.el7.x86_64.rpm mysql-community-server-5.7.29-1.el7.x86_64.rpm 
  
  

• mysql初始化设置

  #初始化
  mysqld --initialize
  
  #更改所属组
  chown mysql:mysql /var/lib/mysql -R
  
  #启动mysql
  systemctl start mysqld.service
  
  #查看生成的临时root密码
  cat  /var/log/mysqld.log
  #临时密码大概在这个位置
  ...
  2023-06-13T12:23:52.391426Z 1 [Note] A temporary password is generated for root@localhost:
  ...
   
  
  
  

• 修改root密码 授权远程访问 设置开机自启动

  
  mysql -u root -p#进入之后输入日志中生成的密码
  #更新密码
  mysql>  alter user user() identified by "hadoop";
  Query OK, 0 rows affected (0.00 sec)
  
  #权限赋予
  mysql> use mysql;
  Reading table information for completion of table and column names
  You can turn off this feature to get a quicker startup with -A
  
  Database changed
  mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'hadoop' WITH GRANT OPTION;
  Query OK, 0 rows affected, 1 warning (0.00 sec)
  
  mysql> FLUSH PRIVILEGES;
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> exit
  Bye
  
  
  #mysql的启动和关闭 状态查看 
  systemctl stop mysqld
  systemctl status mysqld
  systemctl start mysqld
  
  #设置开机自启
  systemctl enable  mysqld
  
  
  

• 如果需要卸载请执行

  #关闭mysql服务
  systemctl stop mysqld.service
  
  #查找安装mysql的rpm包
  [root@node1 ~]# rpm -qa | grep -i mysql      
  mysql-community-libs-5.7.29-1.el7.x86_64
  mysql-community-common-5.7.29-1.el7.x86_64
  mysql-community-client-5.7.29-1.el7.x86_64
  mysql-community-server-5.7.29-1.el7.x86_64
  
  #卸载
  [root@node1 ~]# yum remove mysql-community-libs-5.7.29-1.el7.x86_64 mysql-community-common-5.7.29-1.el7.x86_64 mysql-community-client-5.7.29-1.el7.x86_64 mysql-community-server-5.7.29-1.el7.x86_64
  
  #查看是否卸载干净
  rpm -qa | grep -i mysql
  
  #查找mysql相关目录 删除
  [root@node1 ~]# find / -name mysql
  /var/lib/mysql
  /var/lib/mysql/mysql
  /usr/share/mysql
  
   rm -rf /var/lib/mysql
   rm -rf /var/lib/mysql/mysql
   rm -rf /usr/share/mysql
  
  #删除默认配置 日志
  rm -rf /etc/my.cnf 
  rm -rf /var/log/mysqld.log
  

安装Hive

• 上传Hive安装包（node1安装即可）

  
  [root@node1 server]# pwd
  /export/server
  
  [root@node1 server]# ls
  apache-hive-3.1.2-bin.tar.gz  hadoop-3.3.0  jdk1.8.0_241
  #解压
  tar zxvf apache-hive-3.1.2-bin.tar.gz
  
  

• 解决Hive和Hadoop之间guava版本差异

  cd /export/server/apache-hive-3.1.2-bin/
  rm -rf lib/guava-19.0.jar
  cp /export/server/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar ./lib/
  

• 修改配置文件

  • hive.env.sh

    cd /export/server/apache-hive-3.1.2-bin/conf
    mv hive-env.sh.template hive-env.sh
    
    vim hive-env.sh
    export HADOOP_HOME=/export/server/hadoop-3.3.0
    export HIVE_CONF_DIR=/export/server/apache-hive-3.1.2-bin/conf
    export HIVE_AUX_JARS_PATH=/export/server/apache-hive-3.1.2-bin/lib
    

  • hive-site.xml

    vim hive-site.xml
    

    <configuration>
    
    <property>
    	<name>javax.jdo.option.ConnectionURLname>
    	<value>jdbc:mysql://node1:3306/hive3?createDatabaseIfNotExist=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8value>
    property>
    
    <property>
    	<name>javax.jdo.option.ConnectionDriverNamename>
    	<value>com.mysql.jdbc.Drivervalue>
    property>
    
    <property>
    	<name>javax.jdo.option.ConnectionUserNamename>
    	<value>rootvalue>
    property>
    
    <property>
    	<name>javax.jdo.option.ConnectionPasswordname>
    	<value>hadoopvalue>
    property>
    
    
    <property>
        <name>hive.server2.thrift.bind.hostname>
        <value>node1value>
    property>
    
    
    <property>
        <name>hive.metastore.urisname>
        <value>thrift://node1:9083value>
    property>
    
     
    <property>
        <name>hive.metastore.event.db.notification.api.authname>
        <value>falsevalue>
    property>
    configuration>
    
    

• 上传mysql jdbc驱动到hive安装包lib下

  
  [root@node1 conf]# cd /export/server/apache-hive-3.1.2-bin/lib/
  [root@node1 lib]# ls -aux | grep mysql-connector
  mysql-connector-java-5.1.32.jar
  
  

• 初始化元数据

  cd /export/server/apache-hive-3.1.2-bin/
  
  bin/schematool -initSchema -dbType mysql -verbos
  

• hdfs创建hive存储目录（存在不操作）

  hadoop fs -mkdir /tmp
  hadoop fs -mkdir -p /user/hive/warehouse
  hadoop fs -chmod g+w /tmp
  hadoop fs -chmod g+w /user/hive/warehouse
  

启动

• 启动metastore服务

  #前台启动  关闭ctrl+c
  /export/server/apache-hive-3.1.2-bin/bin/hive --service metastore
  
  #前台启动开启debug日志
  /export/server/apache-hive-3.1.2-bin/bin/hive --service metastore --hiveconf hive.root.logger=DEBUG,console  
  
  #后台启动 进程挂起  关闭使用jps+ kill -9
  nohup /export/server/apache-hive-3.1.2-bin/bin/hive --service metastore &
  

客户端启用

Hive自带客户端：

• bin/hive

• bin/beeline

• 在安装hive服务器上，首先启动metastore，然后启动hieserver2

    nohup /export/server/apache-hive-3.1.2-bin/bin/hive --service hiveserver2 &
  

• beeline客户端连接

  • 拷贝node1安装包到beeline客户端机器（node3）

    scp -r /export/server/apache-hive-3.1.2-bin/ node3:/export/server/
    

  • node3使用bin/hive连接服务
    

  • node3使用bin/beeline连接服务