23年hadoop单机版+hive

说明

工作需要研究hive功能，线上环境不能动，搭建单机版hadoop+hive测试环境，使用当前最新hadoop 3.3.6+ hive 3.1.3。

分享

• 大数据博客列表
• 开发记录汇总
• 个人java工具库 项目https://gitee.com/wangzonghui/object-tool
  • 包含json、string、集合、excel、zip压缩、pdf、bytes、http等多种工具，欢迎使用。

环境信息

• 系统环境 centos7
• 安装目录：
  • java: /usr/java/
  • hadoop: /opt/module/
  • hive: /opt/module/

安装

jdk

• 准备java8安装包：jdk-8u201-linux-x64.tar.gz

• 解压安装：mkdir -p /usr/java && tar -zxvf jdk-8u201-linux-x64.tar.gz -C /usr/java/

• 配置环境变量，编辑文件：vi /etc/profile.d/my_env.sh 增加jdk环境变量

  
  #JAVA_HOME export是全局变量
  export JAVA_HOME=/usr/java/jdk1.8.0_201
  export PATH=$PATH:$JAVA_HOME/bin
  

• 刷新配置 source /etc/profile

• 验证 java -version

hadoop

• 下载hadoop 3.3.6 下载地址 ，解压安装包: mkdir -p /opt/module/ && tar -zxvf hadoop-3.3.4.tar.gz -C /opt/module/
• hadoop不添加到环境变量，hive配置文件指定物理路径

配置core-site.xml

• 修改文件：vi /opt/module/hadoop-3.3.6/etc/hadoop/core-site.xml，configuration 节点间增加主机和用户组权限配置，完整内容如下：

<configuration>
    <property>
            <name>hadoop.proxyuser.root.hostsname>
            <value>*value>
    property>
    <property>
            <name>hadoop.proxyuser.root.groupsname>
            <value>*value>
    property>
configuration>

mysql

• hive运行需要mysql存储元数据，mysql8版本 8.0.33 下载地址
• 安装过程

# 解压安装包
tar -xvf mysql-*-1.el7.x86_64.rpm-bundle.tar

# 安装 
rpm -ivh mysql-community-common-*-1.el7.x86_64.rpm --nodeps --force && rpm -ivh mysql-community-libs-*-1.el7.x86_64.rpm --nodeps --force && rpm -ivh mysql-community-client-*-1.el7.x86_64.rpm --nodeps --force && rpm -ivh mysql-community-server-*-1.el7.x86_64.rpm --nodeps --force

# 验证，查看安装mysql
rpm -qa | grep mysql

# 输入以下命令，完成mysql初始化
mysqld --initialize
# 设置目录权限
chown mysql:mysql /var/lib/mysql -R

# 启动服务和开机启动
systemctl start mysqld.service && systemctl enable mysqld

# 查看默认初始化密码
cat /var/log/mysqld.log | grep password
# 重置root密码并设置root账号远程登录
mysql -uroot -p --connect-expired-password -e "alter user 'root'@'localhost' identified by '12WE#o89T';
create user 'root'@'%' identified by '12WE#o89T';
USE mysql;
grant all on *.* TO 'root'@'%';
FLUSH PRIVILEGES;"

hive

• hive 3.1.3 下载地址

安装

• 解压安装：mkdir -p /opt/module/ && tar -zxvf apache-hive-3.1.3-bin.tar.gz -C /opt/module/
• 配置环境标量，编辑文件：vi /etc/profile.d/my_env.sh，增加如下内容

export HIVE_HOME=/opt/module/apache-hive-3.1.3-bin
export PATH=$PATH:$HIVE_HOME/bin  

• 刷新配置 source /etc/profile

配置hive-site.xml

• 复制文件，cp /opt/module/apache-hive-3.1.3-bin/conf/hive-default.xml.template /opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml
• 修改下面内容，注意自己MySQL配置，vi /opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml，添加如下内容

  
  <property>
    <name>javax.jdo.option.ConnectionURLname>
    <value>jdbc:mysql://localhost:3306/hive?&createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=falsevalue>
  property>
  
  <property>
    <name>javax.jdo.option.ConnectionUserNamename>
    <value>rootvalue>
  property>
  
  <property>
    <name>javax.jdo.option.ConnectionPasswordname>
    <value>12WE#o89Tvalue>
  property>
  
  <property>
    <name>javax.jdo.option.ConnectionDriverNamename>
    <value>com.mysql.cj.jdbc.Drivervalue>
  property>
  <property>
    <name>datanucleus.schema.autoCreateAllname>
    <value>truevalue> 
  property>
  <property>
    <name>hive.metastore.schema.verificationname>
    <value>falsevalue>
  property>
  
  <property>
    <name>hive.server2.thrift.client.username>
    <value>rootvalue>
    <description>Username to use against thrift clientdescription>
  property>
  
  <property>
    <name>hive.server2.thrift.client.passwordname>
    <value>1234value>
    <description>Password to use against thrift clientdescription>
  property>
  
  <property>
    <name>hive.exec.local.scratchdirname>
    <value>/tmp/hivevalue>
    <description>Local scratch space for Hive jobsdescription>
  property>
  <property>
    <name>hive.downloaded.resources.dirname>
    <value>/tmp/hive/resourcesvalue>
    <description>Temporary local directory for added resources in the remote file system.description>
  property>

• 设置缓存目录权限：chmod 777 /tmp/hive

配置hive-env

• 复制hive-env.sh文件 cp /opt/module/apache-hive-3.1.3-bin/conf/hive-env.sh.template /opt/module/apache-hive-3.1.3-bin/conf/hive-env.sh
• 修改文件添加Hadoop、hive路径，vi /opt/module/apache-hive-3.1.3-bin/conf/hive-env.sh

HADOOP_HOME=/opt/module/hadoop-3.3.6
export HIVE_CONF_DIR=/opt/module/apache-hive-3.1.3-bin/conf

初始化mysql数据库

• 下载mysql连接jar：wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar，放在 cp mysql-connector-java-*.jar /opt/module/apache-hive-3.1.3-bin/lib && ls /opt/module/apache-hive-3.1.3-bin/lib/mysql-connector-java-*.jar
• 初始化mysql数据库：/opt/module/apache-hive-3.1.3-bin/bin/schematool -initSchema -dbType mysql
• 最后提示completed说明初始化成功

Initialization script completed
schemaTool completed

• 初始化报错

  SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
  Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
  at [row,col,system-id]: [3215,96,"file:/opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml"]
      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3101)
      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3050)
      at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2923)
      at org.apache.hadoop.conf.Configuration.addResourceObject(Configuration.java:1035)
      at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:940)
      at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5154)
      at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:5107)
      at org.apache.hive.beeline.HiveSchemaTool.(HiveSchemaTool.java:96)
      at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
  

  解决方法，删除：file:/opt/module/apache-hive-3.1.3-bin/conf/hive-site.xml 3215行注释即可，原因是xml解析异常。

启动验证

hive命令

• hive命令是本地客户端登录，官方推荐使用hiveServer2的beeline方式。
  

hiveserver2方式

• 创建日志目录：mkdir -p /opt/module/logs/,后台启动 hiveserver2 服务：nohup hive --service hiveserver2 > /opt/module/logs/hive3.1.3.log 2>&1 &

• beeline 连接验证

beeline
!connect jdbc:hive2://localhost:10000 root 1234

总结

• hive云数据存储数据库可以不用mysql，习惯用mysql，也可以简化安装用其他数据库。