Hive和Spark生产集群搭建(spark on doris)

1.环境准备

1.1 版本选择

序号 bigdata-001 bigdata-002 bigdata-003 bigdata-004 bigdata-005
MySQL-8.0.31 mysql
Datax Datax Datax Datax Datax Datax
Spark-3.3.1 Spark Spark Spark Spark Spark
Hive-3.1.3 Hive Hive

1.2 主要组件官网

hive官网: https://hive.apache.org/
hive安装包下载:http://archive.apache.org/dist/hive/
spark官网:https://spark.apache.org/
spark安装包下载:https://www.apache.org/dyn/closer.lua/spark/spark-3.3.1/

注意:官网下载的Hive3.1.3和Spark3.3.1默认是不兼容的。因为Hive3.1.3支持的Spark版本是2.4.5,所以需要我们重新编译Hive3.1.3版本。

Hadoop环境安装详见本博客最全Hadoop实际生产集群高可用搭建

2.Hive安装部署

2.1 环境配置

  1. 解压apache-hive-3.1.3-bin.tar.gz到/data/module/目录下面
[hadoop@hadoop1 software]$ tar -zxvf /data/software/apache-hive-3.1.3-bin.tar.gz -C /data/module/
  1. 修改apache-hive-3.1.3-bin.tar.gz的名称为hive
[hadoop@hadoop1 software]$ mv /data/module/apache-hive-3.1.3-bin/ /data/module/hive-3.1.3
  1. 修改/etc/profile.d/my_env.sh,添加环境变量
[hadoop@hadoop1 software]$ sudo vim /etc/profile.d/my_env.sh
  1. 添加内容
#HIVE_HOME
export HIVE_HOME=/data/module/hive-3.1.3
export PATH=$PATH:$HIVE_HOME/bin
export PATH JAVA_HOME HADOOP_HOME HIVE_HOME

2.2 Hive元数据配置到MySQL

  1. 拷贝mysql的jdbc驱动(mysql-connector-java-5.1.48.jar)到hive的lib目录下
[hadoop@hadoop1 software]$ cp /data/software/mysql-connector-java-5.1.48.jar $HIVE_HOME/lib
  1. 配置Metastore到MySql
    在$HIVE_HOME/conf目录下新建hive-site.xml文件
[hadoop@hadoop1 software]$ vim $HIVE_HOME/conf/hive-site.xml

添加如下内容



<configuration>
   
   <property>
       <name>javax.jdo.option.ConnectionURLname>
       <value>jdbc:mysql://xxx:3306/metastore?useSSL=false&createDatabaseIfNotExist=true&characterEncoding=UTF-8value>
property>

   
   <property>
       <name>javax.jdo.option.ConnectionDriverNamename>
       <value>com.mysql.jdbc.Drivervalue>
property>

   
   <property>
       <name>javax.jdo.option.ConnectionUserNamename>
       <value>xxxvalue>
   property>

   
   <property>
       <name>javax.jdo.option.ConnectionPasswordname>
       <value>xxxvalue>
property>

   
   <property>
       <name>hive.metastore.warehouse.dirname>
       <value>/user/hive/warehousevalue>
   property>
   
  
   <property>
       <name>hive.metastore.schema.verificationname>
       <value>falsevalue>
   property>

   <property>
       <name>metastore.storage.schema.reader.implname>
       <value>org.apache.hadoop.hive.metastore.SerDeStorageSchemaReadervalue>
   property>
  
   
   <property>
       <name>hive.metastore.event.db.notification.api.authname>
       <value>falsevalue>
property>

<property>
       <name>hive.cli.print.headername>
       <value>truevalue>
property>
<property>
       <name>hive.cli.print.current.dbname>
       <value>truevalue>
property>

   <property>
       <name>hive.metastore.urisname>
   <value>thrift://xxx:9083,thrift://xxx1:9083value>
   property>

   <property>
       <name>hive.server2.thrift.bind.hostname>
       <value>xxxvalue>
   property>



   
   <property>
       <name>hive.server2.thrift.portname>
       <value>10000value>
   property>

<property>
<name>hive.server2.enable.doAs name>
<value>falsevalue>
property>


<property>
   <name>spark.yarn.jarsname>
   <value>hdfs://hadoopcluster/spark-jars/*value>
property>
 

<property>
   <name>hive.execution.enginename>
   <value>sparkvalue>
property>

<property>
   <name>spark.dynamicAllocation.enabledname>
   <value>truevalue>
property>

<property>
   <name>hive.spark.client.connect.timeoutname>
   <value>100000msvalue>
property>
   <property>
       <name>hive.zookeeper.client.portname>
       <value>2181value>
   property>

   <property>
       <name>hive.zookeeper.quorumname>
       <value>xxxxxvalue>
   property>

 <property>
       <name>hive.server2.support.dynamic.service.discoveryname>
       <value>truevalue>
property>

<property>
       <name>hive.server2.zookeeper.namespacename>
       <value>hiveserver2_zkvalue>
property>



<property>
   <name>hive.server2.sleep.interval.between.start.attemptsname>
   <value>2svalue>
   <description>
     Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec, us/usec, ns/nsec), which is msec if not specified.
     The time should be in between 0 msec (inclusive) and 9223372036854775807 msec (inclusive).
     Amount of time to sleep between HiveServer2 start attempts. Primarily meant for tests
   description>
 property>

<property>
   <name>hive.server2.logging.operation.enabledname>
   <value>falsevalue>
property>

<property>
   <name>hive_timeline_logging_enabledname>
   <value>truevalue>
property>



<property>
 <name>hive.reloadable.aux.jars.pathname>
 <value>/data/module/hive-3.1.3/jarsvalue>
property>





<property>
     <name>hive.users.in.admin.rolename>
     <value>hadoopvalue>
property>
configuration>

2.3 初始化元数据库

  1. 登陆MySQL
[hadoop@hadoop1 software]$ mysql -uroot -pxxx
  1. 新建Hive元数据库
mysql> create database metastore;
mysql> quit;
  1. 初始化Hive元数据库
[hadoop@hadoop1 software]$ schematool -initSchema -dbType mysql -verbose

4) 修改元数据库字符集
Hive元数据库的字符集默认为Latin1,由于其不支持中文字符,故若建表语句中包含中文注释,会出现乱码现象。如需解决乱码问题,须做以下修改。
修改Hive元数据库中存储注释的字段的字符集为utf-8
//字段注释

mysql> alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
//表注释
mysql> alter table TABLE_PARAMS modify column PARAM_VALUE mediumtext character set utf8;
//退出
quit;
  1. hadoop的配置文件core-site.xml和hdfs-site.xml复制到hive的conf中

2.4 启动metastore和hiveserver2

  1. 启动hiveserver2
[hadoop@hadoop1 hive]$ bin/hive --service hiveserver2
  1. 启动beeline客户端(需要多等待一会)
[hadoop@hadoop1 hive]$ bin/beeline -u jdbc:hive2://hadoop1:10000 -n hadoop
  1. 看到如下界面
Connecting to jdbc:hive2://hadoop1:10000
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.2 by Apache Hive
0: jdbc:hive2://hadoop1:10000>

3.Spark安装

3.1 解压缩文件

将spark-3.3.1-bin-hadoop3.tgz文件上传到Linux并解压缩,放置在指定位置,路径中不要包含中文或空格

tar -zxvf spark-3.3.1-bin-hadoop3.tgz -C /data/module
cd /data/module 
mv spark-3.3.1-bin-hadoop3.2  spark-3.3.1

3.2 启动环境

1)进入解压缩后的路径,执行如下指令

bin/spark-shell
  1. 启动成功后,可以输入网址进行Web UI监控页面访问
    http://hadoop1:4040

3.3 Hive on Spark配置

3.3.1 配置SPARK_HOME环境变量

[hadoop@hadoop1 software]$ sudo vim /etc/profile.d/my_env.sh

添加如下内容

# SPARK_HOME
export SPARK_HOME=/data/module/spark-3.3.1
export PATH=$PATH:$SPARK_HOME/bin

source 使其生效

[hadoop@hadoop1 software]$ source /etc/profile.d/my_env.sh

3.3.2 创建spark配置文件并复制到hive中

[hadoop@hadoop1 software]$ vim /data/module/spark-3.3.1
/conf/spark-defaults.conf

添加如下内容(在执行任务时,会根据如下参数执行)

spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir    hdfs://yourhadoopcluster/spark-history
spark.executor.cores    1
spark.executor.memory    4g
spark.executor.memoryOverhead    2g
spark.driver.memory    4g
spark.driver.memoryOverhead    2g
spark.dynamicAllocation.enabled  true
spark.shuffle.service.enabled  true
spark.dynamicAllocation.executorIdleTimeout  60s
spark.dynamicAllocation.initialExecutors    1
spark.dynamicAllocation.minExecutors  1
spark.dynamicAllocation.maxExecutors  12
spark.dynamicAllocation.schedulerBacklogTimeout 1s
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout 5s
spark.dynamicAllocation.cachedExecutorIdleTimeout 30s
spark.shuffle.useOldFetchProtocol    true
spark.history.fs.cleaner.enabled    true
spark.history.fs.cleaner.interval    1d
spark.history.fs.cleaner.maxAge    7d
spark.hadoop.orc.overwrite.output.file true
spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8
spark.driver.extraJavaOptions=-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8

在HDFS创建如下路径,用于存储历史日志

[hadoop@hadoop1 software]$ hadoop fs -mkdir /spark-history

3.3.4 向HDFS上传Spark纯净版jar包

说明1:由于Spark3.3.1非纯净版默认支持的是hive2.3.7版本,直接使用会和安装的Hive3.1.2出现兼容性问题。所以采用Spark纯净版jar包,不包含hadoop和hive相关依赖,避免冲突。

说明2:Hive任务最终由Spark来执行,Spark任务资源分配由Yarn来调度,该任务有可能被分配到集群的任何一个节点。所以需要将Spark的依赖上传到HDFS集群路径,这样集群中任何一个节点都能获取到。

① 上传并解压spark-3.3.1-bin-without-hadoop.tgz

[hadoop@hadoop1 software]$ tar -zxvf /data/software/spark-3.3.1-bin-without-hadoop.tgz

② 上传Spark纯净版jar包到HDFS

[hadoop@hadoop1 software]$ hadoop fs -mkdir /spark-jars

[hadoop@hadoop1 software]$ hadoop fs -put spark-3.3.1-bin-without-hadoop/jars/* /spark-jars

cp /data/module/spark-3.3.1/yarn/spark-3.3.1-yarn-shuffle.jar /data/module/hadoop-3.3.4/share/hadoop/yarn/lib/

6)将spark的jar包拷贝到yarn中

cp /data/module/spark-3.3.1/yarn/spark-3.3.1-yarn-shuffle.jar /data/module/hadoop-3.3.4/share/hadoop/yarn/lib/

3.3.5 修改hive-site.xml文件(以上已配置)

[hadoop@hadoop1 ~]$ vim /data/module/hive/conf/hive-site.xml
添加如下内容
<!--Spark依赖位置(注意:端口号8020必须和namenode的端口号一致)-->
<property>
    <name>spark.yarn.jars</name>
    <value>hdfs://xxx:8020/spark-jars/*</value>
</property>
  
<!--Hive执行引擎-->
<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>

3.3.6 spark-sql操作doris

下载git代码库的spark代码:https://github.com/apache/doris-spark-connector
按照readme介绍打包自己的适配版连接器jar包
将jar包复制到spark的jars目录下,同时hdfs上的spark包目录也上传一份

cp /your_path/spark-doris-connector/target/spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar  $SPARK_HOME/jars
hadoop fs -put /your_path/spark-doris-connector/target/spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar /spark-jars

运行spark-sql 测试:

//测试
CREATE
TEMPORARY VIEW spark_doris1
USING doris
OPTIONS(
  'table.identifier'='demo.t1',
  'fenodes'='xxx:8030',
  'user'='xxx',
  'password'='xxx'
);
CREATE
TEMPORARY VIEW spark_doris2
USING doris
OPTIONS(
  'table.identifier'='demo.t2',
  'fenodes'='xxx:8030',
  'user'='xxx',
  'password'='xxx'
);

INSERT INTO spark_doris1
select * from spark_doris2;

你可能感兴趣的:(Hive相关,hive,spark,hadoop,大数据,yarn)