impala安装部署(rpm安装)

1.下载impala所需的rpm包

bigtop-utils rpm包下载
http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.8.3/RPMS/noarch/
impala rpm包下载地址
http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.15.1/RPMS/x86_64/
下载后的rpm包上传到服务器

[hadoop@hadoop002 impala]$ ll
total 645596
-rw-r--r-- 1 hadoop hadoop     10864 Apr  8 18:50 bigtop-utils-0.7.0+cdh5.8.3+0-1.cdh5.8.3.p0.7.el6.noarch.rpm
-rw-r--r-- 1 hadoop hadoop 202621568 Apr  8 17:26 impala-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop      4656 Apr  8 14:38 impala-catalog-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop 457476312 Apr  8 18:58 impala-debuginfo-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop      4624 Apr  8 14:38 impala-server-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop    882824 Apr  8 14:39 impala-shell-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop      4712 Apr  8 14:39 impala-state-store-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm
-rw-r--r-- 1 hadoop hadoop     65076 Apr  8 14:39 impala-udf-devel-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm

2.安装rpm
2.1 安装 sentry rpm包
下载地址：http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.15.1/RPMS/noarch/

[hadoop@hadoop002 impala]$ sudo rpm -ivh --nodeps  sentry*

2.2 安装 impala rpm包

[hadoop@hadoop002 impala]$ sudo rpm -ivh --nodeps  impala*
[sudo] password for hadoop: 
warning: impala-2.12.0+cdh5.15.1+0-1.cdh5.15.1.p0.4.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:impala-2.12.0+cdh5.15.1+0-1.cdh5.################################# [ 14%]
   2:impala-catalog-2.12.0+cdh5.15.1+0################################# [ 29%]
   3:impala-server-2.12.0+cdh5.15.1+0-################################# [ 43%]
   4:impala-state-store-2.12.0+cdh5.15################################# [ 57%]
   5:impala-udf-devel-2.12.0+cdh5.15.1################################# [ 71%]
   6:impala-shell-2.12.0+cdh5.15.1+0-1################################# [ 86%]
   7:impala-debuginfo-2.12.0+cdh5.15.1################################# [100%]

3.使用rpm安装之后在/usr/lib/impala/lib下面会发现很多无效软连接，删除无效的软连接

[hadoop@hadoop002 lib]$ cd /usr/lib/impala/lib
[hadoop@hadoop002 lib]$ sudo rm -rf avro*.jar
[hadoop@hadoop002 lib]$ sudo rm -rf hive-*.jar
[hadoop@hadoop002 lib]$ sudo rm -rf hbase-*.jar
[hadoop@hadoop002 lib]$ sudo rm -rf parquet-hadoop-bundle.jar
[hadoop@hadoop002 lib]$ sudo rm -rf sentry-*.jar
[hadoop@hadoop002 lib]$ sudo rm -rf zookeeper.jar
[hadoop@hadoop002 lib]$ sudo rm -rf libhadoop.so
[hadoop@hadoop002 lib]$ sudo rm -rf libhadoop.so.1.0.0
[hadoop@hadoop002 lib]$ sudo rm -rf libhdfs.so
[hadoop@hadoop002 lib]$ sudo rm -rf libhdfs.so.0.0.0
[hadoop@hadoop002 lib]$ 

4.重建软连接，脚本如下

#!/bin/bash
HBASE_HOME=/home/hadoop/app/hbase-1.2.0-cdh5.15.1
HADOOP_HOME=/home/hadoop/app/hadoop
HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.15.1
ZK_HOME=/home/hadoop/app/zookeeper-3.4.5-cdh5.15.1
SENTRY_HOME=/usr/lib/sentry

sudo ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-annotations.jar
sudo ln -s $HADOOP_HOME/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-auth.jar
sudo ln -s $HADOOP_HOME/share/hadoop/tools/lib/hadoop-aws-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-aws.jar
sudo ln -s $HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-hdfs.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar
sudo ln -s $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-api.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-client.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-server-common.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-server-nodemanager.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar
sudo ln -s $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.15.1.jar /usr/lib/impala/lib/hadoop-yarn-server-web-proxy.jar
sudo ln -s $HADOOP_HOME/lib/native/libhadoop.so /usr/lib/impala/lib/libhadoop.so
sudo ln -s $HADOOP_HOME/lib/native/libhadoop.so.1.0.0 /usr/lib/impala/lib/libhadoop.so.1.0.0
sudo ln -s $HADOOP_HOME/lib/native/libhdfs.so /usr/lib/impala/lib/libhdfs.so
sudo ln -s $HADOOP_HOME/lib/native/libhdfs.so.0.0.0 /usr/lib/impala/lib/libhdfs.so.0.0.0

sudo ln -s $HBASE_HOME/lib/avro-1.7.6-cdh5.15.1.jar /usr/lib/impala/lib/avro.jar
sudo ln -s $HBASE_HOME/lib/hbase-annotations-1.2.0-cdh5.15.1.jar /usr/lib/impala/lib/hbase-annotations.jar
sudo ln -s $HBASE_HOME/lib/hbase-client-1.2.0-cdh5.15.1.jar /usr/lib/impala/lib/hbase-client.jar
sudo ln -s $HBASE_HOME/lib/hbase-common-1.2.0-cdh5.15.1.jar /usr/lib/impala/lib/hbase-common.jar
sudo ln -s $HBASE_HOME/lib/hbase-protocol-1.2.0-cdh5.15.1.jar /usr/lib/impala/lib/hbase-protocol.jar

sudo ln -s $HIVE_HOME/lib/ant-1.9.1.jar /usr/lib/impala/lib/hive-ant.jar
sudo ln -s $HIVE_HOME/lib/hive-beeline-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-beeline.jar
sudo ln -s $HIVE_HOME/lib/hive-common-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-common.jar
sudo ln -s $HIVE_HOME/lib/hive-exec-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-exec.jar
sudo ln -s $HIVE_HOME/lib/hive-hbase-handler-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-hbase-handler.jar
sudo ln -s $HIVE_HOME/lib/hive-metastore-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-metastore.jar
sudo ln -s $HIVE_HOME/lib/hive-serde-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-serde.jar
sudo ln -s $HIVE_HOME/lib/hive-service-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-service.jar
sudo ln -s $HIVE_HOME/lib/hive-shims-common-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-shims-common.jar
sudo ln -s $HIVE_HOME/lib/hive-shims-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-shims.jar
sudo ln -s $HIVE_HOME/lib/hive-shims-scheduler-1.1.0-cdh5.15.1.jar /usr/lib/impala/lib/hive-shims-scheduler.jar
sudo ln -s $HIVE_HOME/lib/parquet-hadoop-bundle-1.5.0-cdh5.15.1.jar /usr/lib/impala/lib/parquet-hadoop-bundle.jar

sudo ln -s $SENTRY_HOME/lib/sentry-binding-hive-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-binding-hive.jar
sudo ln -s $SENTRY_HOME/lib/sentry-core-common-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-core-common.jar
sudo ln -s $SENTRY_HOME/lib/sentry-core-model-db-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-core-model-db.jar
sudo ln -s $SENTRY_HOME/lib/sentry-policy-common-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-policy-common.jar
sudo ln -s $SENTRY_HOME/lib/sentry-policy-db-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-policy-db.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-cache-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-provider-cache.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-common-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-provider-common.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-db-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-provider-db.jar
sudo ln -s $SENTRY_HOME/lib/sentry-provider-file-1.5.1-cdh5.15.1.jar /usr/lib/impala/lib/sentry-provider-file.jar

sudo ln -s $ZK_HOME/zookeeper-3.4.5-cdh5.15.1.jar /usr/lib/impala/lib/zookeeper.jar

安装配置bigtop-utils
5.1 安装bigtop-utils

[hadoop@hadoop002 impala]$ sudo rpm -ivh bigtop*
[sudo] password for hadoop: 
warning: bigtop-utils-0.7.0+cdh5.8.3+0-1.cdh5.8.3.p0.7.el6.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:bigtop-utils-0.7.0+cdh5.8.3+0-1.c################################# [100%]
[hadoop@hadoop002 impala]$ 

5.2 配置bigtop-utils
[hadoop@hadoop002 bigtop-utils]$ sudo vim /etc/default/bigtop-utils

export JAVA_NATIVE_PATH=/usr/java/jdk1.8.0_161

[hadoop@hadoop002 bigtop-utils]$ source /etc/default/bigtop-utils
6. 配置impala的配置文件
6.1 拷贝 hadoop的hdfs-site.xml 到目录/etc/impala/conf， 并修改。同时创建目录/var/run/hadoop-hdfs/dn
[hadoop@hadoop002 bigtop-utils]$ sudo cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml /etc/impala/conf
[hadoop@hadoop002 bigtop-utils]$ sudo vim /etc/impala/conf/hdfs-site.xml 

    dfs.client.read.shortcircuit
    true


    dfs.domain.socket.path
    /var/run/hadoop-hdfs/dn


  dfs.datanode.hdfs-blocks-metadata.enabled
  true


   dfs.client.use.legacy.blockreader.local
   false


   dfs.datanode.data.dir.perm
   750


   dfs.block.local-path-access.user
   hadoop


   dfs.client.file-block-storage-locations.timeout
   3000

创建目录

[hadoop@hadoop002 lib]$ sudo mkdir -p /var/run/hadoop-hdfs/dn
[hadoop@hadoop002 lib]$ sudo chown hadoop:hadoop /var/run/hadoop-hdfs/dn

6.2 拷贝 hadoop的hdfs-site.xml 到目录/etc/impala/conf
[hadoop@hadoop002 bigtop-utils]$ sudo cp $HADOOP_HOME/etc/hadoop/core-site.xml /etc/impala/conf
[hadoop@hadoop002 bigtop-utils]$ cat /etc/impala/conf/core-site.xml 

　　dfs.client.read.shortcircuit
　　true


　　dfs.client.read.shortcircuit.skip.checksum
　　false


　　dfs.datanode.hdfs-blocks-metadata.enabled
　　true


　　fs.AbstractFileSystem.adl.impl
　　org.apache.hadoop.fs.adl.Adl


　　fs.adl.impl
　　org.apache.hadoop.fs.adl.AdlFileSystem

6.3 拷贝Hive的配置文件
[hadoop@hadoop002 bigtop-utils]$ sudo cp $HIVE_HOME/conf/hive-site.xml /etc/impala/conf
7.复制mysql-connector jar包到/usr/lib/impala/lib并修改配置文件

[hadoop@hadoop002 lib]$ sudo cp mysql-connector-java-5.1.47.jar  /usr/lib/impala/lib

并且修改/usr/bin/catalogd中修改CLASSPATH，添加MySQL驱动包：
export CLASSPATH=“”${IMPALA_HOME}/lib/mysql-connector-java-5.1.40-bin.jar:……”

8. 修改impala的配置文件
[hadoop@hadoop002 lib]$ sudo vim /etc/default/impala 

IMPALA_CATALOG_SERVICE_HOST=hadoop002
IMPALA_STATE_STORE_HOST=hadoop002
IMPALA_STATE_STORE_PORT=24000
IMPALA_BACKEND_PORT=22000
IMPALA_LOG_DIR=/var/log/impala

IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} "
IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}"
IMPALA_SERVER_ARGS=" \
    -log_dir=${IMPALA_LOG_DIR} \
    -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
    -state_store_port=${IMPALA_STATE_STORE_PORT} \
    -use_statestore \
    -state_store_host=${IMPALA_STATE_STORE_HOST} \
    -be_port=${IMPALA_BACKEND_PORT}
    --kudu_master_hosts=hadoop002:7051"

ENABLE_CORE_DUMPS=true

9.修改hadoop的hdfs-site.xml配置文件与上文保持一致并重启hdfs
10. 启动impala服务

[hadoop@hadoop002 sbin]$ sudo service impala-state-store start 
Started Impala State Store Server (statestored):           [  OK  ]
[hadoop@hadoop002 sbin]$ sudo service impala-catalog start
Started Impala Catalog Server (catalogd) :                 [  OK  ]
[hadoop@hadoop002 sbin]$ sudo service impala-server start
Started Impala Server (impalad):                           [  OK  ]
[hadoop@hadoop002 sbin]$ Redirecting stdout to /var/log/impala/impalad.INFO
Redirecting stderr to /var/log/impala/impalad.ERROR

11.启动impala-shell

[hadoop@hadoop002 sbin]$ impala-shell
Starting Impala Shell without Kerberos authentication
Error connecting: TTransportException, Could not connect to hadoop002:21000
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.12.0-cdh5.15.1 (64f4e19) built on Thu Aug  9 09:21:02 PDT 2018)

Every command must be terminated by a ';'.
***********************************************************************************
[Not connected] > 

测试建表语句

CREATE TABLE test_first
(
	id BIGINT,
	name STRING
)
DISTRIBUTE BY HASH INTO 16 BUCKETS
TBLPROPERTIES(
 'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',
'kudu.table_name' = 'test_first',
'kudu.master_addresses' = 'hadoop002:7051',
'kudu.key_columns' = 'id'
);

12.查看impala服务

[hadoop@hadoop002 sbin]$  ps -ef | grep impala
impala   13432     1  0 20:22 ?        00:00:00 /usr/lib/impala/sbin/statestored -log_dir=/var/log/impala -state_store_port=24000
hadoop   14990 22289  0 20:28 pts/1    00:00:00 grep --color=auto impala
[hadoop@hadoop002 sbin]$ 

13.浏览器页面访问
访问impalad的管理界面：http://hadoop002:25000/
访问statestored的管理界面：http://hadoop002:25010/