ambari2.7.3 安装 impala

1.下载ambari-impala-service

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
sudo git clone https://github.com/cas-bigdatalab/ambari-impala-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/IMPALA

2./etc/yum.repos.d新建impala.repo

[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.8.0
gpgkey =https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

3.重启ambari-server

sudo service ambari-server restart

4.安装impala

Actions-->Add service -->impala，然后一步一步安装

 

中间发生一次报错信息：

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in
    BeforeAnyHook().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
    setup_users()
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
    groups = params.user_to_groups_dict[user],
KeyError: u'impala'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-863.json', '/

1 命令行

# cd /var/lib/ambari-server/resources/scripts
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
"ignore_groupsusers_create": "false",
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

详细参数详见

# python configs.py --help

2 rest api

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env"

从返回的items中找到tag，然后再请求（增加tag参数）：

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env&tag=$tag"

返回结果中包含所有的property

"properties" : {
"agent_mounts_ignore_list" : "",
"alerts_repeat_tolerance" : "1",
"enable_external_ranger" : "false",
"fetch_nonlocal_groups" : "true",
"hide_yarn_memory_widget" : "false",
"ignore_bad_mounts" : "false",
"ignore_groupsusers_create" : "false",

修改需要先将上面的配置保存到一个文件，比如/tmp/configurations.tmp，然后修改，然后再调用接口:

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X PUT -d @/tmp/configurations.tmp http://ambari.server:8080/api/v1/clusters/$cluster_name

然后再安装即可；

因为要去cloudera官网下载impala的rpm文件，安装过程比较慢，可能会失败，多试几次就好了

如果需要重装 报错

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA2.6/package/scripts/impala-catalog.py", line 36, in 
    ImpalaCatalog().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA2.6/package/scripts/impala-catalog.py", line 12, in install
    self.installImpala(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA2.6/package/scripts/impala_base.py", line 19, in installImpala
    Package(pack)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
    self._pkg_manager.install_package(package_name, self.__create_context())
  File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
    shell.repository_manager_executor(cmd, self.properties, context)
  File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor
    raise RuntimeError(message)
RuntimeError: Failed to execute command '/usr/bin/yum -y install impala-server', exited with code '1', message: 'https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

To address this issue please refer to the below knowledge base article 



https://access.redhat.com/articles/1320623



If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/



https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/bigtop-utils-0.7.0%2Bcdh5.16.1%2B0-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/avro-libs-1.7.6%2Bcdh5.16.1%2B143-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/x86_64/impala-2.12.0%2Bcdh5.16.1%2B0-1.cdh5.16.1.p0.3.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/x86_64/impala-server-2.12.0%2Bcdh5.16.1%2B0-1.cdh5.16.1.p0.3.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/parquet-1.5.0%2Bcdh5.16.1%2B200-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/parquet-format-2.1.0%2Bcdh5.16.1%2B22-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/sentry-1.5.1%2Bcdh5.16.1%2B559-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/solr-4.10.3%2Bcdh5.16.1%2B532-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.





Error downloading packages:

  avro-libs-1.7.6+cdh5.16.1+143-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  impala-2.12.0+cdh5.16.1+0-1.cdh5.16.1.p0.3.el7.x86_64: [Errno 256] No more mirrors to try.

  parquet-format-2.1.0+cdh5.16.1+22-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  sentry-1.5.1+cdh5.16.1+559-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  solr-4.10.3+cdh5.16.1+532-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  parquet-1.5.0+cdh5.16.1+200-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  impala-server-2.12.0+cdh5.16.1+0-1.cdh5.16.1.p0.3.el7.x86_64: [Errno 256] No more mirrors to try.

  bigtop-utils-0.7.0+cdh5.16.1+0-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

需要执行

yum clean all
yum list update
yum makecache
yum repolist

5.hdfs增加配置

#HDFS config we need add below config to /etc/hadoop/conf/core-site.xml

    dfs.client.read.shortcircuit 
   true



    dfs.client.read.shortcircuit.skip.checksum
        false


 
    dfs.datanode.hdfs-blocks-metadata.enabled 
    true

we need add below config to /etc/hadoop/conf/hdfs-site.xml

    dfs.datanode.hdfs-blocks-metadata.enabled 
    true

 
    dfs.block.local-path-access.user 
    impala


    dfs.client.file-block-storage-locations.timeout.millis
    60000

 6.重启hdfs

7.启动impala

报错：

cat impala-server.log （每台impala机器都要做）
Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils

vim /etc/default/bigtop-utils

JAVA_HOME=/opt/jdk1.8.0_201 （与/etc/profile保持一样）

报错

cat /var/log/impala/impalad.ERROR

hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
ClassNotFoundException: com.ctc.wstx.io.InputBootstrapperjava.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Wrote minidump to /var/log/impala/minidumps/impalad/0a803f89-a804-4bb4-242e02b9-94eba9b1.dmp

把hbase的jar包拷贝到 /usr/lib/impala/lib 并创建软连接

这里注意根据安装的impala的版本对应的cdh的版本选择对应的hbase的版本，否则可能出现版本不兼容的问题

ln -s hbase-annotations-1.2.0-cdh5.11.0.jar hbase-annotations.jar
ln -s hbase-client-1.2.0-cdh5.11.0.jar hbase-client.jar
ln -s hbase-common-1.2.0-cdh5.11.0.jar hbase-common.jar
ln -s hbase-examples-1.2.0-cdh5.11.0.jar hbase-examples.jar
ln -s hbase-external-blockcache-1.2.0-cdh5.11.0.jar hbase-external-blockcache.jar
ln -s hbase-hadoop2-compat-1.2.0-cdh5.11.0.jar hbase-hadoop2-compat.jar
ln -s hbase-hadoop-compat-1.2.0-cdh5.11.0.jar hbase-hadoop-compat.jar
ln -s hbase-it-1.2.0-cdh5.11.0.jar hbase-it.jar
ln -s hbase-prefix-tree-1.2.0-cdh5.11.0.jar hbase-prefix-tree.jar
ln -s hbase-procedure-1.2.0-cdh5.11.0.jar hbase-procedure.jar
ln -s hbase-protocol-1.2.0-cdh5.11.0.jar hbase-protocol.jar
ln -s hbase-resource-bundle-1.2.0-cdh5.11.0.jar hbase-resource-bundle.jar
ln -s hbase-rest-1.2.0-cdh5.11.0.jar hbase-rest.jar
ln -s hbase-rsgroup-1.2.0-cdh5.11.0.jar hbase-rsgroup.jar
ln -s hbase-server-1.2.0-cdh5.11.0.jar hbase-server.jar
ln -s hbase-shell-1.2.0-cdh5.11.0.jar hbase-shell.jar
ln -s hbase-spark-1.2.0-cdh5.11.0.jar hbase-spark.jar
ln -s hbase-thrift-1.2.0-cdh5.11.0.jar hbase-thrift.jar

报错

E0524 17:18:04.260188  1741 impala-server.cc:279] Invalid short-circuit reads co
  - Impala cannot read or execute the parent directory of dfs.domain.socket.path

在配置文件中查看dfs.domain.socket.path 为$path

chmod 775  ../$path

def:

http://www.mamicode.com/info-detail-2589065.html

https://github.com/cas-bigdatalab/ambari-impala-service/(ambari-impala github官网有详细步骤持续更新)