Hue使用（Apache Hadoop UI系统）

一、Hue的安装部署

1. Hue版本：cdh5.3.6-hue-3.7.0

2. 需要编译才能使用

   1. 需要联网

   2. 依赖(针对不同Linux版本)

      ant
      asciidoc
      cyrus-sasl-devel
      cyrus-sasl-gssapi
      gcc
      gcc-c++
      krb5-devel
      libtidy (for unit tests only)
      libxml2-devel
      libxslt-devel
      mvn (from maven package or maven3 tarball)
      mysql
      mysql-devel
      openldap-devel
      python-devel
      sqlite-devel
      openssl-devel (for version 7+)

   3. 检查系统有没有这些包
      sudo rpm -qa | grep package_name
   4. yum 安装
      sudo yum install ant asciidoc cyrus-sasl-devel gcc gcc-c++ krb5-devel libtidy libxml2-devel libxslt-devel mvn mysql-devel openldap-devel python-devel sqlite-devel openssl-devel -y
3. 编译
   
   1. 下载解压：
      tar -zxvf hue-3.7.0-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/
   2. 进入hue家目录：
      cd /opt/cdh-5.3.6/hue-3.7.0-cdh5.3.6
   3. 编译：make apps
      
   4. 卸载open-jdk
      
      1. 查看：sudo rpm -qa | grep java
      2. 卸载：sudo rpm -e --nodeps java_cup-0.10k-5.el6.x86_64 java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 tzdata-java-2016j-1.el6.noarch java-1.7.0-openjdk-1.7.0.121-2.6.8.1.el6_8.x86_64 java-1.7.0-openjdk-devel-1.7.0.121-2.6.8.1.el6_8.x86_64
      3. 

4. 配置

   1. hue的全局配置文件：desktop/conf/hue.ini

   2. 配置文件中所有的标记启用以中括号表示

      [desktop]
          secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mnon this address and port
          http_host=hadoop-senior01.ibeifeng.com
          http_port=8888
          # Time zone name
          time_zone=Asia/Shanghai
      

      

5. 使用
   
   1. 启动：build/env/bin/supervisor
   2. web ui：hadoop-senior01.ibeifeng.com:8888
      

---

二、 Hue框架集成

1. hadoop-HDFS集成

   1. HDFS的配置

      1. hdfs-site.xml

         
         <property>
             <name>dfs.webhdfs.enabledname>
             <value>truevalue>
         property>

      2. core-site.xml

         
         <property>
             <name>hadoop.proxyuser.hue.hostsname>
             <value>*value>
         property>
         <property>
             <name>hadoop.proxyuser.hue.groupsname>
             <value>*value>
         property> 

      3. 重启hdfs

   2. hue的配置：hue的全局配置文件：desktop/conf/hue.ini

      [[hdfs_clusters]]
      fs_defaultfs=hdfs://hadoop-senior01.ibeifeng.com:8020
      webhdfs_url=http://hadoop-senior01.ibeifeng.com:50070/webhdfs/v1
      hadoop_conf_dir=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop
      hadoop_hdfs_home=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6
      hadoop_bin=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/bin
      

      

   3. 通过网页查看HDFS
      

2. Hadoop-yarn集成

   1. 配置hue [[yarn_clusters]]

      resourcemanager_host=hadoop-senior01.ibeifeng.com
      resourcemanager_port=8032
      submit_to=True
      resourcemanager_api_url=http://hadoop-senior01.ibeifeng.com:8088
      proxy_api_url=http://hadoop-senior01.ibeifeng.com:8088
      history_server_api_url=http://hadoop-senior01.ibeifeng.com:19888    

      

   2. 测试一个mr程序：
      bin/yarn jar /opt/datas/wc2.jar org.apache.hadoop.wordcount.WordCount /input /hue
      

3. Hive的集成

   1. hive的metastore
      
      1. Derby存储：只能启动一个实例
      2. 本地MySQL：MySQL与hive在同一台机器上
      3. 远程的MySQL
         
         
         1. a) 修改远程机器的hive配置文件hive-site.xml
            
hive.metastore.uris



         2. b) 启动远程metastore
            bin/hive --service metastore
后台启动：nohup bin/hive --service metastore &

   2. hive的配置
      
      1. hiveserver2两种模式：binary和http，默认为binary
      2. 开启hiveserver2：bin/hiveserver2

   3. hue的配置 [beeswax]

      hive_server_host=hadoop-senior01.ibeifeng.com
      hive_server_port=10000
      hive_conf_dir=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/conf
      hive_home_dir=/opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6       

      

   4. Hue上查看：
      

4. RDBMS的集成，配置hue

   1. sqlite [[[sqlite]]]

      nice_name=SQLite
      name=/opt/cdh-5.3.6/hue-3.7.0-cdh5.3.6/desktop/desktop.db
      engine=sqlite

      

   2. MySql

      nice_name="My SQL DB"
      engine=mysql
      host=hadoop-senior01.ibeifeng.com
      port=3306
      user=root
      password=123456

      
      
      

5. oozie的集成

   1. hue [liboozie]
      oozie_url=http://hadoop-senior01.ibeifeng.com:11000/oozie
      
      启动Oozie

   2. 存在问题，无法检测到sharelib

      1. 原因：同一家公司出的，有默认的检索地址

      2. 解决：oozie-site.xml

         <property>
                       <name>oozie.service.WorkflowAppService.system.libpathname>
             <value>/user/oozie/share/libvalue>
         property>
         

      3. 重新生成sharelib
         bin/oozie-setup.sh sharelib create -fs hdfs://hadoop-senior01.ibeifeng.com:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz

      4. 重启oozie：bin/oozied.sh start
         

---

三、 Hue中运行workflow

1. 一个节点对应一个.xml文件
2. 创建一个workflow，名称为test2
   
3. 在中间添加一个Mapreduce节点mr-node
   
4. mr-node节点中主要配置属性
   
5. 作业XML中hueworkflow.xml只保留了节点中的内容
6. 运行结束后可以看到结果