Apache Ranger源码编译及使用

Ranger源码编译、使用手册

1 Ranger简介

Apache Ranger提供一个集中式安全管理框架,它可以对Hadoop生态的组件如Hive,Hbase进行细粒度的数据访问控制.通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问HDFS文件夹、HDFS文件、数据库、表、字段权限.这些策略可以为不同的用户和组来设置,同时权限可与hadoop无缝对接.

2 准备

2.1 环境说明

1 Ranger源码编译依赖如下linux组件:maven,git,gcc,mysql

2 安装git和gcc时采用yum安装,请配置好本地源

3 安装mysql是请确保清理好系统自带的mysql.

 

2.1.1虚拟机里Linux系统版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

注:官网强调一定是1.7以上版本.

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14    |

+-----------+

注:

1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar

2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享

2.1.4 Maven版本

 [root@localhost bin]# mvn -version

Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;2014-02-15T01:37:52+08:00)

Maven home: /root/maven-3.2.1

Java version: 1.7.0_67,vendor: Oracle Corporation

Java home:/root/jdk1.7.0_67/jre

Default locale: en_US,platform encoding: UTF-8

OS name: "linux",version: "2.6.32-431.el6.x86_64", arch: "amd64", family:"unix"

2.1.5 git版本

 [root@localhost native]# git version

git version 1.7.1

 

2.2 编译准备

2.2.1 安装maven

[root@localhost ~]# cd /root

#下载地址

#https://maven.apache.org/download.cgi 最新版

#http://apache.opencas.org/maven/binaries/apache-maven-3.2.1-bin.tar.gz

tar –zxvf apache-maven-3.2.1-bin.tar.gz

mv apache-maven-3.2.1-bin maven-3.2.1

#修改环境变量,在~/.bash_profile里定义MAVEN_HOME并追加到PATH里

export MAVEN_HOME=/root/maven-3.2.1

:$MAVEN_HOME/bin:$PATH

#source环境变量,测试maven版本

source ~/.bash_profile

mvn –version

2.2.2 安装git

这里通过本地源yum方式安装.

yum install git

注:Linux本地源配置见下:

1) 虚拟机加载Linux ISO镜像

2) 找到rom对应设备名

[root@localhost ~]# lsblk

NAME                        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

sr0                          11:0    1 1024M 0 rom 

sr1                          11:1    1 3.6G  0 rom 

sr2                          11:2    1 1024M 0 rom 

sda                           8:0    0   8G  0 disk

?..sda1                       8:1    0 500M  0 part /boot

?..sda2                        8:2    0 7.5G  0 part

 ?..VolGroup-lv_root (dm-0) 253:0   0  6.7G  0 lvm /

 ?..VolGroup-lv_swap (dm-1) 253:1   0  816M  0 lvm [SWAP]

3) 创建目录

[root@localhost ~]#mkdir –p /mnt/cdrom/

4) 挂载镜像

mount -t iso9660 /dev/sr1 /mnt/cdrom

5) 配置linux更新源,/etc/yum.repos.d,修改成如下:

vi /etc/yum.repos.d/redhat.repo

[rhel-source]

name=Redhat

baseurl=file:///mnt/cdrom/

enabled=1

gpgcheck=1

gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release

6) 更新更新源

yum clean all

yum update list

2.2.3 安装gcc

yum install gcc

2.2.4 安装mysql

1) 安装Mysql服务、客户端

rpm –ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

2) 启动mysql服务

service mysql start

3) 修改mysql初始密码,先找到安装时的初始密码,在修改成自己的密码

[root@localhost ~]#cat /root/.mysql_secret

# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY

mysql –uroot -p RUmKBqcY

set password=password(‘root’)

3 编译

3.1编译中

1) 拷贝ranger源代码

[root@localhost ~]# cd ~

git clone https://github.com/apache/incubator-ranger.git

cd incubator-ranger

git checkout ranger-0.5

2) 编译ranger源代码

cd ~/incubator-ranger

export MAVEN_OPTS="-Xmx512M" 

export JAVA_HOME= /root/jdk1.7.0_67

export PATH=$JAVA_HOME/bin:$PATH

mvn clean compile package assembly:assembly install

ls target/*.tar.gz

[root@localhost ~]#ls/root/incubator-ranger/target/*.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hbase-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-kafka-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-kms.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-knox-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-migration-util.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-solr-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-src.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-storm-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-yarn-plugin.tar.gz

3.2 编译问题

1) 编译过程异常缓慢,一般要3-4天时间

2) 如果出现异常不好定位,可在maven参数里加-X以debug模式诊断

3) 源码编译的相关问题及解决方案

a) Failedto execute goal on project ranger-hdfs-plugin: Could not resolve dependenciesfor project

     security_plugins.ranger-hdfs-plugin:

     ranger-hdfs-plugin:jar:0.5.0:The following artifacts could not be resolved:

org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde,eigenbase:eigenbase-properties:jar:1.1.4,net.hydromatic:linq4j:jar:0.4,net.hydromatic:quidem:jar:0.1.1:

Could not transfer artifactorg.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde from/to conjars(http://conjars.org/repo): conjars.org:Unknownhost conjars.org -> [Help 1]

 

解决方案:手动下载错误提示里的jar并拷贝到相应的m2目录内.

源:http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/

     目标:/root/.m2/repository/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde

  源:http://conjars.org/repo/eigenbase/eigenbase-properties/1.1.4/

  目标:/root/.m2/repository/eigenbase/eigenbase-properties/1.1.4/

  源:http://conjars.org/repo/net/hydromatic/linq4j/0.4/

  目标:/root/.m2/repository/net/hydromatic/linq4j/0.4/

  源:http://conjars.org/repo/net/hydromatic/quidem/0.1.1/

  目标:/root/.m2/repository/net/hydromatic/quidem/0.1.1/

 

b)[ERROR]error: error reading /root/.m2/repository/org/json/json/20090211/json-20090211.jar;zip file is empty

[ERROR] -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:Failed to execute goalorg.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) onproject ranger-hdfs-plugin: Compilation failure

error: error reading/root/.m2/repository/org/json/json/20090211/json-20090211.jar; zip file isempty

 

     atorg.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)

.

解决方案:手动下载错误提示里的json-20090211.jar并拷贝到相应的m2目录内.

c) Runningorg.apache.hadoop.crypto.key.kms.server.TestKeyAuthorizationKeyProvider此步耗时较长,大约20分钟左右

4 配置

Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr

注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr.

4.1 Solr或SolrCloud安装配置

cd /root/incubator-ranger/security-admin/contrib/solr_for_audit_setup

#打开install.properties文件,修改参数的值如下所示:

vi install.properties

JAVA_HOME=/root/jdk1.7.0_67

SOLR_INSTALL=true

SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz

SOLR_INSTALL_FOLDER=/opt/solr

SOLR_RANGER_HOME=/opt/solr/ranger_audit_server

SOLR_DEPLOYMENT=standalone

SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data

SOLR_LOG_FOLDER=/var/log/solr/ranger_audits

SOLR_MAX_MEM=2g

#安装单节点的solr

./ setup.sh

#按照如下安装提示启动、停止solr服务

cat/opt/solr/ranger_audit_server/install_notes.txt

#启动solr

/opt/solr/ranger_audit_server/scripts/start_solr.sh

#验证solr服务是否可正常使用

lsof –i:6083

http://192.168.56.101:6083

#确保防火墙已经关闭

chkconfig iptables off

4.2 Ranger Admin安装配置

cp/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz /root

cd /root

tar –zxvf ranger-0.5.0-admin.tar.gz

cd ranger-0.5.0-admin

#打开Ranger Admin里install.properties文件,修改参数的值如下所示:

vi install.properties

setup_mode=SeparateDB

DB_FLAVOR=MYSQL

db_root_user=root

db_root_password=root

db_host=localhost

db_name=ranger

db_user=root

db_password=root

audit_store=db

audit_db_name=ranger_audit

audit_db_user=root

audit_db_password=root

policymgr_external_url=http://localhost:6080

policymgr_http_enabled=true

unix_user=ranger

unix_group=ranger

#安装Ranger Admin

./setup.sh

#启动Ranger Admin服务

ranger-admin start

#验证Ranger Admin服务,如果出现Ranger的登录界面,说Okay了.注:用户名/密码 admin/admin

lsof –i:6080

http://192.168.56.101:6080

4.3 Ranger-usersync安装配置

cp/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz /root/

cd /root

tar –zxvf ranger-0.5.0-usersync.tar.gz

cd ranger-0.5.0-usersync

#打开usersync Plugin里install.properties文件,修改参数的值如下所示:

vi install.properties

POLICY_MGR_URL=http://localhost:6080

SYNC_SOURCE=unix

#同步周期,1分钟

SYNC_INTERVAL=1

logdir=/var/log/ranger/usersync

#安装usersync Plugin

./setup.sh

#启用usersync Plugin插件

./ranger­-usersync-services.sh start

 

4.4 HDFS-Plugin安装配置

cp/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz /root

cd /root

tar –zxvf ranger-0.5.0-hdfs-plugin.tar.gz

cd ranger-0.5.0-hdfs-plugin

#打开HDFS Plugin里install.properties文件,修改参数的值如下所示:

vi install.properties

POLICY_MGR_URL=http://localhost:6080

SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar

REPOSITORY_NAME=hadoopdev

XAAUDIT.DB.IS_ENABLED=true

XAAUDIT.DB.FLAVOUR=MYSQL

XAAUDIT.DB.HOSTNAME=localhost

XAAUDIT.DB.DATABASE_NAME=ranger_audit

XAAUDIT.DB.USER_NAME=root

XAAUDIT.DB.PASSWORD=root

#组件对应的用户,这里设置为空.一般Hadoop的内置用户是HDFS或则hadoop

CUSTOM_USER=root

CUSTOM_GROUP=root

#启用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]#./enable-hdfs-plugin.sh

Customuser and group are not available, using default user and group.

ERROR:Unable to find the conf directory of component [hadoop]; dir [/root/hadoop/conf] not found.

Exitinginstallation.

:这里报错,需要额外将HADOOP的conf做个软连接到/root/hadoop/conf.

ln-s /root/hadoop-2.7.1/etc/hadoop/ /root/hadoop/conf

#再次启用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh

Custom user and group are not available,using default user and group.

ERROR: Unable to find the lib directory ofcomponent [hadoop];  dir [/root/hadoop/lib] not found.

Exiting installation.

#这里需要将HDFS Plugin内的jar和HADOOP包含的HDFS jar都指向/root/hadoop/lib

cp /root/ranger-0.5.0-hdfs-plugin/lib/ranger-hdfs-plugin-impl/*.jar/root/hadoop-2.7.1/share/hadoop/hdfs/lib/

mkdir /root/hadoop/lib

ln -s /root/hadoop-2.7.1/share/hadoop/hdfs/lib//root/hadoop/lib/

#再一次启用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh

#验证HDFS Plugin服务,这时登丽Ranger的管理员界面验证下HDFS plugin是够加载成功,发现并没有.

 

原因是安装HDFS plugin时install.properties文件里定义的REPOSITORY_NAME(值为hadoopdev)并未通过Ranger Admin在HDFS插件里的服务管理里注册成服务(名hadoopdev).

解决方案:

1 登录Ranger Adming

2 点击HDFS plugin的添加按钮


3 定义服务名为hadoopdev,提交其它信息后保存


#再次验证HDFS plugin插件,则发现已经正常加载

注:

1 如果没有安装和开启Ranger-usersync服务的情况下直接测试HDFS赋权权限是不成功的.

2 Ranger Admin的日志文件见 /root/ranger-0.5.0-admin/ews/logs/xa_portal.log

4.5 Hive-Plugin安装配置

先启动hive的metastore和hiveserver2服务

nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &

nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &

#beeline验证

[root@localhost ~]# beeline -u"jdbc:hive2://192.168.56.101:10000" -n root -p test

Connectingto jdbc:hive2://192.168.56.101:10000

Connectedto: Apache Hive (version 1.2.1)

Driver:Hive JDBC (version 1.2.1)

Transactionisolation: TRANSACTION_REPEATABLE_READ

Beelineversion 1.2.1 by Apache Hive

0:jdbc:hive2://192.168.56.101:10000> show databases;

+----------------+--+

|database_name  |

+----------------+--+

|default        |

| shenl          |

+----------------+--+

 

2) Ranger-Admin里注册hive plugin的服务

 

3) 配置、启用hive plugin

cp/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz /root

cd /root

tar -zxvf ranger-0.5.0-hive-plugin.tar.gz

cd ranger-0.5.0-hive-plugin

#打开Hive Plugin里install.properties文件,修改参数的值如下所示:

vi install.properties

POLICY_MGR_URL=http://192.168.56.101:6080

REPOSITORY_NAME=hivedev

XAAUDIT.DB.IS_ENABLED=true

XAAUDIT.DB.FLAVOUR=MYSQL=MYSQL

XAAUDIT.DB.HOSTNAME=localhost

XAAUDIT.DB.DATABASE_NAME=ranger_audit

XAAUDIT.DB.USER_NAME=root

XAAUDIT.DB.PASSWORD=root

 

[[email protected]]# ./enable-hive-plugin.sh

Customuser and group is available, using custom user and group.

ERROR:Unable to find the conf directory of component [hive]; dir [/root/hive/conf]not found.

Exitinginstallation.

#解决方法:

ln -s /root/apache-hive-0.13.0-bin/conf//root/hive/conf

 

[[email protected]]# ./enable-hive-plugin.sh

Customuser and group is available, using custom user and group.

ERROR:Unable to find the lib directory of component [hive];  dir [/root/hive/lib] not found.

Exitinginstallation.

#解决方法:

cp root/ranger-0.5.0-hive-plugin/lib/ranger-hive-plugin-impl/*.jar/root/apache-hive-1.2.1-bin/lib/

mkdir /root/hive/lib

ln -s/root/apache-hive-1.2.1-bin/lib/ /root/hive/lib/

#将生成的hiveserver2-site.xml拷贝到hive的配置目录下

cp /root/hive/conf/*/root/apache-hive-1.2.1-bin/conf/

#如果hive配置目录里有hiveserver2-site.xml,则需要添加如下内容:

       hive.security.authorization.enabled

        true

   

   

       hive.security.authorization.manager

        org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory

   

   

       hive.security.authenticator.manager

       org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator

   

   

       hive.conf.restricted.list

       hive.security.authorization.enabled,hive.security.authorization.manager,hive.security.authenticator.manager

   


5 使用

5.1 HDFS Plugin验证

 

1 新增linux用户shenl进行测试

useradd shenl

hadoop fs –mkdir /shenl

2 登录Ranger Admin,新增用户shenl


3 登录Ranger Admin,在hadoopdev里注册新的Policy

4 添加该Policy的权限,指定可以对HDFS里的/shenl有R权限


#切换到shenl用户进行put权限测试,应该报错.


5 编辑Policy追加shenl用户的W权限


#此时shenl用户应该可以拥有/shenl目录的put权限

5.2 Hive Plugin验证

1 hive plugin加载验证


2 定义权限策略


3 beeline里权限验证




6 总结

:文档来之我网易博客 http://shenliang1985.blog.163.com/blog/static/2908380520151126102050593/

你可能感兴趣的:(大数据)