Apache Ranger源码编译及使用
Ranger源码编译、使用手册
1 Ranger简介
Apache Ranger提供一个集中式安全管理框架,它可以对Hadoop生态的组件如Hive,Hbase进行细粒度的数据访问控制.通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问HDFS文件夹、HDFS文件、数据库、表、字段权限.这些策略可以为不同的用户和组来设置,同时权限可与hadoop无缝对接.
2 准备
2.1 环境说明
1 Ranger源码编译依赖如下linux组件:maven,git,gcc,mysql
2 安装git和gcc时采用yum安装,请配置好本地源
3 安装mysql是请确保清理好系统自带的mysql.
2.1.1虚拟机里Linux系统版本
[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux
Red Hat Enterprise Linux Server release 6.5 (Santiago)
2.1.2 JDK版本
[root@localhost native]# java -version
java version "1.7.0_67"
注:官网强调一定是1.7以上版本.
Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)
2.1.3mysql版本
[root@localhost native]# mysql -uroot -proot-e"select version()";
Warning: Using a password onthe command line interface can be insecure.
+-----------+
| version() |
+-----------+
| 5.6.14 |
+-----------+
注:
1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar
2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享
2.1.4 Maven版本
[root@localhost bin]# mvn -version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;2014-02-15T01:37:52+08:00)
Maven home: /root/maven-3.2.1
Java version: 1.7.0_67,vendor: Oracle Corporation
Java home:/root/jdk1.7.0_67/jre
Default locale: en_US,platform encoding: UTF-8
OS name: "linux",version: "2.6.32-431.el6.x86_64", arch: "amd64", family:"unix"
2.1.5 git版本
[root@localhost native]# git version
git version 1.7.1
2.2 编译准备
2.2.1 安装maven
[root@localhost ~]# cd /root
#下载地址
#https://maven.apache.org/download.cgi 最新版
#http://apache.opencas.org/maven/binaries/apache-maven-3.2.1-bin.tar.gz
tar –zxvf apache-maven-3.2.1-bin.tar.gz
mv apache-maven-3.2.1-bin maven-3.2.1
#修改环境变量,在~/.bash_profile里定义MAVEN_HOME并追加到PATH里
export MAVEN_HOME=/root/maven-3.2.1
:$MAVEN_HOME/bin:$PATH
#source环境变量,测试maven版本
source ~/.bash_profile
mvn –version
2.2.2 安装git
这里通过本地源yum方式安装.
yum install git
注:Linux本地源配置见下:
1) 虚拟机加载Linux ISO镜像
2) 找到rom对应设备名
[root@localhost ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
sr1 11:1 1 3.6G 0 rom
sr2 11:2 1 1024M 0 rom
sda 8:0 0 8G 0 disk
?..sda1 8:1 0 500M 0 part /boot
?..sda2 8:2 0 7.5G 0 part
?..VolGroup-lv_root (dm-0) 253:0 0 6.7G 0 lvm /
?..VolGroup-lv_swap (dm-1) 253:1 0 816M 0 lvm [SWAP]
3) 创建目录
[root@localhost ~]#mkdir –p /mnt/cdrom/
4) 挂载镜像
mount -t iso9660 /dev/sr1 /mnt/cdrom
5) 配置linux更新源,/etc/yum.repos.d,修改成如下:
vi /etc/yum.repos.d/redhat.repo
[rhel-source]
name=Redhat
baseurl=file:///mnt/cdrom/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release
6) 更新更新源
yum clean all
yum update list
2.2.3 安装gcc
yum install gcc
2.2.4 安装mysql
1) 安装Mysql服务、客户端
rpm –ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-server-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-client-5.6.14-1.el6.x86_64.rpm
2) 启动mysql服务
service mysql start
3) 修改mysql初始密码,先找到安装时的初始密码,在修改成自己的密码
[root@localhost ~]#cat /root/.mysql_secret
# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY
mysql –uroot -p RUmKBqcY
set password=password(‘root’)
3 编译
3.1编译中
1) 拷贝ranger源代码
[root@localhost ~]# cd ~
git clone https://github.com/apache/incubator-ranger.git
cd incubator-ranger
git checkout ranger-0.5
2) 编译ranger源代码
cd ~/incubator-ranger
export MAVEN_OPTS="-Xmx512M"
export JAVA_HOME= /root/jdk1.7.0_67
export PATH=$JAVA_HOME/bin:$PATH
mvn clean compile package assembly:assembly install
ls target/*.tar.gz
[root@localhost ~]#ls/root/incubator-ranger/target/*.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-hbase-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-kafka-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-kms.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-knox-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-migration-util.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-solr-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-src.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-storm-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-yarn-plugin.tar.gz
3.2 编译问题
1) 编译过程异常缓慢,一般要3-4天时间
2) 如果出现异常不好定位,可在maven参数里加-X以debug模式诊断
3) 源码编译的相关问题及解决方案
a) Failedto execute goal on project ranger-hdfs-plugin: Could not resolve dependenciesfor project
security_plugins.ranger-hdfs-plugin:
ranger-hdfs-plugin:jar:0.5.0:The following artifacts could not be resolved:
org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde,eigenbase:eigenbase-properties:jar:1.1.4,net.hydromatic:linq4j:jar:0.4,net.hydromatic:quidem:jar:0.1.1:
Could not transfer artifactorg.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde from/to conjars(http://conjars.org/repo): conjars.org:Unknownhost conjars.org -> [Help 1]
解决方案:手动下载错误提示里的jar并拷贝到相应的m2目录内.
源:http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/
目标:/root/.m2/repository/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde
源:http://conjars.org/repo/eigenbase/eigenbase-properties/1.1.4/
目标:/root/.m2/repository/eigenbase/eigenbase-properties/1.1.4/
源:http://conjars.org/repo/net/hydromatic/linq4j/0.4/
目标:/root/.m2/repository/net/hydromatic/linq4j/0.4/
源:http://conjars.org/repo/net/hydromatic/quidem/0.1.1/
目标:/root/.m2/repository/net/hydromatic/quidem/0.1.1/
b)[ERROR]error: error reading /root/.m2/repository/org/json/json/20090211/json-20090211.jar;zip file is empty
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException:Failed to execute goalorg.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) onproject ranger-hdfs-plugin: Compilation failure
error: error reading/root/.m2/repository/org/json/json/20090211/json-20090211.jar; zip file isempty
atorg.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
.
解决方案:手动下载错误提示里的json-20090211.jar并拷贝到相应的m2目录内.
c) Runningorg.apache.hadoop.crypto.key.kms.server.TestKeyAuthorizationKeyProvider此步耗时较长,大约20分钟左右
4 配置
Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr
注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr.
4.1 Solr或SolrCloud安装配置
cd /root/incubator-ranger/security-admin/contrib/solr_for_audit_setup
#打开install.properties文件,修改参数的值如下所示:
vi install.properties
JAVA_HOME=/root/jdk1.7.0_67
SOLR_INSTALL=true
SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
SOLR_INSTALL_FOLDER=/opt/solr
SOLR_RANGER_HOME=/opt/solr/ranger_audit_server
SOLR_DEPLOYMENT=standalone
SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data
SOLR_LOG_FOLDER=/var/log/solr/ranger_audits
SOLR_MAX_MEM=2g
#安装单节点的solr
./ setup.sh
#按照如下安装提示启动、停止solr服务
cat/opt/solr/ranger_audit_server/install_notes.txt
#启动solr
/opt/solr/ranger_audit_server/scripts/start_solr.sh
#验证solr服务是否可正常使用
lsof –i:6083
http://192.168.56.101:6083
#确保防火墙已经关闭
chkconfig iptables off
4.2 Ranger Admin安装配置
cp/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz /root
cd /root
tar –zxvf ranger-0.5.0-admin.tar.gz
cd ranger-0.5.0-admin
#打开Ranger Admin里install.properties文件,修改参数的值如下所示:
vi install.properties
setup_mode=SeparateDB
DB_FLAVOR=MYSQL
db_root_user=root
db_root_password=root
db_host=localhost
db_name=ranger
db_user=root
db_password=root
audit_store=db
audit_db_name=ranger_audit
audit_db_user=root
audit_db_password=root
policymgr_external_url=http://localhost:6080
policymgr_http_enabled=true
unix_user=ranger
unix_group=ranger
#安装Ranger Admin
./setup.sh
#启动Ranger Admin服务
ranger-admin start
#验证Ranger Admin服务,如果出现Ranger的登录界面,说Okay了.注:用户名/密码 admin/admin
lsof –i:6080
http://192.168.56.101:6080
4.3 Ranger-usersync安装配置
cp/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz /root/
cd /root
tar –zxvf ranger-0.5.0-usersync.tar.gz
cd ranger-0.5.0-usersync
#打开usersync Plugin里install.properties文件,修改参数的值如下所示:
vi install.properties
POLICY_MGR_URL=http://localhost:6080
SYNC_SOURCE=unix
#同步周期,1分钟
SYNC_INTERVAL=1
logdir=/var/log/ranger/usersync
#安装usersync Plugin
./setup.sh
#启用usersync Plugin插件
./ranger-usersync-services.sh start
4.4 HDFS-Plugin安装配置
cp/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz /root
cd /root
tar –zxvf ranger-0.5.0-hdfs-plugin.tar.gz
cd ranger-0.5.0-hdfs-plugin
#打开HDFS Plugin里install.properties文件,修改参数的值如下所示:
vi install.properties
POLICY_MGR_URL=http://localhost:6080
SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
REPOSITORY_NAME=hadoopdev
XAAUDIT.DB.IS_ENABLED=true
XAAUDIT.DB.FLAVOUR=MYSQL
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger_audit
XAAUDIT.DB.USER_NAME=root
XAAUDIT.DB.PASSWORD=root
#组件对应的用户,这里设置为空.一般Hadoop的内置用户是HDFS或则hadoop
CUSTOM_USER=root
CUSTOM_GROUP=root
#启用HDFS Plugin插件
[root@localhost ranger-0.5.0-hdfs-plugin]#./enable-hdfs-plugin.sh
Customuser and group are not available, using default user and group.
ERROR:Unable to find the conf directory of component [hadoop]; dir [/root/hadoop/conf] not found.
Exitinginstallation.
注:这里报错,需要额外将HADOOP的conf做个软连接到/root/hadoop/conf.
ln-s /root/hadoop-2.7.1/etc/hadoop/ /root/hadoop/conf
#再次启用HDFS Plugin插件
[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh
Custom user and group are not available,using default user and group.
ERROR: Unable to find the lib directory ofcomponent [hadoop]; dir [/root/hadoop/lib] not found.
Exiting installation.
#这里需要将HDFS Plugin内的jar和HADOOP包含的HDFS jar都指向/root/hadoop/lib
cp /root/ranger-0.5.0-hdfs-plugin/lib/ranger-hdfs-plugin-impl/*.jar/root/hadoop-2.7.1/share/hadoop/hdfs/lib/
mkdir /root/hadoop/lib
ln -s /root/hadoop-2.7.1/share/hadoop/hdfs/lib//root/hadoop/lib/
#再一次启用HDFS Plugin插件
[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh
#验证HDFS Plugin服务,这时登丽Ranger的管理员界面验证下HDFS plugin是够加载成功,发现并没有.
原因是安装HDFS plugin时install.properties文件里定义的REPOSITORY_NAME(值为hadoopdev)并未通过Ranger Admin在HDFS插件里的服务管理里注册成服务(名hadoopdev).
解决方案:
1 登录Ranger Adming
2 点击HDFS plugin的添加按钮
3 定义服务名为hadoopdev,提交其它信息后保存
#再次验证HDFS plugin插件,则发现已经正常加载
注:
1 如果没有安装和开启Ranger-usersync服务的情况下直接测试HDFS赋权权限是不成功的.
2 Ranger Admin的日志文件见 /root/ranger-0.5.0-admin/ews/logs/xa_portal.log
4.5 Hive-Plugin安装配置
先启动hive的metastore和hiveserver2服务
nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &
nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &
#beeline验证
[root@localhost ~]# beeline -u"jdbc:hive2://192.168.56.101:10000" -n root -p test
Connectingto jdbc:hive2://192.168.56.101:10000
Connectedto: Apache Hive (version 1.2.1)
Driver:Hive JDBC (version 1.2.1)
Transactionisolation: TRANSACTION_REPEATABLE_READ
Beelineversion 1.2.1 by Apache Hive
0:jdbc:hive2://192.168.56.101:10000> show databases;
+----------------+--+
|database_name |
+----------------+--+
|default |
| shenl |
+----------------+--+
2) Ranger-Admin里注册hive plugin的服务
3) 配置、启用hive plugin
cp/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz /root
cd /root
tar -zxvf ranger-0.5.0-hive-plugin.tar.gz
cd ranger-0.5.0-hive-plugin
#打开Hive Plugin里install.properties文件,修改参数的值如下所示:
vi install.properties
POLICY_MGR_URL=http://192.168.56.101:6080
REPOSITORY_NAME=hivedev
XAAUDIT.DB.IS_ENABLED=true
XAAUDIT.DB.FLAVOUR=MYSQL=MYSQL
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger_audit
XAAUDIT.DB.USER_NAME=root
XAAUDIT.DB.PASSWORD=root
[[email protected]]# ./enable-hive-plugin.sh
Customuser and group is available, using custom user and group.
ERROR:Unable to find the conf directory of component [hive]; dir [/root/hive/conf]not found.
Exitinginstallation.
#解决方法:
ln -s /root/apache-hive-0.13.0-bin/conf//root/hive/conf
[[email protected]]# ./enable-hive-plugin.sh
Customuser and group is available, using custom user and group.
ERROR:Unable to find the lib directory of component [hive]; dir [/root/hive/lib] not found.
Exitinginstallation.
#解决方法:
cp root/ranger-0.5.0-hive-plugin/lib/ranger-hive-plugin-impl/*.jar/root/apache-hive-1.2.1-bin/lib/
mkdir /root/hive/lib
ln -s/root/apache-hive-1.2.1-bin/lib/ /root/hive/lib/
#将生成的hiveserver2-site.xml拷贝到hive的配置目录下
cp /root/hive/conf/*/root/apache-hive-1.2.1-bin/conf/
#如果hive配置目录里有hiveserver2-site.xml,则需要添加如下内容:
5 使用
5.1 HDFS Plugin验证
1 新增linux用户shenl进行测试
useradd shenl
hadoop fs –mkdir /shenl
2 登录Ranger Admin,新增用户shenl
3 登录Ranger Admin,在hadoopdev里注册新的Policy
4 添加该Policy的权限,指定可以对HDFS里的/shenl有R权限
#切换到shenl用户进行put权限测试,应该报错.
5 编辑Policy追加shenl用户的W权限
#此时shenl用户应该可以拥有/shenl目录的put权限
5.2 Hive Plugin验证
1 hive plugin加载验证
2 定义权限策略
3 beeline里权限验证
