Centos7.6 + Apache Ranger 2.4.0编译（docker方式）

目录

一、Ranger简介

1、组件列表

2、支持的数据引擎服务

二、主机环境准备

1、关闭防火墙

2、关闭SELINUX

3、安装docker

4、下载Ranger源码包

5、下载Maven安装包

三、编译Ranger源码

1、修改官方包中的build_ranger_using_docker.sh

2、运行脚本编译

3、编译检查

---

一、Ranger简介

        Apache Ranger提供一个集中式安全管理框架, 并解决授权和审计。它可以对Hadoop生态的组件如HDFS、Yarn、Hive、Hbase等进行细粒度的数据访问控制。通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问权限。

1、组件列表

#	Service Name	Listen Port	Core Ranger Service
1	ranger	6080/tcp	Y (ranger engine - 3.0.0-SNAPSHOT version)
2	ranger-postgres	5432/tcp	Y (ranger datastore)
3	ranger-solr	8983/tcp	Y (audit store)
4	ranger-zk	2181/tcp	Y (used by solr)
5	ranger-usersync	-	Y (user/group synchronization from Local Linux/Mac)
6	ranger-kms	9292/tcp	N (needed only for Encrypted Storage / TDE)
7	ranger-tagsync	-	N (needed only for Tag Based Policies to be sync from ATLAS)

2、支持的数据引擎服务

#	Service Name	Listen Port	Service Description
1	Hadoop	8088/tcp 9000/tcp	Apache Hadoop 3.3.0 Protected by Apache Ranger's Hadoop Plugin
2	HBase	16000/tcp 16010/tcp 16020/tcp 16030/tcp	Apache HBase 2.4.6 Protected by Apache Ranger's HBase Plugin
3	Hive	10000/tcp	Apache Hive 3.1.2 Protected by Apache Ranger's Hive Plugin
4	Kafka	6667/tcp	Apache Kafka 2.8.1 Protected by Apache Ranger's Kafka Plugin
5	Knox	8443/tcp	Apache Knox 1.4.0 Protected by Apache Ranger's Knox Plugin

二、主机环境准备

1、关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

2、关闭SELINUX

sed -i.bak$DATE '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config
setenforce 0

3、安装docker

yum install -y docker
systemctl start docker
systemctl enable docker

4、下载Ranger源码包

Apache Ranger官网没有可以直接部署的安装包，必须通过源码进行编译。

官网地址：Apache Ranger - Download Apache Ranger?

wget https://www.apache.org/dist/ranger/2.4.0/apache-ranger-2.4.0.tar.gz --no-check-certificate

5、下载Maven安装包

wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate

三、编译Ranger源码

1、修改官方包中的build_ranger_using_docker.sh

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License

#This script creates the Docker image (if not already created) and runs maven in the container
#1. Install Docker
#2. Checkout Ranger source and go to the root directory
#3. Run this script. If host is linux, then run this script as "sudo $0 ..."
#4. If you are running on Mac, then you don't need to use "sudo"
#5. To delete the image, run "[sudo] docker rmi ranger_dev"

#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn  
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true 
#Notes: To remove build image manually, run "docker rmi ranger_dev" or "sudo docker rmi ranger_dev"

default_command="mvn -Pall -DskipTests=true clean compile package install"
build_image=0
if [ "$1" = "-build_image" ]; then
    build_image=1
    shift
fi

params=$*
if [ $# -eq 0 ]; then
    params=$default_command
fi

image_name="ranger_dev"
remote_home="$HOME"
container_name="--name ranger_build"

if [ ! -d security-admin ]; then
    echo "ERROR: Run the script from root folder of source. e.g. $HOME/git/ranger"
    exit 1
fi

images=`docker images | cut -f 1 -d " "`
[[ $images =~ $image_name ]] && found_image=1 || build_image=1

if [ $build_image -eq 1 ]; then
    echo "Creating image $image_name ..."
    docker rmi -f $image_name

docker build -t $image_name - < FROM centos:centos7.6.1810

RUN mkdir /tools
WORKDIR /tools

#Install default services
RUN yum install -y wget git gcc bzip2 fontconfig python3 java-1.8.0-openjdk-devel.x86_64
RUN ln -sf /usr/bin/python3 /usr/bin/python

ENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk/
ENV PATH $JAVA_HOME/bin:$PATH

RUN wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificate

RUN tar xfz apache-maven-3.9.4-bin.tar.gz
RUN ln -sf /tools/apache-maven-3.9.4 /tools/maven

ENV  PATH /tools/maven/bin:$PATH
ENV MAVEN_OPTS "-Xmx2048m -XX:MaxPermSize=512m"

RUN mkdir -p /scripts

RUN echo "#!/bin/bash" > /scripts/mvn.sh
RUN echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh

RUN chmod -R 777 /scripts
RUN chmod -R 777 /tools

ENTRYPOINT ["/scripts/mvn.sh"]
Dockerfile

fi

src_folder=`pwd`
LOCAL_M2="$HOME/.m2"
mkdir -p $LOCAL_M2
set -x

docker run --rm  -v "${src_folder}:/ranger" -w "/ranger" -v "${LOCAL_M2}:${remote_home}/.m2" $container_name $image_name $params

说明：
考虑实验运行环境为centos7.6，且国内有部分外国源访问不到，所以做了一定修改和裁剪，本次实验中，使用root用户运行该脚本，容器内使用root用户进行编译操作。
1、原脚本的基础镜像为centos:lastest，其对应Centos 8.1，现修改为centos:centos7.6.1810；
2、原脚本安装jkd8时，使用AWS s3's docker-assets里的jdk-8u101-linux-x64.rpm，现修改为使用centos自带的openjdk1.8，即java-1.8.0-openjdk-devel.x86_64；
3、原脚本未安装python3，最终编译时会找不到python3包而报错，现增加安装python3，同时设置默认使用python3，即RUN ln -sf /usr/bin/python3 /usr/bin/python
4、原脚本安装maven时，使用ADD来获取apache-maven-3.6.3-bin.tar.gz并校验包，现修改为使用wget获得最新的apache-maven-3.9.4-bin.tar.gz，且不做额外的包正确性校验，即wget https://dlcdn.apache.org/maven/maven-3/3.9.4/binaries/apache-maven-3.9.4-bin.tar.gz --no-check-certificatewget
5、原脚本的启动脚本创建并使用了非root用户builder，但会与后面运行容器时映射本地卷组有权限限制，考虑到只是临时编译使用，剔除所有builder用户的内容，包含gosu安装、用户创建、用户判断等，只保留echo 'set -x; exec "\$@" ' >> /scripts/mvn.sh
6、原脚本中${remote_home}为空值，会将运行该脚本的用户Home下的.m2映射到容器内根目录的.m2，现修改为容器内工作用户的Home目录，即remote_home="$HOME"

2、运行脚本编译

chmod +x build_ranger_using_docker.sh
./build_ranger_using_docker.sh

说明：
参照脚本使用说明
#Usage: [sudo] ./build_ranger_using_docker.sh [-build_image] mvn  
#Example 1 (default no param): (mvn -Pall -DskipTests=true clean compile package install)
#Example 2 (Regular build): ./build_ranger_using_docker.sh mvn -Pall clean install -DskipTests=true
#Example 3 (Recreate Docker image): ./build_ranger_using_docker.sh mvn -Pall -build_image clean install -DskipTests=true 

3、编译检查

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ranger 2.4.0:
[INFO] 
[INFO] ranger ............................................. SUCCESS [ 12.567 s]
[INFO] Jdbc SQL Connector ................................. SUCCESS [ 13.553 s]
[INFO] Credential Support ................................. SUCCESS [ 14.914 s]
[INFO] Audit Component .................................... SUCCESS [01:09 min]
[INFO] ranger-plugin-classloader .......................... SUCCESS [  9.662 s]
[INFO] Common library for Plugins ......................... SUCCESS [02:03 min]
[INFO] ranger-intg ........................................ SUCCESS [ 40.185 s]
[INFO] Installer Support Component ........................ SUCCESS [  8.196 s]
[INFO] Credential Builder ................................. SUCCESS [ 12.157 s]
[INFO] Embedded Web Server Invoker ........................ SUCCESS [ 33.355 s]
[INFO] Key Management Service ............................. SUCCESS [01:40 min]
[INFO] HBase Security Plugin Shim ......................... SUCCESS [ 52.109 s]
[INFO] HBase Security Plugin .............................. SUCCESS [01:25 min]
[INFO] Hdfs Security Plugin ............................... SUCCESS [ 36.159 s]
[INFO] Hive Security Plugin ............................... SUCCESS [ 41.491 s]
[INFO] Knox Security Plugin Shim .......................... SUCCESS [  9.255 s]
[INFO] Knox Security Plugin ............................... SUCCESS [ 21.750 s]
[INFO] Storm Security Plugin .............................. SUCCESS [ 16.017 s]
[INFO] YARN Security Plugin ............................... SUCCESS [ 13.554 s]
[INFO] Ozone Security Plugin .............................. SUCCESS [ 12.752 s]
[INFO] Ranger Util ........................................ SUCCESS [ 11.776 s]
[INFO] Unix Authentication Client ......................... SUCCESS [ 11.990 s]
[INFO] User Group Synchronizer Util ....................... SUCCESS [  6.909 s]
[INFO] Security Admin Web Application ..................... SUCCESS [08:54 min]
[INFO] KAFKA Security Plugin .............................. SUCCESS [01:17 min]
[INFO] SOLR Security Plugin ............................... SUCCESS [01:18 min]
[INFO] NestedStructure Security Plugin .................... SUCCESS [ 24.474 s]
[INFO] NiFi Security Plugin ............................... SUCCESS [ 12.265 s]
[INFO] NiFi Registry Security Plugin ...................... SUCCESS [ 11.211 s]
[INFO] Presto Security Plugin ............................. SUCCESS [ 24.201 s]
[INFO] Kudu Security Plugin ............................... SUCCESS [ 14.920 s]
[INFO] Unix User Group Synchronizer ....................... SUCCESS [02:08 min]
[INFO] Ldap Config Check Tool ............................. SUCCESS [ 11.640 s]
[INFO] Unix Authentication Service ........................ SUCCESS [ 11.348 s]
[INFO] KMS Security Plugin ................................ SUCCESS [01:13 min]
[INFO] Tag Synchronizer ................................... SUCCESS [ 45.784 s]
[INFO] Hdfs Security Plugin Shim .......................... SUCCESS [  9.535 s]
[INFO] Hive Security Plugin Shim .......................... SUCCESS [01:23 min]
[INFO] YARN Security Plugin Shim .......................... SUCCESS [ 42.092 s]
[INFO] OZONE Security Plugin Shim ......................... SUCCESS [ 23.710 s]
[INFO] Storm Security Plugin shim ......................... SUCCESS [ 10.665 s]
[INFO] KAFKA Security Plugin Shim ......................... SUCCESS [ 10.838 s]
[INFO] SOLR Security Plugin Shim .......................... SUCCESS [ 22.091 s]
[INFO] Atlas Security Plugin Shim ......................... SUCCESS [ 28.752 s]
[INFO] KMS Security Plugin Shim ........................... SUCCESS [ 52.920 s]
[INFO] Presto Security Plugin Shim ........................ SUCCESS [ 26.065 s]
[INFO] ranger-examples .................................... SUCCESS [  0.272 s]
[INFO] Ranger Examples - Conditions and ContextEnrichers .. SUCCESS [ 11.692 s]
[INFO] Ranger Examples - SampleApp ........................ SUCCESS [  5.863 s]
[INFO] Ranger Examples - Ranger Plugin for SampleApp ...... SUCCESS [ 10.167 s]
[INFO] sample-client ...................................... SUCCESS [ 11.777 s]
[INFO] Apache Ranger Examples Distribution ................ SUCCESS [  6.742 s]
[INFO] Ranger Tools ....................................... SUCCESS [ 35.518 s]
[INFO] Atlas Security Plugin .............................. SUCCESS [ 41.615 s]
[INFO] SchemaRegistry Security Plugin ..................... SUCCESS [03:02 min]
[INFO] Sqoop Security Plugin .............................. SUCCESS [ 53.693 s]
[INFO] Sqoop Security Plugin Shim ......................... SUCCESS [ 14.680 s]
[INFO] Kylin Security Plugin .............................. SUCCESS [03:33 min]
[INFO] Kylin Security Plugin Shim ......................... SUCCESS [ 41.171 s]
[INFO] Elasticsearch Security Plugin Shim ................. SUCCESS [ 22.381 s]
[INFO] Elasticsearch Security Plugin ...................... SUCCESS [ 37.204 s]
[INFO] Apache Ranger Distribution ......................... SUCCESS [02:26 min]
[INFO] Unix Native Authenticator .......................... SUCCESS [  4.438 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  49:17 min
[INFO] Finished at: 2023-08-07T10:43:31Z
[INFO] ------------------------------------------------------------------------

在target目录可以看到生成的程序包：

-rw-r--r-- 1 root root 579387182 Aug  7 18:42 ranger-2.4.0-admin.tar.gz
-rw-r--r-- 1 root root  43729654 Aug  7 18:43 ranger-2.4.0-atlas-plugin.tar.gz
-rw-r--r-- 1 root root  34172214 Aug  7 18:43 ranger-2.4.0-elasticsearch-plugin.tar.gz
-rw-r--r-- 1 root root  39122941 Aug  7 18:42 ranger-2.4.0-hbase-plugin.tar.gz
-rw-r--r-- 1 root root  37684529 Aug  7 18:42 ranger-2.4.0-hdfs-plugin.tar.gz
-rw-r--r-- 1 root root  37478412 Aug  7 18:42 ranger-2.4.0-hive-plugin.tar.gz
-rw-r--r-- 1 root root  56846325 Aug  7 18:42 ranger-2.4.0-kafka-plugin.tar.gz
-rw-r--r-- 1 root root 195376717 Aug  7 18:43 ranger-2.4.0-kms.tar.gz
-rw-r--r-- 1 root root  51454934 Aug  7 18:42 ranger-2.4.0-knox-plugin.tar.gz
-rw-r--r-- 1 root root  36625366 Aug  7 18:43 ranger-2.4.0-kylin-plugin.tar.gz
-rw-r--r-- 1 root root     34201 Aug  7 18:43 ranger-2.4.0-migration-util.tar.gz
-rw-r--r-- 1 root root  43393403 Aug  7 18:42 ranger-2.4.0-ozone-plugin.tar.gz
-rw-r--r-- 1 root root  57425250 Aug  7 18:43 ranger-2.4.0-presto-plugin.tar.gz
-rw-r--r-- 1 root root  16563346 Aug  7 18:43 ranger-2.4.0-ranger-tools.tar.gz
-rw-r--r-- 1 root root     36915 Aug  7 18:42 ranger-2.4.0-solr_audit_conf.tar.gz
-rw-r--r-- 1 root root  38256335 Aug  7 18:42 ranger-2.4.0-solr-plugin.tar.gz
-rw-r--r-- 1 root root  36860763 Aug  7 18:43 ranger-2.4.0-sqoop-plugin.tar.gz
-rw-r--r-- 1 root root   6376456 Aug  7 18:43 ranger-2.4.0-src.tar.gz
-rw-r--r-- 1 root root  51760282 Aug  7 18:42 ranger-2.4.0-storm-plugin.tar.gz
-rw-r--r-- 1 root root  31046503 Aug  7 18:42 ranger-2.4.0-tagsync.tar.gz
-rw-r--r-- 1 root root  20128101 Aug  7 18:42 ranger-2.4.0-usersync.tar.gz
-rw-r--r-- 1 root root  35792990 Aug  7 18:42 ranger-2.4.0-yarn-plugin.tar.gz

参考文档：

Ranger Installation Guide - Ranger - Apache Software Foundation