Apache Sentry安装及简单使用

1 环境描述

三台hadoop集群,分别是master、slave1和slave2。下面是这三台机器的软件分布:

  • master:NameNode、ZK、HiveMetaSotre、HiveServer2、SentryServer
  • slave1:DataNode、ZK
  • slave2:DataNode、ZK

2 软件需求

  1. MySql
  2. mysql-jdbc.jar:mysql-connector-java-5.1.38-bin.jar
  3. Hadoop
  4. Hive1.1.0:apache-hive-1.1.0-bin.tar.gz
  5. Sentry1.6.0:apache-sentry-1.6.0-incubating-bin.tar.gz
    注意:这里的Sentry1.6.0版本对应的是Hive1.1.0版本,目前还不支持Hive1.2及以上版本。

3 软件安装

3.1 MySql安装

  • 在线安装:sudo apt-get install mysql-server
  • 修改文件:vim /etc/mysql/my.cnf,内容如下:
#bind-address           = 127.0.0.1
bind-address            = 0.0.0.0
  • 重启mysql:service mysql restart

3.2 Hadoop安装

此处省略

3.2 Hive安装

此处省略

创建hive的元数据mysql数据库
create database hive;
CREATE USER hive IDENTIFIED BY 'hive';
GRANT all ON hive.* TO hive@'%' IDENTIFIED BY'hive';
flush privileges;

3.3 Sentry安装

1. 解压安装

tar -zxvf apache-sentry-1.6.0-incubating-bin.tar.gz -C /opt

2. 修改sentry的配置文件(sentry-site.xml)

mv apache-sentry-1.6.0-incubating-bin apache-sentry-1.6.0
cp mysql-connector-java-5.1.38-bin.jar /opt/apache-sentry-1.6.0/lib
cd /opt/apache-sentry-1.6.0/conf
vim sentry-site.xml

3. 修改sentry-sitem.xml文件

 
   
  1. xml version="1.0" encoding="utf-8"?>
  2. xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. sentry.service.admin.group
  4. admin
  5. sentry.service.allow.connect
  6. hive,admin
  7. sentry.service.reporting
  8. JMX
  9. sentry.service.server.rpc-address
  10. 192.168.70.110
  11. sentry.service.server.rpc-port
  12. 8038
  13. sentry.store.group.mapping
  14. org.apache.sentry.provider.common.HadoopGroupMappingService
  15. sentry.hive.server
  16. server1
  17. sentry.service.web.enable
  18. true
  19. sentry.service.web.port
  20. 51000
  21. sentry.service.web.authentication.type
  22. NONE
  23. sentry.service.web.authentication.kerberos.principal
  24. sentry.service.web.authentication.kerberos.keytab
  25. sentry.service.security.mode
  26. none
  27. sentry.service.server.principal
  28. sentry.service.server.keytab
  29. sentry.store.jdbc.url
  30. jdbc:mysql://192.168.70.110:3306/sentry
  31. sentry.store.jdbc.driver
  32. com.mysql.jdbc.Driver
  33. sentry.store.jdbc.user
  34. sentry
  35. sentry.store.jdbc.password
  36. sentry

4. 配置环境变量
编辑vim /etc/profile

export SENTRY_HOME=/opt/apache-sentry-1.6.0
export PATH=${SENTRY_HOME}/bin:$PATH

使生效source /etc/profile
5. 在mysql中创建sentry数据库

create database sentry;
CREATE USER sentry IDENTIFIED BY 'sentry';
GRANT all ON sentry.* TO sentry@'%' IDENTIFIED BY 'sentry';
flush privileges;

6. 将mysql-jdbc的jar包拷贝到sentry/lib目录下

cp mysql-connector-java-5.1.38-bin.jar  ${SENTRY_HOME}/lib

7. 替换hadoop中jline的jar包

rm ${HADOOP_HOME}/share/hadoop/yarn/lib/jline-0.9.94.jar
cp ${SENTRY_HOME}/lib/jline-2.12.jar ${HADOOP_HOME}/share/hadoop/yarn/lib

8. 初始化sentry数据库

sentry --command schema-tool --conffile  ${SENTRY_HOME}/conf/sentry-site.xml --dbType mysql --initSchema

9. 启动sentry服务

sentry --command service --conffile  ${SENTRY_HOME}/conf/sentry-site.xml

4 Hive集成Sentry

1. 修改hive-site.xml配置文件

 
   
  1. xml version="1.0" encoding="utf-8"?>
  2. xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. hive.sentry.conf.url
  4. file:///opt/apache-hive-1.1.0/conf/sentry-site.xml
  5. hive.stats.collect.scancols
  6. true
  7. hive.metastore.pre.event.listeners
  8. org.apache.sentry.binding.metastore.MetastoreAuthzBinding
  9. hive.metastore.event.listeners
  10. org.apache.sentry.binding.metastore.SentryMetastorePostEventListener
  11. hive.server2.session.hook
  12. org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook
  13. hive.security.authorization.task.factory
  14. org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl
  15. javax.jdo.option.ConnectionURL
  16. jdbc:mysql://192.168.70.110:3306/hive?createDatabaseIfNotExist=true
  17. javax.jdo.option.ConnectionDriverName
  18. com.mysql.jdbc.Driver
  19. javax.jdo.option.ConnectionUserName
  20. hive
  21. javax.jdo.option.ConnectionPassword
  22. hive

2. 修改sentry-site.xml配置文件
注意:该文件是在hive/conf目录下的,不要和sentry/conf目录下的sentry-site.xml文件混淆!

 
   
  1. xml version="1.0" encoding="utf-8"?>
  2. xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. sentry.service.client.server.rpc-address
  4. 192.168.70.110
  5. sentry.service.client.server.rpc-port
  6. 8038
  7. sentry.service.client.server.rpc-connection-timeout
  8. 200000
  9. sentry.service.security.mode
  10. none
  11. sentry.service.server.principal
  12. sentry.service.server.keytab
  13. sentry.provider
  14. org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider
  15. sentry.hive.provider.backend
  16. org.apache.sentry.provider.db.SimpleDBProviderBackend
  17. sentry.metastore.service.users
  18. hive
  19. sentry.hive.server
  20. server1
  21. sentry.hive.testing.mode
  22. true

3. 拷贝sentry/lib下的jar包到hive/lib目录下

cp ${SENTRY_HOME}/lib/sentry*.jar ${HIVE_HOME}/lib
cp ${SENTRY_HOME}/lib/shiro-*.jar ${HIVE_HOME}/lib

注意:shiro-*.jar一定也要拷贝过去,否则会报错。
4. 修改Hive在HDFS上的文件权限

hadoop fs -chown -R hive:hive /user/hive/warehouse
hadoop fs -chmod -R 770 /user/hive/warehouse

由于HDFS上文件是hive:hive权限,所以linux用户上能
5. 启动Hive Metastore服务

hive --service metastore -hiveconf hive.root.logger=DEBUG,console

6. 启动Hive Server2服务

hive --service hiveserver2 -hiveconf hive.root.logger=DEBUG,console

5 Hive权限验证

5.1 准备测试数据

  • 创建测试数据文件:vim /tmp/events.csv,内容如下:
10.1.2.3,US,android,createNote
10.200.88.99,FR,windows,updateNote
10.1.2.3,US,android,updateNote
10.200.88.77,FR,ios,createNote
10.1.4.5,US,windows,updateTag
  • 然后,在hive中运行下面sql语句(注意:以hive用户启动hive命令):
 
   
  1. create database sensitive;
  2. create table sensitive.events (ip STRING, country STRING, client STRING, action STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
  3. load data local inpath '/tmp/events.csv' overwrite into table sensitive.events;
  4. create database filtered;
  5. create view filtered.events as select country, client, action from sensitive.events;
  6. create view filtered.events_usonly as select * from filtered.events where country = 'US';
  • 在master上通过 beeline 连接 hiveserver2,运行下面命令创建角色和组:
beeline -u "jdbc:hive2://master:10000/" -n admin -p admin -d org.apache.hive.jdbc.HiveDriver
  • 在beelin中执行下面的 sql 语句创建 role、group等:
 
   
  1. create role admin_role;
  2. GRANT ALL ON SERVER server1 TO ROLE admin_role;
  3. GRANT ROLE admin_role TO GROUP admin;
  4. create role test_role;
  5. GRANT ALL ON DATABASE filtered TO ROLE test_role;
  6. use sensitive;
  7. GRANT SELECT(ip) on TABLE sensitive.events TO ROLE test_role;
  8. GRANT ROLE test_role TO GROUP test;

上面创建了两个角色:

  • admin_role,具有管理员权限,可以读写所有数据库,并授权给 admin 和 hive 组(对应操作系统上的组)
  • test_role,只能读写 filtered 数据库,和只能读取sensitive库中events 表中的ip字段,并授权给 test 组。

因为系统上没有test用户和组,所以需要手动创建:useradd test

5.2 测试

5.2.1 测试admin_role角色

使用admin用户访问beelin:

beeline -u "jdbc:hive2://master:10000/" -n admin -p admin -d org.apache.hive.jdbc.HiveDriver

admin用户属于admin_role组,具有管理员权限,可以查看所有角色:

0: jdbc:hive2://master:10000/> show roles;
+-------------+--+
|    role     |
+-------------+--+
| admin_role  |
| test_role   |
+-------------+--+
2 rows selected (0.251 seconds)

查看所有权限:

0: jdbc:hive2://master:10000/> show grant role admin_role;
+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+
| database  | table  | partition  | column  | principal_name  | principal_type  | privilege  | grant_option  |    grant_time     | grantor  |
+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+
| *         |        |            |         | admin_role      | ROLE            | *          | false         | 1461507543582000  | --       |
+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+
1 row selected (0.111 seconds)
0: jdbc:hive2://master:10000/> show grant role test_role;
+------------+---------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+
|  database  |  table  | partition  | column  | principal_name  | principal_type  | privilege  | grant_option  |    grant_time     | grantor  |
+------------+---------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+
| sensitive  | events  |            | ip      | test_role       | ROLE            | select     | false         | 1461558337008000  | --       |
| filtered   |         |            |         | test_role       | ROLE            | *          | false         | 1461557354579000  | --       |
+------------+---------+------------+---------+-----------------+-----------------+------------+---------------+-------------------+----------+--+
2 rows selected (0.111 seconds)

hive用户可以查看所有数据库、访问所有表:

0: jdbc:hive2://master:10000/> show databases;
+----------------+--+
| database_name  |
+----------------+--+
| default        |
| filtered       |
| sensitive      |
+----------------+--+
3 rows selected (0.142 seconds)
0: jdbc:hive2://master:10000/> use filtered;
No rows affected (0.144 seconds)
0: jdbc:hive2://master:10000/> select * from filtered.events;
+-----------------+----------------+----------------+--+
| events.country  | events.client  | events.action  |
+-----------------+----------------+----------------+--+
| US              | android        | createNote     |
| FR              | windows        | updateNote     |
| US              | android        | updateNote     |
| FR              | ios            | createNote     |
| US              | windows        | updateTag      |
+-----------------+----------------+----------------+--+
5 rows selected (0.458 seconds)
0: jdbc:hive2://master:10000/> select * from sensitive.events;
+---------------+-----------------+----------------+----------------+--+
|   events.ip   | events.country  | events.client  | events.action  |
+---------------+-----------------+----------------+----------------+--+
| 10.1.2.3      | US              | android        | createNote     |
| 10.200.88.99  | FR              | windows        | updateNote     |
| 10.1.2.3      | US              | android        | updateNote     |
| 10.200.88.77  | FR              | ios            | createNote     |
| 10.1.4.5      | US              | windows        | updateTag      |
+---------------+-----------------+----------------+----------------+--+
5 rows selected (0.731 seconds)

5.2.2 测试test_role角色

使用test用户访问beeline:

beeline -u "jdbc:hive2://master:10000/" -n test -p test -d org.apache.hive.jdbc.HiveDriver

test用户不是管理员,是不能查看所有角色的:

0: jdbc:hive2://master:10000/> show roles;
ERROR : Error processing Sentry command: Access denied to test.Please grant admin privilege to test.
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. SentryAccessDeniedException: Access denied to test (state=08S01,code=1)

test用户可以列出所有数据库:

0: jdbc:hive2://master:10000/> show databases;
+----------------+--+
| database_name  |
+----------------+--+
| default        |
| filtered       |
| sensitive      |
+----------------+--+

test用户可以filtered库:

0: jdbc:hive2://master:10000/> use filtered;
No rows affected (0.256 seconds)
0: jdbc:hive2://master:10000/> select * from events;
+-----------------+----------------+----------------+--+
| events.country  | events.client  | events.action  |
+-----------------+----------------+----------------+--+
| US              | android        | createNote     |
| FR              | windows        | updateNote     |
| US              | android        | updateNote     |
| FR              | ios            | createNote     |
| US              | windows        | updateTag      |
+-----------------+----------------+----------------+--+
5 rows selected (0.199 seconds)

test用户只能查看sensitive库中events表中的ip字段,其他的都都无法查看:

0: jdbc:hive2://master:10000/> use sensitive;
No rows affected (0.112 seconds)
0: jdbc:hive2://master:10000/> select * from events;
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
 User test does not have privileges for QUERY (state=42000,code=40000)
0: jdbc:hive2://master:10000/> select ip from events;
+---------------+--+
|      ip       |
+---------------+--+
| 10.1.2.3      |
| 10.200.88.99  |
| 10.1.2.3      |
| 10.200.88.77  |
| 10.1.4.5      |
+---------------+--+
5 rows selected (0.176 seconds)


来自为知笔记(Wiz)


转载于:https://www.cnblogs.com/xiaojun9/p/2efd58c794ed7049cb863b0ace8565c9.html

你可能感兴趣的:(Apache Sentry安装及简单使用)