Apache Sentry手动安装、使用手册

Sentry手动安装、使用手册

1 Sentry简介

Apache Sentry 是Cloudera公司发布的一个Hadoop开源组件，截止目前还是Apache的孵化项目，它提供了细粒度级、基于角色的授权以及多租户的管理模式。Sentry当前可以和Hive/Hcatalog、Apache Solr 和Cloudera Impala集成，未来会扩展到其他的Hadoop组件，例如HDFS和HBase.

2 准备

2.1 环境说明

1 Sentry安装采用rpm包的方式.

2 Hadoop版本为hadoop-2.5.0-cdh5.3.3,Hive版本为hive-0.13.1-cdh5.3.3,Sentry版本为sentry-1.4.0-cdh5.3.3

3 Sentry下载地址:

http://archive-primary.cloudera.com/cdh5/redhat/6/x86_64/cdh/5.3.3/RPMS/noarch/

 

2.1.1虚拟机里Linux系统版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14    |

+-----------+

注:

1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar

2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享

 

3 安装

3.1安装Mysql

1)安装mysql相关服务

rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm时报如下错误:

file/usr/share/mysql/charsets/macroman.xml from install of MySQL-server-5.6.14-1.el6.x86_64conflicts with file from package mysql-libs-5.1.71-1.el6.x86_64 file/usr/share/mysql/charsets/swe7.xml from install ofMySQL-server-5.6.14-1.el6.x86_64 conflicts with file from packagemysql-libs-5.1.71-1.el6.x86_64

rpm -q mysql-libs-5.1.71-1.el6.x86_64

rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

ARANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !

Youwill find that password in '/root/.mysql_secret'.(生成mysql root密码)

Youmust change that password on your first connect,

noother statement but 'SET PASSWORD' will be accepted.

rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

service mysql start

#这里密码 来自/root/.mysql_secret

mysql -uroot -p9RNrbk9O

 

#首次执行会提示修改mysql root密码

SET PASSWORD=PASSWORD('root');

#创建Hive数据库为Hive的元数据库

create database hive;

GRANT all ON hive.* TO root@'%' IDENTIFIED BY 'root';

#创建sentry数据库为sentry元数据库

create database sentry;

CREATE USER sentry IDENTIFIED BY 'sentry';

GRANT all ON sentry.* TO sentry@'%' IDENTIFIED BY'sentry';

flush privileges;

 

3.2 安装Hive

1)解压Hive压缩包,并配置环境变量

cd /root

tar –zxvf hive-0.13.1-cdh5.3.3.tar.gz

vi ~/.bash_profile

exportHIVE_HOME=/root/hive-0.13.1-cdh5.3.3

#追加HIVE执行文件路径到PATH内

exportPATH=$PATH:$HIVE_HOME/bin

2) 拷贝Mysql驱动到Hive的lib目录下:

cp     /root/mysql-connector-java-5.1.31-bin.jar/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar

 

3)配置Hive的conf.详细的hive-site.xml配置内容见下:

 javax.jdo.option.ConnectionURL
 jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true
  JDBC connect string for aJDBC metastore

 

 javax.jdo.option.ConnectionDriverName
 com.mysql.jdbc.Driver
  Driver class name for aJDBC metastore

 

 javax.jdo.PersistenceManagerFactoryClass
 org.datanucleus.api.jdo.JDOPersistenceManagerFactory
  class implementing the jdopersistence

 

  javax.jdo.option.DetachAllOnCommit
  true
  detaches all objects fromsession so that they can be used after transaction iscommitted

 

 javax.jdo.option.NonTransactionalRead
  true
  reads outside oftransactions

 

 javax.jdo.option.ConnectionUserName
  root
  username to use againstmetastore database

 

 javax.jdo.option.ConnectionPassword
  root
  password to use againstmetastore database

 

 javax.jdo.option.Multithreaded
  true
  Set this to true ifmultiple threads access metastore through JDO concurrently.

 

3.3 安装Sentry

1)安装Sentry相关服务

rpm -ivh --nodepssentry-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm

rpm -ivh --nodeps sentry-hdfs-plugin-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm

rpm -ivh --nodepssentry-store-1.4.0+cdh5.3.3+137-1.cdh5.3.3.p0.8.el6.noarch.rpm

 

2)替换Sentry内的Hadoop、Hive、Impala、Hbase、Zookeeper、Parquet、Avro等jar包

rm -rf /usr/lib/sentry/lib/hive*.jar

rm -rf /usr/lib/sentry/lib/hadoop*.jar

rm -rf /usr/lib/sentry/lib/zookeeper*.jar

rm -rf /usr/lib/sentry/lib/avro*.jar

rm -rf /usr/lib/sentry/lib/server/hive-beeline.jar

cp ~/SentryLibs/* /usr/lib/sentry/lib/

注:这里的Sentrylibs里的jar包是通过/usr/lib/sentry/lib里的文件名分别在Hadoop、Hive、Hbase等里的安装目录里一一找到的.

mv /root/SentryLibs/hive-beeline.jar/usr/lib/sentry/lib/server/  

#拷贝mysql驱动到sentry的lib目录内 

cp/root/hive-0.13.1-cdh5.3.3/lib/mysql-connector-java-5.1.31-bin.jar/usr/lib/sentry/lib/

#如果装了Impala,此步可忽略

rpm -ivh bigtop-utils-0.7.0+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6.noarch.rpm

 

3) 配置Sentry,详细配置见sentry-site.xml内容如下:

    
       sentry.service.security.mode
        none
    
    
       sentry.service.admin.group
        impala,hive
    
    
       sentry.service.allow.connect
        impala,hive
    
    
        sentry.verify.schema.version
        true
    
    
   sentry.service.server.rpc-address
    data1
    
    
   sentry.service.server.rpc-port
    8038
    
    
       sentry.store.jdbc.url
       jdbc:mysql://localhost:3306/sentry
    
    
       sentry.store.jdbc.driver
        com.mysql.jdbc.Driver
    
    
       sentry.store.jdbc.user
        sentry
    
    
       sentry.store.jdbc.password
        sentry
    
    
       sentry.hive.server
        data1
    
    
       sentry.store.group.mapping
       org.apache.sentry.provider.common.HadoopGroupMappingService
    

 

4)初始化Sentry元数据

sentry --command schema-tool --conffile/etc/sentry/conf/sentry-site.xml --dbType mysql --initSchema

                  ..........................

                  No rows affected (0.094 seconds)

                  No rows affected (0.015 seconds)

                  No rows affected (0.075 seconds)

                  1 row affected (0.007 seconds)

                  Closing: 0: jdbc:mysql://localhost:3306/sentry

                  Initialization script completed

                  Sentry schemaTool completed

 

3.4 安装问题

暂无.

4 配置

Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr

注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr.

4.1 Hive集成Sentry配置

1) 拷贝Sentry相关jar包到Hive的lib目录内

 cp /usr/lib/sentry/lib/sentry*.jar$HIVE_HOME/lib/

 cp /usr/lib/sentry/lib/shiro-core-*.jar$HIVE_HOME/lib/

 

2)  Hive conf内新增Sentry-site.xml,配置内容见下:

 

     sentry.service.security.mode
     none


     sentry.service.server.principal
     hivemeta/[email protected]

    
       sentry.service.client.server.rpc-port
       8038
     
     
       sentry.service.client.server.rpc-address
       data1
     
     
       sentry.service.client.server.rpc-connection-timeout
       200000
     
     
              sentry.hive.provider
              org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider
      
     
              sentry.hive.provider.backend
              org.apache.sentry.provider.db.SimpleDBProviderBackend
      
     
              sentry.hive.server
              server1
      
      
              sentry.metastore.service.users
              root
      
     
              sentry.hive.testing.mode
              true
      

 

3)  修改Hive conf内的hive-site.xml,修改成如下:

    hive.metastore.execute.setugi
    true
    
      In unsecure mode, setting this propertyto true will cause the metastore to execute DFS operations using
      the client's reported user and grouppermissions. Note that this property must be set on
      both the client and server sides. Furthernote that its best effort.
      If client sets its to true and serversets it to false, client setting will be ignored.
    
 

  hive.metastore.pre.event.listeners
 org.apache.sentry.binding.metastore.MetastoreAuthzBinding

 

 hive.metastore.event.listeners
 org.apache.sentry.binding.metastore.SentryMetastorePostEventListener

 
 
 

     hive.server2.enable.impersonation
     true


   hive.security.authorization.task.factory
              org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl


   hive.server2.session.hook
   org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook


   hive.sentry.conf.url
    file:///root/hive-0.13.1-cdh5.3.3/conf/sentry-site.xml



 javax.jdo.option.ConnectionURL
 jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true
  JDBC connect string for aJDBC metastore

 

 javax.jdo.option.ConnectionDriverName
 com.mysql.jdbc.Driver
  Driver class name for aJDBC metastore


  javax.jdo.PersistenceManagerFactoryClass
 org.datanucleus.api.jdo.JDOPersistenceManagerFactory
  class implementing the jdopersistence


 javax.jdo.option.DetachAllOnCommit
  true
  detaches all objects fromsession so that they can be used after transaction iscommitted


 javax.jdo.option.NonTransactionalRead
  true
  reads outside of transactions


 javax.jdo.option.ConnectionUserName
  root
  username to use againstmetastore database


  javax.jdo.option.ConnectionPassword
  root
  password to use againstmetastore database


 javax.jdo.option.Multithreaded
  true
  Set this to true ifmultiple threads access metastore through JDO concurrently.

 

4)  验证Sentry的权限控制,详见 5 使用:

 

 

4.2 Impala集成Sentry配置

1)  Impala的conf里新增sentry配置,/etc/impala/conf/sentry-site.xml内容如下:

  sentry.service.client.server.rpc-port
   8038


  sentry.service.client.server.rpc-address
   data1


   sentry.service.client.server.rpc-connection-timeout
   200000


  sentry.service.security.mode
   none

2)  修改impala的参数

vi/etc/default/impala

IMPALA_CATALOG_ARGS 参数里追加 -sentry_config=/etc/impala/conf/sentry-site.xml

IMPALA_SERVER_ARGS 参数里追加 -server_name=server1\

   -sentry_config=/etc/impala/conf/sentry-site.xml

5 使用

5.1 Hive集成Sentry权限验证

1) 启动sentry服务,hive的metastore和hiveserver2服务

sentry --command service--conffile /etc/sentry/conf/sentry-site.xml

nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &

nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &

#hiveserver2里新增hive和test用户,通过beeline方式访问.

groupadd hive; useradd hive -ghive;passwd hive (type hive123)

groupadd test; useradd test -gtest;passwd test (type test123)

beeline -u "jdbc:hive2://data1:10000"-n hive -p test

 

2)验证整体思路及场景概述

思路:

1 HIVE用户属于管理员组,服务整个server的权限

2 Test用户创建两个数据库test_only(all权限),test_select_only(仅有select权限)

具体验证:

1 通过show databases和use database验证访问权限

2 通过test用户的use test_only能drop table,create table验证all权限

3 通过test用户的use test_select_only不能drop table验证仅有select权限

 

#场景一 root用户连入,尝试执行DDL操作.(无权操作)

[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n root

                  scan complete in 17ms

                  Connecting to jdbc:hive2://data1:10000

                  Connected to: Apache Hive (version 0.13.1-cdh5.3.3)

                  Driver: Hive JDBC (version 0.13.1-cdh5.3.3)

                  Transaction isolation: TRANSACTION_REPEATABLE_READ

                  Beeline version 0.13.1-cdh5.3.3 by Apache Hive

0:jdbc:hive2://data1:10000>create database sensitive;

        Error: Error while compiling statement: FAILED:SemanticException No valid privileges

        Required privileges for this query: Server=server1->action=*;(state=42000,code=40000)

        0: jdbc:hive2://data1:10000> create role admin_role;

        Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.SentryAccessDeniedException: Access denied to root (state=08S01,code=1)

0:jdbc:hive2://data1:10000> !q

        Closing: 0: jdbc:hive2://data1:10000

 

#场景二 hive用户连入,对用户hive、test进行授权.

 

[root@data1 conf]#beeline-u "jdbc:hive2://data1:10000" -n hive

                  scan complete in 5ms

                  Connecting to jdbc:hive2://data1:10000

                  Connected to: Apache Hive (version 0.13.1-cdh5.3.3)

                  Driver: Hive JDBC (version 0.13.1-cdh5.3.3)

                  Transaction isolation: TRANSACTION_REPEATABLE_READ

                  Beeline version 0.13.1-cdh5.3.3 by Apache Hive

0:jdbc:hive2://data1:10000>create role admin_role;

        No rows affected (1.383 seconds)

GRANT ALL ONSERVER server1 TO ROLE admin_role;

GRANT ROLEadmin_role TO GROUP hive;

create roletest_role;

GRANT ALL ONDATABASE test_only TO ROLE test_role;

GRANT ROLEtest_role TO GROUP test;

GRANT SELECT ONDATABASE test_select_only TO ROLE test_role;

#场景三 test用户连入,对数据库admin_only、test_select_only进行权限验证.

beeline -u"jdbc:hive2://data1:10000" -n test

0:jdbc:hive2://data1:10000> showdatabases;

                  +----------------+--+

                  | database_name  |

                  +----------------+--+

                  | admin_only     |

                  | default        |

                  | test_only      |

                  +----------------+--+

                  3 rows selected (0.721 seconds)

0:jdbc:hive2://data1:10000>use admin_only;

                  Error: Error while compiling statement: FAILED:SemanticException No valid privileges

                  Required privileges for this query:Server=server1->Db=admin_only->Table=*->action=insert;Server=server1->Db=admin_only->Table=*->action=select;(state=42000,code=40000)           

0:jdbc:hive2://data1:10000>use test_select_only;

                  No rows affected (0.313 seconds)

0:jdbc:hive2://data1:10000>show tables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | select_only  |

                  +--------------+--+

                  1 row selected (0.337 seconds)

0:jdbc:hive2://data1:10000>drop table select_only;

                  Error: Error while processing statement: FAILED: Execution Error, return code 1from org.apache.hadoop.hive.ql.exec.DDLTask.MetaException(message:hive.metastore.execute.setugi can't be false in nonsecure mode) (state=08S01,code=1)

    #这里需要在HIVE服务端添加如下参数

                  

                           hive.metastore.execute.setugi

                           true

                  

                  

                  4 rows selected (2.9 seconds)

0:jdbc:hive2://data1:10000>use test_select_only;

                   Norows affected (0.46 seconds)

0:jdbc:hive2://data1:10000>show tables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | select_only  |

                  +--------------+--+

                  1 row selected (0.507 seconds)

0:jdbc:hive2://data1:10000>drop table select_only;

                  Error: Error while compilingstatement: FAILED: SemanticException No valid privileges

                  Required privileges for this query:Server=server1->Db=test_select_only->Table=select_only->action=*;(state=42000,code=40000)

#场景四 test用户连入,对数据库test_only进行验证,拥有所有权限:可以查看、删除表

0: jdbc:hive2://data1:10000>usetest_only;

                  No rows affected (0.819 seconds)

0:jdbc:hive2://data1:10000> showtables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | test_itself  |

                  +--------------+--+

                  1 row selected (0.426 seconds)

0:jdbc:hive2://data1:10000>drop table test_itself;

                  No rows affected (6.336 seconds)

                  

0: jdbc:hive2://data1:10000> create table test_newone ( ont string);

                   Norows affected (1.377 seconds)

0:jdbc:hive2://data1:10000>show tables;

                  +--------------+--+

                  |   tab_name   |

                  +--------------+--+

                  | test_newone  |

                  +--------------+--+

5.2 Impala集成Sentry权限验证

1)  进入impala-shell,执行权限验证

create role supervisor;

grant all on server to rolesupervisor;

grant role supervisor to groupimpala;

GRANT ALL ON SERVER server1 TOROLE any_operation;

GRANT ROLE any_operation TOGROUP hive;

grant ALL ON database EDA TOtest_role;

grant select ON databasepriselect TO test_role;

 

2)  切换至test用户测试impala权限:

 

6 总结

仔细跟踪log,具体问题具体分析.