使用Spark/Java读取已开启Kerberos认证的HBase

1.赋予drguo用户相应的权限

---

2.KDC中创建drguo用户并导出相应的keytab文件

[root@bigdata28 ~]# kadmin.local 
Authenticating as principal drguo/admin@AISINO.COM with password.
kadmin.local:  addprinc drguo/bigdata28
WARNING: no policy specified for drguo/bigdata28@AISINO.COM; defaulting to no policy
Enter password for principal "drguo/[email protected]": 
Re-enter password for principal "drguo/[email protected]": 
Principal "drguo/[email protected]" created.
kadmin.local:  xst -norandkey -k /home/drguo/drguo_bigdata28.keytab drguo/bigdata28@AISINO.COM
Entry for principal drguo/bigdata28@AISINO.COM with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/home/drguo/drguo_bigdata28.keytab.
Entry for principal drguo/bigdata28@AISINO.COM with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/home/drguo/drguo_bigdata28.keytab.
Entry for principal drguo/bigdata28@AISINO.COM with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/home/drguo/drguo_bigdata28.keytab.
Entry for principal drguo/bigdata28@AISINO.COM with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/home/drguo/drguo_bigdata28.keytab.
Entry for principal drguo/bigdata28@AISINO.COM with kvno 1, encryption type des-hmac-sha1 added to keytab WRFILE:/home/drguo/drguo_bigdata28.keytab.
Entry for principal drguo/bigdata28@AISINO.COM with kvno 1, encryption type des-cbc-md5 added to keytab WRFILE:/home/drguo/drguo_bigdata28.keytab.
kadmin.local:  q

---

3.将krb5.conf与keytab文件拷到本地，方便测试

---

4.使用Spark读取HBase

package drguo.test

import java.io.IOException

import com.google.protobuf.ServiceException
import dguo.test.HBaseKerb
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.client.{HBaseAdmin, HTable}
import org.apache.hadoop.hbase.mapreduce.{TableInputFormat}
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.{SparkConf, SparkContext}

/**
  * Created by drguo on 2018/7/18.
  */
object SparkExecHBase {


  def main(args: Array[String]): Unit = {
//    HBaseKerb.getAllRows("XMJZ")
    System.setProperty("java.security.krb5.conf", "d:/krb5.conf")
    val sparkConf = new SparkConf().setAppName("SparkExecHBase").setMaster("local")
    val sc = new SparkContext(sparkConf)

    val conf = HBaseConfiguration.create()
    conf.set(TableInputFormat.INPUT_TABLE, "XMJZ")
    conf.set("hbase.zookeeper.quorum","172.19.6.28,172.19.6.29,172.19.6.30")
    conf.set("hbase.zookeeper.property.clientPort", "2181")
    conf.set("hadoop.security.authentication", "Kerberos")

    UserGroupInformation.setConfiguration(conf)
    try {
      UserGroupInformation.loginUserFromKeytab("drguo/[email protected]", "d:/drguo_bigdata28.keytab")
      HBaseAdmin.checkHBaseAvailable(conf)
    } catch {
      case e: IOException =>
        e.printStackTrace()
      case e: ServiceException =>
        e.printStackTrace()
    }

    val hbaseRdd = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
//    println(hbaseRdd.toString())
    hbaseRdd.map( x=>x._2).map{result => (result.getRow,result.getValue(Bytes.toBytes("Info"),Bytes.toBytes("ADDTIME")))}.map(row => (new String(row._1),new String(row._2))).collect.foreach(r => (println(r._1+":"+r._2)))

  }

}

---

5.使用Java读取（网上也有不少例子，但大部分都有一些重复、多余的代码）

package dguo.test;

import com.google.protobuf.ServiceException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.security.UserGroupInformation;

import java.io.IOException;

/**
 * Created by drguo on 2018/7/18.
 */
public class HBaseKerb {

    private static Configuration conf = null;
    static {
        System.setProperty("java.security.krb5.conf", "d:/krb5.conf" );
        //使用HBaseConfiguration的单例方法实例化
        conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum", "172.19.6.28,172.19.6.29,172.19.6.30");
        conf.set("hbase.zookeeper.property.clientPort", "2181");
        conf.set("hadoop.security.authentication" , "Kerberos" );

        UserGroupInformation.setConfiguration(conf);

        try {
            UserGroupInformation.loginUserFromKeytab("drguo/[email protected]", "d:/drguo_bigdata28.keytab");
            HBaseAdmin.checkHBaseAvailable(conf);
        } catch (IOException e) {
            e.printStackTrace();
        } catch (ServiceException e) {
            e.printStackTrace();
        }

    }

    public static void getAllRows(String tableName) throws IOException{
        HTable hTable = new HTable(conf, tableName);
        //得到用于扫描region的对象
        Scan scan = new Scan();
        //使用HTable得到resultcanner实现类的对象
        ResultScanner resultScanner = hTable.getScanner(scan);
        for(Result result : resultScanner){
            Cell[] cells = result.rawCells();
            for(Cell cell : cells){
                //得到rowkey
                System.out.println("行键:" + Bytes.toString(CellUtil.cloneRow(cell)));
                //得到列族
                System.out.println("列族" + Bytes.toString(CellUtil.cloneFamily(cell)));
                System.out.println("列:" + Bytes.toString(CellUtil.cloneQualifier(cell)));
                System.out.println("值:" + Bytes.toString(CellUtil.cloneValue(cell)));
            }
        }
    }

    public static void main(String[] args) throws IOException{
        getAllRows("XMJZ");
    }
}

---

PS:

出现下述错误往往是因为System.setProperty(“java.security.krb5.conf”, “d:/krb5.conf”)中的krb5.conf文件没有找到（比如路径错误）或是里面配置的kdc、admin_server地址错误。

Exception in thread “main” java.lang.IllegalArgumentException: Can’t get Kerberos realm
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:319)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:374)
at drguo.test.SparkExecHBase$.main(SparkExecHBase.scala:32)
at drguo.test.SparkExecHBase.main(SparkExecHBase.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84)
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
… 4 more
Caused by: KrbException: Cannot locate default realm
at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
… 10 more