隶属于文章系列：大数据安全实战 https://www.jianshu.com/p/76627fd8399c
步骤
- 创建principle
- 修改core-site.xml
- 修改hdfs-site.xml
- 配置HTTPS
hdfs集成Kerberos
hadoop.security.authentication
kerberos
hadoop.security.authorization
true
dfs.block.access.token.enable
true
dfs.datanode.data.dir.perm
700
dfs.namenode.keytab.file
/etc/hadoop/conf/hdfs-service.keytab
dfs.namenode.kerberos.principal
hdfs/[email protected]
dfs.namenode.kerberos.https.principal
HTTP/[email protected]
dfs.datanode.address
0.0.0.0:61004
dfs.datanode.http.address
0.0.0.0:61006
dfs.http.policy
HTTPS_ONLY
dfs.data.transfer.protection
integrity
dfs.datanode.keytab.file
/etc/hadoop/conf/hdfs-service.keytab
dfs.datanode.kerberos.principal
hdfs/[email protected]
dfs.datanode.kerberos.https.principal
HTTP/[email protected]
dfs.journalnode.keytab.file
/etc/hadoop/conf/hdfs-service.keytab
dfs.journalnode.kerberos.principal
hdfs/[email protected]
dfs.journalnode.kerberos.internal.spnego.principal
HTTP/[email protected]
dfs.webhdfs.enabled
true
dfs.web.authentication.kerberos.principal
HTTP/[email protected]
dfs.web.authentication.kerberos.keytab
/etc/hadoop/conf/hdfs-service.keytab
- 在配置完上面的配置文件，启动后报如下错误
java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP. Using privileged resources in combination with SASL RPC data transfer protection is not supported.
at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1201)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1101)
at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:429)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2406)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2293)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2340)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2522)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2546)
2018-03-13 14:01:27,317 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2018-03-13 14:01:27,318 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
Using privileged resources in combination with SASL RPC data transfer protection is not supported.显示privileged resources（即小端口号）和SASL RPC data transfer protection不能同时使用。
这时候有两条道路选择：
- 继续使用小端口号。这样需要在root用户下使用jscv来启动dataNode，并且需要繁琐的下载依赖包额编译工作来获得jscv,最终放弃
- 使用大的端口号。参考： CSDN博Using privileged resources in combination with SASL RPC data transfer protection is not supported. -
dfs.datanode.address
0.0.0.0:61004
dfs.datanode.http.address
0.0.0.0:61006
这时候又报错：
2018-03-09 20:44:10,993 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab /etc/hadoop/conf/hdfs-service.keytab, for principal HTTP/[email protected]
2018-03-09 20:44:11,000 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab /etc/hadoop/conf/hdfs-service.keytab, for principal HTTP/[email protected]
2018-03-09 20:44:11,003 WARN org.mortbay.log: failed [email protected]:50470: java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
2018-03-09 20:44:11,003 WARN org.mortbay.log: failed Server@10ded6a9: java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
2018-03-09 20:44:11,003 INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException
java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:624)
at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:598)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:877)
at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:760)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:819)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:803)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)
2018-03-09 20:44:11,006 INFO org.mortbay.log: Stopped [email protected]:50470
2018-03-09 20:44:11,107 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2018-03-09 20:44:11,108 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2018-03-09 20:44:11,108 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2018-03-09 20:44:11,108 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.io.FileNotFoundException: /home/kduser/.keystore (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at org.mortbay.resource.FileResource.getInputStream(FileResource.java:275)
at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:624)
at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:598)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:877)
at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:760)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:819)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:803)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)
2018-03-09 20:44:11,110 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2018-03-09 20:44:11,111 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at v-hadoop-kbds.sz.kingdee.net/172.20.178.28
************************************************************/
这需要配置HTTPS。
参考：HDFS的HTTPS配置