HDFS 通过MR或IO流操作时 kerberos认证配置一切正确缺认证异常失败，往往忽悠了一个很重要的问题。

19/05/20 11:09:25 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/05/20 11:09:25 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/05/20 11:09:25 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort XXXXXXXXXX:8020 , LocalHost:localPort XXXXXXXXXX:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over XXXXXXXXXX:8020 after 1 failover attempts. Trying to failover after sleeping for 598ms.
19/05/20 11:09:26 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/05/20 11:09:26 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort XXXXXXXXXX:8020 , LocalHost:localPort XXXXXXXXXX:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over XXXXXXXXXX:8020 after 2 failover attempts. Trying to failover after sleeping for 2548ms.
19/05/20 11:09:28 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/05/20 11:09:28 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort XXXXXXXXXX:8020 , LocalHost:localPort XXXXXXXXXX:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over XXXXXXXXXX:8020 after 3 failover attempts. Trying to failover after sleeping for 4792ms.

错误：

FileSystem fs = FileSystem.get(new URI(dataPath), conf, "InterceptPornsLogs");

List dataFilesList = getFilesUnderFolder(fs, new Path(dataPath), dataTime);

修改：

FileSystem fs = FileSystem.get(conf);

List dataFilesList = getFilesUnderFolder(fs, new Path(dataPath), dataTime);

原因：

new URI(dataPath), conf, "InterceptPornsLogs"

源码 ：

 

public static FileSystem get(URI uri, Configuration conf) throws IOException {
    String scheme = uri.getScheme();
    String authority = uri.getAuthority();
    if (scheme == null && authority == null) {
        return get(conf);
    } else {
        if (scheme != null && authority == null) {
            URI defaultUri = getDefaultUri(conf);
            if (scheme.equals(defaultUri.getScheme()) && defaultUri.getAuthority() != null) {
                return get(defaultUri, conf);
            }
        }

。。。。。

public static FileSystem get(Configuration conf) throws IOException {
    return get(getDefaultUri(conf), conf);
}

这两个构造方法中第一个先通过URI获取信息在绑定Configuration，这一步互信还没建立会一直被阻挡。

所有采用第二个即可，通过getDefaultUri(conf)中的conf去获取URI在进行绑定Configuration。

。