| 问题：datanode |
| 2018-08-16 18:29:53,212 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain java.lang.RuntimeException: Cannot start secure DataNode due to incorrect config. See https://cwiki.apache.org/confluence/display/HADOOP/Secure+DataNode for detail s. at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1509) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1363) at org.apache.hadoop.hdfs.server.datanode.DataNode. at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2769) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2677) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2719) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2863) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2887) 2018-08-16 18:29:53,213 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: Cannot start secure DataNode due to incorrect conf ig. See https://cwiki.apache.org/confluence/display/HADOOP/Secure+DataNode for details. 2018-08-16 18:29:53,224 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: |
| 解决方法： |
| 配置文件问题，请认真检查每一项的name和value是否正确 |
| 问题：nodemanager |
| ************************************************************/ 2018-08-16 18:30:01,445 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT] 2018-08-16 18:30:01,883 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor: Shell execution returned exit code: 24. Privileged Execution Operation Stderr: File /opt/hadoop/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 2380
Stdout: Full command array for failed execution: [/opt/hadoop/bin/container-executor, --checksetup] 2018-08-16 18:30:01,884 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container executor initialization is : 24 org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=24: File /opt/hadoop/etc/hadoo p/container-executor.cfg must be owned by root, but is owned by 2380
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:180) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:206) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:300) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:389) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:929) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:997) Caused by: ExitCodeException exitCode=24: File /opt/hadoop/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 2380
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009) at org.apache.hadoop.util.Shell.run(Shell.java:902) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:152) ... 6 more 2018-08-16 18:30:01,886 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:391) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:929) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:997) Caused by: java.io.IOException: Linux container executor not configured properly (error=24) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:307) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:389) ... 3 more Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=24: File /opt/hadoo p/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 2380
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:180) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:206) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:300) ... 4 more Caused by: ExitCodeException exitCode=24: File /opt/hadoop/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 2380
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009) at org.apache.hadoop.util.Shell.run(Shell.java:902) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:152) ... 6 more 2018-08-16 18:30:01,887 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:391) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:929) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:997) Caused by: java.io.IOException: Linux container executor not configured properly (error=24) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:307) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:389) ... 3 more Caused by: org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=24: File /opt/hadoo p/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 2380
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:180) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:206) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:300) ... 4 more Caused by: ExitCodeException exitCode=24: File /opt/hadoop/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 2380
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009) at org.apache.hadoop.util.Shell.run(Shell.java:902) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationEx ecutor.java:152) ... 6 more 2018-08-16 18:30:01,889 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: |
| 解决方法： |
| Yarn支持两种容器实现方式，一种是yarn容器，一种是Linux容器，Linux容器较比Yarn容器具有更好的扩展性和隔离性。本文将讲述Linux容器的配置。 需要配置的文件有三：
$HADOOP_HOME/etc/hadoop/yarn-site.xml $HADOOP_HOME/etc/hadoop/container-executor.cfg $HADOOP_HOME/bin/container-executor
yarn-site.xml 为了运行Linux Container需要增加以下配置： This is the container executor setting that ensures that all applications are started with the LinuxContainerExecutor.
The POSIX group of the NodeManager. It should match the setting in "container-executor.cfg". This configuration is required for validating the secure access of the container-executor binary.
Whether all applications should be run as the NodeManager process' owner. When false, applications are launched instead as the application owner.
分别设置容器类为LinuxContainerExecutor;允许使用的用户组为：hadoop；第三个为可选项，配置是否限制NodeManager用户作为应用用户。 container-executor.cfg container-executor.cfg是启动container需要使用的配置属性。 需要保证有如下内容： yarn.nodemanager.linux-container-executor.group=hadoop yarn.nodemanager.local-dirs=$HADOOP_HOME/yarn/local yarn.nodemanager.log-dirs=$HADOOP_HOME/yarn/log banned.users=hdfs,yarn,mapred,bin min.user.id=1000
分别是确保容器运行组是hadoop；设置local和log路径，禁用用户；禁用其他系统用户。 相应的需要创建这两个目录： mkdir -p $HADOOP_HOME/yarn/local mkdir -p $HADOOP_HOME/yarn/log
修改后需要修改container-executor.cfg权限： chown root:hadoop $HADOOP_HOME/etc/hadoop/container-executor.cfg
container-executor.cfg所有者必须是root:hadoop,如果没有这一用户或用户组，需要创建。 container-executor container-executor是在$HADOOP_HOME/bin下的可执行文件，是运行Linux Container的最终执行入口。需修改其权限： chown root:hhdfs $HADOOP_HOME/bin/container-executor chmod 6050 $HADOOP_HOME/bin/container-executor
chown root:hhdfs $HADOOP_HOME chown root:hhdfs $HADOOP_HOME/etc chown root:hhdfs $HADOOP_HOME/etc/hadoop |
| 问题：SecondaryNameNode |
| 2018-08-16 18:29:55,438 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2018-08-16 18:29:55,643 FATAL org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start secondary namenode java.io.IOException: Running in secure mode, but config doesn't have a keytab at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:306) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:219) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode. at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:690) 2018-08-16 18:29:55,646 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: ExitException 2018-08-16 18:29:55,648 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG: |
| 解决方法： |
| 在hdfs-site.xml 里配置一下内容
|
| 问题： |
| hadoop3客户端验： 2018-08-17 15:14:47,489 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] mkdir: DestHost:destPort viyavi:9000 , LocalHost:localPort viyavi/192.168.0.24:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
hadoop2客户端验： 18/08/24 11:52:19 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "viyavi/192.168.0.24"; destination host is: "viyavi":9000; |
| 解决方法： |
| KERBEROS 配置文件（/etc/krb5.conf）中关于default_ccache_name 的配置不能使用默认的“KEYRING:persistent:%{uid}” ，应指定具体路径如：/tmp/krb5cc_%{uid}，或注释掉default_ccache_name的配置。 |