hadoop安装遇到的一些障碍

一.start-dfs.sh(开启不成功)

原因:1.dadoop   文件所属者 与 执行时 登陆者不一致(权限不足)

                报错:

WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.

Starting namenodes on [centos7]

centos7: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Starting datanodes

localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Starting secondary namenodes [centos7]

centos7: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /hadoop/hadoop-3.2.0/logs/fairscheduler-statedump.log (权限不够)

at java.io.FileOutputStream.open0(Native Method)

at java.io.FileOutputStream.open(FileOutputStream.java:270)

at java.io.FileOutputStream.(FileOutputStream.java:213)

at java.io.FileOutputStream.(FileOutputStream.java:133)

at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)

at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)

at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)

at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)

at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)

at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)

at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)

at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)

at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)

at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)

at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)

at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)

at org.apache.log4j.LogManager.(LogManager.java:127)

at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)

at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)

at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)

at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)

at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)

at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)

at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)

at org.apache.hadoop.conf.Configuration.(Configuration.java:227)

at org.apache.hadoop.hdfs.tools.GetConf.(GetConf.java:131)

情况一

二.使用root配置的hadoop并启动会出现报错

错误:

        Starting namenodes on [master]

ERROR: Attempting to operate on hdfs namenode as root


ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.


Starting datanodes

ERROR: Attempting to operate on hdfs datanode as root


ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.

        Starting secondary namenodes [slave1]

        ERROR: Attempting to operate on hdfs secondarynamenode as root

        ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

解决方法:

在/hadoop/sbin路径下: 

将start-dfs.sh,stop-dfs.sh两个文件顶部添加以下参数

HDFS_DATANODE_USER=root

HADOOP_SECURE_DN_USER=hdfs

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

start-yarn.sh,stop-yarn.sh顶部也需添加以下

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn

YARN_NODEMANAGER_USER=root

2、添加1后出现以下错误

WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.

Starting namenodes on [mylinux_1]

mylinux_1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Starting datanodes

localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Starting secondary namenodes [mylinux_1]

mylinux_1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

2018-11-26 09:32:18,082 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting resourcemanager

Starting nodemanagers

localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

解决办法:配置免密登录(注意:对本机也需要配置)

ssh-keygen -t rsa

ssh-copy-id -i ~/.ssh/id_rsa.pub root@localhost


三.执行jps 没有 nodemanager

原因是:yarn-site.xml 配置出现了问题

在yarn-site.xml添加

    yarn.nodemanager.resource.cpu-vcores

    你的cpu个数

       

        yarn.nodemanager.resource.memory-mb

        你的内存

   


你可能感兴趣的:(hadoop安装遇到的一些障碍)