Macbook安装Hadoop遇到的问题总结

在自己的mac上安装了hadoop，折腾了一下午总算成功了，总结一下遇到的几个问题。报错信息没有保存，文中的报错信息都是复制的网上的，可能会有出入，后面附上了搜到答案的链接。有问题最好还是上外网搜，没条件用谷歌就用必应的国际版，百度上的答案基本上都是复读机，一个不能用就都不能用。

基本安装流程

参考以下链接做的，如果一帆风顺就不用往后看了。
https://www.jianshu.com/p/a86ccd96b78a

遇到的问题

1、Java版本问题

报错信息不记得了，大致包含export: PATH;: not a valid identifier，具体解决流程包括

1. 安装Java环境，这里不介绍了
2. 部署hadoop之前最好检查一下，命令行输入open ~/.bash_profile，查看JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home这一行
3. 输入open /usr/local/Cellar/hadoop/3.2.1/libexec/etc/hadoop/hadoop-env.sh，查看大致54行，export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home，这里的JAVA_HOME内容需要和上一步的一样，尤其要注意jdk版本是否一致；

2、bash问题

报错信息大致为

[root@master logs]# start-dfs.sh
Starting namenodes on [master]
Last login: Wed Jul  4 16:30:19 CST 2018 on pts/0
bash v3.2+ is required. Sorry.
Starting datanodes
Last login: Wed Jul  4 16:31:04 CST 2018 on pts/0
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master]
Last login: Wed Jul  4 16:31:04 CST 2018 on pts/0
bash v3.2+ is required. Sorry.

可能是使用了zsh，输入chsh -s /bin/bash切换bash，参考链接https://segmentfault.com/q/1010000015478820

3、用户名登录问题

报错信息包括

[root@master sbin]# ./start-dfs.sh
Starting namenodes on [master]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [slave1]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

这个问题困扰了我很久，百度上的方法都是修改start-dfs.sh等4个文件，加上用户名等信息，参考
https://blog.csdn.net/lglglgl/article/details/80553828，但是我实际上操作后还是有同样的问题，在外网上搜到了解决方法，实际操作后可行，步骤包括

1. 打开hadoop-env.sh，打开方式前面有
2. 在文件最后添加如下代码，参考http://www.installationbyravi.co.in/2018/04/error-attempting-to-operate-on-hdfs.html

export HDFS_NAMENODE_USER="root"
export HDFS_DATANODE_USER="root"
export HDFS_SECONDARYNAMENODE_USER="root"
export YARN_RESOURCEMANAGER_USER="root"
export YARN_NODEMANAGER_USER="root"

3. 如果不再出现上述问题说明正确，如果出现类似于

Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: `  done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 244: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 252: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 213: hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 214: hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 263: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 270: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_generic_java_subcmd_handler: command not found

这样大段错误代码，把hadoop-env.sh中的root替换为自己的用户名，命令行里能看到，参考https://stackoverflow.com/questions/54696265/anyone-know-how-to-fix-hadoop-functions-sh-syntax-error-near-unexpected-token

4.ssh问题

报错信息包括

U:sbin U$ ./start-dfs.sh
Starting namenodes on [localhost]
localhost: U@localhost: Permission denied (publickey,password,keyboard-interactive).
Starting datanodes
localhost: U@localhost: Permission denied (publickey,password,keyboard-interactive).
Starting secondary namenodes [U.local]
U.local: [email protected]: Permission denied (publickey,password,keyboard-interactive).
2018-02-25 14:52:15,505 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

解决方法参考https://stackoverflow.com/questions/48978480/hadoop-permission-denied-publickey-password-keyboard-interactive-warning，流程包括

1. 生成ssh key

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

2. 注册这个key

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

3. 登录到localhost，命令行输入ssh localhost