mac搭建大数据环境遇到的问题

使用brew安装hadoop，使用无法正常市用。最终是使用下载解压包的方式。

问题1 Invalid HADOOP_COMMON_HOME

xin:out root# hadoop jar /usr/local/Cellar/hadoop/3.1.2/libexec/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-3.1.2-sources.jar org.apache.hadoop.examples.WordCount /Users/xin/Downloads/out/word.txt /Users/xin/Downloads/out/word-out
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
ERROR: Invalid HADOOP_COMMON_HOME

配置了上述的路径之后，输入hadoop命令没有任何反应，也没有报错。

xin:out root# hadoop version
xin:out root# 

问题2 Connection closed by remote host

ssh无法使用。
一种原因是mac账号问题。
mac 无法使用 ssh root@本机：https://blog.csdn.net/datadev_sh/article/details/90111894

一种是sshd配置问题。

xin:mapreduce xin$ start-all.sh 
WARNING: Attempting to start all Apache Hadoop daemons as xin in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
localhost: ssh_exchange_identification: Connection closed by remote host
Starting datanodes
localhost: ssh_exchange_identification: Connection closed by remote host
Starting secondary namenodes [xin]
xin: ssh_exchange_identification: Connection closed by remote host
2019-05-11 17:27:17,603 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
localhost: ssh_exchange_identification: Connection closed by remote host

① 打开远程登录

所有用户

② 修改PermitRootLogin

只要修改PermitRootLogin，别的都不用修改，且也不用添加其他配置，否则可能导致问题继续。比如 Line 38 的PubkeyAuthentication 不要打开。

问题3 start-dfs.sh Permission denied

xin:.ssh xin$ start-dfs.sh 
Starting namenodes on [localhost]
localhost: xin@localhost: Permission denied (publickey,password,keyboard-interactive).
Starting datanodes
localhost: xin@localhost: Permission denied (publickey,password,keyboard-interactive).
Starting secondary namenodes [xin]
xin: Warning: Permanently added 'xin' (ECDSA) to the list of known hosts.
xin: xin@xin: Permission denied (publickey,password,keyboard-interactive).
2019-05-11 19:44:41,189 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

参考mac 无法使用 ssh root@本机：https://blog.csdn.net/datadev_sh/article/details/90111894

测试

xin:.ssh xin$ ssh xin@localhost
Password:
Last login: Sat May 11 19:43:45 2019 from ::1

也要发一份ssh key到本地

xin:.ssh xin$ ssh-copy-id -i id_rsa.pub localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:

问题4 启动 hbase 找不到或无法加载主类

xin:bin xin$ /usr/local/opt/hbase/bin/start-hbase.sh
错误: 找不到或无法加载主类 org.apache.hadoop.hbase.util.HBaseConfTool
错误: 找不到或无法加载主类 org.apache.hadoop.hbase.zookeeper.ZKServerTool
mv: rename /usr/local/Cellar/hbase/1.2.9/logs/hbase-xin-master-xin.out to /usr/local/Cellar/hbase/1.2.9/logs/hbase-xin-master-xin.out.1: Permission denied
starting master, logging to /usr/local/Cellar/hbase/1.2.9/logs/hbase-xin-master-xin.out
/usr/local/Cellar/hbase/1.2.9/libexec/bin/hbase-daemon.sh: line 189: /usr/local/Cellar/hbase/1.2.9/logs/hbase-xin-master-xin.out: Permission denied
错误: 找不到或无法加载主类 org.apache.hadoop.hbase.master.HMaster
cat: /usr/local/Cellar/hbase/1.2.9/conf/regionservers: No such file or directory
cat: /usr/local/Cellar/hbase/1.2.9/conf/regionservers: No such file or directory

brew 安装的文件都在 hbase/1.2.9/libexec/ 下，而启动 hbase 时都是去 hbase/1.2.9/ 下寻找文件。

/usr/local/Cellar/hbase/1.2.9/libexec/conf
/usr/local/Cellar/hbase/1.2.9/conf

把libexec下的都移出来，移到上一级目录。