hadoop启动后某些节点未启动，hadoop主节点无法启动datanode DataNode

项目场景：

hadoop集群搭建，主节点hadoop01无法启动datanode

要注意的是：它们同时也都是DataNode节点。

---

问题描述：

执行查看出错日志：

cd /home/hadoop/apps/hadoop-2.7.7/logs/

因为日志内容是追加的，我们查看后100行就好：

tail -100 hadoop-hadoop-datanode-hadoop01.log

[hadoop@hadoop01 logs]$ tail -100 hadoop-hadoop-datanode-hadoop01.log

2020-09-09 09:21:22,113 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool (Datanode Uuid unassigned) service to hadoop01/127.0.0.1:9000 starting to offer service
2020-09-09 09:21:22,136 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2020-09-09 09:21:22,139 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2020-09-09 09:21:23,067 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
2020-09-09 09:21:23,097 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/hadoop/data/hadoopdata/data/in_use.lock acquired by nodename 3028@localhost
2020-09-09 09:21:23,099 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/home/hadoop/data/hadoopdata/data/
java.io.IOException: Incompatible clusterIDs in /home/hadoop/data/hadoopdata/data: namenode clusterID = CID-0f9e986e-1234-4742-92bd-e19d053ed284; datanode clusterID = CID-905fd4fb-483b-4c42-9763-f016e8b55fb0
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:777)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:300)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:416)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:395)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:573)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:216)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:637)
at java.lang.Thread.run(Thread.java:745)
2020-09-09 09:21:23,112 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool (Datanode Uuid unassigned) service to hadoop01/127.0.0.1:9000. Exiting.
java.io.IOException: All specified directories are failed to load.
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:574)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1393)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1358)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:216)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:637)
at java.lang.Thread.run(Thread.java:745)
2020-09-09 09:21:23,112 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool (Datanode Uuid unassigned) service to hadoop01/127.0.0.1:9000
2020-09-09 09:21:23,114 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool (Datanode Uuid unassigned)
2020-09-09 09:21:25,114 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2020-09-09 09:21:25,118 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2020-09-09 09:21:25,123 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at localhost/127.0.0.1
************************************************************/

---

原因分析：

namenode与datanode的clusterID在重新格式化namenode以后已经不再匹配，datanode无法启动。

另外：
此错误会导致在hive导入数据时发生如下错误（由于metadata不存在hdfs中，故create table并无报错）：
hive> load data local inpath ‘/root/dbfile’ overwrite into table employees PARTITION (country=‘US’, state=‘IL’);
Loading data to table default.employees partition (country=US, state=IL)
Failed with exception Unable to move source file:/root/dbfile to destination hdfs://localhost:9000/user/hive/warehouse/employees/country=US/state=IL/dbfile
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTas

参考来源：hdfs datanode 启动失败

---

解决方案：

1， 停止hadoop
先停止hadoop(条件允许的话，在所有节点同时执行)：

stop-all.sh 

如果是想在单个节点依次停止，可以执行如：
stop-dfs.sh（所有节点都要执行）
stop-yarn.sh（在ResourceManager节点执行）

2，删除存储数据
将hdfs存储数据的所在目录删掉：

在主节点删除data目录，（如果条件允许，在所有节点都执行也行），重新格式化以后，data目录会自动生成的。

rm -rf data

3， 重新格式化hdfs（相关参数：dfs.name.dir dfs.data.dir）：
一般千万不能再格式化，就像实际的硬盘格式化一样，之前的所有数据基本都会被删除
先在主节点hadoop01上执行格式化：

hadoop namenode -format

4, 重新启动hadoop

在主节点上NameNode (hadoop01)执行：

start-dfs.sh

然后在ResourceManager (hadoop03)节点执行：

start-yarn.sh

5，查看所有节点集群是否成功：

jps

注意：它们同时也DataNode节点。
到此，DataNode重启成功。

---