cdh5.3.10_hadoop环境迁移之主机IP修改

cdh5.3.10_hadoop环境迁移
主机迁移后hadoop集群调整：
1、修改/etc/hosts（三台主机均需要修改）
修改前：
[root@hadoop-master ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost.localdomain localhost4 localhost4.localdomain4 localhost
::1 localhost.localdomain localhost.localdomain localhost6 localhost6.localdomain6 localhost

10.117.130.207 hadoop-master
10.117.130.208 hadoop-slave1
10.117.130.209 hadoop-slave2
[root@hadoop-master ~]# 
修改后：
[root@hadoop-master ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost.localdomain localhost4 localhost4.localdomain4 localhost
::1 localhost.localdomain localhost.localdomain localhost6 localhost6.localdomain6 localhost

10.117.130.110 hadoop-master
10.117.130.111 hadoop-slave1
10.117.130.112 hadoop-slave2
[root@hadoop-master ~]#

2、修改ntp时钟同步（三台主机均需要修改）
修改前：
[root@hadoop-master ~]# cat /etc/ntp.conf 
server 10.117.130.207 prefer
修改后：
[root@hadoop-master ~]# cat /etc/ntp.conf 
server 10.117.130.110 prefer

3、修改mysql中hadoop相关配置（仅在master节点修改）
修改前：
[root@hadoop-master data]# mysql -u root -p 
Enter password: 
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 58
Server version: 5.6.29 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| am                 |
| hive               |
| mysql              |
| oozie              |
| performance_schema |
| rm                 |
| scm                |
| test               |
+--------------------+
9 rows in set (0.00 sec)


mysql> use scm;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A


Database changed
mysql> show tables;
+--------------------------------+
| Tables_in_scm                  |
+--------------------------------+
| AUDITS                         |
| CLIENT_CONFIGS                 |
| CLIENT_CONFIGS_TO_HOSTS        |
| CLUSTERS                       |
| CLUSTERS_AUD                   |
| CLUSTER_ACTIVATED_RELEASES     |
| CLUSTER_ACTIVATED_RELEASES_AUD |
| CLUSTER_MANAGED_RELEASES       |
| CLUSTER_UNDISTRIBUTED_RELEASES |
| CM_PEERS                       |
| CM_VERSION                     |
| COMMANDS                       |
| COMMAND_SCHEDULES              |
| CONFIGS                        |
| CONFIGS_AUD                    |
| CONFIG_CONTAINERS              |
| CREDENTIALS                    |
| GLOBAL_SETTINGS                |
| HOSTS                          |
| HOSTS_AUD                      |
| HOST_TEMPLATES                 |
| HOST_TEMPLATE_TO_ROLE_CONF_GRP |
| METRICS                        |
| PARCELS                        |
| PARCEL_COMPONENTS              |
| PROCESSES                      |
| PROCESS_ACTIVE_RELEASES        |
| RELEASES                       |
| RELEASES_AUD                   |
| REVISIONS                      |
| ROLES                          |
| ROLES_AUD                      |
| ROLE_CONFIG_GROUPS             |
| ROLE_CONFIG_GROUPS_AUD         |
| ROLE_STALENESS_STATUS          |
| SCHEMA_VERSION                 |
| SERVICES                       |
| SERVICES_AUD                   |
| SNAPSHOT_POLICIES              |
| USERS                          |
| USER_ROLES                     |
| USER_SETTINGS                  |
+--------------------------------+
42 rows in set (0.00 sec)

mysql> mysql> select * from HOSTS;
+---------+-------------------------+--------------------------------------+---------------+----------------+----------+--------+---------------------+-------------------+--------------------+------------+-----------+----------------------+-------------+-------------------+----------------+
| HOST_ID | OPTIMISTIC_LOCK_VERSION | HOST_IDENTIFIER                      | NAME          | IP_ADDRESS     | RACK_ID  | STATUS | CONFIG_CONTAINER_ID | MAINTENANCE_COUNT | DECOMMISSION_COUNT | CLUSTER_ID | NUM_CORES | TOTAL_PHYS_MEM_BYTES | PUBLIC_NAME | PUBLIC_IP_ADDRESS | CLOUD_PROVIDER |
+---------+-------------------------+--------------------------------------+---------------+----------------+----------+--------+---------------------+-------------------+--------------------+------------+-----------+----------------------+-------------+-------------------+----------------+
|       1 |                      14 | 135caa69-b742-467b-bf08-6742781a1b2c | hadoop-master | 10.117.130.207 | /default | NA     |                   1 |                 0 |                  0 |          1 |         4 |          12626055168 | NULL        | NULL              | NULL           |
|       2 |                      13 | de481baa-e125-4c35-8538-aa3bf7229f33 | hadoop-slave1 | 10.117.130.208| /default | NA     |                   1 |                 0 |                  0 |          1 |         4 |          12626055168 | NULL        | NULL              | NULL           |
|       3 |                      11 | 6af8fa7f-1ce1-47cd-a1a6-6d8ef91c77f2 | hadoop-slave2 | 10.117.130.209 | /default | NA     |                   1 |                 0 |                  0 |          1 |         4 |          12626055168 | NULL        | NULL              | NULL           |
+---------+-------------------------+--------------------------------------+---------------+----------------+----------+--------+---------------------+-------------------+--------------------+------------+-----------+----------------------+-------------+-------------------+----------------+
3 rows in set (0.00 sec)
mysql> update HOSTS set ip_address= '10.117.130.110' where host_id=1;
Query OK, 1 row affected (0.00 sec)
mysql> update HOSTS set ip_address= '10.117.130.111' where host_id=2;
Query OK, 1 row affected (0.00 sec)
mysql> update HOSTS set ip_address= '10.117.130.112' where host_id=3;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

mysql> select * from HOSTS;
+---------+-------------------------+--------------------------------------+---------------+----------------+----------+--------+---------------------+-------------------+--------------------+------------+-----------+----------------------+-------------+-------------------+----------------+
| HOST_ID | OPTIMISTIC_LOCK_VERSION | HOST_IDENTIFIER                      | NAME          | IP_ADDRESS     | RACK_ID  | STATUS | CONFIG_CONTAINER_ID | MAINTENANCE_COUNT | DECOMMISSION_COUNT | CLUSTER_ID | NUM_CORES | TOTAL_PHYS_MEM_BYTES | PUBLIC_NAME | PUBLIC_IP_ADDRESS | CLOUD_PROVIDER |
+---------+-------------------------+--------------------------------------+---------------+----------------+----------+--------+---------------------+-------------------+--------------------+------------+-----------+----------------------+-------------+-------------------+----------------+
|       1 |                      14 | 135caa69-b742-467b-bf08-6742781a1b2c | hadoop-master | 10.117.130.110 | /default | NA     |                   1 |                 0 |                  0 |          1 |         4 |          12626055168 | NULL        | NULL              | NULL           |
|       2 |                      13 | de481baa-e125-4c35-8538-aa3bf7229f33 | hadoop-slave1 | 10.117.130.111 | /default | NA     |                   1 |                 0 |                  0 |          1 |         4 |          12626055168 | NULL        | NULL              | NULL           |
|       3 |                      11 | 6af8fa7f-1ce1-47cd-a1a6-6d8ef91c77f2 | hadoop-slave2 | 10.117.130.112 | /default | NA     |                   1 |                 0 |                  0 |          1 |         4 |          12626055168 | NULL        | NULL              | NULL           |
+---------+-------------------------+--------------------------------------+---------------+----------------+----------+--------+---------------------+-------------------+--------------------+------------+-----------+----------------------+-------------+-------------------+----------------+
3 rows in set (0.00 sec)


mysql>

mysql> commit;
Query OK, 0 rows affected (0.00 sec)
mysql>

4、重启cdh-hadoop的服务
[root@hadoop-master data]# /opt/cloudera-manager/cm-5.3.10/etc/init.d/cloudera-scm-server restart
鍋滄 cloudera-scm-server锛 [纭畾]
Starting cloudera-scm-server: [纭畾]
[root@hadoop-master data]# /opt/cloudera-manager/cm-5.3.10/etc/init.d/cloudera-scm-agent restart
Stopping cloudera-scm-agent: [纭畾]
姝ｅ湪鍚姩 cloudera-scm-agent锛 [纭畾]
[root@hadoop-master data]
[root@hadoop-slave1 ~]# /opt/cloudera-manager/cm-5.3.10/etc/init.d/cloudera-scm-agent start
姝ｅ湪鍚姩 cloudera-scm-agent锛 [纭畾]
[root@hadoop-slave1 ~]# 
[root@hadoop-slave2 ~]# /opt/cloudera-manager/cm-5.3.10/etc/init.d/cloudera-scm-agent start
姝ｅ湪鍚姩 cloudera-scm-agent锛 [纭畾]
[root@hadoop-slave2 ~]#

5、登录cdh集群web管理页面
http://10.117.130.110:7180/cmf/home
首先保证CDH第一个服务即主机一项是正常的
看到部分服务提示配置信息过旧：需要重新部署客户端配置 此时点击链接按照页面提示重新部署客户端配置即可

6、重新部署cdh部分服务的客户端配置后重启cdh服务，cdh恢复正常

注意：由于cdh三台虚拟主机是镜像过来的，cdh相关服务文件或进程有状态保留，需要查杀相关进程
本次处理中，碰到节点2重启agent客户端进程一直在运行，问题出在9000端口相应的进程占用，导致cdh识别节点2出现问题
[root@hadoop-slave1 ~]# /opt/cloudera-manager/cm-5.3.10/etc/init.d/cloudera-scm-agent start
cloudera-scm-agent is already running
[root@hadoop-slave1 ~]# netstat -nap | grep 9000
tcp 0 0 10.117.130.111:9000 0.0.0.0:* LISTEN 12510/python 
tcp 0 0 10.117.130.111:44223 10.117.130.111:9000 TIME_WAIT - 
[root@hadoop-slave1 ~]# kill -9 0
Connection to hadoop-slave1 closed.
[root@hadoop-slave2 ~]# netstat -nap | grep 9000
[root@hadoop-slave2 ~]# /opt/cloudera-manager/cm-5.3.10/etc/init.d/cloudera-scm-agent start
姝ｅ湪鍚姩 cloudera-scm-agent锛 [纭畾]
[root@hadoop-slave2 ~]# netstat -nap | grep 9000
tcp 0 0 10.117.130.112:9000 0.0.0.0:* LISTEN 3689/python 
tcp 0 0 10.117.130.112:50759 10.117.130.112:9000 TIME_WAIT - 
[root@hadoop-slave2 ~]#