KuduMaster迁移

KuduMaster迁移

主数量必须为奇数。

官方文档

1.添加Kudu Master

• 获取现有master的uuid

  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  
  5315431da71e44a8ba9318330366563c
  

• 新增Kudu Master角色

• 停止所有Kudu进程（备份所有KuduMaster数据(也可以不备份，实施过程中Kudu会自动备份raft文件)）

• 格式化每台新主计算机上的数据目录，并记录生成的UUID

  mkdir -p /kudu/master/fs_wal_dir && mkdir -p /kudu/master/fs_data_dirs && chown -R kudu:kudu /kudu/master/fs_wal_dir && chown -R kudu:kudu /kudu/master/fs_data_dirs && chmod 700 /kudu/master/fs_wal_dir && chmod 700 /kudu/master/fs_data_dirs
  sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  

  • 第1台新增节点执行结果

  [root@cdh02:/root]
  # sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  
  I0717 11:31:04.182476 32125 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_data_dirs/instance:
  uuid: "5b36178fe35241e4b9ef734944c733e9"
  format_stamp: "Formatted at 2019-07-17 03:31:04 on cdh02"
  I0717 11:31:04.182989 32125 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_wal_dir/instance:
  uuid: "5b36178fe35241e4b9ef734944c733e9"
  format_stamp: "Formatted at 2019-07-17 03:31:04 on cdh02"
  I0717 11:31:04.184741 32125 fs_manager.cc:471] Time spent creating directory manager: real 0.002s       user 0.000s     sys 0.001s
  
  [root@cdh02:/root]
  # 
  
  [root@cdh02:/root]
  # sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  5b36178fe35241e4b9ef734944c733e9
  

  • 第2台新增节点执行结果

  [root@cdh03:/root]
  # mkdir -p /kudu/master/fs_wal_dir && mkdir -p /kudu/master/fs_data_dirs && chown -R kudu:kudu /kudu/master/fs_wal_dir && chown -R kudu:kudu /kudu/master/fs_data_dirs && chmod 700 /kudu/master/fs_wal_dir && chmod 700 /kudu/master/fs_data_dirs
  sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  [root@cdh03:/root]
  # sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  I0717 11:32:10.213480 30641 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_data_dirs/instance:
  uuid: "d17c7304ff8f4a22b9c7d547dc540268"
  format_stamp: "Formatted at 2019-07-17 03:32:10 on cdh03"
  I0717 11:32:10.214143 30641 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_wal_dir/instance:
  uuid: "d17c7304ff8f4a22b9c7d547dc540268"
  format_stamp: "Formatted at 2019-07-17 03:32:10 on cdh03"
  I0717 11:32:10.216106 30641 fs_manager.cc:471] Time spent creating directory manager: real 0.002s       user 0.000s     sys 0.002s
  
  [root@cdh03:/root]
  # sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  d17c7304ff8f4a22b9c7d547dc540268
  
  

  • 第3台新增节点执行结果

  # mkdir -p /kudu/master/fs_wal_dir && mkdir -p /kudu/master/fs_data_dirs && chown -R kudu:kudu /kudu/master/fs_wal_dir && chown -R kudu:kudu /kudu/master/fs_data_dirs && chmod 700 /kudu/master/fs_wal_dir && chmod 700 /kudu/master/fs_data_dirs
  sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  [root@cdh04:/root]
  # sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  I0717 11:33:01.965132 29273 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_data_dirs/instance:
  uuid: "6e61d6b48ff04201802e76be355fbaa8"
  format_stamp: "Formatted at 2019-07-17 03:33:01 on cdh04"
  I0717 11:33:01.965773 29273 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_wal_dir/instance:
  uuid: "6e61d6b48ff04201802e76be355fbaa8"
  format_stamp: "Formatted at 2019-07-17 03:33:01 on cdh04"
  I0717 11:33:01.967491 29273 fs_manager.cc:471] Time spent creating directory manager: real 0.002s       user 0.000s     sys 0.001s
  
  [root@cdh04:/root]
  # sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  6e61d6b48ff04201802e76be355fbaa8
  

• 在现有的Kudu Master主机上执行

  sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 00000000000000000000000000000000 5315431da71e44a8ba9318330366563c:cdh01:7051 5b36178fe35241e4b9ef734944c733e9:cdh02:7051 d17c7304ff8f4a22b9c7d547dc540268:cdh03:7051 6e61d6b48ff04201802e76be355fbaa8:cdh04:7051
  
  

• CM上启动现有Kudu Master主机

• 使用以下命令将主数据复制到每个新主服务器，在每台新主计算机上执行。

  sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 00000000000000000000000000000000 cdh01:7051
  
  

• 启动所有添加的新KuduMaster

2.删除KuduMaster节点

• CM页面停止所有Kudu服务

• CM删除不需要的KuduMaster节点

• 在所有其他KuduMaster节点上执行

  sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 00000000000000000000000000000000 5b36178fe35241e4b9ef734944c733e9:cdh02:7051 d17c7304ff8f4a22b9c7d547dc540268:cdh03:7051 6e61d6b48ff04201802e76be355fbaa8:cdh04:7051
  

• 启动所有剩余的KuduMaster服务

• 启动所有的KuduTabletServer服务

• 修改mysql元数据

  UPDATE TABLE_PARAMS
  SET PARAM_VALUE =
    'cdh02,cdh03,cdh04'
  WHERE PARAM_KEY = 'kudu.master_addresses'
  AND PARAM_VALUE = 'cdh01';
  

• 重启Impala的Impala Catalog Server

• 确定服务成功启动后删除旧KuduMaster数据目录

  Remove the data directories and WAL directory on the unwanted masters. This is a precaution to ensure that they cannot start up again and interfere with the new multi-master deployment.

• 检查迁移结果

  sudo -u kudu kudu cluster ksck cdh02,cdh03,cdh04