Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移

基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移

NFS+DRBD+Keepalived+KVM动态迁移演示

学习环境:

主机名 IP 安装服务
KVM01 192.168.1.112 KVM
KVM01 192.168.1.113 KVM
NFS 192.168.1.111 nfs+drbd+keepalived
NFS2 192.168.1.114 nfs+drbd+keepalived

VMware 虚拟机中 linux 开机前需要勾选Inter VT-x/EPT 或 AMD/RVI(v)
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第1张图片

在CentOS7的系统光盘镜像中,已经提供了安装KVM所需软件,通过部署基于光盘镜像的本地yum源,直接使用yum安装所需软件即可。

[root@localhost ~] yum -y groupinstall -y "GNOME Desktop"     #安装GNOME桌面环境
[root@localhost ~] yum -y install qemu-kvm                  #KVM模块
[root@localhost ~] yum -y install qemu-kvm-tools        #KVM调试工具,可不安装
[root@localhost ~] yum -y install qemu-img                  #qemu组件,创建磁盘,启动虚拟机
[root@localhost ~] yum -y install bridge-utils                 #网络支持工具
[root@localhost ~] yum -y install libvirt                           #虚拟机管理工具
[root@localhost ~] yum -y install virt-manager              #图形界面管理虚拟机
[root@localhost ~] yum -y install qemu-kvm qemu-kvm-tools virt-install qemu-img bridge-utils libvirt virt-manager

检测KVM模块安装是否成功

[root@localhost ~] lsmod | grep kvm
kvm_intel             174841  0
kvm                   578518  1 kvm_intel
irqbypass              13503  1 kvm

注意:使用yum安装完桌面后,将系统默认的运行target修改为graphical.target。重启系统后进入图形界面,若不修改系统的默认运行target,重启时可能会出错。(init 5)

开启服务,安装完成后还需要开启libvirtd服务

[root@localhost ~] systemctl start libvirtd
[root@localhost ~] systemctl enable libvirtd

设置KVM网络默认是NAT。

[root@KVM01 ~] tail -4 /etc/hosts
192.168.1.112     KVM01
192.168.1.113     KVM02
192.168.1.111     NFS
192.168.1.114     NFS2
[root@KVM01 ~] scp /etc/hosts 192.168.1.111:/etc
[root@KVM01 ~] scp /etc/hosts 192.168.1.113:/etc
[root@KVM01 ~] scp /etc/hosts 192.168.1.114:/etc

NFS SERVER:配置共享目录 /data/kvm-share
[root@NFS ~] cat /etc/exports
/data/kvm-share 192.168.1.112(rw,sync,no_root_squash)
/data/kvm-share 192.168.1.113(rw,sync,no_root_squash)
[root@NFS ~] mkdir -p /data/kvm-share
[root@NFS ~] systemctl start nfs

  • DRBD部署
    DRBD详细解说
    实验初始配置:所有主机关闭防火墙与selinux
[root@localhost ~] iptables -F
[root@localhost ~] systemctl stop firewalld 
[root@localhost ~] systemctl disable firewalld 

[root@localhost ~] setenforce 0
[root@localhost ~] sed -i '/SELINUX/ s/enforcing/disabled/g' /etc/sysconfig/selinux
[root@NFS ~]rpm -ivh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
[root@NFS ~]yum install -y drbd84-utils kmod-drbd84
[root@NFS ~]modprobe drbd
[root@NFS ~]lsmod | grep drbd
drbd                  397041  0
  • DRBD配置(NFS和NFS2)
[root@NFS ~]cp /etc/drbd.d/global_common.conf /etc/drbd.d/global_common.conf.bak
[root@NFS ~]vim /etc/drbd.d/global_common.conf
global {
  usage-count yes;
  }
common {
  protocol C;
  handlers {
}
  startup {
        wfc-timeout          240;
        degr-wfc-timeout     240;
        outdated-wfc-timeout 240;
  }
  disk {
        on-io-error detach;
  }
  net {
        cram-hmac-alg md5;
        shared-secret "testdrbd";
  }
  syncer {
        rate 30M;
  }
}

[root@NFS ~]vim /etc/drbd.d/r0.res
resource r0 {
on NFS {
  device     /dev/drbd0;           
  disk       /dev/sdb1;
  address    192.168.1.111:7898;		#端口可以自己定义
  meta-disk  internal;
 }
on NFS2 {
  device     /dev/drbd0;          
  disk       /dev/sdb1;
  address    192.168.1.114:7898; 
  meta-disk  internal;
 }
}
  • 在两台机器上添加DRBD磁盘(NFS和NFS2)
    在NFS机器上添加一块10G的硬盘作为DRBD,分区为/dev/sdb1,不做格式化,并在本地系统创建/data目录,不做挂载操作。
fdisk -l
磁盘 /dev/sdb:10.7 GB, 10737418240 字节,20971520 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理):512 字节 / 512 字节
I/O 大小(最小/最佳):512 字节 / 512 字节
fdisk /dev/sdb   
依次输入"n->p->1->回车->回车->w"     //分区创建后,再次使用"fdisk /dev/sdb",输入p,即可查看到创建的分区.
mkdir /data
  • 在两台机器上分别创建DRBD设备并激活r0资源(NFS和NFS2)
[root@NFS ~]mknod /dev/drbd0 b 147 0
[root@NFS ~]drbdadm create-md r0
	Writing meta data...
	initializing activity log
	NOT initialized bitmap
	New drbd meta data block successfully created.
[root@NFS ~]drbdadm create-md r0
	You want me to create a v08 style flexible-size internal meta data block.
	There appears to be a v08 flexible-size internal meta data block
	already in place on /dev/vdd1 at byte offset 10737340416
	Do you really want to overwrite the existing v08 meta-data?
	[need to type 'yes' to confirm] yes                         //这里输入"yes"
	Writing meta data...
	initializing activity log
	NOT initialized bitmap
	New drbd meta data block successfully created.
启动drbd服务(注意:需要主从共同启动方能生效)
[root@NFS ~]systemctl start drbd
[root@NFS ~]ps -ef|grep drbd
	root      81611   7779  0 20:11 pts/0    00:00:00 grep --color=auto drbd
[root@NFS ~]cat /proc/drbd
	version: 8.4.11-1 (api:1/proto:86-101)
	GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-	11-03 01:26:55
 	0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10484380
[root@NFS2 ~]: cat /proc/drbd
	version: 8.4.11-1 (api:1/proto:86-101)
	GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-	11-03 01:26:55
	0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10484380
由上面两台主机的DRBD状态查看结果里的ro:Secondary/Secondary表示两台主机的状态都是备机状态,ds是磁盘状态,显示的状态内容为“不一致”,这是因为DRBD无法判断哪一方为主机,应以哪一方的磁盘数据作为标准。
  • 接着将NFS主机配置为DRBD的主节点
[root@NFS ~]drbdsetup /dev/drbd0 primary --force
[root@NFS ~] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:163528 nr:0 dw:0 dr:165648 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10320852
	[>....................] sync'ed:  1.7% (10076/10236)M
	finish: 0:05:15 speed: 32,704 (32,704) K/sec

[root@NFS2 ~]#  cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:1221508 dw:1221508 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:9262872
	[=>..................] sync'ed: 11.7% (9044/10236)M
	finish: 0:03:51 speed: 40,056 (34,900) want: 41,040 K/sec

ro在主从服务器上分别显示 Primary/Secondary和Secondary/Primary
ds显示UpToDate/UpToDate 表示主从配置成功
  • 挂载DRBD(NFS)
先格式化/dev/drbd0
[root@ NFS ~] mkfs.xfs /dev/drbd0
meta-data=/dev/drbd0             isize=512    agcount=4, agsize=655274 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2621095, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

创建挂载目录,然后执行DRBD挂载
[root@ NFS ~] mkdir /data
[root@ NFS ~] mount /dev/drbd0 /data
[root@NFS ~] df -Th
文件系统                类型      容量  已用  可用 已用% 挂载点
/dev/drbd0              xfs        10G   33M   10G    1% /data

特别注意:
server01节点上不允许对DRBD设备进行任何操作,包括只读,所有的读写操作只能在server01节点上进行。
只有当server01节点挂掉时,server02节点才能提升为server01节点
  • DRBD主备故障切换测试(可不做)
模拟NFS节点发生故障,NFS2接管并提升为NFS
下面是在NFS主节点上操作记录
 [root@ NFS ~] cd /data/
[root@ NFS data] touch test{1..5}
[root@NFS data] ls
test1  test2  test3  test4  test5
[root@ NFS data] cd ../
[root@ NFS /] umount /data
[root@ NFS /] drbdsetup /dev/drbd0 secondary     //将Primary主机设置为DRBD的备节点。在实际生产环境中,直接在NFS2主机上提权(即设置为主节点)即可。
[root@ NFS /] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:Connected ro:Secondary/Secondary ds:UpToDate/UpToDate C r-----
    ns:10497043 nr:0 dw:12663 dr:10489714 al:16 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
注意:这里实际生产环境若NFS主节点宕机,在NFS2状态信息中ro的值会显示为Secondary/Unknown,只需要进行DRBD提权操作即可。
下面是在NFS2 备份节点上操作记录
先进行提权操作,即将server02手动升级为DRBD的主节点
[root@NFS2 ~] drbdsetup /dev/drbd0 primary
[root@NFS2 ~] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:0 nr:10497043 dw:10497043 dr:2120 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
然后挂在drbd
[root@NFS2 ~] mount /dev/drbd0 /data
[root@NFS2 ~] df -Th
文件系统                类型      容量  已用  可用 已用% 挂载点
/dev/mapper/centos-root xfs        50G  4.0G   47G    8% /
devtmpfs                devtmpfs  894M     0  894M    0% /dev
tmpfs                   tmpfs     910M     0  910M    0% /dev/shm
tmpfs                   tmpfs     910M  9.9M  900M    2% /run
tmpfs                   tmpfs     910M     0  910M    0% /sys/fs/cgroup
/dev/sr0                iso9660   4.3G  4.3G     0  100% /media/cdrom
/dev/sda1               xfs      1014M  179M  836M   18% /boot
/dev/mapper/centos-home xfs        27G   37M   27G    1% /home
tmpfs                   tmpfs     182M     0  182M    0% /run/user/0
/dev/drbd0              xfs        10G   33M   10G    1% /data

发现DRBD挂载目录下已经有了之前在远程NFS主机上写入的内容
[root@NFS2 ~] ls /data
test1  test2  test3  test4  test5

NFS部署

  • 在两台主机上安装nfs(NFS和NFS2)
[root@NFS ~] yum install rpcbind nfs-utils
[root@NFS ~] vim /etc/exports
/data/kvm-share 192.168.1.112(rw,sync,no_root_squash)
/data/kvm-share 192.168.1.113(rw,sync,no_root_squash)
~                                                       
[root@ NFS ~] systemctl start rpcbind
[root@ NFS ~] systemctl start nfs
  • 在两台kvm主机上验证(KVM01和KVM02)
[root@KVM01 ~] showmount -e 192.168.1.114
Export list for 192.168.1.114:
/data/kvm-share 192.168.1.113,192.168.1.112 
[root@KVM01 ~] showmount -e 192.168.1.111
Export list for 192.168.1.111:
/data/kvm-share 192.168.1.113,192.168.1.112
[root@KVM02 /] showmount -e 192.168.1.111
Export list for 192.168.1.111:
/data/kvm-share 192.168.1.113,192.168.1.112
[root@KVM02 /] showmount -e 192.168.1.114
Export list for 192.168.1.114:
/data/kvm-share 192.168.1.113,192.168.1.112

keepalived部署

  • 两台机器yum安装keepalived(NFS 和NFS2)
[root@NFS /] yum -y install keepalived
[root@NFS2 /] yum -y install keepalived
  • 更改NFS的配置文件并启动
[root@NFS ~] vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
router_id DRBD_MASTER
}

vrrp_script chk_nfs {
        script "/etc/keepalived/check_nfs.sh"
        interval 2
        weight -5
    }

vrrp_instance VI_1 {
        state BACKUP
        interface ens32
        virtual_router_id 51
        priority 100
        nopreempt
        advert_int 1
        authentication {
        auth_type PASS
        auth_pass 1111
        }
        track_script {
            chk_nfs
        }
    notify_master /etc/keepalived/notify_master.sh  //当此机器为keepalived的master角色时执行这个脚本
    virtual_ipaddress {
        192.168.1.100
    }
}
[root@NFS ~] vim /etc/keepalived/check_nfs.sh
#!/bin/sh
###检查nfs可用性:进程和是否能够挂载
/usr/bin/systemctl status nfs &>/dev/null
if [ $? -ne 0 ];then
        ###卸载drbd设备
        umount /dev/drbd0
        ###将drbd主降级为备
        drbdadm secondary r0
        #关闭keepalived
        /usr/bin/systemctl stop keepalived
fi
[root@NFS ~] chmod 755 /etc/keepalived/check_nfs.sh
[root@NFS ~] vim /etc/keepalived/notify_master.sh
#!bin/bash
/usr/sbin/drbdadm primary r0 &>/dev/null
/usr/bin/mount /dev/drbd0 /data  &>/dev/null
/usr/bin/systemctl restart nfs &>/dev/null`
[root@NFS ~] chmod 755 /etc/keepalived/notify_master.sh
[root@NFS ~] systemctl restart keepalived
查看vip
[root@NFS ~] ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.1.111/24 brd 192.168.1.255 scope global noprefixroute ens32
    inet 192.168.1.100/32 scope global ens32
  • 更改NFS2的配置文件
[root@NFS2 ~] vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived
global_defs {
router_id DRBD_MASTER
}
vrrp_script chk_nfs {
        script "/etc/keepalived/check_nfs.sh"
        interval 2
        weight -5
    }
vrrp_instance VI_1 {
        state BACKUP
        interface ens32
        virtual_router_id 51
        priority 99
        advert_int 1
        authentication {
        auth_type PASS
        auth_pass 1111
        }
        track_script {
            chk_nfs
        }
    notify_master /etc/keepalived/notify_master.sh     //当此机器为keepalived的master角色时执行这个脚本
    virtual_ipaddress {
        192.168.1.100
    }
}
[root@NFS2 ~] vim /etc/keepalived/check_nfs.sh
#!/bin/sh
###检查nfs可用性:进程和是否能够挂载
/usr/bin/systemctl status nfs &>/dev/null
if [ $? -ne 0 ];then
        ###卸载drbd设备
        umount /dev/drbd0
        ###将drbd主降级为备
        drbdadm secondary r0
        #关闭keepalived
        /usr/bin/systemctl stop keepalived
fi
[root@NFS2 ~] chmod 755 /etc/keepalived/check_nfs.sh
[root@NFS2 ~] vim /etc/keepalived/notify_master.sh
#!bin/bash
/usr/sbin/drbdadm primary r0 &>/dev/null
/usr/bin/mount /dev/drbd0 /data  &>/dev/null
/usr/bin/systemctl restart nfs &>/dev/null
[root@NFS2 ~] chmod 755 /etc/keepalived/notify_master.sh
[root@NFS2 ~] systemctl restart keepalived

验证nfs高可用

  • 模拟nfs服务停机
[root@NFS ~] systemctl stop nfs
[root@NFS ~] ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.1.111/24 brd 192.168.1.255 scope global noprefixroute ens32
//VIP不在了

[root@NFS ~] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:10501164 nr:6171 dw:22955 dr:10496179 al:16 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
//drbd变为Secondary/Primary节点
  • 查看NFS2(有了vip 变为primary/secondary)
[root@NFS2 ~] ip a | grep ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.1.114/24 brd 192.168.1.255 scope global noprefixroute ens32
    inet 192.168.1.100/32 scope global ens32

[root@NFS2 ~] cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:6171 nr:10501164 dw:10507335 dr:7581 al:10 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@NFS2 ~] ls /data/
aaaa  test1  test2  test3  test4  test5

KVM

Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第2张图片
两台KVM连接共享存储:双击localhost(qemu) 点击 存储 点击 加号

Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第3张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第4张图片

Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第5张图片
源主机:192.168.1.112 在共享存储上创建虚拟机

Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第6张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第7张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第8张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第9张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第10张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第11张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第12张图片Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第13张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第14张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第15张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第16张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第17张图片

过程比较慢,需要耐心等待。
KVM迁移:打开virt-manager 选择文件 选择Add Connection
源主机连接目标主机
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第18张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第19张图片

Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第20张图片

[root@KVM01 ~]# yum -y install openssh-askpass
[root@KVM02 ~]# yum -y install openssh-askpass
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第21张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第22张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第23张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第24张图片
[root@KVM01 ~]# virsh shutdown centos7.0
域 centos7.0 被关闭

[root@KVM01 ~]# virsh edit centos7.0

[root@KVM01 ~]# virsh start centos7.0
域 centos7.0 已开始

[root@KVM01 ~]# virsh list --all
Id 名称 状态
2 centos7.0 running

Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第25张图片
Linux运维昱笔记(十三):基于NFS+DRBD+Keepalived实现KVM虚拟主机动态迁移_第26张图片
迁移成功

你可能感兴趣的:(Linux)