rhcs+drbd+postgresql实战

一、环境说明：

测试环境使用虚拟机2台，OS：centos6.4。内核：2.6.32-358.el6.x86_64

系统安装采用最小化安装。

主机名和IP    

hostname	ip address	remarks
pg1-rg	172.26.184.38	primary
pg2-rg	172.26.184.39	secondary
	172.26.184.42	VIP

Disk

2台server都是2块磁盘。xvda做操作系统使用，xvdb和xvdc分区/dev/xvdb1、/dev/xvdc1做drbd使用。

Disk	partition	remarks
xvda	/dev/xvda1	boot
xvda	/dev/xvda2	os
xvdb	/dev/xvdb1	drbd
xvdc	/dev/xvdc1	drbd（扩容）

二、系统初始化

1. 关闭selinux

   vi /etc/sysconfig/selinux

   找到行 SELINUX=enforcing

   修改为 SELINUX=disabled

2. 关闭iptables

   chkconfig ipables off

   service iptables stop

3. 设置主机名解析

   vi /etc/hosts

   文件最后加入以下：

   172.26.184.38   pg1-rg
   172.26.184.39   pg2-rg
   

4. 配置安装源--这里配置的是本地yum repository

   vi  /etc/yum.repos.d/custom.repo

   [centos]
   name=centos6.4iso
   baseurl=http://172.26.184.241/centos/6.4/os/x86_64
   enabled=1
   gpgcheck=0

   [epel]
   name=epel
   baseurl=http://172.26.184.241/epel/6/x86_64/
   enabled=1
   gpgcheck=0

   [postgresql91]
   name=postgresql91
   baseurl=http://172.26.184.241/postgresql91
   enabled=1
   gpgcheck=0

三、DRBD安装配置（2台主机)

1. 安装依赖软件包

   yum install kernel-devel kernel-headers flex wget make gcc glibc perl -y

2. 下载最新drbd源码包，并解压

   cd /tmp

   wget http://oss.linbit.com/drbd/drbd-latest.tar.gz

   tar -zxvf drbd-latest.tar.gz -C /usr/local/src/

3. 编译安装drbd

   cd /usr/local/src/drbd-8.4.4
   ./configure --prefix=/opt/drbd --localstatedir=/var --sysconfdir=/etc --with-km --with-rgmanager

   make && make install

4. 安装drbd模块

   cd /usr/local/src/drbd-8.4.4/drbd

   make clean
   make
   make install
   cp drbd.ko /lib/modules/`uname -r`/kernel/lib/
   modprobe drbd
   lsmod | grep drbd     #如果看到以下信息，说明内核加载drbd模块成功。

      drbd                  340679  8
      libcrc32c               1246  1 drbd

5. 配置drbd

   cd /etc/drbd.d

   全局配置
   

   vi global_common.conf

   global {
           usage-count yes;
           # minor-count dialog-refresh disable-ip-verification
   }
   common {
           syncer { rate 100M; }
           handlers {
                   # These are EXAMPLE handlers only.
                   # They may have severe implications,
                   # like hard resetting the node under certain circumstances.
                   # Be careful when chosing your poison.
                   # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                   # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                   # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                   # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                   # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                   # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                   # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                   # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
           }
           startup {
                   # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
                   degr-wfc-timeout 120;    # 2 minutes.
           }
           options {
                   # cpu-mask on-no-data-accessible
           }
           disk {
                   # size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
                   # disk-drain md-flushes resync-rate resync-after al-extents
                   # c-plan-ahead c-delay-target c-fill-target c-max-rate
                   # c-min-rate disk-timeout
           }
           net {
                   # protocol timeout max-epoch-size max-buffers unplug-watermark
                   # connect-int ping-int sndbuf-size rcvbuf-size ko-count
                   # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
                   # after-sb-1pri after-sb-2pri always-asbp rr-conflict
                   # ping-timeout data-integrity-alg tcp-cork on-congestion
                   # congestion-fill congestion-extents csums-alg verify-alg
                   # use-rle
                   cram-hmac-alg "sha1";
                   shared-secret "FooFunFactory";
                   after-sb-0pri disconnect;
                   after-sb-1pri disconnect;
                   after-sb-2pri disconnect;
                   rr-conflict disconnect;
           }
   }

6. 资源配置
   

   vi drbd.res

   resource r0 {
     protocol C;
     device    /dev/drbd0;
     disk      /dev/xvdb1;
     meta-disk internal;
     on pg1-rg {
       address    172.26.184.38:7788;
     }
     on pg2-rg {
       address    172.26.184.39:7788;
     }
   }
   resource r1 {
     protocol C;
     device    /dev/drbd1;
     disk      /dev/xvdc1;
     meta-disk internal;
     on pg1-rg {
       address    172.26.184.38:7789;
     }
     on pg2-rg {
       address    172.26.184.39:7789;
     }
   }

7. 创建节点启动drbd服务
   

   drbdadm create-md r0
   chkconfig --add drbd
   service drbd start

8. drbd默认 2个都是secondary，所以在主节点设置为primary

   drbdadm -- --overwrite-data-of-peer primary all  （第一次设置主节点）
   drbdadm primary all （以后设置节点为主节点）

9. drbd常用命令

   查看资源的连接状态
   drbdadm cstate all

   查看节点状态

   drbdadm role all
   

   查看同步进度

   cat /proc/drbd     或者  /opt/drbd/sbin/drbd-overview

10. #############报错信息#############
    报错1:
    configure: error: Cannot build utils without flex, either install flex or pass the --without-utils option.
    解决办法:  
    yum  -y  intalls  flex
    报错2：
    make:  [check-kdir] error 1
    解决办法：yum install -y kernel-devel
    报错3：
    SORRY, kernel makefile not found.
       You need to tell me a correct KDIR,
    　　Or install the neccessary kernel source packages.
    报错4：'drbd' not defined in your config (for this host).
    这个因为没有加载到drbd模块
    启动报错：
    Command 'drbdmeta 0 v08 /dev/sdb1 internal create-md' terminated with exit code 40
    这是因为sdb1已经有文件系统了，已经有数据存在了
    解决方法：
    [root@node1 ~]# dd if=/dev/zero bs=1M count=1 of=/dev/sdb1
    请检查这个KDIR目录指的路径是否正确
    ##########################
    

四、postgresql安装（以下在主机操作，备机仅需安装postgresql）

1. 准备数据目录（格式化drbd设备）

   pvcreate /dev/drbd0

   vgcreate drbd /dev/drbd0
   lvcreate -l +100%FREE -n pgsql drbd
   mkfs.ext4 /dev/drbd/pgsql
   mkdir -p /var/lib/pgsql/9.1/data
   mount /dev/drbd/pgsql /var/lib/pgsql/9.1/data

2. 安装postgresql（本文主要介绍psql的高可用性，psql配置略过）

   yum install postgresql91-server -y
   chown -R postgres.postgres /var/lib/pgsql/
   chmod -R 700 /var/lib/pgsql/
   service postgresql-9.1 initdb
   service postgresql-9.1 start

五、RHCS套件安装配置

1. yum groupinstall "High Availability" -y

2. 编辑配置文件

   vi /etc/cluster/cluster.conf

   
   

3. cluster.conf配置中，在切换DRBD时需要自己编写故障切换脚本，详见附件。

   vi /etc/init.d/drbd_cluster.sh    #内容见附件

   chmod +x /etc/init.d/drbd_cluster.sh
   

4. 启动cluster

   service cman start

   service rgmanager start
   

5. 检查点状态
   

   clustat -l
   

6. 模拟故障切换

   
   

   在主机（pg1-rg）上确认状态。

   df

   

   
   

   
   

   检查postgresql服务状态

   

   在pg1-rg上停止postgresql-9.1服务

   service postgresql-9.1 stop

   一段时间后，pg2-rg自动接管服务。