centos 7 corosync + pacemaker 搭建（pcs）

centos 7 corosync + pacemaker 搭建（pcs）

node1：co1     11.100.46.4

node2：co2     11.100.46.7

node3：co3    11.100.46.9

webservice：

vip：11.100.46.11 ofc：heartbeat：Ipaddr

web：systemd

 

一、创建系统集成环境

1、建立无密码登录以及名称解析

# ssh-keygen

# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

# echo ‘11.100.46.4 co1’ >> /etc/hosts

# echo ‘11.100.46.7 co2’ >> /etc/hosts

# echo ‘11.100.46.9 co3’ >> /etc/hosts

# scp /etc/hosts [email protected]:/etc/

# scp /etc/hosts [email protected]:/etc/

 

2、关闭防火墙、sellinux以及修改hostname

co1:

# systemctl stop firewalld.service

# systemctl disabled firewalld

# mv -v /etc/selinux/config{,.bak}

# cat /etc/selinux/config.bak | sed ‘s/SELINUX=enforcing/SELINUX=disabled/’ > /etc/selinux/config

# setenforce Permissive

# getenforce

# yum install iptables-services

# iptables -F

# iptables -X

# iptables -L -n

# iptables-save > /etc/sysconfig/iptables

# systemctl restart iptables

# echo ‘co1’ > /etc/hostname

# hostname co1

 

 

co3 bash:

systemctl stop firewalld.service

systemctl disable firewalld

mv -v /etc/selinux/config{,.bak}

cat /etc/selinux/config.bak | sed ‘s/SELINUX=enforcing/SELINUX=disabled/’ > /etc/selinux/config

setenforce Permissive

getenforce

yum -y install iptables-services

iptables -F

iptables -X

iptables -L -n

iptables-save > /etc/sysconfig/iptables

systemctl restart iptables

echo ‘co3’ > /etc/hostname

hostname co3

 

3、创建时间同步

# yum -y install ntp

# mv /etc/ntp.conf{,.bak} -v

# cat /etc/ntp.conf.bak | sed ‘/^server.*$/d’ | sed ‘/^#broadcast 192.168.1.255/i server 172.16.31.125’ > /etc/ntp.conf

# echo ‘server 172.16.31.125’ >> /etc/ntp/step-tickers

# scp /etc/ntp.conf root@co2:/etc/;scp /etc/ntp.conf root@co3:/etc/

# scp /etc/ntp/step-tickers root@co2:/etc/ntp/;scp /etc/ntp/step-tickers root@co3:/etc/ntp

# ssh co2 ‘yum -y install ntp’;ssh co3 ‘yum -y install ntp’

# systemctl start ntpd;ssh co2 ‘systemctl start ntpd’;ssh co3 ‘systemctl start ntpd’;

# date;ssh co2 date;ssh co3 date

 

二、配置nfs-server与apache

1、安装nfs

# yum -y install rpcbind nfs-utils

# systemctl enable rpcbind

# systemctl start rpcbind

# rpcinfo

# mkdir /htdocs/www -pv

# groupadd -r -g 888 apache

# useradd -M -u 888 -g 888 -s /sbin/nologin -r apache

# chmod -R u=rwx,g=rx,o=rx /htdocs/www/

# chown -R apache:apache /htdocs/www/

# echo ‘/htdocs/www 11.100.46.7(rw,all_squash,anonuid=888,anongid=888) 11.100.46.4(rw,all_squash,anonuid=888,anongid=888) 11.100.46.9(rw,all_squash,anonuid=888,anongid=888)’ > /etc/exports

# service nfs start

# exportfs -ar

 

2、测试挂载：

# showmount -e

Export list for localhost.localdomain:

/htdocs/www 11.100.46.9,11.100.46.4,11.100.45.7

# mkdir -pv /htdocs/www;ssh co2 ‘mkdir -pv /htdocs/www’;ssh co3 ‘mkdir -pv /htdocs/www’

# yum -y install nfs-utils;ssh co2 ‘yum -y install nfs-utils’;ssh co3 ‘yum -y install nfs-utils’

# mount -t nfs 11.100.46.13:/htdocs/www /htdocs/www/;ssh co2 ‘mount -t nfs 11.100.46.13:/htdocs/www /htdocs/www/’; ssh co3 ‘mount -t nfs 11.100.46.13:/htdocs/www /htdocs/www/’

# ls /htdocs/www/; ssh co2 ‘ls /htdocs/www/’; ssh co3 ‘ls /htdocs/www/’

 

3、安装httpd

# echo ‘this is NFS-server on 11.100.46.13’ > /htdocs/www/index.html

# yum -y install httpd; ssh co2 ‘yum -y install httpd’; ssh co3 ‘yum -y install httpd’

# mv -v /etc/httpd/conf/httpd.conf{,.bak}

# systemctl start httpd;ssh co2 ‘systemctl start httpd’; ssh co3 ‘systemctl start httpd’

# cat /etc/httpd/conf/httpd.conf.bak | sed ‘s/Directory \”\/var\/www\”/Directory \”\/htdocs\/www\”/’| sed ‘s/DocumentRoot \”\/var\/www\/html\”/DocumentRoot \”\/htdocs\/www\/\”/g’ > /etc/httpd/conf/httpd.conf

# scp /etc/httpd/conf/httpd.conf root@co2:/etc/httpd/conf/httpd.conf; scp /etc/httpd/conf/httpd.conf root@co3:/etc/httpd/conf/httpd.conf

# systemctl restart httpd;ssh co2 ‘systemctl restart httpd’; ssh co3 ‘systemctl restart httpd’

 

4、取消挂载关闭httpd

# umount 11.100.46.13:/htdocs/www/;ssh co2 ‘umount 11.100.46.13:/htdocs/www/’; ssh co3 ‘umount 11.100.46.13:/htdocs/www/’

# systemctl stop httpd;ssh co2 ‘systemctl stop httpd’; ssh co3 ‘systemctl stop httpd’

# ls /htdocs/www/; ssh co2 ‘ls /htdocs/www/’; ssh co3 ‘ls /htdocs/www/’

 

三、安装及启动集群

 

1、安装并启动pcsd

# yum -y install pcs; ssh co2 ‘yum -y install pcs’; ssh co3 ‘yum -y install pcs’

# systemctl start pcsd; ssh co2 ‘systemctl start pcsd’; ssh co3 ‘systemctl start pcsd’

# systemctl enable pcsd; ssh co2 ‘systemctl enable pcsd’; ssh co3 ‘systemctl enable pcsd’

# echo “cisco” | passwd –stdin hacluster;ssh co2 ‘echo “cisco” | passwd –stdin hacluster’;ssh co3 ‘echo “cisco” | passwd –stdin hacluster’

 

2、配置corosysnc

# pcs cluster auth co1 co2 co3 -u hacluster -p cisco

co1: Authorized

co2: Authorized

co3: Authorized

 

# pcs cluster setup –name cpe co1 co2

Destroying cluster on nodes: co1, co2, co3…

co1: Stopping Cluster (pacemaker)…

co2: Stopping Cluster (pacemaker)…

co3: Stopping Cluster (pacemaker)…

co3: Successfully destroyed cluster

co2: Successfully destroyed cluster

co1: Successfully destroyed cluster

 

Sending cluster config files to the nodes…

co1: Succeeded

co2: Succeeded

co3: Succeeded

 

Synchronizing pcsd certificates on nodes co1, co2, co3…

co1: Succeeded

co2: Succeeded

co3: Succeeded

 

Restarting pcsd on the nodes in order to reload the certificates…

co1: Success

co2: Success

co3: Success

 

3、启动集群

在co1、co2、co3上任意运行

# pcs cluster start –all

co1: Starting Cluster…

co2: Starting Cluster…

co3: Starting Cluster…

 

上面的命令相当于：

# systemctl start corosync

# systemctl start pacemaker

 

4、检查集群状态

# corosync-cfgtool -s //no faults 为正常

Printing ring status.

Local node ID 1

RING ID 0

    id    = 11.100.46.4

    status    = ring 0 active with no faults

 

# corosync-cmapctl | grep members //查看各节点是否正常

runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(11.100.46.4)

runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.1.status (str) = joined

runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(11.100.46.7)

runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.2.status (str) = joined

runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0

runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(11.100.46.9)

runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1

runtime.totem.pg.mrp.srp.members.3.status (str) = joined

 

# pcs status //查看集群状态

Cluster name: mycluster

Stack: corosync

Current DC: co1 (version 1.1.15-11.el7-e174ec8) – partition with quorum

Last updated: Fri May 19 17:45:08 2017        Last change: Thu May 18 01:08:45 2017 by root via cibadmin on co1

 

3 nodes and 5 resources configured

 

Online: [ co1 co2 co3 ]

 

Full list of resources:

 

webip    (ocf::heartbeat:IPaddr2):    Started co1

web_store    (ocf::heartbeat:Filesystem):    Started co1

web_service    (systemd:httpd):    Started co1

mysql    (systemd:mysqld):    Started co2

mysql_store    (ocf::heartbeat:Filesystem):    Started co2

 

Daemon Status:

corosync: active/disabled

pacemaker: active/disabled

pcsd: active/disabled

 

# crm_verify -L -V //检查集群配置，无输出为正常

 

四、配置资源

 

1、获取帮助

# pcs resource –help

# pcs resource describe ocf:heartbeat:IPaddr //查看资源使用方式

# pcs resource list LSB //查看支持的资源

# pcs resource list ocf:heartbeat //查看资源描述

 

2、创建资源

# pcs resource create webip ocf:heartbeat:IPaddr ip=”11.100.46.11″ op monitor interval=20s timeout=10s

# pcs resource create webstore ocf:heartbeat:Filesystem device=”11.100.46.13:/htdocs/www” directory=”/htdocs/www” fstype=”nfs” op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=40s

# pcs resource create webserver systemd:httpd op monitor interval=30s timeout=20s

 

3、定义资源启动先后

# pcs resource add webservice webip webstore –before webserver

 

4、定义webip、weistort、webserver 组主资源

# pcs resource group add webservice webip webstore webserver

 

5、定义资源位置约束性

# pcs constraint location add webservice_pref_co1 webservice co1 100

# pcs constraint location add webservice_pref_co1 webservice co2 101

# pcs constraint location add webservice_pref_co1 webservice co1 INFINITY

 

6、定义资源粘性

# pcs property set default-resource-stickiness=15

 

7、查看资源位置约束性

# pcs constraint location show

Location Constraints:

Resource: web_service

Enabled on: co1 (score:100)

Enabled on: co3 (score:90)

Resource: web_store

Enabled on: co1 (score:100)

Enabled on: co3 (score:90)

Resource: webip

Enabled on: co1 (score:100)

Enabled on: co3 (score:90)

Enabled on: co2 (score:100)

Enabled on: co2 (score:100)

Enabled on: co3 (score:90)

Enabled on: co3 (score:90)

 

8、资源上下线

# pcs cluster standby co2

# pcs cluster unstandby co2

 

9、查看主要参数

# pcs property list –all

 

10、管理资源启动停止

# pcs resource show webip

Resource: webip (class=ocf provider=heartbeat type=IPaddr2)

Attributes: ip=11.100.46.11

Meta Attrs: target-role=Started

Operations: monitor interval=10s timeout=20s (webip-monitor-10s)

# pcs resource delete webip

 

# pcs status

Cluster name: mycluster

Stack: corosync

Current DC: co1 (version 1.1.15-11.el7-e174ec8) – partition with quorum

Last updated: Fri May 19 18:03:46 2017        Last change: Thu May 18 01:08:45 2017 by root via cibadmin on co1

 

3 nodes and 5 resources configured

 

Online: [ co1 co2 co3 ]

 

Full list of resources:

 

webip    (ocf::heartbeat:IPaddr2):    Started co1

web_store    (ocf::heartbeat:Filesystem):    Started co1

web_service    (systemd:httpd):    Started co1

mysql    (systemd:mysqld):    Started co2

mysql_store    (ocf::heartbeat:Filesystem):    Started co2

 

Daemon Status:

corosync: active/disabled

pacemaker: active/disabled

pcsd: active/disabled

 

11、监控crm

# crm_mon

Stack: corosync

Current DC: co1 (version 1.1.15-11.el7-e174ec8) – partition with quorum

Last updated: Fri May 19 18:04:31 2017 Last change: Thu May 18 01:08:45 2017 by root via cibadmin on co1

 

3 nodes and 5 resources configured

 

Online: [ co1 co2 co3 ]

 

Active resources:

 

webip (ocf::heartbeat:IPaddr2):    Started co1

web_store    (ocf::heartbeat:Filesystem): Started co1

web_service (systemd:httpd): Started co1

mysql (systemd:mysqld):    Started co2

mysql_store (ocf::heartbeat:Filesystem): Started co2