运维自动化之svn+puppet实现监控系统的版本控制与自动部署监控系统
之前发布了一篇“运维自动化之使用PHP+MYSQL+SHELL打造私有监控系统”地址为：http://dl528888.blog.51cto.com/2382721/1034992大家给了我很多的建议，现在我在把我如何的对监控系统做版本控制与自动化部署来给大家描述一下，希望对大家有益。
在描述之前，先把流程图给大家，希望能使大家清楚流程：
整体流程如下：
1、运维人员在修改监控系统的时候，需要先登录svn服务器里修改监控的配置；
2、在svn里修改完监控的配置，选择更新的时候，会触发svn的hook里post-commit里的内容，根据脚本里的内容，会在puppet master服务器里的/usr/local/monitor/shell目录里留下最新svn的副本；
3、由于puppet client设置的连接puppet master的时间间隔为300秒，所以在300秒后能发现puppet client会从puppet master获得最新的配置，并更新自己本机的配置。
这样一个自动化的监控系统部署更新就完成了。
下面是是svn与puppet部分的详细说明
一、svn部分
安装svn主要是为了进行监控脚本的版本控制，如果不进行版本控制的话，比如我要修改甘肃web的配置的某个部分配置，过几天又修改了这个配置，在过2周呢？我还能记住之前修改什么了吗?我估计我是记不住了，而且我负责的平台过多，直接维护的服务器超过100+，平台更新的次数很频繁，而且根据用户的需求，修改的内容也不尽相同，所以进行版本控制是十分有必要了，废话不说，现在开始svn的配置。
svn服务端的安装我就不再说明，我之前已经发布了一篇“自动化建立svn服务端”
地址为：http://dl528888.blog.51cto.com/2382721/958090，大家可以使用这个来搭建svn的服务端，十分的方便，关于svn的客户端，我使用的是TortoiseSVN具体如何使用这类就不介绍了，大家可以参照 http://www.cnblogs.com/blsong/archive/2010/09/02/1816124.html来学习。
我的svn主要是为了配置我之前发布是“运维自动化之使用PHP+MYSQL+SHELL打造私有监控系统”这篇博文的shell脚本部分与下面的puppet配置部分，使用svn的hook（post-commit）功能，在svn更新的时候，就出发钩子，然后更新到我知道的目录，之后puppet会从这个里取得配置配置，在设置puppet的客户端连接服务端获取连接为300秒，也就是5分钟，这样的话，只有我一更新我的svn，那么puppet的客户端就会爱5分钟的时候，更新配置，减少人工操作，还能版本控制。
下面是我的svn里hook的post-commit的配置
REPOS="$1"
REV="$2"
export LANG=zh_CN.UTF-8
exec /usr/bin/svn update /usr/local/monitor/shell --username *** --password ** --no-auth-cache
echo "please check client config after in the 300s"|/bin/mail -s "svn has been update" [email protected]
现在展示一下我的svn界面与内容
在看看我的更新记录
这个svn由于是新建立的，所以更新的日志比较少。
二、puppet部分
为了安装最新的puppe，如果您想使用yum安装的话，需要在主机上添加epel库
- rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
如果此源失效，可以使用
- wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
- rpm -ivh epel-release-5-4.noarch.rpm
下面开始安装puppet服务端与客户端
1、在服务端
（1）使用yum安装puppet
- yum -y install puppet-server openssl*
（2）开启puppet服务端的服务
- service puppetmaster start
（3）添加到开机启动
- chkconfig --level 345 puppet on
- chkconfig --level 345 puppetmaster on
（4）在hosts里添加包含本身以及需要自动化的客户端机器主机名与ip地址
（5）接下来需要去客户端安装puppet、启动服务与进行认证了
2、在客户端
- yum -y install puppet openssl*
- service puppet start
- chkconfig --level 345 puppet on
然后在与服务端进行认证
- puppetd --server master --test
3、接下来在服务端进行允许认证
- puppetca -s -a
这样就完成了认证，之后就可以在puppet的client里进行获取服务端的配置，并自动的在本机更新了。
下面是我的配置，提供给大家参考（我已经贵州的puppet为例）
贵州的puppet结构为：
贵州备用为puppetmaster服务端
贵州web、贵州引擎、贵州引擎、贵州备用为客户端（主要贵州备用既为服务端与客户端是没有问题）
先看一下我的puppet服务端贵州备用的文件结构
- [root@beiyong /]# tree /etc/puppet/
- /etc/puppet/
- |-- auth.conf
- |-- fileserver.conf #puppet文件服务器配置文件
- |-- manifests #puppet主文件所在目录
- | |-- guizhou_beiyong.pp #该模块对应的文件资源，可能是要发送给slave的配置文件等
- | |-- guizhou_jiaohuan.pp
- | |-- guizhou_web.pp
- | |-- guizhou_yinqing.pp
- | `-- site.pp #puppet主文件（入口文件）
- `-- puppet.conf
- 1 directory, 8 files
客户端连接的顺序：site.pp=》*.pp
比如贵州web的接收顺序为：site.pp=》guizhou_web.pp
首先，slave向发起master连接请求，进行证书验证；
接着，证书验证通过后，master会直接找到入口文件manifests目录下的site.pp文件，该文件可能包含一些全局变量，参数缺省值（当各个模块没有设置这些参数时，它们的缺省值）以及其它pp文件的调用（在该例子中，会调用manifests下的各个pp文件）；
然后，master通过manifests下的各个pp文件定位到该slave要执行的模块（site.pp是各个模块的入口），汇总这些模块代码返回给slave；
最后，slave根据master发过来的manifest，配置信息。
注意：在配置之前一定要求的服务端与客户端的时间一致，并且hosts里都加入了对方的ip与主机名，并且能互相ping通，最好在改好hosts文件后重启服务器，同时selinux与iptables关闭。
4、先看一下我fileserver的设置
- [root@beiyong /]# cat /etc/puppet/fileserver.conf
- # This file consists of arbitrarily named sections/modules
- # defining where files are served from and to whom
- # Define a section 'files'
- # Adapt the allow/deny settings to your needs. Order
- # for allow/deny does not matter, allow always takes precedence
- # over deny
- # [files]
- # path /var/lib/puppet/files
- # allow *.example.com
- # deny *.evil.example.com
- # allow 192.168.0.0/24
- [puppet]
- path /usr/local/monitor/shell
- allow 172.16.6.0/24
这个path就是puppet master的配置文件里从哪里获得需要给客户端传输程序的地址，同时也是svn的hook里post-commit更新的地址。
allow后面的网段是我服务器网卡的ip段，也就是需要我这个网段的ip能从这个path里或者程序。
5、贵州web的配置
- [root@beiyong /]# cat /etc/puppet/manifests/guizhou_web.pp
- node web {
- file { "/usr/local/monitor/shell/GuiZhou_alter80.sh":
- source=>"puppet:///puppet/GuiZhou_web/GuiZhou_web_alter80.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_alter90.sh":
- source=>"puppet:///puppet/GuiZhou_web/GuiZhou_web_alter90.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_web_check.sh":
- source=>"puppet:///puppet/GuiZhou_web/GuiZhou_web_check.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/web_tomcat.sh":
- source=>"puppet:///puppet/GuiZhou_web/web_tomcat.sh",
- mode=>755,
- owner=>"lbs",
- group=>"lbs",
- }
- file { "/usr/local/monitor/shell/web_yz_tomcat.sh":
- source=>"puppet:///puppet/GuiZhou_web/web_yz_tomcat.sh",
- mode=>755,
- owner=>"lbs",
- group=>"lbs",
- }
- cron { GuiZhou_web_alert80_check:
- command=>"/usr/local/monitor/shell/GuiZhou_web_alter80.sh",
- hour=>'*/6',
- minute=>0,
- }
- cron { GuiZhou_web_alert90_check:
- command=>"/usr/local/monitor/shell/GuiZhou_web_alter90.sh",
- hour=>'*/1',
- minute=>0,
- }
- cron { GuiZhou_web_check:
- command=>"/usr/local/monitor/shell/GuiZhou_web_check.sh",
- minute=>'*/3',
- }
- cron { GuiZhou_web_sendmail_restart:
- command=>"/etc/init.d/sendmail restart",
- hour=>'23',
- minute=>'00',
- }
- }
6、贵州交换的配置
- [root@beiyong /]# cat /etc/puppet/manifests/guizhou_jiaohuan.pp
- node jiaohuan {
- file { "/usr/local/monitor/shell/GuiZhou_jiaohuan_alter80.sh":
- source=>"puppet:///puppet/GuiZhou_jiaohuan/GuiZhou_jiaohuan_alter80.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_jiaohuan_alter90.sh":
- source=>"puppet:///puppet/GuiZhou_jiaohuan/GuiZhou_jiaohuan_alter90.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_jiaohuan_check.sh":
- source=>"puppet:///puppet/GuiZhou_jiaohuan/GuiZhou_jiaohuan_check.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_jiaohuan_ping.sh":
- source=>"puppet:///puppet/GuiZhou_jiaohuan/GuiZhou_jiaohuan_ping.sh",
- mode=>755,
- }
- cron { GuiZhou_jiaohuan_alert80_check:
- command=>"/usr/local/monitor/shell/GuiZhou_jiaohuan_alter80.sh",
- hour=>'*/6',
- minute=>0,
- }
- cron { GuiZhou_jiaohuan_alert90_check:
- command=>"/usr/local/monitor/shell/GuiZhou_jiaohuan_alter90.sh",
- hour=>'*/1',
- minute=>0,
- }
- cron { GuiZhou_jiaohuan_check:
- command=>"/usr/local/monitor/shell/GuiZhou_jiaohuan_check.sh",
- minute=>'*/3',
- }
- cron { GuiZhou_jiaohuan_ping:
- command=>"/usr/local/monitor/shell/GuiZhou_jiaohuan_ping.sh",
- minute=>'*/15',
- }
- cron { GuiZhou_jiaohuan_sendmail_restart:
- command=>"/etc/init.d/sendmail restart",
- hour=>'23',
- minute=>'00',
- }
- }
7、贵州引擎的配置
- [root@beiyong /]# cat /etc/puppet/manifests/guizhou_yinqing.pp
- node savecenter {
- file { "/usr/local/monitor/shell/GuiZhou_yinqing_alter80.sh":
- source=>"puppet:///puppet/GuiZhou_yinqing/GuiZhou_yinqing_alter80.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_yinqing_alter90.sh":
- source=>"puppet:///puppet/GuiZhou_yinqing/GuiZhou_yinqing_alter90.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_yinqing_check.sh":
- source=>"puppet:///puppet/GuiZhou_yinqing/GuiZhou_yinqing_check.sh",
- mode=>755,
- owner=>lbs,
- group=>lbs,
- }
- cron { GuiZhou_yinqing_alert80_check:
- command=>"/usr/local/monitor/shell/GuiZhou_yinqing_alter80.sh",
- hour=>'*/6',
- minute=>0,
- }
- cron { GuiZhou_yinqing_alert90_check:
- command=>"/usr/local/monitor/shell/GuiZhou_yinqing_alter90.sh",
- hour=>'*/1',
- minute=>0,
- }
- cron { GuiZhou_yinqing_check:
- command=>"/usr/local/monitor/shell/GuiZhou_yinqing_check.sh",
- user=>lbs,
- minute=>'*/3',
- }
- cron { GuiZhou_yinqing_sendmail_restart:
- command=>"/etc/init.d/sendmail restart",
- hour=>'23',
- minute=>'00',
- }
- }
8、贵州备用的配置
- [root@beiyong /]# cat /etc/puppet/manifests/guizhou_beiyong.pp
- node beiyong {
- file { "/usr/local/monitor/shell/GuiZhou_beiyong_alter80.sh":
- source=>"puppet:///puppet/GuiZhou_beiyong/GuiZhou_beiyong_alter80.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_beiyong_alter90.sh":
- source=>"puppet:///puppet/GuiZhou_beiyong/GuiZhou_beiyong_alter90.sh",
- mode=>755,
- }
- file { "/usr/local/monitor/shell/GuiZhou_beiyong_check.sh":
- source=>"puppet:///puppet/GuiZhou_beiyong/GuiZhou_beiyong_check.sh",
- mode=>755,
- }
- cron { GuiZhou_beiyong_alert80_check:
- command=>"/usr/local/monitor/shell/GuiZhou_beiyong_alter80.sh",
- hour=>'*/6',
- minute=>0,
- }
- cron { GuiZhou_beiyong_alert90_check:
- command=>"/usr/local/monitor/shell/GuiZhou_beiyong_alter90.sh",
- hour=>'*/1',
- minute=>0,
- }
- cron { GuiZhou_beiyong_check:
- command=>"/usr/local/monitor/shell/GuiZhou_beiyong_check.sh",
- minute=>'*/3',
- }
- cron { GuiZhou_beiyong_sendmail_restart:
- command=>"/etc/init.d/sendmail restart",
- hour=>'23',
- minute=>'00',
- }
- }
9、在查看一个site.pp的配置
- [root@beiyong /]# cat /etc/puppet/manifests/site.pp
- import "guizhou_web.pp"
- import "guizhou_jiaohuan.pp"
- import "guizhou_yinqing.pp"
- import "guizhou_beiyong.pp"
现在服务端的配置已经展现完成，在去客户端查看一下配置是否生效
1、贵州web
- [root@web puppet]# crontab -l
- # HEADER: This file was autogenerated at Fri Oct 12 15:59:14 +0800 2012 by puppet.
- # HEADER: While it can still be managed manually, it is definitely not recommended.
- # HEADER: Note particularly that the comments starting with 'Puppet Name' should
- # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
- # Puppet Name: GuiZhou_web_sendmail_restart
- 0 23 * * * /etc/init.d/sendmail restart
- # Puppet Name: GuiZhou_web_alert90_check
- 0 */1 * * * /usr/local/monitor/shell/GuiZhou_web_alter90.sh
- # Puppet Name: GuiZhou_web_alert80_check
- #0 */6 * * * /usr/local/monitor/shell/GuiZhou_web_alter80.sh
- # Puppet Name: GuiZhou_web_check
- */3 * * * * /usr/local/monitor/shell/GuiZhou_web_check.sh
2、贵州交换
- [root@jiaohuan ~]# crontab -l
- # HEADER: This file was autogenerated at Fri Oct 12 16:03:24 +0800 2012 by puppet.
- # HEADER: While it can still be managed manually, it is definitely not recommended.
- # HEADER: Note particularly that the comments starting with 'Puppet Name' should
- # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
- # Puppet Name: GuiZhou_jiaohuan_sendmail_restart
- 0 23 * * * /etc/init.d/sendmail restart
- # Puppet Name: GuiZhou_jiaohuan_alert80_check
- 0 */6 * * * /usr/local/monitor/shell/GuiZhou_jiaohuan_alter80.sh
- # Puppet Name: GuiZhou_jiaohuan_alert90_check
- 0 */1 * * * /usr/local/monitor/shell/GuiZhou_jiaohuan_alter90.sh
- # Puppet Name: GuiZhou_jiaohuan_ping
- #*/15 * * * * /usr/local/monitor/shell/GuiZhou_jiaohuan_ping.sh
- # Puppet Name: GuiZhou_jiaohuan_check
- */3 * * * * /usr/local/monitor/shell/GuiZhou_jiaohuan_check.sh
3、贵州引擎
由于贵州引擎有root账号与lbs用户的配置，所以我们先查看root用户的
- [root@savecenter ~]# crontab -l
- # HEADER: This file was autogenerated at Fri Oct 12 16:16:41 +0800 2012 by puppet.
- # HEADER: While it can still be managed manually, it is definitely not recommended.
- # HEADER: Note particularly that the comments starting with 'Puppet Name' should
- # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
- # Puppet Name: GuiZhou_yinqing_sendmail_restart
- 0 23 * * * /etc/init.d/sendmail restart
- # Puppet Name: GuiZhou_yinqing_alert80_check
- 0 */6 * * * /usr/local/monitor/shell/GuiZhou_yinqing_alter80.sh
- # Puppet Name: GuiZhou_yinqing_alert90_check
- 0 */1 * * * /usr/local/monitor/shell/GuiZhou_yinqing_alter90.sh
在查看lbs用户的
- [root@savecenter ~]# crontab -u lbs -l
- # HEADER: This file was autogenerated at Fri Oct 12 16:16:41 +0800 2012 by puppet.
- # HEADER: While it can still be managed manually, it is definitely not recommended.
- # HEADER: Note particularly that the comments starting with 'Puppet Name' should
- # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
- #MIN HOUR DAY MONTH DAYOFWEEK COMMAND
- # Puppet Name: GuiZhou_yinqing_check
- */3 * * * * /usr/local/monitor/shell/GuiZhou_yinqing_check.sh
4、贵州备用
- [root@beiyong /]# crontab -l
- # HEADER: This file was autogenerated at Fri Oct 12 16:03:37 +0800 2012 by puppet.
- # HEADER: While it can still be managed manually, it is definitely not recommended.
- # HEADER: Note particularly that the comments starting with 'Puppet Name' should
- # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
- # Puppet Name: GuiZhou_beiyong_check
- */3 * * * * /usr/local/monitor/shell/GuiZhou_beiyong_check.sh
- # Puppet Name: GuiZhou_beiyong_sendmail_restart
- 0 23 * * * /etc/init.d/sendmail restart
- # Puppet Name: GuiZhou_beiyong_alert90_check
- 0 */1 * * * /usr/local/monitor/shell/GuiZhou_beiyong_alter90.sh
- # Puppet Name: GuiZhou_beiyong_alert80_check
- 0 */6 * * * /usr/local/monitor/shell/GuiZhou_beiyong_alter80.sh
现在puppet的配置已经完成，当前的配置已经在我们的生产服务器里部署。
如果大家想要在客户端的puppe设置一个时间来连接puppetmaster获取文件？
可以再客户端里的/etc/puppet/puppet.conf文件里加上：
- runinterval =1800
默认是30分钟，可以修改此处的值，单位为秒。
修改后需要重启服务。
需要注意的：
1、在客户端申请证书的时候，建议把客户端与服务器的时间与时区调整为一样的，可以使用ntpdate 210.72.145.44来更新；
2、在/etc/resolv.conf里，一定要把#search localdomain这一行给注销，否则在申请证书的时候，会带有.localdomain，在证书申请完成，服务端给予授权的时候，你在获得资源的时候，会出现hostname not match with the server certificate错误。
在hosts里添加包含本身以及需要自动化的服务端机器主机名与ip地址
3、大家可能发现我写的puppet里的资源配置非常的简单，没有那么花哨、各个模块依赖的，对于这个我的解释是：简单的配置大家都能看懂，运行的时候如果出现问题，那么我们排错的话，也很简单，只有能够顺利的运行，就是一个好的资源配置。
4、主机名使用，建议使用FQDN方式
5、每次生成证书的id均不相同。
由于本文的脚本内容过多，所以关于puppet的安装与运行过程的问题FAQ在下一页，
名字为：
运维自动化之puppet自动化部署的FAQ
地址为： http://dl528888.blog.51cto.com/2382721/1040569
BTW:如果大家认为我写的不错，希望能给我的博客投个票，谢谢！
http://blog.51cto.com/contest2012/2382721