Puppetmaster高可用和可扩展的方案设计

Puppet是当前devops中常用于管理系统配置和应用部署，多数会使用其C/S架构的方式来进行部署，其中puppetmaster是集群中配置管理的核心节点。在实际的生产环境中，如果因为master节点性能不够或者发生意外宕机，可能会影响到实际业务，因此维护一个高可用和可扩展的puppetmaster池子是一个首要任务。

这里我使用了一种常规的方案：前端使用apache/nginx做负载均衡，使用packmaker/keepalived来做健康检查和故障切换，来做HA，后端起多个puppetmaster实例做横向扩展，来提高处理能力。

方案验证

这里，我将在在每台Master Node上起两个puppetmaster实例，前端使用Apache作负载均衡，keepalived做健康检查。唯一的难点是证书同步问题，在部署中将把我们将证书设成自动认证，只接受fqdn是*.clustername.ustack.com的机器，就不需同步证书了。

IP	主机名	角色	vip
192.168.1.53	ha-puppet1.ustack.com	puppet master	192.168.1.103
192.168.1.54	ha-puppet2.ustack.com	puppet master	192.168.1.104

配置细节

客户端 配置文件

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
pluginsync=true

[agent]
server = ha-puppet.ustack.com
report = true
pluginsync = true
listen = true
runinterval = 300

Master node的配置选项

[master]
autosign       = $confdir/autosign.conf { mode = 664 }

autosign.conf

*.ustack.com

服务器端配置信息

192.168.2.53

loadbalancer配置

apache的proxy监听在8140端口，后面可以配置多个puppetmaster进程

<Proxy balancer://puppetmaster>
  BalancerMember http://127.0.0.1:18140
  BalancerMember http://127.0.0.1:18141
</Proxy>

Listen 8140
<VirtualHost *:8140>

	SSLEngine on
	SSLProtocol -ALL +SSLv3 +TLSv1
	SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

	SSLCertificateFile      /var/lib/puppet/ssl/certs/ha-puppet.ustack.com.pem
	SSLCertificateKeyFile   /var/lib/puppet/ssl/private_keys/ha-puppet.ustack.com.pem
	SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
	SSLCACertificateFile    /var/lib/puppet/ssl/ca/ca_crt.pem
	# CRL checking should be enabled; if you have problems with Apache complaining about the CRL, disable the next line
	SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
	SSLVerifyClient optional
	SSLVerifyDepth  1
	SSLOptions +StdEnvVars +ExportCertData

	# The following client headers allow the same configuration to work with Pound.
	RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
	RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
	RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
	RequestHeader unset X-Forwarded-For

	# you probably want to tune these settings
	PassengerHighPerformance on
	PassengerMaxPoolSize 12
	PassengerPoolIdleTime 1500
	# PassengerMaxRequests 1000
	PassengerStatThrottleRate 120

	RackAutoDetect On
	DocumentRoot /etc/puppet/rack/public/
	<Directory /etc/puppet/rack>
		Options None
		AllowOverride None
		Order allow,deny
		allow from all
	</Directory>
	<Location />
	  SetHandler balancer-manager
	  Order allow,deny
	  Allow from all
	</Location>

	ProxyPass / balancer://puppetmaster/
	ProxyPassReverse / balancer://puppetmaster/
	ProxyPreserveHost On

	CustomLog /var/log/httpd/balance-8140-access.log combined
	ErrorLog /var/log/httpd/balance-8140-error.log
	CustomLog /var/log/httpd/balancer_ssl_requests.log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

第一个puppetmaster的vhost配置文件，puppetmaster实例的数量可以水平扩展：

Listen 18140
<VirtualHost *:18140>

        SSLEngine off

        # The following client headers allow the same configuration to work with Pound.
        SetEnvIf set X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
        SetEnvIf set X-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

        # you probably want to tune these settings
        PassengerHighPerformance on
        PassengerMaxPoolSize 12
        PassengerPoolIdleTime 1500
        # PassengerMaxRequests 1000
        PassengerStatThrottleRate 120

        RackAutoDetect On
        DocumentRoot /etc/puppet/rack/18140/public/
        <Directory /etc/puppet/rack/18140/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
        CustomLog /var/log/httpd/puppetmaster-18140-access.log combined
        ErrorLog /var/log/httpd/puppetmaster-18140-error.log
</VirtualHost>

rack配置文件

拷贝rack配置文件给第一个puppetmaster：

rsync -avxH /etc/puppet/rack/{,18140}/

Keepalived配置文件

! Configuration File for keepalived
global_defs {
   notification_email {
     [email protected]

   }
   notification_email_from [email protected]
   smtp_server localhost
   smtp_connect_timeout 30
   router_id route-45
}

vrrp_script chk_http_port {
        script "/usr/bin/killall -0 httpd"
        interval 2
        weight 2
}

vrrp_instance 45 {
    virtual_router_id 45


    priority 100
    state BACKUP

    interface eth0

    virtual_ipaddress {
        192.168.2.103
    }

        track_script {
            chk_http_port
        }
}

vrrp_instance 46 {
    virtual_router_id 46


    priority 101
    state MASTER

    interface eth0

    virtual_ipaddress {
        192.168.2.104
    }

        track_script {
            chk_http_port
        }
}

192.168.2.54

apache的配置文件和53相同，唯一区别就是keepalived的配置文件上的IP地址互为热备，在此就不再赘述。

验证

在dnspod上绑定192.168.1.103/4到ha-puppet.ustack.com。

并在ha-puppet1上使用以下manifests文件：

node /default/ {

        notify {'Hello,I am Master 1':}
}

node 'nginx.novalocal' inherits default{}

在ha-puppet2上使用以下manifests文件：

node /default/ {

        notify {'Hello,I am Master 2':}
}

node 'nginx.novalocal' inherits default{}

测试结果，前两次是：

notice: Hello,I am Master 2
notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 2]/message: defined 'message' as 'Hello,I am Master 2'
notice: Finished catalog run in 0.12 seconds

随后出现：

notice: Hello,I am Master 1
notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 1]/message: defined 'message' as 'Hello,I am Master 1'
notice: Finished catalog run in 0.10 seconds

结论

以上方案验证通过，手动部署很简单，难点在于把它们设计成puppet module来进行部署时，需要考虑到所有的服务器需要使用同一个证书，因此需要在启动第一台puppetmaster的时候，根据设定的fqdn生成证书，然后修改hostname，同时把证书和配置文件同步到其他服务器上去。