通过Rex管理所有的web服务器节点
为什么要用到Rex，因为目前线上业务量越来越多，服务器数量也越来越多，维护的项目也越来越多，无论是安装软件，修改配置，优化，管理，升级等操作通过写一堆脚本来集中操作已经很难达到我的要求。这时候就需要将所有不同业务类型不同项目所对应的集群环境都统一起来，集中进行管理。
常用的集中管理软件有puppet salt ansible Rex等。在进行综合比对后，我选择了Rex。原因是Rex是基于SSH来进行集成管理的，不需要再各个服务节点安装客户端，简洁，轻量，是我选择Rex的一部分原因，当然其它的集成管理软件没有怎么研究过，不过Rex已经能满足我的所有要求了，模块Rex既可以做为一个库来调用，也可以作为一个集中管理平台来使用，通过rex命令来进行一切操作。我对perl非常熟练，对于一些特殊的应用场景我会结合把Rex当做一个普通的模块，通过在自己的脚本中调用Rex模块提供的一些方法来进行远程操作，灵活性，可用性都比较强。这也是我选择Rex的一个主要原因。
介绍一下我线上的环境：
我要管理7个项目，每个项目都有一个集群环境（平均都是4个节点左右),每个节点上的web服务都是用tomcat7.0+jdk1.7的环境；session共享使用的是两台memcache组成的集群环境，数据库采用的是二台高性能的物理机+一个iscsi的盘柜组成的RAC环境。四台代理服务器（都是采用的nginx），图片存储采用的drbd+heartbeat+nfs的形式，因为是商城网站图片量现在越来越多，访问量越来越多，nfs图片服务器很容易因为I/O问题变的非常不稳定，目前测试用Moosefs和fastDFS，准备将图片存储迁移到分布式存储。
我接手之后就是这么个环境，当然整体的网络结构设计的很糟糕，很差劲。无论是从安全性，稳定性，高可用性方面去看都存在缺陷。但是上头不愿意让我对架构进行改动，只好对现有的环境进行集中管理。（哎，做运维很难！！）
废话不多说，因为实验环境有限，我只能为7个项目准备7个节点，也就是一个项目对应的集群就是一个节点也就是它自己！（公司穷，没办法，内部测试用两台pc机加一个组装的盘柜搞了一个集群虚拟化，为了节约资源只能这样玩)
1，因为是内部测试，为了方便，每个节点对应的域名都在hosts文件中解析；
[root@localhost ~]# cat /etc/hosts 127.0.0.1 mail.weike.com mx localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #192.168.0.128 test1 #192.168.0.129 test2 192.168.0.164 shopwap 192.168.0.193 gerenwap 192.168.0.196 appwap 192.168.0.224 geren 192.168.0.226 chaoshi 192.168.0.228 quanguo 192.168.0.215 boss
则七个节点对应的域名为 shopwap gerenwap appwap geren chaoshi quanguo boss（实际上线上的项目远比这多，有的是单个tomcat节点跑多个实例）
2，安装Rex，在安装之前需要确定你已经安装了perl环境(如果没有就通过yum install perl cpan进行安装)
[root@localhost src]# perl -v This is perl 5, version 16, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi (with 28 registered patches, see perl -V for more detail) Copyright 1987-2012, Larry Wall
3，对于所有的perl模块安装我们都用cpanm命令来安装，到cpan或者metacpan 上去找一个叫App-cpanminus 的源码包，然后下载下来进行解压编译安装
[root@StorageServer1 src]# ls -l |grep -i app -rw-r--r-- 1 root root 316814 Sep 16 09:52 App-cpanminus-1.7039.tar.gz [root@StorageServer1 src]# tar zxvf App-cpanminus-1.7039.tar.gz App-cpanminus-1.7039/ App-cpanminus-1.7039/bin/ App-cpanminus-1.7039/Changes App-cpanminus-1.7039/cpanfile App-cpanminus-1.7039/lib/ App-cpanminus-1.7039/LICENSE App-cpanminus-1.7039/Makefile.PL App-cpanminus-1.7039/MANIFEST App-cpanminus-1.7039/MANIFEST.SKIP App-cpanminus-1.7039/META.json App-cpanminus-1.7039/META.yml App-cpanminus-1.7039/README App-cpanminus-1.7039/t/ App-cpanminus-1.7039/t/happy_cpantesters.t App-cpanminus-1.7039/lib/App/ App-cpanminus-1.7039/lib/App/cpanminus/ App-cpanminus-1.7039/lib/App/cpanminus.pm App-cpanminus-1.7039/lib/App/cpanminus/fatscript.pm App-cpanminus-1.7039/bin/cpanm [root@StorageServer1 src]# cd App-cpanminus-1.7039 [root@StorageServer1 App-cpanminus-1.7039]# perl Makefile.PL && make && make install Checking if your kit is complete... Looks good Warning: prerequisite Test::More 0 not found. Writing Makefile for App::cpanminus cp lib/App/cpanminus/fatscript.pm blib/lib/App/cpanminus/fatscript.pm cp lib/App/cpanminus.pm blib/lib/App/cpanminus.pm cp bin/cpanm blib/script/cpanm /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/cpanm Manifying blib/man1/cpanm.1 Manifying blib/man3/App::cpanminus::fatscript.3pm Manifying blib/man3/App::cpanminus.3pm Installing /usr/local/share/perl5/App/cpanminus.pm Installing /usr/local/share/perl5/App/cpanminus/fatscript.pm Installing /usr/local/share/man/man1/cpanm.1 Installing /usr/local/share/man/man3/App::cpanminus::fatscript.3pm Installing /usr/local/share/man/man3/App::cpanminus.3pm Installing /usr/local/bin/cpanm Appending installation info to /usr/lib64/perl5/perllocal.pod [root@StorageServer1 App-cpanminus-1.7039]#
3,安装完成现在可以使用cpanm来安装Rex了（因为源地址默认都是国外的一些站点，有可能会被屏蔽，所以用163提供的源地址),设置下环境变量：
[root@StorageServer1 ~]# cat .bashrc # .bashrc # User specific aliases and functions alias vi='vim' alias rm='rm -i' alias cp='cp -i' alias mv='mv -i' alias cpanm='cpanm --mirror http://mirrors.163.com/cpan --mirror-only' # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi [root@StorageServer1 ~]#
4安装rex
root@StorageServer1 ~]# cpanm Rex --> Working on Rex Fetching http://www.cpan.org/authors/id/F/FE/FERKI/Rex-1.3.3.tar.gz ... OK ==> Found dependencies: ExtUtils::MakeMaker
安装过程中它自己会解决模块相关的依赖关系，可能会有一些底层的库需要手动去安装，比如我在安装过程中就失败了
! Installing the dependencies failed: Module 'XML::LibXML' is not installed, Module 'XML::Simple' is not installed
! Bailing out the installation for Rex-1.3.3.
提示有个XML::LibXML相关的模块安装不成功，这个和它依赖的相关的底层xml库有关系，所以我将libxml相关的底层库都安装一下就行了。
[root@StorageServer1 ~]# yum install libxml*
然后再安装rex
root@StorageServer1 ~]# cpanm Rex --> Working on Rex Fetching http://www.cpan.org/authors/id/F/FE/FERKI/Rex-1.3.3.tar.gz ... OK ==> Found dependencies: ExtUtils::MakeMaker
5，安装完成后现在就开始进行管理。
用户：管理用户我用tomcat（因为root权限不能给测试人员使用）；
模块：建立7个模块，每个模块对应一个不同的项目；
模板：建立两个模板模块，将对所有项目节点的统一操作都写进模板里，比如批量更新软件，设置时间，删除软件等等。
主机名配置文件：以ini形式保存所有项目的主机名。七个项目建立七个组，每个组中包含的主机名就是属于该项目集群环境中的几点服务器的主机名（因为集群就一个节点所以只有一个主机名)
Rexfile：配置文件，rex命令所有的操作都是读取该文件，所有类似于全局设置的选项都保存在该文件里。
第一步：建立模板：两个模板对应的名称为：
Template::Service ：该模板主要存放服务等相关的操作比如操作tomcat，启动 关闭 等
Template::File: 该模块主要发布项目和一些文件，包括文件内容同步等相关工作
建立tomcat用户并切换到tomcat目录下，然后创建rex目录，进入rex目录在创建两个模板；如下：
[
tomcat@StorageServer1 ~]$ pwd /home/tomcat [tomcat@StorageServer1 ~]$ mkdir rex [tomcat@StorageServer1 ~]$ cd rex [tomcat@StorageServer1 rex]$ rexify Template::Service --create-module Creating module Template::Service... mkdir Template/Service Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Template/Service. [tomcat@StorageServer1 rex]$ rexify Template::File --create-module Creating module Template::File... mkdir Template/File Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Template/File. [tomcat@StorageServer1 rex]$ ls Template [tomcat@StorageServer1 rex]$ ls -ld Template/* drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:18 Template/File drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:18 Template/Service
第二步：在rex目录下建立file.ini文件 里面存放的主要是操作的主机组如下
[tomcat@StorageServer1 rex]$ ls file.ini Template [tomcat@StorageServer1 rex]$ cat file.ini [quanguo_servers] quanguo [chaoshi_servers] chaoshi [geren_servers] geren [boss_servers] boss [shopwap_servers] shopwap [appwap_servers] appwap [gerenwap_servers] gerenwap [all_servers] @quanguo_servers @chaoshi_servers @geren_servers @boss_servers @shopwap_servers @appwap_servers @gerenwap_servers [tomcat@StorageServer1 rex]$
主机名所对应的ip可以查看hosts文件
第三步：在rex目录下建立模块，总共七个项目，我需要建立七个模块；
[tomcat@StorageServer1 rex]$ rexify Servers::Appwap --create-module Creating module Servers::Appwap... mkdir Servers/Appwap Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Appwap. [tomcat@StorageServer1 rex]$ rexify Servers::Boss --create-module Creating module Servers::Boss... mkdir Servers/Boss Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Boss. [tomcat@StorageServer1 rex]$ rexify Servers::Chaoshi --create-module Creating module Servers::Chaoshi... mkdir Servers/Chaoshi Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Chaoshi. [tomcat@StorageServer1 rex]$ rexify Servers::Geren --create-module Creating module Servers::Geren... mkdir Servers/Geren Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Geren. [tomcat@StorageServer1 rex]$ rexify Servers::Gerenwap --create-module Creating module Servers::Gerenwap... mkdir Servers/Gerenwap Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Gerenwap. [tomcat@StorageServer1 rex]$ rexify Servers::Quanguo --create-module Creating module Servers::Quanguo... mkdir Servers/Quanguo Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Quanguo. [tomcat@StorageServer1 rex]$ rexify Servers::Shopwap --create-module Creating module Servers::Shopwap... mkdir Servers/Shopwap Creating template file: __module__.pm Creating template file: meta.yml Your module has been created in Servers/Shopwap. [tomcat@StorageServer1 rex]$
查看创建的七个模块
[tomcat@StorageServer1 rex]$ ls Servers/ -l total 28 drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:27 Appwap drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:27 Boss drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:27 Chaoshi drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:27 Geren drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:28 Gerenwap drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:28 Quanguo drwxr-xr-x 2 tomcat tomcat 4096 Sep 16 12:28 Shopwap [tomcat@StorageServer1 rex]$ ls file.ini Servers Template [tomcat@StorageServer1 rex]$
第五步：在rex目录下创建Rexfile主配置文件
[tomcat@StorageServer1 rex]$ ls file.ini Rexfile Servers Template
稍后会讲Rexfile中的内容是什么意思（现在可以忽略Rexfie中的内容只需要创建个Rexfile文件就行)；
因为我所有项目都是部署的tomcat7.0+jdk1.7 所以我可以将重启 关闭 启动等服务操作写到Template::Sevice模块中如下：
[tomcat@StorageServer1 rex]$ cat Template/Service/__module__.pm
package Template::Service;
use v5.10;
use warnings;
use Rex -base;
task "shutdown_tomcat",group=>"all_servers",sub {
my $server = Rex::get_current_connection()->{server};
for my $proc(ps()){
if($proc->{"command"}=~/java.*?tomcat/i){
run "./shutdown.sh",
cwd =>"/usr/local/tomcat/bin",
only_if =>"pgrep java";
say "[$server]: tomcat服务关闭完成";
last;
}else{
next;
}
}
given($server){
when(/\bchaoshi\b/i){
file "/usr/local/tomcat/webapps/osms",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
when(/\bquanguo\b/i){
file "/usr/local/tomcat/webapps/olsm",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
when(/\bgeren\b/i){
file "/usr/local/tomcat/webapps/imss",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
when(/\bboss\b/i){
file "/usr/local/tomcat/webapps/boss",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
when(/\bshopwap\b/i){
file "/usr/local/tomcat/webapps/shopwap",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
when(/\bappwap\b/i){
file "/usr/local/tomcat/webapps/moser",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
when(/\bgerenwap\b/i){
file "/usr/local/tomcat/webapps/imsswap",ensure=>"absent";
file "/usr/local/tomcat/work/Catalina",ensure=>"absent";
file "/usr/local/tomcat/webapps/ROOT",ensure=>"absent";
say "[$server:缓存清理完成]";
break;}
default{ say "没有该服务";}
}
say "[$server]: 服务已经关闭";
};
task "start_tomcat",group=>"all_servers",sub {
my $server = Rex::get_current_connection()->{server};
if(is_file("/usr/local/tomcat/bin/startup.sh")){
run "./startup.sh",sub {
my($stdout,$stderr)=@_;
say "[$server]: $stdout";
},
cwd =>"/usr/local/tomcat/bin",
unless =>"pgrep java";
}else{
say "[$server]:没有正确安装tomcat,请认真检查服务状态";
exit 0;
}
for my $proc(ps()){
if($proc->{"command"}=~/java.*tomcat/i){
say "[$server]: tomcat已经启动,进程pid:".$proc->{"pid"};
last;
}else{
next;
}
}
};
task "restart_tomcat",group=>"all_servers",sub {
my $server = Rex::get_current_connection()->{server};
needs 'shutdown_tomcat';
sleep 5;
needs 'start_tomcat';
say "[$server]: 服务重启完成";
};
task "show_time",group=>"all_servers",sub {
say run "date";
};
1;
里面定义了四个任务，分别为tomcat的启动，关闭，重启，和系统当前时间
第四步：调用该模板，查看当前任务，我们讲过所有的操作都是通过rex命令来完成的，rex必须要在有Rexfile的目录下执行并且读取Rexfile中的代码；因此我要通过在Rexfile写代码调用这个模板。如下所示
[tomcat@StorageServer1 rex]$ cat Rexfile
use Rex -feature => ['1.3'];
use strict;
use warnings;
use Rex::Group::Lookup::INI;
#use Data::Dumper;
#加载模版
require Template::Service;
require Template::File;
#设置并发连接数，默认是1
parallelism 'max';
#将服务器分组并以文件的形式来保存主机名
groups_file "file.ini";
#将定义好的方法,属性,以及变量都保存在改模版中(此模板也是一个模块)
auth for =>"all_servers"=>
user=>"tomcat",
private_key=>"/home/tomcat/.ssh/id_rsa",
public_key=> "/home/tomcat/.ssh/id_rsa.pub";
desc "启动所有web服务";
task "web_start",group=>"all_servers",sub {
Template::start_tomcat();
};
desc "关闭所有web服务";
task "web_stop",group=>"all_servers",sub {
Template::shutdown_tomcat();
};
desc "重启所有web服务";
task "web_restart",group=>"all_servers",sub {
Template::Service::restart_tomcat();
};
desc "查看所有服务器节点时间";
task "show_times",group=>"all_servers",sub {
Template::show_time();
};
通过rex -T来查看当前的任务 如下
[tomcat@StorageServer1 rex]$ rex -T Tasks show_times 查看所有服务器节点时间 web_restart 重启所有web服务 web_start 启动所有web服务 web_stop 关闭所有web服务 Server Groups all_servers appwap, boss, chaoshi, geren, gerenwap, quanguo, shopwap appwap_servers appwap boss_servers boss chaoshi_servers chaoshi geren_servers geren gerenwap_servers gerenwap quanguo_servers quanguo shopwap_servers shopwap [tomcat@StorageServer1 rex]$
比如我现在要重启所有项目的web服务 我只需要通过rex+任务名就行了，
在执行之前我要说明下：因为rex是通过ssh来进行集成管理的，所以我需要用到账户和密码，ssh登陆可以通过口令登陆
也可以通过公钥认证的方式，我这里使用的是公钥认证的方式，需要将本地生成的公钥传输到各个节点上，不然登陆的话会提示让你输入密码等交互操作
为了解决这个问题，我写了个脚本名为login.pl执行以下就行了 操作如下
首先生成key
[tomcat@StorageServer1 rex]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/tomcat/.ssh/id_rsa): Created directory '/home/tomcat/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/tomcat/.ssh/id_rsa. Your public key has been saved in /home/tomcat/.ssh/id_rsa.pub. The key fingerprint is: 2b:15:90:70:0a:06:99:a6:2d:e1:5f:28:99:11:31:a9 tomcat@StorageServer1 The key's randomart image is: +--[ RSA 2048]----+ |.B* ..o. | |+=.. o.. | |=o+ o . | |E=.. . . | | .o . S | | . . . | | . . | | . | | | +-----------------+
然后在写个login.pl脚本 赋予执行权限 在执行(因为用到了Expect模块如果没安装的话用cpanm安装下就行了)
[tomcat@StorageServer1 rex]$ vi login.pl
#!/usr/bin/perl
use v5.16;
use warnings;
use Expect;
my @serversqw(quanguo chaoshi geren boss shopwap appwap imsswap);
for my $host(@servers){
my $exp=Expect->new();
$exp->spawn("ssh-copy-id -i /home/tomcat/.ssh/id_rsa.pub $host");
$exp->exp_internal(1);
$exp->log_stdout(0);
$exp->debug(3);
$exp->expect(2,[ qr/connecting \(yes\/no\)\?/i,sub {
my $self=shift;
$self->send("yes\n");
exp_continue; }],
[ qr/password: /i,sub {
my $self=shift;
$self->send("tomcat12300.\n");
exp_continue; }]
);
[tomcat@StorageServer1 rex]$ chmod a+x login.pl
[tomcat@StorageServer1 rex]$ ./login.pl
重启所有tomcat服务