使用Monit监控进程与系统状态

背景介绍：
随着线上服务器数量的增加，各种开源软件和工具的广泛使用，一些服务自动停止或无响应的情况时有发生。
而其中有很大一部分都是由于软件自身的稳定性或者机器硬件资源的限制而造成的，按道理来讲，这些情况都应该设法找到本质原因，然后避免再次出现。

但现实是残酷的，不少软件本身的稳定性有待提升，机器的硬件资源提升会触及成本，因此在集群的环境中，具备冗余，使得执行简单的服务重启成为了最现实的选择。

这本身不是什么困难的事情，实现的方法有很多，比如在Zabbix或Nagios的报警中增加Action或Commands，或自己写脚本放到计划任务中执行都可以。

但本文要介绍的，是专门来做这种事情的一个工具：Monit。
它最大的特点是配置文件简单易读，同时支持进程和系统状态的监控，并灵活的提供了各种检测的方式，周期，并进行报警和响应（重启服务，执行命令等）

系统环境：
OS: CentOS 6.4 x86_64 Minimal

具体配置：
1. 安装EPEL仓库
# yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

2. 安装Monit软件包
# yum install monit

3. 配置Monit通用参数，包括开启HTTP统计界面，邮件报警等
# vim /etc/monit.conf

###############################################################################
## Monit control file
###############################################################################
##
## Comments begin with a '#' and extend through the end of the line. Keywords
## are case insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/'.
##
## Below you will find examples of some frequently used statements. For
## information about the control file and a complete list of statements and
## options, please have a look in the Monit manual.
##
##
###############################################################################
## Global section
###############################################################################
##
## Start Monit in the background (run as a daemon):
#
set daemon 120            # check services at 2-minute intervals
    with start delay 60   # optional: delay the first check by 4-minutes (by
#                         # default Monit check immediately after Monit start)
#
#
## Set syslog logging with the 'daemon' facility. If the FACILITY option is
## omitted, Monit will use 'user' facility by default. If you want to log to
## a standalone log file instead, specify the full path to the log file
#
# set logfile syslog facility log_daemon                      
#
#
### Set the location of the Monit id file which stores the unique id for the
### Monit instance. The id is generated and stored on first Monit start. By
### default the file is placed in $HOME/.monit.id.
#
set idfile /var/run/monit/.monit.id
#
### Set the location of the Monit state file which saves monitoring states
### on each cycle. By default the file is placed in $HOME/.monit.state. If
### the state file is stored on a persistent filesystem, Monit will recover
### the monitoring state across reboots. If it is on temporary filesystem, the
### state will be lost on reboot which may be convenient in some situations.
#
set statefile /var/run/monit/.monit.state
#
## Set the list of mail servers for alert delivery. Multiple servers may be
## specified using a comma separator. By default Monit uses port 25 - it is
## possible to override this with the PORT option.
#
set mailserver localhost
# set mailserver mail.bar.baz,               # primary mailserver
#                backup.bar.baz port 10025,  # backup mailserver on port 10025
#                localhost                   # fallback relay
#
#
## By default Monit will drop alert events if no mail servers are available.
## If you want to keep the alerts for later delivery retry, you can use the
## EVENTQUEUE statement. The base directory where undelivered alerts will be
## stored is specified by the BASEDIR option. You can limit the maximal queue
## size using the SLOTS option (if omitted, the queue is limited by space
## available in the back end filesystem).
#
set eventqueue
    basedir /var/run/monit  # set the base directory where events will be stored
#   slots 100               # optionally limit the queue size
#
#
## Send status and events to M/Monit (for more informations about M/Monit
## see http://mmonit.com/).
#
# set mmonit http://monit:[email protected]:8080/collector
#
#
## Monit by default uses the following alert mail format:
##
## --8<--
## From: monit@$HOST                                     # sender
## Subject: Monit Alert - Event:$EVENT Service:$SERVICE  # subject
##
## Event:$EVENT Service:$SERVICE             #
##                                           #
##  Date:        $DATE                   #
##  Action:      $ACTION                 #
##  Host:        $HOST                   # body
##  Description: $DESCRIPTION            #
##                                           #
## Your faithful employee,                   #
## Monit                                     #
## --8<--
##
## You can override this message format or parts of it, such as subject
## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.
## are expanded at runtime. For example, to override the sender, use:
#
set mail-format {
  from: [email protected]
  subject: [$SERVICE] $EVENT
  message:
[$SERVICE] $EVENT
  Date:        $DATE
  Action:      $ACTION
  Host:        heylinux.com
  Description: $DESCRIPTION
Your faithful employee,                  
Monit }
#
#
## You can set alert recipients whom will receive alerts if/when a
## service defined in this file has errors. Alerts may be restricted on
## events by using a filter as in the second example below.
#
set alert [email protected]
# set alert [email protected]                       # receive all alerts
# set alert [email protected] only on { timeout }  # receive just service-
#                                                # timeout alert
#
#
## Monit has an embedded web server which can be used to view status of
## services monitored and manage services from a web interface. See the
## Monit Wiki if you want to enable SSL for the web server.
#
set httpd port 2812 and
    use address localhost  # only accept connection from localhost
    allow localhost        # allow localhost to connect to the server and
#     allow admin:monit      # require user 'admin' with password 'monit'
#     allow @monit           # allow users of group 'monit' to connect (rw)
#     allow @users readonly  # allow users of group 'users' to connect readonly
#
#
###############################################################################
## Services
###############################################################################
##
## Check general system resources such as load average, cpu and memory
## usage. Each test specifies a resource, conditions and the action to be
## performed should a test fail.
#
#  check system myhost.mydomain.tld
#    if loadavg (1min) > 4 then alert
#    if loadavg (5min) > 2 then alert
#    if memory usage > 75% then alert
#    if cpu usage (user) > 70% then alert
#    if cpu usage (system) > 30% then alert
#    if cpu usage (wait) > 20% then alert
#
#   
## Check a file for existence, checksum, permissions, uid and gid. In addition
## to alert recipients in the global section, customized alert can be sent to
## additional recipients by specifying a local alert handler. The service may
## be grouped using the GROUP option. More than one group can be specified by
## repeating the 'group name' statement.
#   
#  check file apache_bin with path /usr/local/apache/bin/httpd
#    if failed checksum and
#       expect the sum 8f7f419955cefa0b33a2ba316cba3659 then unmonitor
#    if failed permission 755 then unmonitor
#    if failed uid root then unmonitor
#    if failed gid root then unmonitor
#    alert [email protected] on {
#           checksum, permission, uid, gid, unmonitor
#        } with the mail-format { subject: Alarm! }
#    group server
#
#   
## Check that a process is running, in this case Apache, and that it respond
## to HTTP and HTTPS requests. Check its resource usage such as cpu and memory,
## and number of children. If the process is not running, Monit will restart
## it by default. In case the service is restarted very often and the
## problem remains, it is possible to disable monitoring using the TIMEOUT
## statement. This service depends on another service (apache_bin) which
## is defined above.
#   
#  check process apache with pidfile /usr/local/apache/logs/httpd.pid
#    start program = "/etc/init.d/httpd start" with timeout 60 seconds
#    stop program  = "/etc/init.d/httpd stop"
#    if cpu > 60% for 2 cycles then alert
#    if cpu > 80% for 5 cycles then restart
#    if totalmem > 200.0 MB for 5 cycles then restart
#    if children > 250 then restart
#    if loadavg(5min) greater than 10 for 8 cycles then stop
#    if failed host www.tildeslash.com port 80 protocol http
#       and request "/somefile.html"
#       then restart
#    if failed port 443 type tcpssl protocol http
#       with timeout 15 seconds
#       then restart
#    if 3 restarts within 5 cycles then timeout
#    depends on apache_bin
#    group server
#   
#   
## Check filesystem permissions, uid, gid, space and inode usage. Other services,
## such as databases, may depend on this resource and an automatically graceful
## stop may be cascaded to them before the filesystem will become full and data
## lost.
#
#  check filesystem datafs with path /dev/sdb1
#    start program  = "/bin/mount /data"
#    stop program  = "/bin/umount /data"
#    if failed permission 660 then unmonitor
#    if failed uid root then unmonitor
#    if failed gid disk then unmonitor
#    if space usage > 80% for 5 times within 15 cycles then alert
#    if space usage > 99% then stop
#    if inode usage > 30000 then alert
#    if inode usage > 99% then stop
#    group server
#
#
## Check a file's timestamp. In this example, we test if a file is older
## than 15 minutes and assume something is wrong if its not updated. Also,
## if the file size exceed a given limit, execute a script
#
#  check file database with path /data/mydatabase.db
#    if failed permission 700 then alert
#    if failed uid data then alert
#    if failed gid data then alert
#    if timestamp > 15 minutes then alert
#    if size > 100 MB then exec "/my/cleanup/script" as uid dba and gid dba
#
#
## Check directory permission, uid and gid.  An event is triggered if the
## directory does not belong to the user with uid 0 and gid 0.  In addition,
## the permissions have to match the octal description of 755 (see chmod(1)).
#
#  check directory bin with path /bin
#    if failed permission 755 then unmonitor
#    if failed uid 0 then unmonitor
#    if failed gid 0 then unmonitor
#
#
## Check a remote host availability by issuing a ping test and check the
## content of a response from a web server. Up to three pings are sent and
## connection to a port and an application level network check is performed.
#
#  check host myserver with address 192.168.1.1
#    if failed icmp type echo count 3 with timeout 3 seconds then alert
#    if failed port 3306 protocol mysql with timeout 15 seconds then alert
#    if failed url http://user:[email protected]:8080/?querystring
#       and content == 'action="j_security_check"'
#       then alert
#
#
###############################################################################
## Includes
###############################################################################
##
## It is possible to include additional configuration parts from other files or
## directories.
#
#  include /etc/monit.d/*
#
#
# Include all files from /etc/monit.d/
include /etc/monit.d/*

4. 举例配置针对Nginx,PHP-FPM,MySQL以及根分区用量的监控
在步骤3的配置文件中，可以看到在已经注释的代码中有很多的配置示例，这些足够我们用来参考了。
这里，给大家分享一下我在自己的VPS中所创建的相关监控项：
# vim /etc/monit.d/nginx

1 check process nginx with pidfile /webserver/nginx/run/nginx.pid
2 start program = "/webserver/init.d/nginx start"with timeout 10 seconds
3 stop program  = "/webserver/init.d/nginx stop"
4 iffailed host heylinux.com port 80 protocol http
5 with timeout 10 seconds
6 thenrestart
7 if3 restarts within 5 cycles thentimeout
8 group webserver

# vim /etc/monit.d/php-fpm

1 check process php-fpm with pidfile /webserver/php/logs/php-fpm.pid
2 start program = "/webserver/init.d/php-fpm start"with timeout 10 seconds
3 stop program  = "/webserver/init.d/php-fpm stop"
4 ifcpu > 80% for5 cycles thenrestart
5 ifloadavg(5min) greater than 4 for5 cycles thenrestart
6 if3 restarts within 5 cycles thentimeout
7 group webserver

# vim /etc/monit.d/mysql

1 check process mysql with pidfile /webserver/mysql/run/mysqld.pid
2 start program = "/webserver/init.d/mysqld start"with timeout 10 seconds
3 stop program  = "/webserver/init.d/mysqld stop"
4 iffailed port 3306 protocol mysql
5 with timeout 10 seconds
6 thenrestart
7 if3 restarts within 5 cycles thentimeout
8 group webserver

# vim /etc/monit.d/rootfs

1 check filesystem rootfs with path /dev/xvde
2 ifspace usage > 80% for5 timeswithin 15 cycles thenalert
3 group os

5. 启动Monit
# mkdir /var/run/monit
# /etc/init.d/monit start

6. 模拟Nginx进程故障，测试Monit响应行为与报警
停止Nginx进程
# /webserver/init.d/nginx stop
# /webserver/init.d/nginx status

1 nginx is stopped

观察日志输出
# tailf /var/log/monit

1 [CST Apr 12 01:11:55] error    : skipping /var/run/monit/.monit.id- unknown dataformat
2 [CST Apr 12 01:11:55] error    : Aborting event /var/run/monit/.monit.state - invalid size 5
3 [CST Apr 12 01:41:56] error    : 'nginx'process is not running
4 [CST Apr 12 01:41:56] info     : 'nginx'trying to restart
5 [CST Apr 12 01:41:56] info     : 'nginx'start: /webserver/init.d/nginx

检查Nginx是否被Monit启动
# /webserver/init.d/nginx status

1 nginx (pid 22419 22417) is running...

查看所收到的邮件，总共有两封

提示服务异常的邮件

提示服务恢复的邮件

查看Monit所有监控项的状态
# monit status

01 The Monit daemon 5.1.1 uptime: 1h 8m
02
03 Filesystem 'rootfs'
04 status                            accessible
05 monitoring status                 monitored
06 permission                        660
07 uid                               0
08 gid                               6
09 filesystem flags                  0x1000
10 block size                        4096 B
11 blocks total                      2580302 [10079.3 MB]
12 blocks freefornon superuser     1800023 [7031.3 MB] [69.8%]
13 blocks freetotal                 1931088 [7543.3 MB] [74.8%]
14 inodes total                      655360
15 inodes free607619 [92.7%]
16 data collected                    Sat Apr 12 02:17:58 2014
17
18 Process 'php-fpm'
19 status                            running
20 monitoring status                 monitored
21 pid                               13768
22 parent pid                        1
23 uptime                            6h 14m
24 children                          5
25 memory kilobytes                  3124
26 memory kilobytes total            220032
27 memory percent                    0.5%
28 memory percent total              36.3%
29 cpu percent                       0.0%
30 cpu percent total                 5.8%
31 data collected                    Sat Apr 12 02:17:58 2014
32
33 Process 'nginx'
34 status                            running
35 monitoring status                 monitored
36 pid                               22417
37 parent pid                        1
38 uptime                            36m
39 children                          1
40 memory kilobytes                  1244
41 memory kilobytes total            29256
42 memory percent                    0.2%
43 memory percent total              4.8%
44 cpu percent                       0.0%
45 cpu percent total                 0.0%
46 port response time0.144s to heylinux.com:80 [HTTP via TCP]
47 data collected                    Sat Apr 12 02:17:58 2014
48
49 Process 'mysql'
50 status                            running
51 monitoring status                 monitored
52 pid                               21502
53 parent pid                        21026
54 uptime                            1h 13m
55 children                          0
56 memory kilobytes                  44988
57 memory kilobytes total            44988
58 memory percent                    7.4%
59 memory percent total              7.4%
60 cpu percent                       0.2%
61 cpu percent total                 0.2%
62 port response time0.001s to localhost:3306 [MYSQL via TCP]
63 data collected                    Sat Apr 12 02:17:58 2014
64
65 System 'ec2-tokyo.localdomain'
66 status                            running
67 monitoring status                 monitored
68 load average                      [0.12] [0.08] [0.03]
69 cpu                               5.9%us 0.5%sy 0.4%wa
70 memory usage                      314260 kB [51.9%]
71 data collected                    Sat Apr 12 02:17:58 2014