系统或进程监控工具monit

   monit是一款功能强大的系统状态、进程、文件、目录和设备的监控软件，它可以自动重启那些已经挂掉的程序，非常适合监控系统关键的进程和资源，如：nginx、apache、mysql和cpu占有率等。

   monit安装之后，可以在/etc目录下找到配置文件，monit主要配置如下：

###############################################################################
## Monit control file monit控制文件
###############################################################################

##

## Comments begin with a '#' and extend through the end of the line. Keywords

## are case insensitive. All path's MUST BE FULLY QUALIFIED, starting with '/'.

## 注释以一个‘#’开始一直到本行结束。关键字不区分大小写。路径必须是以‘/’开头的全路径。

## Below you will find examples of some frequently used statements. For

## information about the control file and a complete list of statements and
## options, please have a look in the Monit manual.
##
##
###############################################################################
## Global section 全局配置
###############################################################################
##
## Start Monit in the background (run as a daemon): 后台运行monit
#
set daemon 30 # check services at 30 seconds intervals 每30s检查一次服务
# with start delay 240 # optional: delay the first check by 4-minutes (by
# # default Monit check immediately after Monit start)
#可选参数：第一次检查延迟4分钟，默认monit启动之后立即检查一次。
#
## Set syslog logging. If you want to log to a standalone log file instead,
## specify the full path to the log file
#设置系统日志作为日志记录路径。如果想将日志记录在其它地方，需要详细指定日志文件的全路径。
set logfile syslog

#
#
## Set the location of the Monit lock file which stores the process id of the
## running Monit instance. By default this file is stored in $HOME/.monit.pid
#设置monit进程号存储文件，默认存储在$HOME/.monit.pid
# set pidfile /var/run/monit.pid
#
## Set the location of the Monit id file which stores the unique id for the
## Monit instance. The id is generated and stored on first Monit start. By
## default the file is placed in $HOME/.monit.id.
#
# set idfile /var/.monit.id
#
## Set the location of the Monit state file which saves monitoring states
## on each cycle. By default the file is placed in $HOME/.monit.state. If
## the state file is stored on a persistent filesystem, Monit will recover
## the monitoring state across reboots. If it is on temporary filesystem, the
## state will be lost on reboot which may be convenient in some situations.
#
# set statefile /var/.monit.state
#
## Set the list of mail servers for alert delivery. Multiple servers may be
## specified using a comma separator. If the first mail server fails, Monit
# will use the second mail server in the list and so on. By default Monit uses
# port 25 - it is possible to override this with the PORT option.

# 设置报警推送邮件服务器列表，多个服务器以逗号隔开，可以用PORT选项来指定服务端口，默认端口是25.
# set mailserver mail.bar.baz, # primary mailserver
# backup.bar.baz port 10025, # backup mailserver on port 10025
# localhost # fallback relay
#
#
## By default Monit will drop alert events if no mail servers are available.
## If you want to keep the alerts for later delivery retry, you can use the
## EVENTQUEUE statement. The base directory where undelivered alerts will be
## stored is specified by the BASEDIR option. You can limit the queue size
## by using the SLOTS option (if omitted, the queue is limited by space
## available in the back end filesystem).
# 默认情况下没有可用的邮件服务器的时候报警事件会被丢弃，如果想延时重发可以使用EVENTQUEUE

# 配置。未被投递的报警会被存储在BASEDIR选项指定的路径中，可以通过SLOTS选项限制发送队列的

# 长度。
# set eventqueue
# basedir /var/monit # set the base directory where events will be stored
# slots 100 # optionally limit the queue size
#
#
## Send status and events to M/Monit (for more informations about M/Monit
## see http://mmonit.com/). By default Monit registers credentials with
## M/Monit so M/Monit can smoothly communicate back to Monit and you don't
## have to register Monit credentials manually in M/Monit. It is possible to
## disable credential registration using the commented out option below.
## Though, if safety is a concern we recommend instead using https when
## communicating with M/Monit and send credentials encrypted.
#
# set mmonit http://monit:[email protected]:8080/collector
# # and register without credentials # Don't register credentials
#
#
## Monit by default uses the following format for alerts if the the mail-format
## statement is missing:: monit默认使用一下格式发送通知消息。
## --8<--
## set mail-format {
## from: monit@$HOST
## subject: monit alert -- $EVENT $SERVICE
## message: $EVENT Service $SERVICE
## Date: $DATE
## Action: $ACTION
## Host: $HOST
## Description: $DESCRIPTION
##
## Your faithful employee,
## Monit
## }
## --8<--
##
## You can override this message format or parts of it, such as subject
## or sender using the MAIL-FORMAT statement. Macros such as $DATE, etc.
## are expanded at runtime. For example, to override the sender, use:
#
# set mail-format { from: [email protected] }
#
#
## You can set alert recipients whom will receive alerts if/when a
## service defined in this file has errors. Alerts may be restricted on
## events by using a filter as in the second example below.
# 可以设置通知消息的收件人
# set alert [email protected] # receive all alerts
#
## Do not alert when Monit starts, stops or performs a user initiated action.
## This filter is recommended to avoid getting alerts for trivial cases.
# 可以设置过滤条件避免在不重要情况下发送通知消息
# set alert [email protected] not on { instance, action }
#
#
## Monit has an embedded HTTP interface which can be used to view status of
## services monitored and manage services from a web interface. The HTTP
## interface is also required if you want to issue Monit commands from the
## command line, such as 'monit status' or 'monit restart service' The reason
## for this is that the Monit client uses the HTTP interface to send these
## commands to a running Monit daemon. See the Monit Wiki if you want to
## enable SSL for the HTTP interface.
# monit有一个内置的http服务可以用来查看被监控的服务的状态并通过界面管理被监控的服务。
set httpd port 2812 and
use address localhost # http服务地址和端口设置

allow localhost # allow localhost to connect to the server and 允许本地连接

allow 10.133.0.0/16 #允许此号段访问

allow admin:monit # require user 'admin' with password 'monit' 访问用户名密码

###############################################################################
## Services
###############################################################################
##
## Check general system resources such as load average, cpu and memory
## usage. Each test specifies a resource, conditions and the action to be
## performed should a test fail.
#系统整体情况监测
# check system $HOST 系统名称可以是ip或者域名
# if loadavg (1min) > 4 then alert 如果在1分钟之内的平均负载大于4则报警
# if loadavg (5min) > 2 then alert 如果在5分钟之内的平均负载大于2则报警
# if cpu usage > 95% for 10 cycles then alert 如果在10个监测周期内cpu使用率大于95%报警
# if memory usage > 75% then alert 如果内存使用率超过75%报警
# if swap usage > 25% then alert 如果交换内存使用率超过25%则报警
#
#
## Check if a file exists, checksum, permissions, uid and gid. In addition
## to alert recipients in the global section, customized alert can be sent to
## additional recipients by specifying a local alert handler. The service may
## be grouped using the GROUP option. More than one group can be specified by
## repeating the 'group name' statement.
# 检查一个文件是否存在，校验、所属用户、所属组是否正确。GROUP 选项可以将服务分组。
# check file apache_bin with path /usr/local/apache/bin/httpd
# if failed checksum and
# expect the sum 8f7f419955cefa0b33a2ba316cba3659 then unmonitor
# if failed permission 755 then unmonitor
# if failed uid root then unmonitor
# if failed gid root then unmonitor
# alert [email protected] on {
# checksum, permission, uid, gid, unmonitor
# } with the mail-format { subject: Alarm! }
# group server
#
#
## Check that a process is running, in this case Apache, and that it respond
## to HTTP and HTTPS requests. Check its resource usage such as cpu and memory,
## and number of children. If the process is not running, Monit will restart
## it by default. In case the service is restarted very often and the
## problem remains, it is possible to disable monitoring using the TIMEOUT
## statement. This service depends on another service (apache_bin) which
## is defined above.
#
# check process apache with pidfile /usr/local/apache/logs/httpd.pid
# start program = "/etc/init.d/httpd start" with timeout 60 seconds
# stop program = "/etc/init.d/httpd stop"
# if cpu > 60% for 2 cycles then alert
# if cpu > 80% for 5 cycles then restart
# if totalmem > 200.0 MB for 5 cycles then restart
# if children > 250 then restart
# if loadavg(5min) greater than 10 for 8 cycles then stop
# if failed host www.tildeslash.com port 80 protocol http
# and request "/somefile.html"
# then restart
# if failed port 443 type tcpssl protocol http
# with timeout 15 seconds
# then restart
# if 3 restarts within 5 cycles then unmonitor
# depends on apache_bin
# group server
#
#
## Check filesystem permissions, uid, gid, space and inode usage. Other services,
## such as databases, may depend on this resource and an automatically graceful
## stop may be cascaded to them before the filesystem will become full and data
## lost.
#
# check filesystem datafs with path /dev/sdb1
# start program = "/bin/mount /data"
# stop program = "/bin/umount /data"
# if failed permission 660 then unmonitor
# if failed uid root then unmonitor
# if failed gid disk then unmonitor
# if space usage > 80% for 5 times within 15 cycles then alert
# if space usage > 99% then stop
# if inode usage > 30000 then alert
# if inode usage > 99% then stop
# group server
#
#
## Check a file's timestamp. In this example, we test if a file is older
## than 15 minutes and assume something is wrong if its not updated. Also,
## if the file size exceed a given limit, execute a script
#
# check file database with path /data/mydatabase.db
# if failed permission 700 then alert
# if failed uid data then alert
# if failed gid data then alert
# if timestamp > 15 minutes then alert
# if size > 100 MB then exec "/my/cleanup/script" as uid dba and gid dba
#
#
## Check directory permission, uid and gid. An event is triggered if the
## directory does not belong to the user with uid 0 and gid 0. In addition,
## the permissions have to match the octal description of 755 (see chmod(1)).
#
# check directory bin with path /bin
# if failed permission 755 then unmonitor
# if failed uid 0 then unmonitor
# if failed gid 0 then unmonitor
#
#
## Check a remote host availability by issuing a ping test and check the
## content of a response from a web server. Up to three pings are sent and
## connection to a port and an application level network check is performed.
#
# check host myserver with address 192.168.1.1
# if failed ping then alert
# if failed port 3306 protocol mysql with timeout 15 seconds then alert
# if failed port 80 protocol http
# and request /some/path with content = "a string"
# then alert
#
#
## Check a network link status (up/down), link capacity changes, saturation
## and bandwidth usage.
#
# check network public with interface eth0
# if failed link then alert
# if changed link then alert
# if saturation > 90% then alert
# if download > 10 MB/s then alert
# if total upload > 1 GB in last hour then alert
#
#
## Check custom program status output.
#
# check program myscript with path /usr/local/bin/myscript.sh
# if status != 0 then alert
#
#
###############################################################################
## Includes
###############################################################################
##
## It is possible to include additional configuration parts from other files or
## directories.
#
# include /etc/monit.d/*
#

# set daemon mode timeout to 1 minute
set daemon 60
# Include all files from /etc/monit.d/
include /etc/monit.d/*

monit 监控没有pid文件的进程，monit提供了新的方法 MATCHING，例如进程cmdline匹配到了test_abc，我就监控这个进程，监控项叫test_abc 

    check process test_abc with MATCHING test_abc
          if changed pid then alert

monit参考：

https://wiki.gentoo.org/wiki/Monit/zh-cn#.E5.AE.89.E8.A3.85_monit

https://mmonit.com/wiki/Monit/ConfigurationExamples

https://mmonit.com/monit/documentation/monit.html#SERVICE-METHODS