ELK中利用elastalert监控日志中的异常，发送邮件警告

1. 准备工作

 sudo yum update #如果该服务器有正在运行的服务会有风险，可考虑后面跟包名 指定更新

python版本最好2.7版本以上（现在centos7好像都能满足）

2.安装elastalert（按照官网的操作走）

先切换到自己的工作目录
我这里是 /usr/local/dev/
git 克隆源码

 git clone https://github.com/Yelp/elastalert.git

 cd elastalert

安装

sudo python setup.py install

出错了
Running blist-1.3.6/setup.py -q bdist_egg –dist-dir /tmp/easy_install-Gc6gbe/blist-1.3.6/egg-dist-tmp-Ik7LL2
The required version of setuptools (>=1.1.6) is not available,
and can’t be installed while this script is running. Please
install a more recent version first, using
‘easy_install -U setuptools’.

(Currently using setuptools 0.9.8 (/usr/lib/python2.7/site-packages))

ok按照它提示的，，，执行：

sudo easy_install -U setuptools

再执行安装（sudo python setup.py install），又出错了

warning: no files found matching 'blist.rst'
blist/_blist.c:38:20: 致命错误：Python.h：没有那个文件或目录
 #include

这其实是缺少开发包，执行命令：

sudo yum install python-devel

再执行安装（sudo python setup.py install）
到这里基本能安装成功。
有时候会报一些包找不到的错误，可能是网络问题，重复执行安装命令即可（我在阿里云上遇到了）。

继续执行安装命令

pip install -r requirements.txt

找不到pip ，先安装pip并更新

sudo yum -y install python-pip
sudo pip install --upgrade pip

安装完继续

sudo pip install -r requirements.txt

在elasticsearch中创建elastalert的日志索引

sudo elastalert-create-index

根据自己的情况，填入elasticsearch的相关信息，关于
elastalert_status部分直接回车默认的即可。
如下所示：

Enter elasticsearch host: log.example.com
Enter elasticsearch port: 9200
Use SSL? t/f: t
Enter optional basic-auth username (or leave blank): es_admin
Enter optional basic-auth password (or leave blank):
Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request):
New index name? (Default elastalert_status)
Name of existing index to copy? (Default None)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
New index elastalert_status created
Done!

3.创建配置文件

sudo cp config.yaml.example config.yaml

sudo vi config.yaml

根据自己的具体情况进行修改

比如我这里的elasticsearch是ssl 并需要用户登陆的，配置大致如下：

run_every:
  minutes: 1

buffer_time:
  minutes: 15

es_host: log.example.com

es_port: 9200

use_ssl: True

es_send_get_body_as: GET

es_username: es_admin

es_password: es_password

writeback_index: elastalert_status

alert_time_limit:
  days: 2

以同样的方式配置规则

cd example_rules/

sudo cp example_frequency.yaml my_rule.yaml

sudo vi my_rule.yaml

这里就是根据具体的elasticsearch的信息进行配置

es_host: log.example.com
es_port: 9200
use_ssl: True
es_username: es_admin
es_password: es_password
#name属性要求唯一，这里最好能标示自己的产品
name: My-Product Exception Alert
#类型，我选择任何匹配的条件都发送邮件警告
type: any
#需要监控的索引，支持通配
index: logstash-*
#下面两个随意配置
num_events: 50
timeframe:
  hours: 4
#根据条件进行过滤查询（这里我只要出现异常的日志，并且排除业务异常（自定义异常））
filter:
- query:
    query_string:
      query: "message: *exception* AND message: (!*BusinessException*) AND message: (!*ServiceException*)"
#email的警告方式
alert:
- "email"

#增加邮件内容，这里我附加一个日志访问路径
alert_text: "Ref Log https://log.example.com:5601/app/kibana"
#SMTP协议的邮件服务器相关配置（我这里是腾讯企业邮箱）
smtp_host: smtp.exmail.qq.com
smtp_port: 25
#用户认证文件，需要user和password两个属性
smtp_auth_file: smtp_auth_file.yaml
email_reply_to: no-reply@example.com
from_addr: no-reply@example.com 

#需要接受邮件的邮箱地址列表
email:
- "[email protected]"
- "[email protected]"

接下来创建smtp_auth_file.yaml

sudo touch smtp_auth_file.yaml
sudo vi smtp_auth_file.yaml

##配置文件内容
user: "[email protected]"
password: "password"

现在可以简单测试一下配置文件是否正确

sudo elastalert-test-rule ./my_rule.yaml

如果有问题，就检查配置文件。
然后启动运行一下试试：

sudo python -m elastalert.elastalert --verbose --rule my_rule.yaml

然后故意产生一下异常试试，比如

if(true){
    throw new NullPointerException("测试日志异常监控服务");
}

执行一下，一般没啥问题，邮件应该就能收到了，有问题就检查配置文件。

4.我是要监控服务，所以要让elastalert以服务的形式运行
首先创建elastalert的配置目录和相关文件

sudo mkdir /etc/elastalert
cd /etc/elastalert
-- 复制配置文件

sudo cp /usr/local/dev/elastalert/config.yaml config.yaml
sudo mkdir rules
cd rules

-- 复制规则文件
sudo cp /usr/local/dev/elastalert/example_rules/my_rule.yaml my_rule.yaml

-- 复制邮件用户认证文件
sudo cp /usr/local/dev/elastalert/example_rules/smtp_auth_file.yaml smtp_auth_file.yaml

接下来修改配置文件
修改 config.yaml 中

rules_folder: /etc/elastalert/rules

修改 my_rule.yaml中

smtp_auth_file: /etc/elastalert/rules/umu_smtp_auth_file.yaml

接下来就是创建systemd服务了

cd /etc/systemd/system
sudo touch elastalert.service

sudo vi elastalert.service

elastalert.service 内容

[Unit]
Description=elastalert
After=elasticsearch.service

[Service]
Type=simple
User=root
Group=root
Restart=on-failure
WorkingDirectory=/usr/local/dev/elastalert
ExecStart=/usr/bin/elastalert --config /etc/elastalert/config.yaml --rule /etc/elastalert/rules/my_rule.yaml

[Install]
WantedBy=multi-user.target

保存退出。
启动服务

sudo systemctl start elastalert

查看状态

sudo systemctl status elastalert

不出意外的话，你将看到绿色的Active: active (running)

至此，日志的异常监控服务就搭建好了，好的，收工！