Elastalert是Yelp公司用python写的一个报警框架,github地址为
https://github.com/Yelp/elastalert
文档参见:
http://elastalert.readthedocs.io/en/latest/elastalert.html
安装要求:
· Elasticsearch
· ISO8601 or Unix timestampeddata
· Python 2.7
· pip, see requirements.txt
· Packages on Ubuntu 14.x: python-pip python-dev libffi-dev libssl-dev
注意:
1.如果使用源码编译2.7,则python-dev不需要安装
2.安装libssl-dev通常需要安装同版本的libssl(下面安装步骤中有说明)
Pip install -rrequirements.txt
Python setup.py install
在线机器上打包:
pip list #查看安装的包
pip freeze >requirements.txt
pip install --download /path_to/packages -rrequirements.txt
离线机器上安装:
安装依赖项
更新openssl
rpm -Uvh --replacefilesopenssl-1.0.1e-57.el6.x86_64.rpm
rpm -Uvh --replacefilesopenssl-devel-1.0.1e-57.el6.x86_64.rpm
安装libffi-devel
rpm -ivh libffi-devel-3.0.5-3.2.el6.x86_64.rpm
升级Python2.6.6 -> 2.7.10
cd Python-2.7.10
./configure --enable-shared --enable-loadable-sqlite-extensions--prefix=/usr/local/python27 --with-zlib --with-ssl
vi ./Modules/Setup #找到下边这一行内容,去掉注释
#zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz
make && make install
备份2.6.6
mv /usr/bin/python /usr/bin/python2.6.6
建立软链接
ln -s /usr/local/python27/bin/python2.7 /usr/bin/python
查看版本
python -V
Python 2.7.10
可能遇到的问题
执行 python -V 遇到的问题:
python: error while loading shared libraries: libpython2.7.so.1.0: cannot open sharedobject file: No such file or directory
# 原因:linux系统默认没有把/usr/local/python27/lib路径加入动态库搜索路径
解决:
vi /etc/ld.so.conf
# 添加如下一行内容
/usr/local/python27/lib
ldconfig # 使新添加的路径生效
安装Setuptools 25
cd /Home/elastalert-master/required/setuptools-25.0.0/
python setup.py install
安装Pip 1.5.4
cd /Home/elastalert-master/required/pip-1.5.4/
python setup.py install
安装依赖
/usr/local/python27/bin/pip install --no-index--find-links=./packages/ -r requirements.txt
#该命令可能会失败,需要执行下面的命令:
#############################################################
安装pycparser
cd pycparser-2.18
python setup.py install
#############################################################
安装Elastalert
cd /Home/elastalert-master/
python setup.py install
拷贝elastalert文件
cp /usr/local/python27/bin/elast* /usr/bin/
config.yaml是一个全局的配置,比如rules_folder,es_host, es_port等。启动时使用选项–config
config.yaml是一个全局的配置,比如rules_folder, es_host, es_port等。启动时使用选项–config # This is the folder that contains the rule yaml files # Any .yaml file will be loaded as a rule rules_folder: /home/yuxiuwen/Downloads/elastalert-master/rules # How often ElastAlert will query Elasticsearch # The unit can be anything from weeks to seconds run_every: minutes: 1 # ElastAlert will buffer results from the most recent # period of time, in case some log sources are not in real time buffer_time: minutes: 15 # The Elasticsearch hostname for metadata writeback # Note that every rule can have its own Elasticsearch host es_host: 10.2.24.70 # The Elasticsearch port es_port: 9200 # The AWS region to use. Set this when using AWS-managed elasticsearch #aws_region: us-east-1 # The AWS profile to use. Use this if you are using an aws-cli profile. # See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html # for details #profile: test # Optional URL prefix for Elasticsearch #es_url_prefix: elasticsearch # Connect with TLS to Elasticsearch #use_ssl: True # Verify TLS certificates #verify_certs: True # GET request with body is the default option for Elasticsearch. # If it fails for some reason, you can pass 'GET', 'POST' or 'source'. # See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport # for details #es_send_get_body_as: GET # Option basic-auth username and password for Elasticsearch #es_username: someusername #es_password: somepassword # Use SSL authentication with client certificates client_cert must be # a pem file containing both cert and key for client #verify_certs: True #ca_certs: /path/to/cacert.pem #client_cert: /path/to/client_cert.pem #client_key: /path/to/client_key.key # The index on es_host which is used for metadata storage # This can be a unmapped index, but it is recommended that you run # elastalert-create-index to set a mapping writeback_index: elastalert_status # If an alert fails for some reason, ElastAlert will retry # sending the alert until this time period has elapsed alert_time_limit: days: 2
rule.yaml是位于rules_folder下的规则,有如下的配置属性:
位于rules_folder下的rules都会被启用,example_rules目录下有很多示例配置。
一个frequency.yaml的配置示例:
# Alert when the rate of events exceeds a threshold # (Optional) # Elasticsearch host es_host: 190.2.245.70 # (Optional) # Elasticsearch port es_port: 9200 # (OptionaL) Connect with SSL to Elasticsearch #use_ssl: True # (Optional) basic-auth username and password for Elasticsearch #es_username: someusername #es_password: somepassword # (Required) # Rule name, must be unique name: Deploy error # (Required) # Type of alert. # the frequency rule type alerts when num_events events occur with timeframe time type: frequency # (Required) # Index to search, wildcard supported index: logstash-deploy-* # (Required, frequency specific) # Alert when this many documents matching the query occur within a timeframe num_events: 1 # (Required, frequency specific) # num_events must occur within this amount of time to trigger an alert timeframe: minutes: 1 # (Required) # A list of Elasticsearch filters used for find events # These filters are joined with AND and nested in a filtered query # For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html filter: - query: query_string: query: "deploy_msg: error" smtp_host: smtp.baogang.info smtp_port: 25 smtp_auth_file: /home/yxw/Downloads/elastalert-master/smtp_auth_file.yaml email_reply_to: [email protected] from_addr: [email protected] # (Required) # The alert is use when a match is found alert: - "email" # (required, email specific) # a list of email addresses to send alerts to email: - "[email protected]" - "[email protected]"
建立文件smtp_auth_file.yaml,内容如下,密码中尽量不要使用特殊字符:
user: "[email protected]"
password: "passWord"
运行前要先建立一个索引,执行elastalert-create-index:ElastAlert会把执行记录存放到一个ES 索引中,该命令就是用来 创建这个索引的,默认情况下,索引名叫elastalert_status。
使用elastalert-test-rule检测规则,如下:
[root@ rules]# elastalert-test-rule --config ../config.yaml example_frequency.yaml Successfully loaded Deploy error Got 3 hits from the last 1 day Available terms in first hit: category deploy_msg source beat.hostname beat.name beat.version input_type @timestamp partition deploy_name file_name logtype deploy_op @version logtime offset message type deploy_type INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent. To send them but remain verbose, use --verbose instead. INFO:elastalert:Queried rule Deploy error from 2017-11-29 01:04 EST to 2017-11-29 01:19 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 01:19 EST to 2017-11-29 01:34 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 01:34 EST to 2017-11-29 01:49 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 01:49 EST to 2017-11-29 02:04 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 02:04 EST to 2017-11-29 02:19 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 02:19 EST to 2017-11-29 02:34 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 02:34 EST to 2017-11-29 02:49 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 02:49 EST to 2017-11-29 03:04 EST: 0 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 03:04 EST to 2017-11-29 03:19 EST: 1 / 1 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 03:19 EST to 2017-11-29 03:34 EST: 2 / 1 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 03:34 EST to 2017-11-29 03:49 EST: 3 / 1 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 03:49 EST to 2017-11-29 04:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 04:04 EST to 2017-11-29 04:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 04:19 EST to 2017-11-29 04:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 04:34 EST to 2017-11-29 04:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 04:49 EST to 2017-11-29 05:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 05:04 EST to 2017-11-29 05:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 05:19 EST to 2017-11-29 05:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 05:34 EST to 2017-11-29 05:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 05:49 EST to 2017-11-29 06:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 06:04 EST to 2017-11-29 06:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 06:19 EST to 2017-11-29 06:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 06:34 EST to 2017-11-29 06:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 06:49 EST to 2017-11-29 07:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 07:04 EST to 2017-11-29 07:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 07:19 EST to 2017-11-29 07:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 07:34 EST to 2017-11-29 07:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 07:49 EST to 2017-11-29 08:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 08:04 EST to 2017-11-29 08:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 08:19 EST to 2017-11-29 08:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 08:34 EST to 2017-11-29 08:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 08:49 EST to 2017-11-29 09:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 09:04 EST to 2017-11-29 09:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 09:19 EST to 2017-11-29 09:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 09:34 EST to 2017-11-29 09:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 09:49 EST to 2017-11-29 10:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 10:04 EST to 2017-11-29 10:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 10:19 EST to 2017-11-29 10:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 10:34 EST to 2017-11-29 10:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 10:49 EST to 2017-11-29 11:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 11:04 EST to 2017-11-29 11:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 11:19 EST to 2017-11-29 11:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 11:34 EST to 2017-11-29 11:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 11:49 EST to 2017-11-29 12:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 12:04 EST to 2017-11-29 12:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 12:19 EST to 2017-11-29 12:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 12:34 EST to 2017-11-29 12:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 12:49 EST to 2017-11-29 13:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 13:04 EST to 2017-11-29 13:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 13:19 EST to 2017-11-29 13:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 13:34 EST to 2017-11-29 13:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 13:49 EST to 2017-11-29 14:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 14:04 EST to 2017-11-29 14:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 14:19 EST to 2017-11-29 14:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 14:34 EST to 2017-11-29 14:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 14:49 EST to 2017-11-29 15:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 15:04 EST to 2017-11-29 15:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 15:19 EST to 2017-11-29 15:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 15:34 EST to 2017-11-29 15:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 15:49 EST to 2017-11-29 16:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 16:04 EST to 2017-11-29 16:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 16:19 EST to 2017-11-29 16:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 16:34 EST to 2017-11-29 16:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 16:49 EST to 2017-11-29 17:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 17:04 EST to 2017-11-29 17:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 17:19 EST to 2017-11-29 17:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 17:34 EST to 2017-11-29 17:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 17:49 EST to 2017-11-29 18:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 18:04 EST to 2017-11-29 18:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 18:19 EST to 2017-11-29 18:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 18:34 EST to 2017-11-29 18:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 18:49 EST to 2017-11-29 19:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 19:04 EST to 2017-11-29 19:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 19:19 EST to 2017-11-29 19:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 19:34 EST to 2017-11-29 19:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 19:49 EST to 2017-11-29 20:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 20:04 EST to 2017-11-29 20:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 20:19 EST to 2017-11-29 20:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 20:34 EST to 2017-11-29 20:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 20:49 EST to 2017-11-29 21:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 21:04 EST to 2017-11-29 21:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 21:19 EST to 2017-11-29 21:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 21:34 EST to 2017-11-29 21:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 21:49 EST to 2017-11-29 22:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 22:04 EST to 2017-11-29 22:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 22:19 EST to 2017-11-29 22:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 22:34 EST to 2017-11-29 22:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 22:49 EST to 2017-11-29 23:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 23:04 EST to 2017-11-29 23:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 23:19 EST to 2017-11-29 23:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 23:34 EST to 2017-11-29 23:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-29 23:49 EST to 2017-11-30 00:04 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-30 00:04 EST to 2017-11-30 00:19 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-30 00:19 EST to 2017-11-30 00:34 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-30 00:34 EST to 2017-11-30 00:49 EST: 3 / 0 hits INFO:elastalert:Queried rule Deploy error from 2017-11-30 00:49 EST to 2017-11-30 01:04 EST: 3 / 0 hits INFO:elastalert:Alert for Deploy error at 2017-11-29T08:19:01Z: INFO:elastalert:Deploy error At least 1 events occurred between 2017-11-29 03:18 EST and 2017-11-29 03:19 EST @timestamp: 2017-11-29T08:19:01Z @version: 1 _id: AWAG3klUUHQv5Ow74no_ _index: logstash-sh-run-deploy-2017.11.29 _type: log beat: { "hostname": "PVVMBG0003", "name": "PVVMBG0003", "version": "5.2.0" } category: DEPLOY deploy_msg: this is elastalert test error test deploy_name: test deploy_op: tmboot deploy_type: S file_name: s.NGDeploy-obj.log.20171129 input_type: log logtime: 2017-11-29 16:19:01 logtype: deploy message:2017-11-29 16:19:01 NAME[test];TYPE[S];OPERATION[tmboot];MESSAGES[this is elastalert test error test] num_hits: 3 num_matches: 3 offset: 45290 partition: BGNG source: /BGNG/Deployed/s.NGDeploy-obj.log.20171129 type: log INFO:elastalert:Ignoring match for silenced rule Deploy error INFO:elastalert:Ignoring match for silenced rule Deploy error Would have written the following documents to writeback index (default is elastalert_status): silence - {'rule_name': 'Deploy error', '@timestamp': datetime.datetime(2017, 11, 30, 6, 4, 44, 889391, tzinfo=tzutc()), 'exponent': 0, 'until': datetime.datetime(2017, 11, 30, 6, 5, 44, 889381, tzinfo=tzutc())} elastalert_status - {'hits': 3, 'matches': 3, '@timestamp': datetime.datetime(2017, 11, 30, 6, 4, 44, 894657, tzinfo=tzutc()), 'rule_name': 'Deploy error', 'starttime': datetime.datetime(2017, 11, 29, 6, 4, 42, 155850, tzinfo=tzutc()), 'endtime': datetime.datetime(2017, 11, 30, 6, 4, 42, 155850, tzinfo=tzutc()), 'time_taken': 2.71931791305542}
会显示一天内的命中次数,并分时段显示,以及命中后需要写入索引(elastalert_status)中什么内容。
python -m elastalert.elastalert --verbose --rule example_rules/example_frequency.yaml
python -m elastalert.elastalert --verbose --config ./config.yaml
说明:
--verbose 输出调试信息
--debug 只输出调试,不触发报警
--config 指定配置文件
--rule 指定规则文件
realert
: This option allows you to ignore repeating alerts for a period of time. If the rule uses a query_key
, this option will be applied on a per key basis. All matches for a given rule, or for matches with the same query_key
, will be ignored for the given time. All matches with a missing query_key
will be grouped together using a value of _missing
. This is applied to the time the alert is sent, not to the time of the event. It defaults to one minute, which means that if ElastAlert is run over a large time period which triggers many matches, only the first alert will be sent by default. If you want every alert, set realert to 0 minutes. (Optional, time, default 1 minute)