使用elastalert进行错误报警

关于elastalert

elastalert是yelp出品的一个基于elasticsearch的报警服务，使用python编写。整体的思路还是基于轮询的方法，规则的话，内置frequency、spike、flatline、blacklist/whitelist、any、change。报警的话，提供了Email、HipChat、Slack、Telegram等。

dockerfile

# Elastalert Docker image running on ubuntu
# Based off of ivankrizsan/elastalert:latest .
FROM ubuntu:14.04

MAINTAINER Tom Ganem
ENV SET_CONTAINER_TIMEZONE false
ENV ELASTALERT_VERSION 0.0.95
ENV CONTAINER_TIMEZONE Asia/Shanghai
ENV ELASTALERT_URL https://github.com/Yelp/elastalert/archive/v${ELASTALERT_VERSION}.tar.gz
ENV ELASTALERT_DIRECTORY_NAME elastalert
ENV ELASTALERT_HOME /opt/${ELASTALERT_DIRECTORY_NAME}
ENV RULES_DIRECTORY /opt/${ELASTALERT_DIRECTORY_NAME}/rules


WORKDIR /opt

RUN apt-get update && \
    apt-get install tar curl python-dev tzdata -y

RUN curl -Lo get-pip.py https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

RUN mkdir -p ${ELASTALERT_HOME}

RUN curl -Lo elastalert.tar.gz ${ELASTALERT_URL} && \
    tar xvf *.tar.gz -C ${ELASTALERT_HOME} --strip-components 1 && \
    rm *.tar.gz

WORKDIR ${ELASTALERT_HOME}

RUN mkdir -p ${RULES_DIRECTORY}
RUN sed -i -e "s|'elasticsearch'|'${ELASTALERT_VERSION_CONSTRAINT}'|g" setup.py
RUN python setup.py install && \
    pip install -e .
RUN pip install elasticsearch

COPY ./docker-entrypoint.sh ${ELASTALERT_HOME}/docker-entrypoint.sh
ENTRYPOINT ["/opt/elastalert/docker-entrypoint.sh"]
CMD ["python", "elastalert/elastalert.py", "--verbose"]

关于docker-entrypoint.sh

#!/bin/sh

rules_directory=${RULES_FOLDER:-/opt/elastalert/rules}
es_port=${ELASTICSEARCH_PORT:-9200}

# Render rules files
for file in $(find . -name '*.yaml' -or -name '*.yml');
do
    cat $file | sed "s|es_host: [[:print:]]*|es_host: ${ELASTICSEARCH_HOST}|g" | sed "s|es_port: [[:print:]]*|es_port: $es_port|g" | sed "s|rules_folder: [[:print:]]*|rules_folder: $rules_directory|g" > config
    cat config > $file
    rm config
done

echo "Creating Elastalert index in Elasticsearch..."
elastalert-create-index --index elastalert_status --old-index "" --no-auth;

exec "$@"

主要是从环境变量替换config文件里头的相关变量。

配置文件

rules_folder: /opt/elastalert/rules
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The elasticsearch hostname for metadata writeback
# Note that every rule can have it's own elasticsearch host
es_host: 192.168.99.101
es_port: 9200

smtp_host: smtp.126.com
smtp_port: 25
smtp_auth_file: /opt/elastalert/smtp_cfg.yaml
from_addr: [email protected]

use_ssl: False

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

rules

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com

# (Optional)
# Elasticsearch port
# es_port: 14900

# (OptionaL) Connect with SSL to elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name: Example rule

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index: logstash-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 50

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 4

# (Required)
# A list of elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- query:
    query_string:
      query: "field: value"

# (Required)
# The alert is use when a match is found
alert:
- "email"

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "[email protected]"

启动

docker run -e "ELASTICSEARCH_HOST=192.168.99.101" -e "ELASTICSEARCH_PORT=9200" -e "RULES_FOLDER=/opt/elastalert/rules" -v $PWD/rules:/opt/elastalert/rules -v $PWD/smtp_cfg.yaml:/opt/elastalert/smtp_cfg.yaml -v $PWD/config.yaml:/opt/elastalert/config.yaml -it esalert /bin/bash

关于smtp的550错误

是用户被锁定，需要在网易邮箱里头设置开启smtp，同时设定授权码，然后用授权码替换密码发邮件

docs

• elastalert-docs

• yelp-elastalert

• Alerting with the ELK Stack and Elastalert

• smtp-550

• docker-elastalert