ES Beats安装手册

Beats安装手册

---

Overview

• 该文档用于ES 5.0以上的beats系列centos/ubuntu安装手册
• 更新时间:2017/03/18
• Write By:Brian

Bundled Beats

Packetbeat

Function

• 通过实时的网络包抓取分析来监控服务，主要针对服务有:

ICMP (v4 and v6)
DNS
HTTP
AMQP 0.9.1
Cassandra
Mysql
PostgreSQL
Redis
Thrift-RPC
MongoDB
Memcache

Install

• deb

sudo apt-get install libpcap0.8
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.2.2-amd64.deb
sudo dpkg -i packetbeat-5.2.2-amd64.deb

• rpm

sudo yum install libpcap
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-5.2.2-x86_64.rpm
sudo rpm -vi packetbeat-5.2.2-x86_64.rpm

Config

• 配置文件位置:/etc/packetbeat/packetbeat.yml

# 配置用于捕获数据包的网络设备，any表示所有设备
#
packetbeat.interfaces.device: any
# 配置你需要监控的网络协议及端口，如果端口是标准的，则允许不配置
packetbeat.protocols.dns:
  ports: [53]
  include_authorities: true
  include_additionals: true
#
packetbeat.protocols.http:
  ports: [80, 8080, 8081, 5000, 8002]
#
packetbeat.protocols.memcache:
  ports: [11211]
#
packetbeat.protocols.mysql:
  ports: [3306]
#
packetbeat.protocols.pgsql:
  ports: [5432]
#
packetbeat.protocols.redis:
  ports: [6379]
#
packetbeat.protocols.thrift:
  ports: [9090]
#
packetbeat.protocols.mongodb:
  ports: [27017]
#
packetbeat.protocols.cassandra:
  ports: [9042]
## 配置数据端，默认为ES输出
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["192.168.1.42:9200"]
  username: "elastic"
  password: "BBDelastic123"
# 若需要发送至logstash，请注释ES输出配置，配置logstash路径
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

PS：更多详细信息ES官网文档

Load Template Index

• 默认情况下，启动会自动加载默认的index的json文件，自动创建mapping信息，如果需要配置自定义的index信息，配置如下:

output.elasticsearch:
  hosts: ["localhost:9200"]
  template.name: "packetbeat"
  template.path: "packetbeat.template.json"
  template.overwrite: false

• 默认的，初始index已经存在，并且不会自动覆盖，如果需要覆盖配置，添加

template.overwrite: true

• 手动上传(自定义index需要执行)
  deb or rpm:

curl -XPUT 'http://localhost:9200/_template/packetbeat' -d@/etc/packetbeat/packetbeat.template.json

Start

deb:

sudo /etc/init.d/packetbeat start

rpm:

sudo /etc/init.d/packetbeat start

Loading Sample Kibana Dashboads

• 官网已经制作了一些现有的Dashboad，可以直接加载使用

cd /usr/share/packetbeat/
#
./scripts/import_dashboards -es http://192.168.33.60:9200
OR
./scripts/import_dashboards -es https://xyz.found.io -user user -pass password

Filebeat

Function

• 用于收集日志目录及特定的日志文件

Install

• deb

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.2.2-amd64.deb
sudo dpkg -i filebeat-5.2.2-amd64.deb

• rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.2.2-x86_64.rpm
sudo rpm -vi filebeat-5.2.2-x86_64.rpm

Config

• 配置文件目录或文件（支持通配符）

vim /etc/filebeat/filebeat.yml
#
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

• 配置输出方式ES或logstash

output.elasticsearch:
  hosts: ["192.168.1.42:9200"]
  username: "elastic"
  password: "BBDelastic123"

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

PS:需要了解更多资料点击lostash配置

Load Template Index

• 默认情况下，启动会自动加载默认的index的json文件，自动创建mapping信息，如果需要配置自定义的index信息，配置如下:

output.elasticsearch:
  hosts: ["localhost:9200"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

• 默认的，初始index已经存在，并且不会自动覆盖，如果需要覆盖配置，添加

template.overwrite: true

• 手动上传(自定义index需要执行)
  deb or rpm:

curl -XPUT 'http://localhost:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

Start

• deb

sudo /etc/init.d/filebeat start

• rpm

sudo /etc/init.d/filebeat start

Loading The Kibana index

• 同packetbeat

cd /usr/share/packetbeat/
#
./scripts/import_dashboards -es http://192.168.33.60:9200
OR
./scripts/import_dashboards -es https://xyz.found.io -user user -pass password

Metricbeat

Function

• 定期的收集metrics，可以用于监控：

Apache
HAProxy
MongoDB
MySQL
Nginx
PostgreSQL
Redis
System
Zookeeper
...and so on...

Install

• deb

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.2.2-amd64.deb
sudo dpkg -i metricbeat-5.2.2-amd64.deb

• rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.2.2-x86_64.rpm
sudo rpm -vi metricbeat-5.2.2-x86_64.rpm

Config

• 配置监控项及输出方式（强烈推荐查看/etc/metricbeat/metricbeat.full.yml，官网配套的完整配置文件）

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - filesystem
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false
## apache配置样例
- module: apache
  metricsets: ["status"]
  enabled: true
  period: 1s
  hosts: ["http://127.0.0.1"]

output.elasticsearch:
  hosts: ["192.168.1.42:9200"]
  username: "elastic"
  password: "BBDelastic123"
或
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Loading The Kibana index

• 同其他Beats(默认安装不需要)（略）

Start

• deb

sudo /etc/init.d/metricbeat start

• rpm

sudo /etc/init.d/metricbeat start

Loading The Kibana index

• 同packetbeat

cd /usr/share/packetbeat/
#
./scripts/import_dashboards -es http://192.168.33.60:9200
OR
./scripts/import_dashboards -es https://xyz.found.io -user user -pass password

Heartbeat

Function

• 就像名字一样，用于检测服务的状态，但是不像是metricbeat只是检测服务器的up或者down，它可以告诉你你的服务是否可用。适用于：

ICMP
TCP
HTTP

• Install
• deb

curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-5.2.2-amd64.deb
sudo dpkg -i heartbeat-5.2.2-amd64.deb

• rpm

curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-5.2.2-x86_64.rpm
sudo rpm -vi heartbeat-5.2.2-x86_64.rpm

Config

• 配置监控项，支持配置多个分组（如：按照业务类型ping不同主机）

heartbeat.monitors:
- type: icmp
  schedule: '*/5 * * * * * *'
  hosts: ["myhost"]
- type: tcp
  schedule: '@every 5s'
  hosts: ["myhost:12345"]
  mode: any

• 配置输出端

output.elasticsearch:
  hosts: ["192.168.1.42:9200"]
  username: "elastic"
  password: "BBDelastic123"
或
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

PS:更多信息查阅：heartbeat配置

Loading The Kibana index

• 同其他Beats(默认安装不需要)（略）

Start

• deb

sudo /etc/init.d/ start

• rpm

sudo /etc/init.d/heartbeat start