ELK日志分析初
ELK是一个开源的日志分析系统
ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash, Kibana , 它们都是开源软件。新增了一个FileBeat，它是一个轻量级的日志收集处理工具(Agent)，Filebeat占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具。
官方文档
Filebeat：
https://www.elastic.co/cn/products/beats/filebeat
https://www.elastic.co/guide/en/beats/filebeat/5.6/index.html
Logstash：
https://www.elastic.co/cn/products/logstash
https://www.elastic.co/guide/en/logstash/5.6/index.html
Kibana ：
https://www.elastic.co/cn/products/kibana
https://www.elastic.co/guide/en/kibana/5.5/index.html
Elasticsearch：
https://www.elastic.co/cn/products/elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/index.html
elasticsearch中文社区：
https://elasticsearch.cn/
概念
Elasticsearch 日志检索和存储
Logstash 收集分析处理
Kibana 可视化展示
Elasticsearch 基于Lucene的搜索服务器
Elasticsearch 是一个开源的分布式、高扩展高实时、RESTful风格的搜索与数据分析引擎。 它的底层是开源库Apache Lucene（搜索引擎）。
单机安装
[root@es-0001 ~]# vim /etc/hosts
192.168.1.21 es-0001
192.168.1.22 es-0002
192.168.1.23 es-0003
192.168.1.24 es-0004
192.168.1.25 es-0005
[root@es-0001 ~]# yum install -y java-1.8.0-openjdk elasticsearch
[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml
55: network.host: 0.0.0.0
[root@es-0001 ~]# systemctl enable --now elasticsearch
[root@es-0001 ~]# curl http://127.0.0.1:9200/
{
"name" : "War Eagle",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.3.4",
"build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build_timestamp" : "2016-06-30T11:24:31Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
集群安装用ansible
cluster.name: my-es
node.name: {{ ansible_hostname }}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["es-0001", "es-0002"]---
- hosts: es
tasks:
- copy:
src: hosts
dest: /etc/hosts
owner: root
group: root
mode: 0644
- name: install elasticsearch
yum:
name: java-1.8.0-openjdk,elasticsearch
state: installed
- template:
src: elasticsearch.yml
dest: /etc/elasticsearch/elasticsearch.yml
owner: root
group: root
mode: 0644
notify: reload elasticsearch
tags: esconf
- service:
name: elasticsearch
enabled: yes
handlers:
- name: reload elasticsearch
service:
name: elasticsearch
state: restarted
在任意一台集群上查看集群状态
curl http://127.0.0.1:9200/_cluster/health?pretty
集群管理
API管理
插件管理（本质网页）
-
在 es-0001 上安装 apache，并部署 head 插件
-
通过 ELB 映射 8080 端口，发布 es-0001 的 web 服务到互联网
-
es-0001 访问授权
1.部署Apache放上去
动静 读写 管理分离
2.放在本机
[root@es-0001 ~]# yum install -y httpd
[root@es-0001 ~]# systemctl enable --now httpd
[root@es-0001 ~]# tar zxf head.tar.gz -C /var/www/html [root@es-0001 ~]# vim /etc/httpd/conf/httpd.conf
# 配置文件最后追加
ProxyRequests off
ProxyPass /es/ http://127.0.0.1:9200/
ProxyPassReverse /es/ http://127.0.0.1:9200/
Options None
AuthType Basic
AuthName "Elasticsearch Admin"
AuthUserFile "/var/www/webauth"
Require valid-user
[root@es-0001 ~]# htpasswd -cm /var/www/webauth admin
New password:
Re-type new password:
Adding password for user admin
[root@es-0001 ~]# vim /etc/elasticsearch/elasticsearch.yml
# 配置文件最后追加
http.cors.enabled : true
http.cors.allow-origin : "*"
http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length
[root@es-0001 ~]# systemctl restart elasticsearch httpd通过网页插件访问es集群
API简单管理
htpp请求三部分
Method Request-URL http-version
http请求方法
get post head
PUT
DELETE
POST
GET
curl
curl -X 请求方式
-H 自定义请求头
集群状态查询
# 查询支持的关键字 [root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/ # 查具体的信息 [root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/master # 显示详细信息 ?v [root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/master?v # 显示帮助信息 ?help [root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/_cat/master?help
创建索引
-
指定索引的名称，指定分片数量，指定副本数量
-
创建索引使用 PUT 方法，创建完成以后通过 head 插件验证
[root@es-0001 ~]# curl -XPUT -H "Content-Type: application/json" \
http://127.0.0.1:9200/tedu -d '{
"settings":{
"index":{
"number_of_shards": 5,
"number_of_replicas": 1
}
}
}'增加数据
[root@es-0001 ~]# curl -XPUT -H "Content-Type: application/json" \
http://127.0.0.1:9200/tedu/teacher/1 -d '{
"职业": "诗人",
"名字": "李白",
"称号": "诗仙",
"年代": "唐"
}' 查询数据
[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/tedu/teacher/_search?pretty
[root@es-0001 ~]# curl -XGET http://127.0.0.1:9200/tedu/teacher/1?pretty修改数据
[root@es-0001 ~]# curl -XPOST -H "Content-Type: application/json" \
http://127.0.0.1:9200/tedu/teacher/1/_update -d '{
"doc": {"年代":"公元701"}
}'删除数据
# 删除一条
[root@es-0001 ~]# curl -XDELETE http://127.0.0.1:9200/tedu/teacher/1
# 删除索引
[root@es-0001 ~]# curl -XDELETE http://127.0.0.1:9200/tedu导入数据
[root@ecs-proxy ~]# gunzip logs.jsonl.gz
[root@ecs-proxy ~]# curl -XPOST -H "Content-Type: application/json" http://192.168.1.21:9200/_bulk --data-binary @logs.jsonl kibana安装
[root@kibana ~]# vim /etc/hosts
192.168.1.21 es-0001
192.168.1.22 es-0002
192.168.1.23 es-0003
192.168.1.24 es-0004
192.168.1.25 es-0005
192.168.1.26 kibana
[root@kibana ~]# yum install -y kibana
[root@kibana ~]# vim /etc/kibana/kibana.yml
02 server.port: 5601
07 server.host: "0.0.0.0"
28 elasticsearch.hosts: ["http://es-0002:9200", "http://es-0003:9200"]
113 i18n.locale: "zh-CN"
[root@kibana ~]# systemctl enable --now kibana使用 ELB 发布服务，通过 WEB 浏览器访问验证，访问5601端口