收集 Kubernetes event事件

简介

Event是什么?

Event作为kubernetes的一个对象资源，记录了集群运行所遇到的各种大事件,有助于排错，但大量的事件如果都存储在etcd中，会带来较大的性能与容量压力，所以etcd中默认只保存最近1小时的。

查看Event

[root@T01 elasticsearch]# kubectl get event
LAST SEEN   TYPE     REASON    OBJECT                          MESSAGE
5m16s       Normal   Pulled    pod/nginxtest-bbccd685f-gtf9x   Container image "nginx:1.10" already present on machine
5m15s       Normal   Created   pod/nginxtest-bbccd685f-gtf9x   Created container nginxtest
5m15s       Normal   Started   pod/nginxtest-bbccd685f-gtf9x   Started container nginxtest

[root@T01 elasticsearch]# kubectl get event -o wide
LAST SEEN   TYPE     REASON    OBJECT                          SUBOBJECT                    SOURCE         MESSAGE                                                   FIRST SEEN   COUNT   NAME
5m22s       Normal   Pulled    pod/nginxtest-bbccd685f-gtf9x   spec.containers{nginxtest}   kubelet, t01   Container image "nginx:1.10" already present on machine   5h40m        5       nginxtest-bbccd685f-gtf9x.15c919914460c103
5m21s       Normal   Created   pod/nginxtest-bbccd685f-gtf9x   spec.containers{nginxtest}   kubelet, t01   Created container nginxtest                               5h40m        5       nginxtest-bbccd685f-gtf9x.15c9199145e21995
5m21s       Normal   Started   pod/nginxtest-bbccd685f-gtf9x   spec.containers{nginxtest}   kubelet, t01   Started container nginxtest                               5h40m        5       nginxtest-bbccd685f-gtf9x.15c919914bd75bfe

收集event的方案

• 使用开源项目eventrouter进行收集
• 项目地址: https://github.com/heptiolabs/eventrouter

再容器内部收集,直接然后发送到es

大概流程

• 启动eventrouter容器,挂载/data/log/eventrouter目录
• 启动filebeat容器,挂载/data/log/eventrouter目录
• filebeat收集/data/log/eventrouter目录下的日志
• filebeat数据发送到elasticsearch
• kibana添加索引,并展示数据

IP	角色
192.168.109.128	Kubernetes
192.168.109.128	kibana
192.168.109.128	elasticsearch

es,kibana准备

$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md


## elasticsearch
$ yum -y install java
$ yum -y install elasticsearch-6.3.2

$ systemctl start elasticsearch 
$ systemctl enable elasticsearch 

## kibana
$ yum -y install kibana-6.3.2
$ chown kibana. /var/log/kibana/

$ vim /etc/kibana/kibana.yml 
server.port: 5601
server.host: "192.168.109.128"
elasticsearch.url: "http://192.168.109.128:9200"
kibana.defaultAppId: "discover"
elasticsearch.pingTimeout: 3000
elasticsearch.shardTimeout: 0
elasticsearch.startupTimeout: 9000
pid.file: /tmp/kibana.pid
logging.dest: /var/log/kibana/kibana.log
logging.verbose: false
ops.interval: 5000

$ systemctl start kibana    
$ systemctl enable kibana
$ systemctl status kibana

yaml文件

$ cat eventrouter-infilebeat.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eventrouter 
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: eventrouter 
rules:
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eventrouter 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eventrouter
subjects:
- kind: ServiceAccount
  name: eventrouter
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: eventrouter-cm
  namespace: kube-system
data:
  config.json: |- 
    {
      "sink": "glog"
    }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |-
    filebeat.prospectors:
    - input_type: log
      paths:
        - "/data/log/eventrouter/*"
    output.elasticsearch:
      hosts: ["192.168.109.128:9200"]
      index: "filebeat-k8s-pre-event-%{+yyyy.MM.dd}"
    setup.template.name: "filebeat-k8s-pre-event"
    setup.template.pattern: "filebeat-k8s-pre-event-"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eventrouter
  namespace: kube-system
  labels:
    app: eventrouter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eventrouter
  template:
    metadata:
      labels:
        app: eventrouter
        tier: control-plane-addons
    spec:
      containers:
        - name: kube-eventrouter
          image: baiyongjie/eventrouter:v0.2
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "/eventrouter -v 3 -log_dir /data/log/eventrouter"
          volumeMounts:
          - name: eventrouter-cm
            mountPath: /etc/eventrouter
          - name: log-path
            mountPath: /data/log/eventrouter
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:6.3.2
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "filebeat -c /etc/filebeat/filebeat.yml"
          volumeMounts:
          - name: filebeat-config
            mountPath: /etc/filebeat/
          - name: log-path
            mountPath: /data/log/eventrouter
      serviceAccount: eventrouter
      volumes:
        - name: eventrouter-cm
          configMap:
            name: eventrouter-cm
        - name: filebeat-config
          configMap:
            name: filebeat-config
        - name: log-path
          emptyDir: {}
          
$ kubectl apply -f eventrouter-infilebeat.yaml
serviceaccount/eventrouter created
clusterrole.rbac.authorization.k8s.io/eventrouter created
clusterrolebinding.rbac.authorization.k8s.io/eventrouter created
configmap/eventrouter-cm created
configmap/filebeat-config created
deployment.apps/eventrouter created

$ kubectl get pods -n kube-system |grep event
eventrouter-7bb898ff4b-2jp4r   2/2     Running   0          29s

查看es索引

$ curl http://192.168.109.128:9200/_cat/indices
yellow open filebeat-k8s-pre-event-2019.09.30 GL1lIT6VRp-qvI-reyjiNA 5 1 134 0 32kb 32kb

在kibana添加索引并查看

添加索引.png

kibana展示.png

模拟nginx pod重启

$ kubectl exec -it nginxtest-bbccd685f-gtf9x  -- /bin/bash
root@nginxtest-bbccd685f-gtf9x:/# nginx -s stop
2019/09/30 09:02:46 [notice] 18#18: signal process started
root@nginxtest-bbccd685f-gtf9x:/# command terminated with exit code 137


$ kubectl describe pods nginxtest-bbccd685f-gtf9x  | grep -A 20 Events: 
Events:
  Type     Reason   Age                  From          Message
  ----     ------   ----                 ----          -------
  Normal   Pulled   83s (x5 over 5h36m)  kubelet, t01  Container image "nginx:1.10" already present on machine
  Normal   Created  82s (x5 over 5h36m)  kubelet, t01  Created container nginxtest
  Normal   Started  82s (x5 over 5h36m)  kubelet, t01  Started container nginxtest

kibana查看.png

(轻易科技ops部)