kubernetes上部署Fluentd+Elasticsearch+kibana日志收集系统

kubernetes上部署Fluentd+Elasticsearch+kibana日志收集系统

1  工具介绍

Fluentd：用于收集、处理、传输日志数据。
Elasticsearch：用于实时查询和解析数据。
Kibana：用于数据可视化。

2  工作流程

Fluentd收集日志，将日志过滤处理后输出到Elasticsearch中，Elasticsearch的watcher插件将异常信息通过邮件发送给相关负责人。Kibana配置上ES服务接口，通过Kibana服务访问即可。

3  安装

三个工具全部运行在kubernettes上。

fluentd.yaml配置文件

apiVersion: v1
kind: Pod
metadata:
  name: fluentd
spec:
  containers:
  - name: fluentd-es
    image: index.tenxcloud.com/google_containers/fluentd-elasticsearch:1.9
    imagePullPolicy: Never
    env:
      - name: FLUENTD_ARGS
        value: -qq
    volumeMounts:
      - name: containers
        mountPath: /var/lib/docker/containers
      - name: varlog
        mountPath: /varlog
      - name: podlog
        mountPath: /var/lib/kubelet/pods
  volumes:
    - name: containers
      hostPath:
        path: /var/lib/docker/containers
    - name: varlog
      hostPath:
        path: /var/log
    - name: podlog
      hostPath:
        path: /var/lib/kubelet/pods

Elasticsearch-rc.yaml配置文件

apiVersion: v1
kind: ReplicationController
metadata:
  name: elasticsearch
  labels:
    name: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    name: elasticsearch-logging
  template:
    metadata:
      name: es
      labels:
        name: elasticsearch-logging
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
        - image: index.tenxcloud.com/google_containers/elasticsearch:1.4
          imagePullPolicy: Never
          name: elasticsearch-logging
          ports:
            - name: es-port
              containerPort: 9200
            - name: es-tran-port
              containerPort: 9300
          volumeMounts:
            - name: es-persistent-storage
              mountPath: /data
      volumes:
        - name: es-persistent-storage
          emptyDir: {}

Elasticsearch-service.yaml配置文件

apiVersion: v1

kind: Service
metadata:
  name: elasticsearch-logging
  labels:
    name: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
spec:
  # if your cluster supports it, uncomment the following to automatically create
  # an external load-balanced IP for the frontend service.
  type: NodePort
  ports:
    # the port that this service should serve on
  - port: 9200
    targetPort: 9200
    nodePort: 30009
#  - containerPort: 9200
  selector:
    name: elasticsearch-logging

kibana-rc.yaml文件

apiVersion: v1
kind: ReplicationController
metadata:
  name: kibana-logging
  labels:
    name: kibana-logging
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    name: kibana-logging
  template:
    metadata:
      name: kibana
      labels:
        name: kibana-logging
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
        - image: gcr.io/google_containers/kibana:1.3
          name: kibana-logging
          imagePullPolicy: Never
          env:
            - name: "ES_SCHEME"
              value: "http"
            - name: "ES_HOST"
              value: "/api/v1/proxy/namespaces/default/services/elasticsearch-logging/" 
          ports:
            - name: kibana-port
              containerPort: 80

kibana-service.yaml配置文件

apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  labels:
    name: kibana-logging
    kubernetes.io/cluster-service: "true"
spec:
  # if your cluster supports it, uncomment the following to automatically create
  # an external load-balanced IP for the frontend service.
  # type: LoadBalancer
  ports:
    # the port that this service should serve on
  - port: 5601
    targetPort: 80
#  - containerPort: 9200
  selector:
    name: kibana-logging