k8s+log-pilot日志收集

github 地址：https://github.com/AliyunContainerService/log-pilot

介绍

log-pilot是一个很棒的 docker 日志工具。可以从dockerlog-pilot主机收集日志并将它们发送到您的集中式日志系统，例如 elasticsearch、graylog2、awsog 等。log-pilot不仅可以收集 docker stdout，还可以收集 docker 容器内的日志文件。

系统架构

image.png

准备工作

• 已经成功部署了一个kubernetes集群

部署Elasticsearch

为了方便测试，在单机部署了一个Elasticsearch+Kibana

# 部署Elasticsearch
docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:6.8.23
# 部署Kibana （ELASTICSEARCH_URL请换成自己本地ip）
docker run --name kibana -e ELASTICSEARCH_URL=http://192.168.0.71:9200 -p 5601:5601 -d kibana:6.8.23

成功访问Kibana

image.png

部署log-pilot

注意：ELASTICSEARCH_HOST 、ELASTICSEARCH_PORT 需要换成刚刚创建Elasticsearch地址
如果是集群，ELASTICSEARCH_HOSTS 变量写多个地址即可，中间用逗号隔开

image.png

pilot-elasticsearch-kubernetes.yml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-pilot
  labels:
    k8s-app: log-pilot
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      k8s-app: log-pilot
  template:
    metadata:
      labels:
        k8s-app: log-pilot
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: log-pilot
        image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.5-fluentd
        env:
          - name: "LOGGING_OUTPUT"
            value: "elasticsearch"
          - name: "ELASTICSEARCH_HOST"
            value: "192.168.0.71" #changeme
          - name: "ELASTICSEARCH_PORT"
            value: "9200" #changeme
          - name: "NODE_NAME"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        volumeMounts:
        - name: sock
          mountPath: /var/run/docker.sock
        - name: root
          mountPath: /host
          readOnly: true
        - name: pos
          mountPath: /pilot/pos
        - name: localtime
          mountPath: /etc/localtime
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
              #terminationGracePeriodSeconds: 30
      volumes:
      - name: sock
        hostPath:
          path: /var/run/docker.sock
      - name: root
        hostPath:
          path: /
      - name: pos
        emptyDir: {}
      - name: localtime
        hostPath:
          path: /etc/localtime

在k8s集群部署

kubectl apply -f pilot-elasticsearch-kubernetes.yml

部署成功

image.png

部署一个测试服务

tomcat.yml

apiVersion: v1
kind: Pod
metadata:
  name: tomcat
spec:
  tolerations:
  - key: "node-role.kubernetes.io/master"
    effect: "NoSchedule"
  containers:
  - name: tomcat
    image: "tomcat:7.0"
    env:
    - name: aliyun_logs_catalina
      value: "stdout"   # 告诉 log-pilot 这个容器要收集 stdout 日志。
    - name: aliyun_logs_access
      value: "/usr/local/tomcat/logs/catalina.*.log"  # 则表示要收集容器内 /usr/local/tomcat/logs/ 目录下所有名字匹配 catalina.*.log的文件日志。
    volumeMounts:
      - name: tomcat-log
        mountPath: /usr/local/tomcat/logs
  volumes:
    - name: tomcat-log
      emptyDir: {}

启动服务后，会自动按天创建两个索引

image.png

接下来就可以愉快的玩耍了！！！

image.png