kubernetes filebeat日志采集方案

常见的方案：

1. 日志输出到stdout stderr
   相关的路径有两个/var/lib/docker/container 和 /var/log/container, 一般云平台都默认支持
2. 日志输出到文件，emptyDir方式挂载，pod中再通过sidecar方式部署一个采集agent
3. 日志输出到文件，hostPath方式挂载，agent通过deamonset方式部署

分析：
方案1 我司日志都是打到文件，修改的话成本高
方案2 agent变化时如何全部生效可能会带来麻烦，维护成本高
方案3 存储可能会有浪费，相比其他问题成本还是低很多。常见有fluentd等

确定方案：
使用方案3，agent使用filebeat，因为比较熟悉，最新是7.3版本, 支持不少新功能

具体实施：
1.配置hostPath挂载日志
挂载主机的/var/log/containers2/[namespace]/[svcName]到容器的/home/logs目录，容器在entrypoint脚本中增加软链，将应用通用的/home/abc/logs目录链接到/home/logs/${HOSTNAME}下，可以解决deploy多pod时日志区分的问题。

2. 配置filebeat
   参考了官网的配置

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  # java日志以日期开头，node日志以[日期开头,nginx access日志以日期开头，nginx error日志日期以/分割
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /var/log/containers2/*/*/*/*.log
      multiline.pattern: '^\[?[0-9]{4}[-\/][0-9]{2}[-\/][0-9]{2}'
      multiline.negate: true
      multiline.match: after
      ignore_older: 5m
      close_inactive: 1m
      clean_removed: true
      processors:
        - script:
            lang: javascript
            id: k8s_metadata
            source: >
              function process(event) {
                event.Tag("js");
                var path = event.Get('log.file.path');
                path = path.split('/');
                event.Put('k8s.namespace', path[4]);
                event.Put('k8s.svcname', path[5]);
                event.Put('k8s.podname', path[6]);
              }

    output.kafka:
      hosts: ['xx:9092', 'xxx:9092', 'xxx:9092']
      topic: 'xxx'
      required_acks: 1
      compression: gzip
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.2.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
---
``