loki-promtail 指抓取所选pod的日志

版本：grafana/promtail:v0.4.0

Promtail 默认安装，会抓取所有pod的日志，如果我不想从群集中的所有Pod抓取日志，而仅抓取所选应用程序的Pod日志，需要自己修改的配置。

例如：我们想跟prometheus一样，增加 annotations: prometheus.io/scrape: "true" 来选定抓取的pod：

$ cat nginx-deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        prometheus.io/scrape: "true"
    spec:
      nodeSelector:
        kubernetes.io/hostname: 10.2.2.123
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

在promtail.yaml的scrape_config下增加：

  - action: keep
    regex: true
    source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]

完整的promtail配置文件如下： 

# cat /etc/promtail/promtail.yaml 
client:
  backoff_config:
    maxbackoff: 5s
    maxretries: 20
    minbackoff: 100ms
  batchsize: 102400
  batchwait: 1s
  external_labels: {}
  timeout: 10s
positions:
  filename: /run/promtail/positions.yaml
server:
  http_listen_port: 3101
target_config:
  sync_period: 10s

scrape_configs:
- job_name: kubernetes-pods-app
  pipeline_stages:
    - docker: {}
    
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    regex: true
    source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  - action: drop
    regex: .+
    source_labels:
    - __meta_kubernetes_pod_label_name
  - source_labels:
    - __meta_kubernetes_pod_label_app
    target_label: __service__
  - source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: __host__
  - action: drop
    regex: ''
    source_labels:
    - __service__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - action: replace
    replacement: $1
    separator: /
    source_labels:
    - __meta_kubernetes_namespace
    - __service__
    target_label: job
  - action: replace
    source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_name
    target_label: instance
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container_name
  - replacement: /var/log/pods/*$1/*.log
    separator: /
    source_labels:
    - __meta_kubernetes_pod_uid
    - __meta_kubernetes_pod_container_name
    target_label: __path__

参考：

https://github.com/grafana/loki/blob/master/docs/clients/promtail/configuration.md#relabel_config

https://github.com/grafana/loki/issues/808

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config