说明
promtail 是loki的日志收集agent，也是类似于 promtheus 的服务发现机制，应该是最云原生的日志agent了。赞一个
安装
# helm pull loki/promtail --version=2.0.1
# helm show values promtail-2.0.1.tgz
生成 promtail 配置文件：
配置参考：https://grafana.com/docs/loki/latest/clients/promtail/configuration/
# cat > promtail-config.yaml <====================================================================================================================================
备注：
docker: {}
docker 阶段将匹配并解析此格式的日志行：
`{"log":"level=info ts=2019-04-30T02:12:41.844179Z caller=filetargetmanager.go:180 msg=\"Adding target\"\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}`
自动将时间提取到日志 timestamp 中，stream 传输到标签中，并将日志字段输出到 output 中，因为 docker以这种方式包装您的应用程序日志，这将对其进行解包，以便仅对日志内容进行进一步的管道处理 。
Docker 阶段只是如下定义的包装：
- json:
output: log
stream: stream
timestamp: time
- labels:
stream:
- timestamp:
source: timestamp
format: RFC3339Nano
- output:
source: output
kubernetes_sd_config：
kubernetes_sd_config 发现规则跟 prometheus 一样的包括：
node：
__meta_kubernetes_node_name: The name of the node object.
_meta_kubernetes_node_label: Each label from the node object.
_meta_kubernetes_node_labelpresent: true for each label from the node object.
_meta_kubernetes_node_annotation: Each annotation from the node object.
_meta_kubernetes_node_annotationpresent: true for each annotation from the node object.
_meta_kubernetes_node_address: The first address for each node address type, if it exists. service：
__meta_kubernetes_namespace: The namespace of the service object.
_meta_kubernetes_service_annotation: Each annotation from the service object.
_meta_kubernetes_service_annotationpresent: true for each annotation of the service object.
__meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
__meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
_meta_kubernetes_service_label: Each label from the service object.
_meta_kubernetes_service_labelpresent: true for each label of the service object.
__meta_kubernetes_service_name: The name of the service object.
__meta_kubernetes_service_port_name: Name of the service port for the target.
__meta_kubernetes_service_port_protocol: Protocol of the service port for the target.pod：
__meta_kubernetes_namespace: The namespace of the pod object.
__meta_kubernetes_pod_name: The name of the pod object.
__meta_kubernetes_pod_ip: The pod IP of the pod object.
_meta_kubernetes_pod_label: Each label from the pod object.
_meta_kubernetes_pod_labelpresent: true for each label from the pod object.
_meta_kubernetes_pod_annotation: Each annotation from the pod object.
_meta_kubernetes_pod_annotationpresent: true for each annotation from the pod object.
__meta_kubernetes_pod_container_init: true if the container is an InitContainer
__meta_kubernetes_pod_container_name: Name of the container the target address points to.
__meta_kubernetes_pod_container_port_name: Name of the container port.
__meta_kubernetes_pod_container_port_number: Number of the container port.
__meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
__meta_kubernetes_pod_ready: Set to true or false for the pod’s ready state.
__meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknown in the lifecycle.
__meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
__meta_kubernetes_pod_host_ip: The current host IP of the pod object.
__meta_kubernetes_pod_uid: The UID of the pod object.
__meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
__meta_kubernetes_pod_controller_name: Name of the pod controller.endpoints：
__meta_kubernetes_namespace: The namespace of the endpoints object.
__meta_kubernetes_endpoints_name: The names of the endpoints object.
对于直接从端点列表中发现的所有目标（未从基础 pod 中另外推断出的那些目标），将附加以下标签：
__meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
__meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
__meta_kubernetes_endpoint_ready: Set to true or false for the endpoint’s ready state.
__meta_kubernetes_endpoint_port_name: Name of the endpoint port.
__meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
__meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
__meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
注意：如果端点属于服务，则会附加角色：服务发现的所有标签。
注意：对于由 Pod 支持的所有目标，将附加角色的所有标签：Pod 发现的所有标签。
- ingress：
__meta_kubernetes_namespace: The namespace of the ingress object.
__meta_kubernetes_ingress_name: The name of the ingress object.
_meta_kubernetes_ingress_label: Each label from the ingress object.
_meta_kubernetes_ingress_labelpresent: true for each label from the ingress object.
_meta_kubernetes_ingress_annotation: Each annotation from the ingress object.
_meta_kubernetes_ingress_annotationpresent: true for each annotation from the ingress object.
__meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https if TLS config is set. Defaults to http.
__meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.
====================================================================================================================================
安装 promtail
# helm install promtail promtail-2.0.1.tgz -f promtail-config.yaml -n grafana
# kubectl get all -n grafana
NAME READY STATUS RESTARTS AGE
pod/loki-0 1/1 Running 0 136m
pod/promtail-2b5rt 1/1 Running 0 49s
pod/promtail-58llq 1/1 Running 0 49s
pod/promtail-bvp6q 1/1 Running 0 49s
pod/promtail-cr5hv 1/1 Running 0 49s
pod/promtail-dfq44 1/1 Running 0 49s
pod/promtail-jm9wc 1/1 Running 0 49s
pod/promtail-kl5wr 1/1 Running 0 49s
pod/promtail-mrxjc 1/1 Running 0 49s
pod/promtail-qklwp 1/1 Running 0 49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/loki ClusterIP 10.96.116.194 3100/TCP 136m
service/loki-headless ClusterIP None 3100/TCP 136m
service/promtail-headless ClusterIP None 3101/TCP 51s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/promtail 9 9 9 9 9 51s
NAME READY AGE
statefulset.apps/loki 1/1 136m
修改为 hostNetwork：
# kubectl edit ds -n grafana promtail
...
spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
Promtail 公开了几个 URL，可用于了解其服务发现的工作方式：
- /service-discovery 服务发现页面，显示在 relabeling 之前和之后所有发现的目标及其标签，以及删除目标的原因。
- /targets 目标页面，仅显示正在主动抓取的目标及其相应的标签，文件和位置。
- /ready 健康检查接口
- /metrics metrics 接口
- /static/ 一些静态页，没啥用
---
apiVersion: v1
kind: Service
metadata:
labels:
app: promtail
name: promtail
namespace: grafana
spec:
ports:
- name: http-metrics
port: 3101
protocol: TCP
targetPort: http-metrics
selector:
app: promtail
release: promtail
sessionAffinity: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: traefik
name: promtail
namespace: grafana
spec:
rules:
- host: promtail.hupu.io
http:
paths:
- backend:
serviceName: promtail
servicePort: 3101
path: /
查看默认的任务：
# kubectl get cm -n grafana promtail -o yaml |grep job_name
- job_name: kubernetes-pods-name
- job_name: kubernetes-pods-app
- job_name: kubernetes-pods-direct-controllers
- job_name: kubernetes-pods-indirect-controller
- job_name: kubernetes-pods-static
查看配置详情：
# kubectl get cm -n grafana promtail -o yaml
试运行 Promtail：
注意：还可以添加标签
# cat my.log | promtail --stdin --client.url http://127.0.0.1:3100/loki/api/v1/push --dry-run
# cat my.log | promtail --stdin --client.url http://127.0.0.1:3100/loki/api/v1/push -server.grpc-listen-port 9999 -positions.file /tmp/2.yaml --stdin -log.level debug
# cat my.log | promtail --stdin --client.url http://127.0.0.1:3100/loki/api/v1/push --client.external-labels=k1=v1,k2=v2
发现 promtail 性能比 filebeat 好，promtail 还收集了所有的 .log 日志， filebeat 只收集了一部分：
top - 21:54:40 up 104 days, 1:06, 1 user, load average: 30.98, 31.73, 32.47
Tasks: 472 total, 2 running, 470 sleeping, 0 stopped, 0 zombie
%Cpu(s): 67.4/10.8 78[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
KiB Mem : 65806668 total, 3318964 free, 34830892 used, 27656812 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 30111884 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22705 root 20 0 18.5g 5.8g 25760 S 389.1 9.3 1317:56 java
46827 root 20 0 2557116 371756 17212 S 238.3 0.6 17288:44 filebeat
44872 root 20 0 19.0g 9.2g 18904 S 169.6 14.6 25432:15 java
55659 root 20 0 8576336 102280 22444 S 160.4 0.2 243:31.13 promtail
监控逻辑
- 您的应用向 Promtail 跟踪的文件发出一条日志行
- Promtail 读取新行并增加其计数器
- Promtail 将日志行转发给 distributor，在该 distributor 中应增加收到的计数器
- distributor 将日志行转发到 ingester，请求持续时间计数器应在此增加