Loki+Grafana(外)采集Kubernetes(K8s)集群(基于containerd)
一、Loki简介
1、简介
Loki Loki是一个开源、分布式的日志聚合系统，由Grafana Labs推出。Loki的设计目标是为了高效地处理大规模的日志数据，并具有良好的可扩展性。Loki的最大优点是它具有低资源占用和高效的查询速度。这是因为Loki不需要在处理日志数据时进行索引，而是将数据存储在类似于散列表的内存数据结构中，可以快速地定位和检索日志数据。此外，Loki可以与Prometheus集成，实现更强大的监控功能。
2、组件
- distributor：是负责获取日志数据并将其转发给ingester的无状态组件。Distributor对数据进行预处理，检查其有效性，并确保其来自已配置的租户，这有助于系统扩展并保护其免受潜在的Dos攻击。这里Grafana很好地解释了Promtail(推荐的分发代理)如何处理数据。
- ingester：是Loki架构的关键组件。从distributors接收到的数据由ingester写入云云原生存储服务。ingesters还与查询协作，返回内存中的数据以响应读请求。
- Queriers：负责解析LogQL查询请求，并从ingesters或持久化存储中获取数据。
- query fronted：可选组件-提供API接口，可用于加速读处理。该组件通过将读请求排队、将大请求拆分为多个小请求以及缓存数据来优化读处理。
二、实验环境
1、k8s环境
版本
v1.26.5，容器为containerd
二进制安装Kubernetes(K8s)集群(基于containerd)—从零安装教程（带证书）
| 主机名 | IP | 系统版本 | 安装服务 |
|---|---|---|---|
| master01 | 10.10.10.21 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| master02 | 10.10.10.22 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| master03 | 10.10.10.23 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| node01 | 10.10.10.24 | rhel7.5 | nginx、kubelet、proxy |
| node02 | 10.10.10.25 | rhel7.5 | nginx、kubelet、proxy |
2、Prometheus+Grafana环境
Prometheus+Grafana监控系统
| 主机名 | IP | 系统版本 | 安装服务 |
|---|---|---|---|
| jenkins | 10.10.10.10 | rhel7.5 | docker、Prometheus、Grafana |
三、Loki安装
https://github.com/grafana/loki/releases
https://grafana.com/docs/loki/latest/installation/docker/
1、下载镜像
[root@jenkins ~]# docker pull grafana/loki:2.8.2
[root@jenkins ~]# docker pull grafana/promtail:2.8.2
push到本地harbor：
[root@jenkins ~]# docker tag grafana/loki:2.8.2 harbor.wielun.com/library/grafana/loki:2.8.2
[root@jenkins ~]# docker tag grafana/loki:2.8.2 harbor.wielun.com/library/grafana/promtail:2.8.2
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/loki:2.8.2
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/promtail:2.8.2
2、启动loki
[root@jenkins ~]# mkdir -p /etc/loki/conf
[root@jenkins ~]# wget https://raw.githubusercontent.com/grafana/loki/v2.8.2/cmd/loki/loki-local-config.yaml -O loki-config.yaml
[root@jenkins ~]# mv loki-config.yaml /etc/loki/conf/
[root@jenkins ~]# docker run -d --name loki \
--privileged=true \
--restart always -p 3100:3100 \
-v /etc/loki/conf:/mnt/config \
grafana/loki:2.8.2 \
-config.file=/mnt/config/loki-config.yaml
[root@jenkins ~]# cat /etc/loki/conf/loki-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
# reporting_enabled: false
3、启动promtail
[root@jenkins ~]# wget https://raw.githubusercontent.com/grafana/loki/v2.8.2/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
[root@jenkins ~]# mv promtail-config.yaml /etc/loki/conf/
[root@jenkins ~]# docker run -d --name promtail \
--privileged=true \
--restart always \
-v /etc/loki/conf/:/mnt/config \
-v /var/log:/var/log \
grafana/promtail:2.8.2 \
-config.file=/mnt/config/promtail-config.yaml
[root@jenkins ~]# cat /etc/loki/conf/promtail-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://10.10.10.10:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
4、测试结果
http://10.10.10.10:3100/metrics
http://10.10.10.10:3100/ready
四、grafana查看
1、添加Loki数据源
2、查看本机日志
五、监控K8s日志
官网地址：https://grafana.com/docs/loki/latest/clients/promtail/installation/
1、上传镜像
发现使用2.8.2的镜像启动不成功，这里使用2.5.0版本
[root@jenkins ~]# docker pull grafana/promtail:2.5.0
[root@jenkins ~]# docker tag grafana/promtail:2.5.0 harbor.wielun.com/library/grafana/promtail:2.5.0
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/promtail:2.5.0
2、创建Endpoints
[root@master01 ~]# kubectl create ns logging
[root@master01 ~]# cat loki.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: loki
namespace: logging
subsets:
- addresses:
- ip: 10.10.10.10
ports:
- port: 3100
---
apiVersion: v1
kind: Service
metadata:
name: loki
namespace: logging
spec:
clusterIP: None
ports:
- name: mysql
port: 3100
protocol: TCP
targetPort: 3100
type: ClusterIP
[root@master01 ~]# kubectl apply -f loki.yaml
3、创建promtail
[root@master01 ~]# cat promtail.yaml
--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: promtail-daemonset
namespace: logging
labels:
app: promtail
spec:
selector:
matchLabels:
name: promtail
template:
metadata:
labels:
name: promtail
spec:
serviceAccount: promtail-serviceaccount
containers:
- name: promtail-container
image: harbor.wielun.com/library/grafana/promtail:2.5.0
imagePullPolicy: IfNotPresent
args:
- -config.file=/etc/promtail/promtail.yaml
env:
- name: 'HOSTNAME' # needed when using kubernetes_sd_configs
valueFrom:
fieldRef:
fieldPath: 'spec.nodeName'
volumeMounts:
- name: logs
mountPath: /var/log
- name: promtail-config
mountPath: /etc/promtail
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
volumes:
- name: logs
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: promtail-config
configMap:
name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-config
namespace: logging
labels:
app: promtail
data:
promtail.yaml: |
server:
http_listen_port: 9080
grpc_listen_port: 0
clients:
- url: http://loki.logging.svc.cluster.local:3100/loki/api/v1/push
positions:
filename: /tmp/positions.yaml
target_config:
sync_period: 10s
scrape_configs:
- job_name: pod-logs
kubernetes_sd_configs:
- role: pod
pipeline_stages:
- docker: {}
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_pod_name
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: promtail-clusterrole
labels:
app: promtail
namespace: logging
rules:
- apiGroups: [""]
resources:
- nodes
- services
- pods
verbs:
- get
- watch
- list
--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: promtail-serviceaccount
labels:
app: promtail
namespace: logging
--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: promtail-clusterrolebinding
labels:
app: promtail
namespace: logging
subjects:
- kind: ServiceAccount
name: promtail-serviceaccount
namespace: default
roleRef:
kind: ClusterRole
name: promtail-clusterrole
apiGroup: rbac.authorization.k8s.io
[root@master01 ~]# kubectl apply -f promtail.yaml
4、查看结果
[root@master01 ~]# kubectl get pod -n logging
NAME READY STATUS RESTARTS AGE
promtail-daemonset-2r8sv 1/1 Running 0 56s
promtail-daemonset-f4p8q 1/1 Running 0 56s
promtail-daemonset-tddsf 1/1 Running 0 56s
promtail-daemonset-wdn47 1/1 Running 0 56s
5、dashboard使用
https://grafana.com/grafana/dashboards/?dataSource=loki
