[kubernetes]-filebeat收集ingress日志
需求: 想把ingress的日志都拿出来看或者采集起来，通过kubectl log -f的方式不方便看之前的日志 无法持久化
思路:将stdout和stderr的日志持久化到/var/log/nginx 再通过filebeat采集到kibana
即用filebeat镜像作为sidecar，把nginx-ingress-controller里的/var/log/nginx 的日志采集出来
个人笔记 可能会有错误
主要配置 重定向nginx日志。
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
use-forwarder-headers: "true"
proxy-body-size: "100m"
# 配置nginx日志重定向目标
access-log-path: /var/log/nginx/access.log
error-log-path: /var/log/nginx/error.log
这里如果写了不存在的目录 会报目录不存在。建议老老实实用/var/log/nginx 否则还需要用initcontainer创建对应用户的文件夹 感觉比较麻烦
完整配置
创建base.yaml 作用说明我写了注释。
base.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
#tcp:
# 9000: "devops/test-jenkins:80"
metadata:
name: default-http-backend
labels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: default-http-backend
template:
metadata:
labels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
spec:
# hostAliases:
# - ip: "192.168.1.203"
# hostnames:
# - "test-jenkins.ihaozhuo.com"
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: gcr.io/google_containers/defaultbackend:1.4
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: ingress-nginx
labels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
spec:
ports:
- port: 80
targetPort: 8080
selector:
app.kubernetes.io/name: default-http-backend
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
use-forwarder-headers: "true"
proxy-body-size: "100m"
# 配置nginx日志重定向目标
access-log-path: /var/log/nginx/access.log
error-log-path: /var/log/nginx/error.log
---
# logrotate配置文件 偷懒就写个通配，可以根据自己需要调整
apiVersion: v1
data:
nginx.log: |
/var/log/nginx/*.log {
rotate 5
maxsize 50M
minsize 50M
compress
sharedscripts
missingok
create 0644 www-data www-data
}
kind: ConfigMap
metadata:
name: nginx-ingress-logrotate
namespace: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
# 定义日志收集相关配置的一个通配configmap 如果有需要controller的日志也可以采集
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-filebeat-config
namespace: ingress-nginx
labels:
config: nginx-filebeat-config
data:
filebeat.yml: |-
filebeat.prospectors:
- type: log
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["info","nginx_ingress_controller"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["error","nginx_ingress_controller"]
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.template.name: "filebeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.kibana:
output.elasticsearch:
hosts: ["192.168.1.165:9200"]
index: "jsonlog-%{+yyyy.MM.dd}"
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "-"
# Here: "-"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
创建deploy.yaml
deploy.yaml
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
# hostNetwork网络
hostNetwork: true
# 节点选择器 标签app=nginx
nodeSelector:
app: ingress
# 0906
containers:
- image: registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6 # 提前下载下来到私有镜像库的镜像(官方的可能会被墙)
name: filebeat
args: [
"-c", "/srv/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
# securityContext:
# runAsUser: 0
resources:
limits:
cpu: 400m
memory: 200Mi
requests:
cpu: 200m
memory: 200Mi
volumeMounts:
- name: logrotateconfig # 将configmap的内容放到容器本地目录
mountPath: /etc/logrotate.d/nginx.log
subPath: nginx.log
- name: config # 将configmap的内容放到容器本地目录
mountPath: /srv/filebeat/
- name: data
mountPath: /usr/share/filebeat/data
- name: logdata
mountPath: /var/log/nginx
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
# 设置ingress-controller日志输出地址 偷懒和nginx日志放一起了
- --log_dir=/var/log/nginx/
- --logtostderr=false
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
- name: logdata # 将configmap的内容放到容器本地目录
mountPath: /var/log/nginx
# 0906
volumes:
- name: data
emptyDir: {}
- name: logdata #定义logdata为EmptyDir类型挂载目录
emptyDir: {}
- name: config
configMap:
name: nginx-filebeat-config #使用前面定义的configmap
items:
- key: filebeat.yml
path: filebeat.yml
- name: logrotateconfig
configMap:
name: nginx-ingress-logrotate
items:
- key: nginx.log
path: nginx.log
# 需要的话加上仓库的secret
# imagePullSecrets:
# - name: aliyun-ns-test
# 0906
tcp-service.yaml 开发443 80 和额外的9000端口
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
- name: proxied-tcp-9000
port: 9000
targetPort: 9000
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
应用yaml
kubectl apply -f ./
kubectl get pods -w -n ingress-nginx
![[kubernetes]-filebeat收集ingress日志_第1张图片](http://img.e-com-net.com/image/info8/07ae1ad20f51469bbb152244635b27c6.jpg)
进入容器检查
文件都已经挂载了
![[kubernetes]-filebeat收集ingress日志_第2张图片](http://img.e-com-net.com/image/info8/8b64ca36950043959b62205ab47d588e.jpg)
kibana中采集到的日志
![[kubernetes]-filebeat收集ingress日志_第3张图片](http://img.e-com-net.com/image/info8/2e0b420b359b444698088f3b74f892eb.jpg)
pod内查看如下
![[kubernetes]-filebeat收集ingress日志_第4张图片](http://img.e-com-net.com/image/info8/297105386f534a85a79631316ac58be8.jpg)
日志轮转方面
用filebeat6的镜像稍作修改
# 1 更换apt源
cat > /etc/apt/sources.list <手动轮转结果如下 文件会被压缩 不管是否达到配置里的要求。 我用filebeat采集的话 其实用cat 清空也行
![[kubernetes]-filebeat收集ingress日志_第5张图片](http://img.e-com-net.com/image/info8/f0d1101207784734ade6d4918b94bd33.jpg)
![[kubernetes]-filebeat收集ingress日志_第6张图片](http://img.e-com-net.com/image/info8/2307761e9c004b6e97c785e09b3351ac.jpg)
找到对应的filebeat容器，进行commit，然后push成新镜像filebeat:6.2
docker commit -a "install vim logrotate crontab" 761e161026c7 registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6.2
docker push registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6.2
docker tag registry.cn-shanghai.aliyuncs.com/yjk-datag/filebeat:6.2 woshidaxu/filebeat:6.2
docker push woshidaxu/filebeat:6.2
修改过后的镜像可以下载 也可以替换deploy.yaml里filebeat的tag 方便下次重新部署 我这个镜像logrotate是3.9以下的
docker pull woshidaxu/filebeat:6.2
遇到的问题
error: skipping "/var/log/nginx/access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
需要在configmap中加入su root list
# nginx.log
/var/log/nginx/*.log {
su root list
rotate 5
maxsize 50M
minsize 50M
compress
# sharedscripts
missingok
create 0644 www-data www-data
}
如果自己做镜像 可以用logrotate 3.9以上版本可以用下面的配置 可以实现按小时切割 。logrotate3.9以上支持小时 分钟。ubuntu16 apt-cache madison logrotate 了一下只能3.8
/var/log/nginx/*.log {
su root list
size 50M
notifempty
copytruncate
rotate 5
missingok
compress
dateext
dateformat .%Y%m%d-%H
create 0644 www-data www-data
}
可参考文档 网上貌似都是这篇复制的吧？我和他差别比较多 我直接挂出来采集了。没有用hostpath
https://segmentfault.com/a/1190000022200374