k8s集群日志收集

                                            k8s集群日志收集

1. 收集哪些日志
   K8S系统的组件日志
   K8S Cluster里面部署的应用程序日志

2. 日志方案 Filebeat+ELK

      

Filebeat(日志采集工具)+Logstach(数据处理引擎)+Elasticserch(数据存储、全文检索、分布式搜索引擎)+Kibana（展示数据、绘图、搜索）

      

3 容器中的日志怎么收集
      

收集方案：Pod中附加专用日志收集的容器
优点：低耦合
缺点：每个Pod启动一个日志收集代理，增加资源消耗和运维维护成本
            
4 部署DaemonSet采取k8s组件日志/var/log/messages
      
4.1 ELK安装在harbor节点
      
安装、配置ELK
      

1)安装openjdk
yum -y install java-1.8.0-openjdk
            
2）安装Logstash
https://www.elastic.co/guide/en/logstash/7.6/installing-logstash.html
            
配置网络源
/etc/yum.repos.d/logstash.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
                        

3)安装ELK组件
yum -y install logstash elasticsearch kibana
            
4)配置组件
            
修改kibana配置
vi /etc/kibana/kibana.yml
            
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
                        

修改logstash配置
            
vi /etc/logstash/conf.d/logstash-to-es.conf

input {
beats {
port => 5044
}

}

filter {

}

output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "k8s-log-%{+YYYY.MM.dd}" #索引格式
}
stdout { codec => rubydebug }

}

            
启动服务
systemctl start kibana #启动kibana
systemctl start elasticsearch #启动elasticsearch
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-es.conf &
            
192.168.1.143:5601 #访问kibana
            
5） 创建pod(master节点执行)收集组件日志

            

kubectl apply -f k8s-logs.yaml
                        
apiVersion: v1
kind: ConfigMap
metadata:
name: k8s-logs-filebeat-config
namespace: kube-system

data:
filebeat.yml: |-
filebeat.prospectors:

• type: log
  paths:

  • /messages
    fields:
    app: k8s
    type: module
    fields_under_root: true

  output.logstash:
  hosts: ['192.168.1.143:5044']

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: k8s-logs
namespace: kube-system
spec:
selector:
matchLabels:
project: k8s
app: filebeat
template:
metadata:
labels:
project: k8s
app: filebeat
spec:
containers:

• name: filebeat
  image: docker.elastic.co/beats/filebeat:6.4.2
  args: [
  "-c", "/etc/filebeat.yml",
  "-e",
  ]
  resources:
  requests:
  cpu: 100m
  memory: 100Mi
  limits:
  cpu: 500m
  memory: 500Mi
  securityContext:
  runAsUser: 0
  volumeMounts:
  • name: filebeat-config
    mountPath: /etc/filebeat.yml
    subPath: filebeat.yml
  • name: k8s-logs
    mountPath: /messages
    volumes:
• name: k8s-logs
  hostPath:
  path: /var/log/messages
  type: File
• name: filebeat-config
  configMap:

6）kibana添加索引

kibana界面->Index Patterns-》k8s-log-*->@timestamp->Create index pattern->Discover

            
7）查看日志

                        
6 收集nginx日志
            
1）创建名称空间
cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: test
            
2）创建pod的configmap（保存filebeat的配置文件）
            

kubectl apply -f filebeat-nginx-configmap.yaml
            
cat filebeat-nginx-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-nginx-config
namespace: test

data:
filebeat.yml: |-
filebeat.prospectors:

• type: log
  paths:

  • /usr/local/nginx/logs/access.log
    fields: #添加额外字段，表示字段来源和类型，日志收集时做配置
    app: www
    type: nginx-access
    fields_under_root: true #将收集的日志放在kibana界面顶级

• type: log
  paths:

  • /usr/local/nginx/logs/error.log
    fields:
    app: www
    type: nginx-error
    fields_under_root: true

  output.logstash: #收集的日志数据输出到logstash中
  hosts: ['192.168.1.143:5044']

            
3）创建secret用于保存harbor用户的密码
            
kubectl create secret docker-registry harborsecret123 --docker-server=192.168.1.143 --docker-username='zhanghai' --docker-password='123@Abcde' -n test
            
4）修改logstash配置
            
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-es.conf &
            
cat logstash-to-es.conf
input {
beats {
port => 5044
}
}

filter {
}

output {
if [app] == "www" {
if [type] == "nginx-access" {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "nginx-access-%{+YYYY.MM.dd}"
}
}
else if [type] == "nginx-error" {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "nginx-error-%{+YYYY.MM.dd}"
}
}
else if [type] == "tomcat-catalina" {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "tomcat-catalina-%{+YYYY.MM.dd}"
}
}
else if [app] == "k8s" {
if [type] == "module" {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "k8s-log-%{+YYYY.MM.dd}"
}
}
}
}
}

            
5）创建nginx的pod
              
            
kubectl apply -f nginx-deployment.yaml
            
cat nginx-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: php-demo
namespace: test
spec:
replicas: 3
selector:
matchLabels:
project: www
app: php-demo
template:
metadata:
labels:
project: www
app: php-demo
spec:
imagePullSecrets:

• name: harborsecret123
  containers:

• name: nginx
  image: 192.168.1.143/project/php-demo:latest
  imagePullPolicy: Always
  ports:

  • containerPort: 80
    name: web
    protocol: TCP
    resources:
    requests:
    cpu: 0.5
    memory: 256Mi
    limits:
    cpu: 1
    memory: 1Gi
    resources:
    requests:
    cpu: 0.5
    memory: 256Mi
    limits:
    cpu: 1
    memory: 1Gi
    livenessProbe:
    httpGet:
    path: /status.php
    port: 80
    initialDelaySeconds: 6
    timeoutSeconds: 20
    volumeMounts:
  • name: nginx-logs
    mountPath: /usr/local/nginx/logs

• name: filebeat
  image: docker.elastic.co/beats/filebeat:6.4.2
  args: [
  "-c", "/etc/filebeat.yml",
  "-e",
  ]
  resources:
  limits:
  memory: 500Mi
  requests:
  cpu: 100m
  memory: 100Mi
  securityContext:
  runAsUser: 0
  volumeMounts:

  • name: filebeat-config
    mountPath: /etc/filebeat.yml
    subPath: filebeat.yml

  • name: nginx-logs
    mountPath: /usr/local/nginx/logs

    volumes:

• name: nginx-logs
  emptyDir: {}
• name: filebeat-config
  configMap:
  name: filebeat-nginx-config

            

5）配置kibana前台
            
kibana界面->Kibana Index Patterns-》nginx-access-*->@timestamp->Create index pattern->Discover

            

7. 收集tomcat日志
               
   1）创建pod的configmap
               
   cat filebeat-tomcat-configmap.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
   name: filebeat-config
   namespace: test

data:
filebeat.yml: |-
filebeat.prospectors:

• type: log
  paths:
  • /usr/local/tomcat/logs/catalina.*
    fields:
    app: www
    type: tomcat-catalina
    fields_under_root: true
    multiline: #多行匹配，已[开头为记录一行日志
    pattern: '^['
    negate: true
    match: after
    output.logstash:
    hosts: ['192.168.1.143:5044']
                
    kubectl apply -f filebeat-tomcat-configmap.yaml

            
2）创建tomcat的pod
            
kubectl apply -f tomcat-deployment.yaml
cat tomcat-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: tomcat-java-demo
namespace: test
spec:
replicas: 3
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
imagePullSecrets:

• name: harborsecret123
  containers:

• name: tomcat
  image: 192.168.1.143/project/tomcat-java-demo:latest
  imagePullPolicy: Always
  ports:

  • containerPort: 8080
    name: web
    protocol: TCP
    resources:
    requests:
    cpu: 0.5
    memory: 1Gi
    limits:
    cpu: 1
    memory: 2Gi
    livenessProbe:
    httpGet:
    path: /
    port: 8080
    initialDelaySeconds: 60
    timeoutSeconds: 20
    readinessProbe:
    httpGet:
    path: /
    port: 8080
    initialDelaySeconds: 60
    timeoutSeconds: 20
    volumeMounts:
  • name: tomcat-logs
    mountPath: /usr/local/tomcat/logs
• name: filebeat
  image: docker.elastic.co/beats/filebeat:6.4.2
  args: [
  "-c", "/etc/filebeat.yml",
  "-e",
  ]
  resources:
  limits:
  memory: 500Mi
  requests:
  cpu: 100m
  memory: 100Mi
  securityContext:
  runAsUser: 0
  volumeMounts:
  • name: filebeat-config
    mountPath: /etc/filebeat.yml
    subPath: filebeat.yml
  • name: tomcat-logs
    mountPath: /usr/local/tomcat/logs
    volumes:
• name: tomcat-logs
  emptyDir: {}
• name: filebeat-config
  configMap:
  name: filebeat-config

            

3）配置kibana前台
            

kibana界面->Kibana Index Patterns-tomcat-catalina-*->@timestamp->Create index pattern->Discover

                  
特别声明:本文素材来源于网络,仅作为分享学习之用,如有侵权,请联系删除!