k8s资源限制

[toc]

一、网络策略

1.1 部署tomcat和nginx

1.1.1 创建命名空间

• 创建linux和app命名空间并打label

kubectl create ns linux
kubectl create ns app
kubectl label ns linux nsname=linux
kubectl label ns app nsname=app

1.1.2 tomcat和nginx的yaml

• tomcat的deployment和service

kind: Deployment
# 这里需要注意自己的k8s版本，通过kubectl explain deployment查看自己的k8s版本支持的版本号
# 这里以kubernetes v1.22.2为例
apiVersion: apps/v1
metadata:
  labels:
    app: app-tomcat-deployment-label
  name: app-tomcat-deployment
  namespace: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app:app-tomcat-selector
  template:
    metadata:
      labels:
        app: app-tomcat-selector
    spec:
      containers:
      - name: app-tomcat-container
        image: tomcat:7.0.109-jdk8-openjdk
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
          
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: app-tomcat-service-label
  name: app-tomcat-service
  namespace: app
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30015
  selector:
    app: app-tomcat-selector

• nginx的deployment和service

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: app-nginx-deployment-label
  name: app-nginx-deployment
  namespace: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-nginx-selector
  template:
    metadata:
      labels:
        app: app-nginx-selector
        project: app
    spec:
      containers:
      - name: app-nginx-container
        image: nginx:1.20.2-alpine
        imagePullPolicy: Always
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
        - containerPort: 443
          protocol: TCP
          name: https

---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: app-nginx-service-label
  name: app-nginx-service
  namespace: app
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30014
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30453
  selector:
    app: app-nginx-selector

linux命名空间下的yaml同样配置，需修改下yaml文件中的命名空间和标签

1.1.3 部署tomcat和nginx

# 部署当前命名空间下的所有yaml文件
kubectl apply -f .

1.1.4 配置动静分离

• 在tomcat中创建页面

# TOMCAT_CONTAINER需要替换为tomcat真实的容器名
kubectl -n app exec -it TOMCAT_CONTAINER bash 
cd webapps
mkdir app
echo "app app" > app/index.jsp

• 配置nginx转发

kubectl -n app exec -it NGINX_CONTAINER sh
# nginx是基于alpine系统的，安装vim命令
apk add vim
# 配置匹配目录转发至tomcat规则
vim /etc/nginx/conf.d/default.conf
# 添加如下规则
location /app {
    # TOMCAT_SERVICE需要替换为tomcat真实的service名字
    proxy_pass http://TOMCAT_SERVICE; 
}
# 检查nginx配置文件语法
nginx -t
# 检查正确后，重新加载配置文件，让配置文件生效
nginx -s reload

遇到复制粘贴不了的情况

1645584428139.png

在命令模式下设置即可

1645584683595.png

linux命名空间同样配置，需要修改页面为"linux app"

1.1.5 验证配置

• nginx页面

1645585087768.png

• tomcat页面

1645585231992.png

• 通过nginx端口访问tomcat

1645585279621.png

默认情况下pod跨命名空间也可以访问

1.2 配置网络策略规则

1.2.1 入规则标签限制

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tomcat-access--networkpolicy
  namespace: app
spec:
  policyTypes:
  - Ingress
  podSelector:
    matchLabels:
      app: app-tomcat-selector # 对匹配到的目的Pod应用以下规则
  ingress: # 入栈规则，如果指定目标端口就是匹配全部端口和协议，协议TCP, UDP, or SCTP
  - from:
    - podSelector:
        matchLabels:
          app: app-nginx-selector # 如果存在多个matchLabel条件，要同时满足条件A、条件B、条件X

添加了网络策略后，默认禁止了跨namespace访问目标pod

没有在配置中允许的pod，同namespace也无法访问

不允许从宿主机访问pod

该策略只允许通namespace含有特定标签的源pod访问目标pod

该策略不影响各namespace的pod与非明确禁止的pod之间的相互访问

• 可以使用describe查看networkpolicy

kubectl -n app describe networkpolicies.networking.k8s.io tomcat-access--networkpolicy
Name:         tomcat-access--networkpolicy
Namespace:    app
Created on:   2022-02-23 03:18:02 +0000 UTC
Labels:       
Annotations:  
Spec:
  PodSelector:     app=app-tomcat-selector
  Allowing ingress traffic:
    To Port:  (traffic allowed to all ports)
    From:
      PodSelector: app=app-nginx-selector
  Not affecting egress traffic
  Policy Types: Ingress

• 添加规则后从linux命名空间下访问tomcat的地址

1645597541156.png

访问tomcat的pod地址不通，是因为网络规则限制；访问nginx的pod地址可以通过

1.2.2 入规则标签端口限制

• 在1.2.1的yaml文件后边添加如下内容，位于from下边，跟podSelector同级

ports: #入栈规则，如果指定目标端口就是匹配全部端口和协议,协议TCP, UDP, or SCTP
    - protocol: TCP
      #port: 8080 #允许通过TCP协议访问目标pod的8080端口,但是其它没有允许的端口将全部禁止访问
      port: 80

• 添加规则后从app命名空间下nginx中访问tomcat的地址，不放开8080端口，同命名空间下，符合podSelector匹配规则也访问不通；放开8080端口，可以访问通

1645598181215.png

1.2.3 入规则多端口限制

• 1.2.1的yaml文件，用如下内容替换from内容

  - from:
    - podSelector: #匹配源pod,matchLabels: {}为不限制源pod即允许所有pod
        matchLabels: {}
    ports: #入栈规则，如果指定目标端口就是匹配全部端口和协议，协议TCP, UDP, or SCTP
    - protocol: TCP
      port: 8080
    - protocol: TCP
      port: 8081
    - protocol: TCP
      port: 8082

• 同namespace下的所有pod可以访问目标pod的多个端口

1645599041496.png

1645599069009.png

1.2.4 入规则互访

• 1.2.1的yaml文件，用如下内容替换Ingress内容

  - Ingress
  podSelector:
    matchLabels: {} 
  ingress:
  - from:
    - podSelector: 
        matchLabels: {}

• 同namespace下所有pod可以相互访问

1645599069009.png

1645599983023.png

1.2.5 入规则IP限制

• 1.2.1的yaml文件，用如下内容替换from内容

- from:
    - ipBlock:
        cidr: 10.200.0.0/16 #白名单，允许访问的地址范围，没有允许的将禁止访问目标pod
        except:
        - 10.200.218.0/24 #在以上范围内禁止访问的源IP地址
        - 10.200.230.239/32 #在以上范围内禁止访问的源IP地址
    ports: 
    - protocol: TCP
      port: 8080
    - protocol: TCP
      port: 8081

• 白名单内地址可以访问通，明确限制的地址访问不通

1645600536617.png

1645600464526.png

这个规则只限制ip地址，不限制命名空间，不通命名空间下的pod，只要ip地址在白名单地址段，也可以访问目标pod

1.2.6 入规则命名空间限制

• 1.2.1的yaml文件，用如下内容替换Ingress内容

- Ingress
  podSelector: #目标pod
    matchLabels: {} #允许访问app namespace 中的所有pod
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          nsname: linux #只允许指定的namespace访问
    - namespaceSelector:
        matchLabels:
          nsname: app #只允许指定的namespace访问
    ports:
    - protocol: TCP
      port: 8080
    - protocol: TCP
      port: 8081
    - protocol: TCP
      port: 8082

• 允许多个namespace访问指定namespace中的所有pod和多个端口，没有被允许的namespace中的pod访问不通

1645601040168.png

1645601079875.png

1.2.7 出规则IP端口限制

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-access-networkpolicy
  namespace: app
spec:
  policyTypes:
  - Egress
  podSelector: #目标pod选择器
    matchLabels:  #基于label匹配目标pod
      app: app-tomcat-selector #匹配app namespace中app的值为app-tomcat-selector的pod,然后基于egress中的指定网络策略进行出口方向的网络限制
  egress:
  - to:
    - ipBlock:
        cidr: 10.200.0.0/16 #允许匹配到的pod出口访问的目的CIDR地址范围
    - ipBlock:
        cidr: 172.31.7.106/32 #允许匹配到的pod出口访问的目的主机
    ports:
    - protocol: TCP
      port: 80 #允许匹配到的pod访问目的端口为80的访问
    - protocol: TCP
      port: 53 #允许匹配到的pod访问目的端口为53 即DNS的解析
    - protocol: UDP
      port: 53 #允许匹配到的pod访问目的端口为53 即DNS的解析

• 目标pod可以访问指定地址段的多个端口，没有放开的端口访问不通

1645602269891.png

1.2.8 出规则标签端口限制

• 1.2.7的yaml文件，用如下内容替换spec内容

spec:
  policyTypes:
  - Egress
  podSelector:
    matchLabels:
      app: app-nginx-selector
  egress:
  - to:
    - podSelector: 
        matchLabels:
          app: app-tomcat-selector
    ports:
    - protocol: TCP
      port: 8080
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53

• 限制目标pod只能访问匹配标签的pod的多个端口

1645603454703.png

1.2.9 出规则命名空间限制

• 1.2.7的yaml文件，用如下内容替换egress内容

egress:
  - to:
    - namespaceSelector:
        matchLabels:
          nsname: python #指定允许访问的目的namespace
    - namespaceSelector:
        matchLabels:
          nsname: linux #指定允许访问的目的namespace
    ports:
    - protocol: TCP
      port: 8080
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53

• 指定目标pod可以访问多个命名空间的多个端口，未被允许的端口访问不通

1645604351218.png

calicoctl get networkpolicy -n app

这条命令可以使用calico的命令行工具查看网络规则，使用方法和kubectl一样，可以describe，可以-o yaml

注：网络规则需要网络组件支持，calico支持，flannel不支持

二、Ingress-NGINX

2.1 ExternalName

• ExternalName的yaml文件

apiVersion: v1
kind: Service
metadata:
  name: my-external-test-name
  namespace: default
spec:
  type: ExternalName  #service类型
  externalName: www.baidu.com   #外部域名

• 在service的ExternalName类型中绑定baidu的域名，pod内直接访问域名即可

1645666300469.png

• CLusterIP和Endpoint的yaml文件

apiVersion: v1
kind: Service
metadata:
  name: mysql-production-server-name
  namespace: default
spec:
  ports:
    - port: 6379
---
kind: Endpoints
apiVersion: v1
metadata:
  name: mysql-production-server-name
  namespace: default
subsets:
  - addresses:
      - ip: 192.168.204.182
    ports:
      - port: 6379

• 修改redis配置文件

1645667181373.png

• 重启redis服务

systemctl restart redis

• 进入pod中，安装epel-release，安装redis客户端，通过域名连接redis

1645668178651.png

在centos 7安装redis报错

1645666852755.png

安装epel源

1645666927600.png

再安装

1645666991690.png

2.2 创建ingress-nginx

• 启动ingress-nginx的pod

1645668585935.png

• 网页访问测试

1645668637961.png

访问任意一台主机的80端口，会得到这个报错，证明部署成功

2.3 匹配规则

2.3.1 匹配单主机

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web
  namespace: linux
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/use-regex: "true" ##指定后面rules定义的path可以使用正则表达式
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##连接超时时间,默认为5s
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##后端服务器响应超时时间,默认为60s
    nginx.ingress.kubernetes.io/proxy-body-size: "50m" ##客户端上传文件，最大大小，默认为20m
    nginx.ingress.kubernetes.io/app-root: /index.html

spec:
  rules:
  - host: www.test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: linux-tomcat-app1-service
            port:
              number: 80

• 访问如下

1645670786676.png

需要现在linux命名空间下创建两个tomcat的service和deployment，在tomcat中创建目录和访问页，在主机上的hosts文件添加域名将解析（可以是负载均衡器的地址或者集群任意一个node的地址）

2.3.2 匹配多主机

• 在2.3.1的yaml中添加如下内容

  - host: mobile.test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: linux-tomcat-app2-service
            port:
              number: 80

• 访问如下

1645671237081.png

2.3.3 匹配路径

• 2.3.1的yaml中,host替换为如下内容

  - host: www.test.com
    http:
      paths:
      - pathType: Prefix
        path: "/app1"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80

      - pathType: Prefix
        path: "/app2"
        backend:
          service:
            name: magedu-tomcat-app2-service
            port:
              number: 80

• 访问如下

1645671419515.png

1645671438059.png

2.3.4 匹配单域名

• 签发域名证书

openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 3560 -nodes -subj '/CN=www.test.com'
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=www.test.com'
openssl x509 -req -sha256 -days 3650 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

• 证书上传至k8s

kubectl -n linux create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key

• 匹配规则yaml文件

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-web
  namespace: linux
  annotations:
    kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
    nginx.ingress.kubernetes.io/ssl-redirect: 'true' #SSL重定向，即将http请求强制重定向至https，等于nginx中的全站https
spec:
  tls:
  - hosts:
    - www.test.com
    secretName: tls-secret

  rules:
  - host: www.test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80

• 访问如下

1645673459174.png

2.3.5 匹配多域名

• 签发多域名证书

openssl req -new -newkey rsa:4096 -keyout mobile.key -out mobile.csr -nodes -subj '/CN=mobile.test.com'
openssl x509 -req -sha256 -days 3650 -in mobile.csr -CA ca.crt -CAkey ca.key -set_serial 01  -out mobile.crt

针对2.3.4中签发的ca证书，签发多个域名

• 证书上传至k8s

kubectl  create secret generic mobile-tls-secret --from-file=tls.crt=mobile.crt --from-file=tls.key=mobile.key -n linux

• 2.3.4 的yaml中，spec替换为如下内容

spec:
  tls:
  - hosts:
    - www.test.com
    secretName: tls-secret
  - hosts:
    - mobile.test.com
    secretName: mobile-tls-secret
  rules:
  - host: www.test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app1-service
            port:
              number: 80


  - host: mobile.test.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: magedu-tomcat-app2-service
            port:
              number: 80

• 访问如下

1645673459174.png

1645673765032.png

三、资源限制

3.1 限制资源

• 限制容器cpu和内存

        resources:
          limits:
            cpu: "1.2"
            memory: "512Mi"
          requests:
            memory: "100Mi"
            cpu: "500m"

requests的值一定要小于等于limits的值，不然创建时会报错；加上limits，容器最大允许使用的资源上限就是limits中设置的值

• limitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: limitrange-magedu
  namespace: linux
spec:
  limits:
  - type: Container       #限制的资源类型
    max:
      cpu: "2"            #限制单个容器的最大CPU
      memory: "2Gi"       #限制单个容器的最大内存
    min:
      cpu: "500m"         #限制单个容器的最小CPU
      memory: "512Mi"     #限制单个容器的最小内存
    default:
      cpu: "500m"         #默认单个容器的CPU限制
      memory: "512Mi"     #默认单个容器的内存限制
    defaultRequest:
      cpu: "500m"         #默认单个容器的CPU创建请求
      memory: "512Mi"     #默认单个容器的内存创建请求
    maxLimitRequestRatio:
      cpu: 2              #限制CPU limit/request比值最大为2
      memory: 2         #限制内存limit/request比值最大为1.5
  - type: Pod
    max:
      cpu: "4"            #限制单个Pod的最大CPU
      memory: "4Gi"       #限制单个Pod最大内存
  - type: PersistentVolumeClaim
    max:
      storage: 50Gi        #限制PVC最大的requests.storage
    min:
      storage: 30Gi        #限制PVC最小的requests.storage

创建后可以指定命名空间使用describe命令查看；如果设置的资源超过limitrange中规定的值，kubectl创建的时候不会报错，但是pod由于资源超过限制，起不来，不会被看到，但是没有报错信息；需要使用describe命令查看deployment输出yaml或者json格式内容，message有提示信息

• ResourceQuato

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota-magedu
  namespace: linux
spec:
  hard:
    requests.cpu: "8" # 最小CPU
    limits.cpu: "8" # 最大CPU
    requests.memory: 4Gi # 最小内存
    limits.memory: 4Gi # 最大内存
    requests.nvidia.com/gpu: 4 # 最小英伟达GPU
    pods: "2" # 最多pod数量
    services: "6" # 最多service数量

ResourceQuato是针对命名空间的资源限制

四、RBAC授权

4.1 创建用户授权

• 在linux命名空间下创建账户linux

kubectl -n linux create serviceaccount linux

k8s在创建用户的时候，会给用户分配一个token，这个token会一直伴随着用户，除非用户被删除重建，否则token不会变，secret名字可以使用get secret进行查看，describe查看token字符串

• 创建角色

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: linux
  name: linux-role
rules:
- apiGroups: ["*"]
  resources: ["pods","pods/exec"]
  verbs: ["*"]
- apiGroups: ["extensions", "apps/v1"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

可以使用describe进行查看

• 创建角色绑定

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: role-bind-magedu
  namespace: magedu
subjects:
- kind: ServiceAccount
  name: magedu
  namespace: magedu
roleRef:
  kind: Role
  name: magedu-role
  apiGroup: rbac.authorization.k8s.io

可以使用describe进行查看

4.2 限制用户授权

• 创建角色仅有查看权限，替换4.1创建角色中的rules为如下内容

rules:
- apiGroups: ["*"]
  resources: ["pods","pods/exec"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions", "apps/v1"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]

使用token登陆dashboard时，没有权限删除

4.3 生成kubeconfig文件

• 创建csr文件

cat >> linux-csr.json <

• 签发证书

cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem  -ca-key=/etc/kubernetes/ssl/ca-key.pem -config= /etc/kubeasz/clusters/cluster1/ssl/ca-config.json  -profile=kubernetes linux-csr.json | cfssljson -bare  linux

• 生成普通用户kubeconfig文件

kubectl config set-cluster cluster1 --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.204.101:6443 --kubeconfig=magedu.kubeconfig

• 设置客户端认证参数

kubectl config set-credentials linux \
--client-certificate=linux.pem \
--client-key=linux-key.pem \
--embed-certs=true \
--kubeconfig=linux.kubeconfig

• 设置上下文参数（多集群使用上下文区分）

kubectl config set-context cluster1 \
--cluster=cluster1 \
--user=linux \
--namespace=linux \
--kubeconfig=linux.kubeconfig

https://kubernetes.io/zh/docs/concepts/configuration/organize-cluster-access-kubeconfig/

• 设置默认上下文

kubectl config use-context cluster1 --kubeconfig=magedu.kubeconfig

• 查看token

kubectl -n linux describe secret `kubectl -n linux get secret | grep linux | awk '{print $1}'`

• 编辑linux.kubeconfig，最后边追加写入token

    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkxpeWdUSzlaQ1AtMldqNjhwXzdaeVFJWEFHdnhLNEI5bGJ2UTZtaEcyMDQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYWdlZHUiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoibWFnZWR1LXRva2VuLWx6Y3g4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1hZ2VkdSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjkwZjZkZTI5LWJlZjMtNGVlOC04MGMxLWI2OWZjZGE2N2IxZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptYWdlZHU6bWFnZWR1In0.G2JTuh3B9ncIt8sumk22oD13rUyzvHPtukudvtYZ7W6L_Tn1jRY8YldZKXB9PnejChm2_O3GWh84CCWr2kpad2ELH3nTFMdzFNig-sfoaubt19ZabdLRch1Pd9wu-4YWxPPjUxi3ZQvnxo-TIJ26k_Y5MVQuc81HW2NgvzFGTg4jh6Uusd12uz9HT7Z_JQn7CgSZLg2OrbAuq7OgUVqBqOpoVkN1CXsD6qu_xC7c_dvVsYmU9u-W8VFu4ScKNK1G1P77wsuIiBNN543wJ53dXTePDOrWJbkZvyDBfNFd4PaCCBCl9GVA8GlWVOWiV3A_xFh3D-ZTlFDRLLgJZAI5cQ

五、日志收集

5.1 镜像准备

• 制作自定义基础系统镜像包centos

添加filebeat到基础镜像包，后续收集日志使用

• 基于centos制作自定义jdk镜像

添加环境变量，以便能够全局执行java命令

• 基于jdk制作tomcat镜像

创建日志挂载目录

• 基于tomcat制作业务镜像

添加filebeat.yml文件，让filebeat读取配置并生效

5.2 搭建中间件

5.2.1 elasticsearch

• 安装elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.0-amd64.deb
dpkg -i elasticsearch-7.16.0-amd64.deb

修改配置文件/etc/elasticsearch/elasticsearch.yml（版本：elasticsearch-7.16.0-amd64.deb）

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-es
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: IP
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["IP"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["IP"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true
#
# ---------------------------------- Security ----------------------------------
#
#                                 *** WARNING ***
#
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
#
# To protect your data, we strongly encourage you to enable the Elasticsearch security features.
# Refer to the following documentation for instructions.
#
# https://www.elastic.co/guide/en/elasticsearch/reference/7.16/configuring-stack-security.html

• 重启elasticsearch

systemctl restart elasticsearch.service

5.2.2 logstash

• 安装logstach

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.16.0-amd64.deb
dpkg -i logstash-7.16.0-amd64.deb

修改配置文件 /etc/logstash/logstash.yml（版本：logstash-7.16.0-amd64.deb）

这里边只有数据路径和日志路径，可以不用修改

在/etc/logstash/conf.d/目录下编写json配置文件，文件名字可以自定义（kafka-to-es.conf），指定输入的kafka集群和topic，根据类日志型判断输然后出到es集群中

input {
  kafka {
    bootstrap_servers => "KAFKA_IP:9092"
    topics => ["TOPIC"]
    codec => "json"
  }
}

output{
  if [fields][type] == "tomcat-accesslog" {
    elasticsearch {
      hosts => ["ES_IP:9200"]
      index => "accesslog-%{+YYYY.MM.dd}"
    }
  }

  if [fields][type] == "tomcat-catalina" {
    elasticsearch {
      hosts => ["ES_IP:9200"]
      index => "catalinalog-%{+YYYY.MM.dd}"
    }
  }
}

• 重启logstash

systemctl restart logstash.service

5.2.3 kibana

• 安装kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.16.0-amd64.deb
dpkg -i kibana-7.16.0-amd64.deb

修改配置文件/etc/kibana/kibana.yml（版本：kibana-7.16.0-amd64.deb）

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "192.168.204.113"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
server.maxPayload: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "elasticsearch"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://192.168.204.113:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"

# The default application to load.
kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Kibana can also authenticate to Elasticsearch via "service account tokens".
# If may use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
i18n.locale: "zh-CN"

• 重启kibana

systemctl restart kibana.service

5.2.4 zookeeper

• 安装java

apt install -y openjdk-8-jdk

• 安装zookeeper

wget https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.5.9/apache-zookeeper-3.5.9-bin.tar.gz
tar xf apache-zookeeper-3.5.9-bin.tar.gz

复制配置文件并重命名为zoo.cfg，修改配置文件

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/data/zookeeper/zkdata
dataLogDir=/data/zookeeper/zklogs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=IP:2888:3888

• 启动zookeeper

./bin/kafka-server-start.sh -daemon ./config/server.properties

5.2.5 kafka

• 安装kafka

wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz
tar xf kafka_2.13-3.0.0.tgz

修改配置文件

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://192.168.204.111:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=1073741824


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/data/kafka/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=IP:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

• 启动kafka

./bin/zkServer.sh start

5.2.6 filebeat

• 安装filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.1-x86_64.rpm
rpm -ivh filebeat-7.12.1-x86_64.rpm

由于镜像是基于centos系统的，所以下载的镜像是rpm包

修改配置文件/etc/filebeat/filebeat.yml（版本：filebeat-7.12.1-x86_64.rpm）

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /apps/tomcat/logs/catalina.out
  fields:
    type: tomcat-catalina
- type: log
  enabled: true
  paths:
    - /apps/tomcat/logs/localhost_access_log.*.txt
  fields:
    type: tomcat-accesslog
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:

# 连接kafka报错时，日志可以打开debug模式，方便定位失败原因
#logging.level

output.kafka:
  hosts: ["192.168.204.111:9092"]
  required_acks: 1
  topic: "magedu-n60-app1"
  compression: gzip
  max_message_bytes: 1000000

• 启动filebeat

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

-e选项是将日志标准输出，并且不写日志文件；-c选项是执行配置文件

5.3 查看日志

• 日志收集逻辑图

1646293470844.png

业务容器中产生业务日志，通过filebeat客户端收集，传输到kafka集群，kafka依赖zookeeper集群，经由logstash进行逻辑判断，对业务类型进行分类，存入elasticsearch集群中，由kibana展示出来

• kafka查看topic

/opt/kafka_2.12-3.0.0/bin/kafka-topics.sh --list --bootstrap-server localhost:9092 

1646363948681.png

• kafka消费者指定topic查看日志

/opt/kafka_2.12-3.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TOPIC

1646362018263.png

filebeat客户端收集日志到kafka的时候，需要注意kafka的版本，kafka版本太低，会报错

• 查看es日志

1646361828162.png

谷歌浏览器插件elasticsearch head，如果谷歌浏览器找不到，或者找到插件不可用，可以用谷歌双核浏览器，应用市场中可以搜索到

• 通过kibana展示日志

1646362242337.png

1646362344096.png