shuyang
shuyang

实验文档4：kubernetes集群的监控和日志分析

改造dubbo-demo-web项目为Tomcat启动项目

Tomcat官网

准备Tomcat的镜像底包

准备tomcat二进制包

运维主机HDSS7-200.host.com上：
Tomcat8下载链接

复制/opt/src 
1
2
3
4
 
[root@hdss7-200 src]# ls -l|grep tomcat
-rw-r--r-- 1 root root   9690027 Apr 10 22:57 apache-tomcat-8.5.40.tar.gz
[root@hdss7-200 src]# mkdir -p /data/dockerfile/tomcat8 && tar xf apache-tomcat-8.5.40.tar.gz -C /data/dockerfile/tomcat
[root@hdss7-200 src]# cd /data/dockerfile/tomcat8 && rm -fr apache-tomcat-8.5.40/webapps/*

 

简单配置tomcat

  1. 关闭AJP端口
复制/data/dockerfile/tomcat/apache-tomcat-8.5.40/conf/server.xml 
1

  1. 配置日志
  • 删除3manager，4host-manager的handlers
复制/data/dockerfile/tomcat/apache-tomcat-8.5.40/conf/logging.properties 
1
 
handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler,java.util.logging.ConsoleHandler
  • 日志级别改为INFO
复制/data/dockerfile/tomcat/apache-tomcat-8.5.40/conf/logging.properties 
1
2
3
 
1catalina.org.apache.juli.AsyncFileHandler.level = INFO
2localhost.org.apache.juli.AsyncFileHandler.level = INFO
java.util.logging.ConsoleHandler.level = INFO
  • 注释掉所有关于3manager，4host-manager日志的配置
复制/data/dockerfile/tomcat/apache-tomcat-8.5.40/conf/logging.properties 
1
2
3
4
5
6
7
8
9
 
#3manager.org.apache.juli.AsyncFileHandler.level = FINE
#3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
#3manager.org.apache.juli.AsyncFileHandler.prefix = manager.
#3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

#4host-manager.org.apache.juli.AsyncFileHandler.level = FINE
#4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs
#4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.
#4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

准备Dockerfile

复制/data/dockerfile/tomcat/Dockerfile 
1
2
3
4
5
6
7
8
9
10
11
 
From stanleyws/jre8:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ 
    echo 'Asia/Shanghai' >/etc/timezone
ENV CATALINA_HOME /opt/tomcat
ENV LANG zh_CN.UTF-8
ADD apache-tomcat-8.5.40/ /opt/tomcat
ADD config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
WORKDIR /opt/tomcat
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]
  • config.yml
  • jmx_javaagent-0.3.1.jar
  • entrypoint.sh

vi config.yml

复制 
1
2
3
 
—
rules:
  - pattern: '.*'

制作镜像并推送

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
[root@hdss7-200 tomcat]# docker build . -t harbor.od.com/base/tomcat:v8.5.40
Sending build context to Docker daemon 9.502 MB
Step 1 : FROM stanleyws/jre8:8u112
 ---> fa3a085d6ef1
Step 2 : RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo 'Asia/Shanghai' >/etc/timezone
 ---> Using cache
 ---> 5da5ab0b1a48
Step 3 : ENV CATALINA_HOME /opt/tomcat
 ---> Running in edf8f2bbeae3
 ---> 05c7b829c8ab
Removing intermediate container edf8f2bbeae3
Step 4 : ENV LANG zh_CN.UTF-8
 ---> Running in 50516133f65b
 ---> 421d67c4c188
Removing intermediate container 50516133f65b
Step 5 : ADD apache-tomcat-8.5.40/ /opt/tomcat
 ---> 460a5d0ef93b
Removing intermediate container 0be2b4897d23
Step 6 : ADD config.yml /opt/prom/config.yml
 ---> 04c9d4679f0d
Removing intermediate container 0e88a4ed2088
Step 7 : ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar
 ---> 06e4a78e9cbf
Removing intermediate container 1e6b2919be00
Step 8 : WORKDIR /opt/tomcat
 ---> Running in a51676d15a01
 ---> e8d164847eb3
Removing intermediate container a51676d15a01
Step 9 : ADD entrypoint.sh /entrypoint.sh
 ---> 0522db421536
Removing intermediate container 7f0be318fde8
Step 10 : CMD /entrypoint.sh
 ---> Running in c2df6e511c00
 ---> 8d735515bb42
Removing intermediate container c2df6e511c00
Successfully built 8d735515bb42

[root@hdss7-200 tomcat8]# docker push harbor.od.com/base/tomcat:v8.5.40
The push refers to a repository [harbor.od.com/base/tomcat]
498eaadf86a8: Pushed 
fab679acf269: Pushed 
0e65f86c3a75: Pushed 
938c8a5617cc: Pushed 
052016a734be: Mounted from app/dubbo-demo-web 
0690f10a63a5: Mounted from app/dubbo-demo-web 
c843b2cf4e12: Mounted from app/dubbo-demo-web 
fddd8887b725: Mounted from base/jre8 
42052a19230c: Mounted from base/jre8 
8d4d1ab5ff74: Mounted from base/jre8 
v8.5.40: digest: sha256:407c6abd7c4fa5119376efa71090b49637d7a52ef2fc202f4019ab4c91f6dc50 size: 2409

改造dubbo-demo-web项目

修改dubbo-client/pom.xml

复制/d/workspace/dubbo-demo-web/dubbo-client/pom.xml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
war


  org.springframework.boot
  spring-boot-starter-web
 
 
 org.springframework.boot
 spring-boot-starter-tomcat
 
 



 org.apache.tomcat 
 tomcat-servlet-api
 8.0.36
 provided

修改Application.java

复制/d/workspace/dubbo-demo-web/dubbo-client/src/main/java/com/od/dubbotest/Application.java 
1
2
3
4
5
6
 
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.context.annotation.ImportResource;

@ImportResource(value={"classpath*:spring-config.xml"})
@EnableAutoConfiguration(exclude={DataSourceAutoConfiguration.class})

创建ServletInitializer.java

复制/d/workspace/dubbo-demo-web/dubbo-client/src/main/java/com/od/dubbotest/ServletInitializer.java 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
package com.od.dubbotest;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.context.web.SpringBootServletInitializer;
import com.od.dubbotest.Application;

public class ServletInitializer extends SpringBootServletInitializer {

 @Override
 protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
 return builder.sources(Application.class);
 }
}

新建Jenkins的pipeline

配置New job

  • 使用admin登录
  • New Item
  • create new jobs

  • Enter an item name

    tomcat-demo

  • Pipeline -> OK

  • Discard old builds

    Days to keep builds : 3
    Max # of builds to keep : 30

  • This project is parameterized

  1. Add Parameter -> String Parameter

    Name : app_name
    Default Value :
    Description : project name. e.g: dubbo-demo-web

  2. Add Parameter -> String Parameter

    Name : image_name
    Default Value :
    Description : project docker image name. e.g: app/dubbo-demo-web

  3. Add Parameter -> String Parameter

    Name : git_repo
    Default Value :
    Description : project git repository. e.g: [email protected]:stanleywang/dubbo-demo-web.git

  4. Add Parameter -> String Parameter

    Name : git_ver
    Default Value : tomcat
    Description : git commit id of the project.

  5. Add Parameter -> String Parameter

    Name : add_tag
    Default Value :
    Description : project docker image tag, date_timestamp recommended. e.g: 190117_1920

  6. Add Parameter -> String Parameter

    Name : mvn_dir
    Default Value : ./
    Description : project maven directory. e.g: ./

  7. Add Parameter -> String Parameter

    Name : target_dir
    Default Value : ./dubbo-client/target
    Description : the relative path of target file such as .jar or .war package. e.g: ./dubbo-client/target

  8. Add Parameter -> String Parameter

    Name : mvn_cmd
    Default Value : mvn clean package -Dmaven.test.skip=true
    Description : maven command. e.g: mvn clean package -e -q -Dmaven.test.skip=true

  9. Add Parameter -> Choice Parameter

    Name : base_image
    Default Value :

    • base/tomcat:v7.0.94
    • base/tomcat:v8.5.40
    • base/tomcat:v9.0.17
      Description : project base image list in harbor.od.com.

  10. Add Parameter -> Choice Parameter

    Name : maven
    Default Value :

    • 3.6.0-8u181
    • 3.2.5-6u025
    • 2.2.1-6u025
      Description : different maven edition.

  11. Add Parameter -> String Parameter

    Name : root_url
    Default Value : ROOT
    Description : webapp dir.

实验文档4：kubernetes集群的监控和日志分析_第1张图片

Pipeline Script

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 
pipeline {
  agent any 
    stages {
    stage('pull') { //get project code from repo 
      steps {
 sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_ver}"
 }
 }
 stage('build') { //exec mvn cmd
 steps {
 sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"
 }
 }
 stage('unzip') { //unzip target/*.war -c target/project_dir
 steps {
 sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && unzip *.war -d ./project_dir"
 }
 }
 stage('image') { //build image and push to registry
 steps {
 writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
ADD ${params.target_dir}/project_dir /opt/tomcat/webapps/${params.root_url}"""
 sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build -t harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag} . && docker push harbor.od.com/${params.image_name}:${params.git_ver}_${params.add_tag}"
 }
 }
 }
}

构建应用镜像

使用Jenkins进行CI，并查看harbor仓库
实验文档4：kubernetes集群的监控和日志分析_第2张图片
实验文档4：kubernetes集群的监控和日志分析_第3张图片

准备k8s的资源配置清单

不再需要单独准备资源配置清单

应用资源配置清单

k8s的dashboard上直接修改image的值为jenkins打包出来的镜像
文档里的例子是：harbor.od.com/app/dubbo-demo-web:tomcat_190119_1920

浏览器访问

http://demo.od.com?hello=wangdao

检查tomcat运行情况

任意一台运算节点主机上：

复制 
1
2
3
4
5
6
7
8
9
 
[root@hdss7-22 ~]# kubectl get pods -n app
NAME                                                    READY     STATUS    RESTARTS   AGE
dubbo-demo-consumer-v025-htfx8                          2/2       Running   0          1h
[root@hdss7-22 ~]# kubectl exec -ti dubbo-demo-consumer-v025-htfx8 -n app bash
dubbo-demo-consumer-v025-htfx8:/opt/tomcat#  ls -lsr logs/
total 16
-rw-r----- 1 root root 7435 Jan 19 19:26 catalina.2019-01-19.log
-rw-r----- 1 root root 629 Jan 19 19:26 localhost.2019-01-19.log
-rw-r----- 1 root root 249 Jan 15 19:27 localhost_access_log.2019-01-19.txt

 

使用Prometheus和Grafana监控kubernetes集群

部署kube-state-metrics

运维主机HDSS7-200.host.com

准备kube-state-metrics镜像

kube-state-metrics官方quay.io地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
 
[root@hdss7-200 ~]# docker pull quay.io/coreos/kube-state-metrics:v1.5.0
v1.5.0: Pulling from coreos/kube-state-metrics
cd784148e348: Pull complete 
f622528a393e: Pull complete 
Digest: sha256:b7a3143bd1eb7130759c9259073b9f239d0eeda09f5210f1cd31f1a530599ea1
Status: Downloaded newer image for quay.io/coreos/kube-state-metrics:v1.5.0
[root@hdss7-200 ~]# docker tag 91599517197a harbor.od.com/k8s/kube-state-metrics:v1.5.0
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/kube-state-metrics:v1.5.0
docker push harbor.od.com/k8s/kube-state-metrics:v1.5.0
The push refers to a repository [harbor.od.com/k8s/kube-state-metrics]
5b3c36501a0a: Pushed 
7bff100f35cb: Pushed 
v1.5.0: digest: sha256:0d9bea882f25586543254d9ceeb54703eb6f8f94c0dd88875df216b6a0f60589 size: 739

 

准备资源配置清单

  • RBAC
  • Deployment

vi /data/k8s-yaml/kube-state-metrics/rbac.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
 
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: kube-state-metrics
 namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 labels:
 addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: kube-state-metrics
rules:
- apiGroups:
 - ""
 resources:
 - configmaps
 - secrets
 - nodes
 - pods
 - services
 - resourcequotas
 - replicationcontrollers
 - limitranges
 - persistentvolumeclaims
 - persistentvolumes
 - namespaces
 - endpoints
 verbs:
 - list
 - watch
- apiGroups:
 - extensions
 resources:
 - daemonsets
 - deployments
 - replicasets
 verbs:
 - list
 - watch
- apiGroups:
 - apps
 resources:
 - statefulsets
 verbs:
 - list
 - watch
- apiGroups:
 - batch
 resources:
 - cronjobs
 - jobs
 verbs:
 - list
 - watch
- apiGroups:
 - autoscaling
 resources:
 - horizontalpodautoscalers
 verbs:
 - list
 - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 labels:
 addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: kube-state-metrics
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: kube-state-metrics
subjects:
- kind: ServiceAccount
 name: kube-state-metrics
 namespace: kube-system

vi /data/k8s-yaml/kube-state-metrics/deployment.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
 labels:
 grafanak8sapp: "true"
 app: kube-state-metrics
 name: kube-state-metrics
 namespace: kube-system
spec:
 selector:
 matchLabels:
 grafanak8sapp: "true"
 app: kube-state-metrics
 strategy:
 rollingUpdate:
 maxSurge: 25%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 creationTimestamp: null
 labels:
 grafanak8sapp: "true"
 app: kube-state-metrics
 spec:
 containers:
 - image: harbor.od.com/k8s/kube-state-metrics:v1.5.0
 name: kube-state-metrics
 ports:
 - containerPort: 8080
 name: http-metrics
 protocol: TCP
 readinessProbe:
 failureThreshold: 3
 httpGet:
 path: /healthz
 port: 8080
 scheme: HTTP
 initialDelaySeconds: 5
 periodSeconds: 10
 successThreshold: 1
 timeoutSeconds: 5
 imagePullPolicy: IfNotPresent
 imagePullSecrets:
 - name: harbor
 restartPolicy: Always
 serviceAccount: kube-state-metrics
 serviceAccountName: kube-state-metrics

应用资源配置清单

任意一台运算节点上：

复制 
1
2
3
4
5
6
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/rbac.yaml 
serviceaccount/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/deployment.yaml 
deployment.extensions/kube-state-metrics created

 

检查启动情况

复制 
1
2
 
[root@hdss7-21 ~]# kubectl get pods -n kube-system|grep kube-state-metrics
kube-state-metrics-645bd94c55-wdtqh      1/1     Running   0          94s

部署node-exporter

运维主机HDSS7-200.host.com上：

准备node-exporter镜像

node-exporter官方dockerhub地址
node-expoerer官方github地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
[root@hdss7-200 ~]# docker pull prom/node-exporter:v0.15.0
v0.15.0: Pulling from prom/node-exporter
0de338cf4258: Pull complete 
f508012419d8: Pull complete 
d764f7880123: Pull complete 
Digest: sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
Status: Downloaded newer image for prom/node-exporter:v0.15.0
[root@hdss7-200 ~]# docker tag b3e7f67a1480 harbor.od.com/k8s/node-exporter:v0.15.0
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/node-exporter:v0.15.0
docker push harbor.od.com/k8s/node-exporter:v0.15.0
The push refers to a repository [harbor.od.com/k8s/node-exporter]
0bf893ee7433: Pushed 
17ab2671f87e: Pushed 
b8873621dfbc: Pushed 
v0.15.0: digest: sha256:4e13dd75f00a6114675ea3e62e61dbd79dcb2205e8f3bbe1f8f8ef2fd3e28113 size: 949

 

准备资源配置清单

复制/data/k8s-yaml/node-exporter/node-exporter-ds.yaml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
 
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: node-exporter
  namespace: kube-system
 labels:
 daemon: "node-exporter"
 grafanak8sapp: "true"
spec:
 selector:
 matchLabels:
 daemon: "node-exporter"
 grafanak8sapp: "true"
 template:
 metadata:
 name: node-exporter
 labels:
 daemon: "node-exporter"
 grafanak8sapp: "true"
 spec:
 volumes:
 - name: proc
 hostPath: 
 path: /proc
 type: ""
 - name: sys
 hostPath:
 path: /sys
 type: ""
 containers:
 - name: node-exporter
 image: harbor.od.com/k8s/node-exporter:v0.15.0
 args:
 - --path.procfs=/host_proc
 - --path.sysfs=/host_sys
 ports:
 - name: node-exporter
 hostPort: 9100
 containerPort: 9100
 protocol: TCP
 volumeMounts:
 - name: sys
 readOnly: true
 mountPath: /host_sys
 - name: proc
 readOnly: true
 mountPath: /host_proc
 imagePullSecrets:
 - name: harbor
 restartPolicy: Always
 hostNetwork: true

应用资源配置清单

任意运算节点上：

复制 
1
2
 
[root@hdss7-21 ~]# kubectl -f http://k8s-yaml.od.com/node-exporter/node-exporter-ds.yaml
daemonset.extensions/node-exporter created

 

部署cadvisor

运维主机HDSS7-200.host.com上：

准备cadvisor镜像

cadvisor官方dockerhub地址
cadvisor官方github地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
[root@hdss7-200 ~]# docker pull google/cadvisor:v0.28.3
v0.28.3: Pulling from google/cadvisor
49388a8c9c86: Pull complete 
21c698e54ae5: Pull complete 
fafc7cbc1edf: Pull complete 
Digest: sha256:09c8d73c9b799d30777763b7029cfd8624b8a1bd33af652ec3b51a6b827d492a
Status: Downloaded newer image for google/cadvisor:v0.28.3
[root@hdss7-200 ~]# docker tag d60fd8aeb74c harbor.od.com/k8s/cadvisor:v0.28.3
[root@hdss7-200 ~]# docker push !$
docker push harbor.od.com/k8s/cadvisor:v0.28.3
The push refers to a repository [harbor.od.com/k8s/cadvisor]
daab541dbbf0: Pushed 
ba67c95cca3d: Pushed 
ef763da74d91: Pushed 
v0.28.3: digest: sha256:319812db86e7677767bf6a51ea63b5bdb17dc18ffa576eb6c8a667e899d1329d size: 951

 

准备资源配置清单

  • DaemonSet

vi /data/k8s-yaml/cadvisor/daemonset.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cadvisor
  namespace: kube-system
 labels:
 app: cadvisor
spec:
 selector:
 matchLabels:
 name: cadvisor
 template:
 metadata:
 labels:
 name: cadvisor
 spec:
 hostNetwork: true
 tolerations:
 - key: node-role.kubernetes.io/master
 effect: NoSchedule
 key: enabledDiskSchedule
 value: "true"
 effect: NoSchedule
 containers:
 - name: cadvisor
 image: harbor.od.com/k8s/cadvisor:v0.28.3
 imagePullPolicy: IfNotPresent
 volumeMounts:
 - name: rootfs
 mountPath: /rootfs
 readOnly: true
 - name: var-run
 mountPath: /var/run
 readOnly: true
 - name: sys
 mountPath: /sys
 readOnly: true
 - name: docker
 mountPath: /var/lib/docker
 readOnly: true
 ports:
 - name: http
 containerPort: 4194
 protocol: TCP
 readinessProbe:
 tcpSocket:
 port: 4194
 initialDelaySeconds: 5
 periodSeconds: 10
 args:
 - --housekeeping_interval=10s
 - --port=4194
 imagePullSecrets:
 - name: harbor
 terminationGracePeriodSeconds: 30
 volumes:
 - name: rootfs
 hostPath:
 path: /
 - name: var-run
 hostPath:
 path: /var/run
 - name: sys
 hostPath:
 path: /sys
 - name: docker
 hostPath:
 path: /data/docker

修改运算节点软连接

所有运算节点上：

复制 
1
2
3
4
5
6
7
8
9
 
[root@hdss7-21 ~]# mount -o remount,rw /sys/fs/cgroup/
[root@hdss7-21 ~]# ln -s /sys/fs/cgroup/cpu,cpuacct/ /sys/fs/cgroup/cpuacct,cpu
[root@hdss7-21 ~]# ll /sys/fs/cgroup/ | grep cpu
total 0
lrwxrwxrwx 1 root root 11 Jan 28 22:41 cpu -> cpu,cpuacct
lrwxrwxrwx 1 root root 11 Jan 28 22:41 cpuacct -> cpu,cpuacct
lrwxrwxrwx 1 root root 27 May 5 11:15 cpuacct,cpu -> /sys/fs/cgroup/cpu,cpuacct/
drwxr-xr-x 8 root root 0 Apr 26 11:06 cpu,cpuacct
drwxr-xr-x 7 root root 0 Jan 28 22:41 cpuset

 

应用资源配置清单

任意运算节点上：

复制 
1
2
3
4
 
[root@hdss7-21 ~]# kubectl -f http://k8s-yaml.od.com/cadvisor/deamon.yaml
daemonset.apps/cadvisor created
[root@hdss7-21 ~]# netstat -luntp|grep 4194
tcp6       0      0 :::4194                 :::*                    LISTEN      49153/cadvisor

 

部署blackbox-exporter

运维主机HDSS7-200.host.com上：

准备blackbox-exporter镜像

blackbox-exporter官方dockerhub地址
blackbox-exporter官方github地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
[root@hdss7-200 ~]# docker pull prom/blackbox-exporter:v0.14.0
v0.14.0: Pulling from prom/blackbox-exporter
697743189b6d: Pull complete 
f1989cfd335b: Pull complete 
8918f7b8f34f: Pull complete 
a128dce6256a: Pull complete 
Digest: sha256:c20445e0cc628fa4b227fe2f694c22a314beb43fd8297095b6ee6cbc67161336
Status: Downloaded newer image for prom/blackbox-exporter:v0.14.0
[root@hdss7-200 ~]# docker tag d3a00aea3a01 harbor.od.com/k8s/blackbox-exporter:v0.14.0
[root@hdss7-200 ~]# docker push harbor.od.com/k8s/blackbox-exporter:v0.14.0
docker push harbor.od.com/k8s/blackbox-exporter:v0.14.0
The push refers to a repository [harbor.od.com/k8s/blackbox-exporter]
256c4aa8ebe5: Pushed 
4b6cc55de649: Pushed 
986894c42222: Mounted from infra/prometheus 
adab5d09ba79: Mounted from infra/prometheus 
v0.14.0: digest: sha256:b127897cf0f67c47496d5cbb7ecb86c001312bddd04192f6319d09292e880a5f size: 1155

 

准备资源配置清单

  • ConfigMap
  • Deployment
  • Service
  • Ingress

vi /data/k8s-yaml/blackbox-exporter/configmap.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: blackbox-exporter
 name: blackbox-exporter
 namespace: kube-system
data:
 blackbox.yml: |-
 modules:
 http_2xx:
 prober: http
 timeout: 2s
 http:
 valid_http_versions: ["HTTP/1.1", "HTTP/2"]
 valid_status_codes: [200,301,302]
 method: GET
 preferred_ip_protocol: "ip4"
 tcp_connect:
 prober: tcp
 timeout: 2s

vi /data/k8s-yaml/blackbox-exporter/deployment.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: blackbox-exporter
  namespace: kube-system
 labels:
 app: blackbox-exporter
 annotations:
 deployment.kubernetes.io/revision: 1
spec:
 replicas: 1
 selector:
 matchLabels:
 app: blackbox-exporter
 template:
 metadata:
 labels:
 app: blackbox-exporter
 spec:
 volumes:
 - name: config
 configMap:
 name: blackbox-exporter
 defaultMode: 420
 containers:
 - name: blackbox-exporter
 image: harbor.od.com/k8s/blackbox-exporter:v0.14.0
 args:
 - --config.file=/etc/blackbox_exporter/blackbox.yml
 - --log.level=debug
 - --web.listen-address=:9115
 ports:
 - name: blackbox-port
 containerPort: 9115
 protocol: TCP
 resources:
 limits:
 cpu: 200m
 memory: 256Mi
 requests:
 cpu: 100m
 memory: 50Mi
 volumeMounts:
 - name: config
 mountPath: /etc/blackbox_exporter
 readinessProbe:
 tcpSocket:
 port: 9115
 initialDelaySeconds: 5
 timeoutSeconds: 5
 periodSeconds: 10
 successThreshold: 1
 failureThreshold: 3
 imagePullPolicy: IfNotPresent
 imagePullSecrets:
 - name: harbor
 restartPolicy: Always

vi /data/k8s-yaml/blackbox-exporter/service.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
 
kind: Service
apiVersion: v1
metadata:
  name: blackbox-exporter
  namespace: kube-system
spec:
 selector:
 app: blackbox-exporter
 ports:
 - protocol: TCP
 port: 9115
 name: http

vi /data/k8s-yaml/blackbox-exporter/ingress.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blackbox-exporter
  namespace: kube-system
spec:
 rules:
 - host: blackbox.od.com
 http:
 paths:
 - backend:
 serviceName: blackbox-exporter
 servicePort: 9115

解析域名

HDSS7-11.host.com上

复制/var/named/od.com.zone 
1
 
blackbox	60 IN A 10.4.7.10

 

应用资源配置清单

任意运算节点上：

复制 
1
2
3
4
5
6
7
8
 
[root@hdss7-21 ~]# kubectl -f http://k8s-yaml.od.com/blackbox-exporter/configmap.yaml
configmap/blackbox-exporter created
[root@hdss7-21 ~]# kubectl -f http://k8s-yaml.od.com/blackbox-exporter/deployment.yaml
deployment.extensions/blackbox-exporter created
[root@hdss7-21 ~]# kubectl -f http://k8s-yaml.od.com/blackbox-exporter/service.yaml
service/blackbox-exporter created
[root@hdss7-21 ~]# kubectl -f http://k8s-yaml.od.com/blackbox-exporter/ingress.yaml
ingress.extensions/blackbox-exporter created

 

浏览器访问

http://blackbox.od.com

实验文档4：kubernetes集群的监控和日志分析_第4张图片

部署prometheus

运维主机HDSS7-200.host.com上：

准备prometheus镜像

prometheus官方dockerhub地址
prometheus官方github地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
 
[root@hdss7-200 ~]# docker pull prom/prometheus:v2.9.1
v2.9.1: Pulling from prom/prometheus
697743189b6d: Pull complete 
f1989cfd335b: Pull complete 
b60c2f039ea7: Pull complete 
6a189f2c500c: Pull complete 
bd6be4aea906: Pull complete 
81b69caae2b5: Pull complete 
5e7226eda004: Pull complete 
564568254ec8: Pull complete 
9bd07902fc4b: Pull complete 
Digest: sha256:e02bb3dec47631b4d31cede2d35ff901d892b57f33144406ee7994e8c94fb2d7
Status: Downloaded newer image for prom/prometheus:v2.9.1
[root@hdss7-200 ~]# docker tag 4737a2d79d1a harbor.od.com/infra/prometheus:v2.9.1
[root@hdss7-200 ~]# docker push harbor.od.com/infra/prometheus:v2.9.1
The push refers to a repository [harbor.od.com/infra/prometheus]
a67e5326fa35: Pushed 
02c0c0b3065f: Pushed 
b7b1f5015c12: Pushed 
38be466f60e1: Pushed 
9c3fb6c27da7: Pushed 
a06c616e78e9: Pushed 
05c0d5c2ae72: Pushed 
986894c42222: Pushed 
adab5d09ba79: Pushed 
v2.9.1: digest: sha256:2357e59541f5596dd90d9f4218deddecd9b4880c1e417a42592b00b30b47b0a9 size: 2198

 

准备资源配置清单

运维主机HDSS7-200.host.com上：

复制/data/k8s-yaml 
1
 
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/prometheus && mkdir -p /data/nfs-volume/prometheus/etc && cd /data/k8s-yaml/prometheus

 

  • RBAC
  • Deployment
  • Service
  • Ingress

vi /data/k8s-yaml/prometheus/rbac.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
 
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: prometheus
 namespace: infra
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 labels:
 addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: prometheus
rules:
- apiGroups:
 - ""
 resources:
 - nodes
 - nodes/metrics
 - services
 - endpoints
 - pods
 verbs:
 - get
 - list
 - watch
- apiGroups:
 - ""
 resources:
 - configmaps
 verbs:
 - get
- nonResourceURLs:
 - /metrics
 verbs:
 - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 labels:
 addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: prometheus
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: prometheus
subjects:
- kind: ServiceAccount
 name: prometheus
 namespace: infra

vi /data/k8s-yaml/prometheus/deployment.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "5"
 labels:
 name: prometheus
 name: prometheus
 namespace: infra
spec:
 progressDeadlineSeconds: 600
 replicas: 1
 revisionHistoryLimit: 7
 selector:
 matchLabels:
 app: prometheus
 strategy:
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 1
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: prometheus
 spec:
 nodeName: 10.4.7.21
 containers:
 - image: harbor.od.com/infra/prometheus:v2.9.1
 args:
 - --config.file=/data/etc/prometheus.yml
 - --storage.tsdb.path=/data/prom-db
 - --storage.tsdb.retention=72h
 command:
 - /bin/prometheus
 name: prometheus
 ports:
 - containerPort: 9090
 protocol: TCP
 resources:
 limits:
 cpu: 500m
 memory: 2500Mi
 requests:
 cpu: 100m
 memory: 100Mi
 volumeMounts:
 - mountPath: /data
 name: data
 imagePullPolicy: IfNotPresent
 imagePullSecrets:
 - name: harbor
 securityContext:
 runAsUser: 0
 dnsPolicy: ClusterFirst
 restartPolicy: Always
 serviceAccount: prometheus
 serviceAccountName: prometheus
 volumes:
 - name: data
 nfs:
 server: hdss7-200
 path: /data/nfs-volume/prometheus

vi /data/k8s-yaml/prometheus/service.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
 
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: infra
spec:
 ports:
 - port: 9090
 protocol: TCP
 name: prometheus
 selector:
 app: prometheus
 type: ClusterIP

 

vi /data/k8s-yaml/prometheus/ingress.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
 name: prometheus
 namespace: infra
spec:
 rules:
 - host: prometheus.od.com
 http:
 paths:
 - backend:
 serviceName: prometheus
 servicePort: 9090

 

 

准备prometheus的配置文件

运算节点HDSS7-21.host.com上：

  • 拷贝证书
复制 
1
2
3
4
5
6
7
 
[root@hdss7-21 ~]# mkdir -pv /data/k8s-volume/prometheus/{etc,prom-db}
mkdir: created directory ‘/data/k8s-volume/prometheus/etc’
mkdir: created directory ‘/data/k8s-volume/prometheus/prom-db’
[root@hdss7-21 ~]# cd /data/k8s-volume/prometheus/etc
[root@hdss7-21 ~]# scp [email protected]:/opt/certs/ca.pem .
[root@hdss7-21 ~]# scp [email protected]:/opt/certs/client.pem .
[root@hdss7-21 ~]# scp [email protected]:/opt/certs/client-key.pem .
  • 准备配置
复制/data/k8s-volume/prometheus/etc/prometheus.yml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
 
global:
  scrape_interval:     15s
  evaluation_interval: 15s
scrape_configs:
- job_name: 'etcd'
 tls_config:
 ca_file: /data/etc/ca.pem
 cert_file: /data/etc/client.pem
 key_file: /data/etc/client-key.pem
 scheme: https
 static_configs:
 - targets:
 - '10.4.7.12:2379'
 - '10.4.7.21:2379'
 - '10.4.7.22:2379'
- job_name: 'kubernetes-apiservers'
 kubernetes_sd_configs:
 - role: endpoints
 scheme: https
 tls_config:
 ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
 bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
 relabel_configs:
 - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
 action: keep
 regex: default;kubernetes;https
- job_name: 'kubernetes-pods'
 kubernetes_sd_configs:
 - role: pod
 relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
 action: keep
 regex: true
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
 action: replace
 target_label: __metrics_path__
 regex: (.+)
 - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
 action: replace
 regex: ([^:]+)(?::\d+)?;(\d+)
 replacement: $1:$2
 target_label: __address__
 - action: labelmap
 regex: __meta_kubernetes_pod_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: kubernetes_namespace
 - source_labels: [__meta_kubernetes_pod_name]
 action: replace
 target_label: kubernetes_pod_name
- job_name: 'kubernetes-kubelet'
 kubernetes_sd_configs:
 - role: node
 relabel_configs:
 - action: labelmap
 regex: __meta_kubernetes_node_label_(.+)
 - source_labels: [__meta_kubernetes_node_name]
 regex: (.+)
 target_label: __address__
 replacement: ${1}:10255
- job_name: 'kubernetes-cadvisor'
 kubernetes_sd_configs:
 - role: node
 relabel_configs:
 - action: labelmap
 regex: __meta_kubernetes_node_label_(.+)
 - source_labels: [__meta_kubernetes_node_name]
 regex: (.+)
 target_label: __address__
 replacement: ${1}:4194
- job_name: 'kubernetes-kube-state'
 kubernetes_sd_configs:
 - role: pod
 relabel_configs:
 - action: labelmap
 regex: __meta_kubernetes_pod_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: kubernetes_namespace
 - source_labels: [__meta_kubernetes_pod_name]
 action: replace
 target_label: kubernetes_pod_name
 - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
 regex: .*true.*
 action: keep
 - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
 regex: 'node-exporter;(.*)'
 action: replace
 target_label: nodename
- job_name: 'blackbox_http_pod_probe'
 metrics_path: /probe
 kubernetes_sd_configs:
 - role: pod
 params:
 module: [http_2xx]
 relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_blackbox_scheme]
 action: keep
 regex: http
 - source_labels: [__address__, __meta_kubernetes_pod_annotation_blackbox_port, __meta_kubernetes_pod_annotation_blackbox_path]
 action: replace
 regex: ([^:]+)(?::\d+)?;(\d+);(.+)
 replacement: $1:$2$3
 target_label: __param_target
 - action: replace
 target_label: __address__
 replacement: blackbox-exporter.kube-system:9115
 - source_labels: [__param_target]
 target_label: instance
 - action: labelmap
 regex: __meta_kubernetes_pod_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: kubernetes_namespace
 - source_labels: [__meta_kubernetes_pod_name]
 action: replace
 target_label: kubernetes_pod_name
- job_name: 'blackbox_tcp_pod_probe'
 metrics_path: /probe
 kubernetes_sd_configs:
 - role: pod
 params:
 module: [tcp_connect]
 relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_blackbox_scheme]
 action: keep
 regex: tcp
 - source_labels: [__address__, __meta_kubernetes_pod_annotation_blackbox_port]
 action: replace
 regex: ([^:]+)(?::\d+)?;(\d+)
 replacement: $1:$2
 target_label: __param_target
 - action: replace
 target_label: __address__
 replacement: blackbox-exporter.kube-system:9115
 - source_labels: [__param_target]
 target_label: instance
 - action: labelmap
 regex: __meta_kubernetes_pod_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: kubernetes_namespace
 - source_labels: [__meta_kubernetes_pod_name]
 action: replace
 target_label: kubernetes_pod_name
- job_name: 'traefik'
 kubernetes_sd_configs:
 - role: pod
 relabel_configs:
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
 action: keep
 regex: traefik
 - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
 action: replace
 target_label: __metrics_path__
 regex: (.+)
 - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
 action: replace
 regex: ([^:]+)(?::\d+)?;(\d+)
 replacement: $1:$2
 target_label: __address__
 - action: labelmap
 regex: __meta_kubernetes_pod_label_(.+)
 - source_labels: [__meta_kubernetes_namespace]
 action: replace
 target_label: kubernetes_namespace
 - source_labels: [__meta_kubernetes_pod_name]
 action: replace
 target_label: kubernetes_pod_name

应用资源配置清单

任意运算节点上：

复制 
1
2
3
4
5
6
7
8
9
10
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/rbac.yaml
serviceaccount/prometheus created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/deployment.yaml
deployment.extensions/prometheus created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/service.yaml
service/prometheus created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/prometheus/ingress.yaml
ingress.extensions/prometheus created

 

解析域名

HDSS7-11.host.com上

复制/var/named/od.com.zone 
1
 
prometheus	60 IN A 10.4.7.10

 

浏览器访问

http://prometheus.od.com

Prometheus监控内容

Targets（jobs）

etcd

监控etcd服务

key value
etcd_server_has_leader 1
etcd_http_failed_total 1
… …

kubernetes-apiserver

监控apiserver服务

kubernetes-kubelet

监控kubelet服务

kubernetes-kube-state

监控基本信息

  • node-exporter

    监控Node节点信息

  • kube-state-metrics

    监控pod信息

traefik

监控traefik-ingress-controller

key value
traefik_entrypoint_requests_total{code=”200”,entrypoint=”http”,method=”PUT”,protocol=”http”} 138
traefik_entrypoint_requests_total{code=”200”,entrypoint=”http”,method=”GET”,protocol=”http”} 285
traefik_entrypoint_open_connections{entrypoint=”http”,method=”PUT”,protocol=”http”} 1
… …

注意：在traefik的pod控制器上加annotations，并重启pod，监控生效
配置范例：

复制 
1
2
3
4
5
 
"annotations": {
  "prometheus_io_scheme": "traefik",
  "prometheus_io_path": "/metrics",
  "prometheus_io_port": "8080"
}

 

blackbox*

监控服务是否存活

  • blackbox_tcp_pod_porbe

    监控tcp协议服务是否存活

key value
probe_success 1
probe_ip_protocol 4
probe_failed_due_to_regex 0
probe_duration_seconds 0.000597546
probe_dns_lookup_time_seconds 0.00010898

注意：在pod控制器上加annotations，并重启pod，监控生效
配置范例：

复制 
1
2
3
4
 
"annotations": {
  "blackbox_port": "20880",
  "blackbox_scheme": "tcp"
}

 

  • blackbox_http_pod_probe

    监控http协议服务是否存活

key value
probe_success 1
probe_ip_protocol 4
probe_http_version 1.1
probe_http_status_code 200
probe_http_ssl 0
probe_http_redirects 1
probe_http_last_modified_timestamp_seconds 1.553861888e+09
probe_http_duration_seconds{phase=”transfer”} 0.000238343
probe_http_duration_seconds{phase=”tls”} 0
probe_http_duration_seconds{phase=”resolve”} 5.4095e-05
probe_http_duration_seconds{phase=”processing”} 0.000966104
probe_http_duration_seconds{phase=”connect”} 0.000520821
probe_http_content_length 716
probe_failed_due_to_regex 0
probe_duration_seconds 0.00272609
probe_dns_lookup_time_seconds 5.4095e-05

注意：在pod控制器上加annotations，并重启pod，监控生效
配置范例：

复制 
1
2
3
4
5
 
"annotations": {
  "blackbox_path": "/",
  "blackbox_port": "8080",
  "blackbox_scheme": "http"
}

 

kubernetes-pods*

监控JVM信息

key value
jvm_info{version=”1.7.0_80-b15”,vendor=”Oracle Corporation”,runtime=”Java(TM) SE Runtime Environment”,} 1.0
jmx_config_reload_success_total 0.0
process_resident_memory_bytes 4.693897216E9
process_virtual_memory_bytes 1.2138840064E10
process_max_fds 65536.0
process_open_fds 123.0
process_start_time_seconds 1.54331073249E9
process_cpu_seconds_total 196465.74
jvm_buffer_pool_used_buffers{pool=”mapped”,} 0.0
jvm_buffer_pool_used_buffers{pool=”direct”,} 150.0
jvm_buffer_pool_capacity_bytes{pool=”mapped”,} 0.0
jvm_buffer_pool_capacity_bytes{pool=”direct”,} 6216688.0
jvm_buffer_pool_used_bytes{pool=”mapped”,} 0.0
jvm_buffer_pool_used_bytes{pool=”direct”,} 6216688.0
jvm_gc_collection_seconds_sum{gc=”PS MarkSweep”,} 1.867
… …

注意：在pod控制器上加annotations，并重启pod，监控生效
配置范例：

复制 
1
2
3
4
5
 
"annotations": {
  "prometheus_io_scrape": "true",
  "prometheus_io_port": "12346",
  "prometheus_io_path": "/"
}

 

修改traefik服务接入prometheus监控

dashboard上：
kube-system名称空间->daemonset->traefik-ingress-controller->spec->template->metadata下，添加

复制 
1
2
3
4
5
 
"annotations": {
  "prometheus_io_scheme": "traefik",
  "prometheus_io_path": "/metrics",
  "prometheus_io_port": "8080"
}

 

删除pod，重启traefik，观察监控

继续添加blackbox监控配置项

复制 
1
2
3
4
5
6
7
8
 
"annotations": {
  "prometheus_io_scheme": "traefik",
  "prometheus_io_path": "/metrics",
  "prometheus_io_port": "8080",
  "blackbox_path": "/",
 "blackbox_port": "8080",
 "blackbox_scheme": "http"
}

 

修改dubbo-service服务接入prometheus监控

dashboard上：
app名称空间->deployment->dubbo-demo-service->spec->template=>metadata下，添加

复制 
1
2
3
4
5
6
7
 
"annotations": {
  "prometheus_io_scrape": "true",
  "prometheus_io_path": "/",
  "prometheus_io_port": "12346",
  "blackbox_port": "20880",
 "blackbox_scheme": "tcp"
}

 

删除pod，重启traefik，观察监控

修改dubbo-consumer服务接入prometheus监控

app名称空间->deployment->dubbo-demo-consumer->spec->template->metadata下，添加

复制 
1
2
3
4
5
6
7
8
 
"annotations": {
  "prometheus_io_scrape": "true",
  "prometheus_io_path": "/",
  "prometheus_io_port": "12346",
  "blackbox_path": "/hello",
 "blackbox_port": "8080",
 "blackbox_scheme": "http"
}

 

删除pod，重启traefik，观察监控

部署Grafana

运维主机HDSS7-200.host.com上：

准备grafana镜像

grafana官方dockerhub地址
grafana官方github地址
grafana官网

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
[root@hdss7-200 ~]# docker pull grafana/grafana:6.1.4
6.1.4: Pulling from grafana/grafana
27833a3ba0a5: Pull complete 
9412d126b236: Pull complete 
1b7d6aaa6217: Pull complete 
530d1110a8c8: Pull complete 
fdcf73917f64: Pull complete 
f5009e3ea28a: Pull complete 
Digest: sha256:c2100550937e7aa0f3e33c2fc46a8c9668c3b5f2f71a8885e304d35de9fea009
Status: Downloaded newer image for grafana/grafana:6.1.4
[root@hdss7-200 ~]# docker tag d9bdb6044027 harbor.od.com/infra/grafana:v6.1.4
[root@hdss7-200 ~]# docker push harbor.od.com/infra/grafana:v6.1.4
docker push harbor.od.com/infra/grafana:v6.1.4
The push refers to a repository [harbor.od.com/infra/grafana]
b57e9e94fc2d: Pushed 
3d4e16e25cba: Pushed 
9642e67d431a: Pushed 
af52591a894f: Pushed 
0a8c2d04bf65: Pushed 
5dacd731af1b: Pushed 
v6.1.4: digest: sha256:db87ab263f90bdae66be744ac7935f6980c4bbd30c9756308e7382e00d4eeae8 size: 1576

 

准备资源配置清单

  • RBAC
  • Deployment
  • Service
  • Ingress

vi /data/k8s-yaml/grafana/rbac.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: grafana
rules:
- apiGroups:
 - "*"
 resources:
 - namespaces
 - deployments
 - pods
 verbs:
 - get
 - list
 - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 labels:
 addonmanager.kubernetes.io/mode: Reconcile
 kubernetes.io/cluster-service: "true"
 name: grafana
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: grafana
subjects:
- kind: User
 name: k8s-node

vi /data/k8s-yaml/grafana/deployment.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: grafana
 name: grafana
 name: grafana
 namespace: infra
spec:
 progressDeadlineSeconds: 600
 replicas: 1
 revisionHistoryLimit: 7
 selector:
 matchLabels:
 name: grafana
 strategy:
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 1
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: grafana
 name: grafana
 spec:
 containers:
 - image: harbor.od.com/infra/grafana:v6.1.4
 imagePullPolicy: IfNotPresent
 name: grafana
 ports:
 - containerPort: 3000
 protocol: TCP
 volumeMounts:
 - mountPath: /var/lib/grafana
 name: data
 imagePullSecrets:
 - name: harbor
 nodeName: 10.4.7.21
 restartPolicy: Always
 securityContext:
 runAsUser: 0
 volumes:
 - nfs:
 server: hdss7-200
 path: /data/nfs-volume/grafana
 name: data

vi /data/k8s-yaml/grafana/service.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
 
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: infra
spec:
 ports:
 - port: 3000
 protocol: TCP
 selector:
 app: grafana
 type: ClusterIP

vi /data/k8s-yaml/grafana/ingress.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana
  namespace: infra
spec:
 rules:
 - host: grafana.od.com
 http:
 paths:
 - path: /
 backend:
 serviceName: grafana
 servicePort: 3000

 

应用资源配置清单

任意运算节点上：

复制 
1
2
3
4
5
6
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/grafana/deployment.yaml 
deployment.extensions/grafana created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/grafana/service.yaml 
service/grafana created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/grafana/ingress.yaml 
ingress.extensions/grafana created

 

解析域名

HDSS7-11.host.com上

复制/var/named/od.com.zone 
1
 
grafana	60 IN A 10.4.7.10

 

浏览器访问

http://grafana.od.com

  • 用户名：admin
  • 密 码：admin

登录后需要修改管理员密码
实验文档4：kubernetes集群的监控和日志分析_第5张图片

配置grafana页面

外观

Configuration -> Preferences

  • UI Theme

    Light

  • Home Dashboard

    Default

  • Timezone

    Local browser time

save

插件

Configuration -> Plugins

  • Kubernetes App

安装方法一：

复制 
1
 
grafana-cli plugins install grafana-kubernetes-app

 

安装方法二：
下载地址

复制/data/nfs-volume/grafana/plugins 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
[root@hdss7-21 plugins]# wget https://grafana.com/api/plugins/grafana-kubernetes-app/versions/1.0.1/download -O grafana-kubernetes-app.zip
--2019-04-28 16:15:33--  https://grafana.com/api/plugins/grafana-kubernetes-app/versions/1.0.1/download
Resolving grafana.com (grafana.com)... 35.241.23.245
Connecting to grafana.com (grafana.com)|35.241.23.245|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/grafana/kubernetes-app/legacy.zip/31da38addc1d0ce5dfb154737c9da56e3b6692fc [following]
--2019-04-28 16:15:37-- https://codeload.github.com/grafana/kubernetes-app/legacy.zip/31da38addc1d0ce5dfb154737c9da56e3b6692fc
Resolving codeload.github.com (codeload.github.com)... 13.229.189.0, 13.250.162.133, 54.251.140.56
Connecting to codeload.github.com (codeload.github.com)|13.229.189.0|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3084524 (2.9M) [application/zip]
Saving to: ‘grafana-kubernetes-app.zip’

100%[===================================================================================================================>] 3,084,524 116KB/s in 12s 

2019-04-28 16:15:51 (250 KB/s) - ‘grafana-kubernetes-app.zip’ saved [3084524/3084524]
[root@hdss7-200 plugins]# unzip grafana-kubernetes-app.zip

 

  • Clock Pannel

安装方法一：

复制 
1
 
grafana-cli plugins install grafana-clock-panel

 

安装方法二：
下载地址

  • Pie Chart

安装方法一：

复制 
1
 
grafana-cli plugins install grafana-piechart-panel

 

安装方法二：
下载地址

  • D3 Gauge

安装方法一：

复制 
1
 
grafana-cli plugins install briangann-gauge-panel

 

安装方法二：
下载地址

  • Discrete

安装方法一：

复制 
1
 
grafana-cli plugins install natel-discrete-panel

 

安装方法二：
下载地址

  • 重启grafana的pod

  • 依次enable插件

配置grafana数据源

Configuration -> Data Sources
选择prometheus

  • HTTP
key value
URL http://prometheus.od.com
Access Server(Default)
  • Save & Test

实验文档4：kubernetes集群的监控和日志分析_第6张图片

配置Kubernetes集群Dashboard

kubernetes -> +New Cluster

  • Add a new cluster
key value
Name myk8s
  • HTTP
key value
URL https://10.4.7.10:7443
Access Server(Default)
  • Auth
key value
TLS Client Auth 勾选
With Ca Cert 勾选

将ca.pem、client.pem和client-key.pem粘贴至文本框内

  • Prometheus Read
key value
Datasource Prometheus
  • Save

注意：

  • K8S Container中，所有Pannel的

    pod_name -> container_label_io_kubernetes_pod_name

  • K8S-Cluster
  • K8S-Node
  • K8S-Deployment
  • K8S-Container

见附件

配置自定义dashboard

根据Prometheus数据源里的数据，配置如下dashboard：

  • etcd dashboard
  • traefik dashboard
  • generic dashboard
  • JMX dashboard
  • blackbox dashboard
  • ETCD
  • TRAEFIK
  • GENERIC
  • JMX
  • BlackBox

见附件

使用ELK Stack收集kubernetes集群内的应用日志

部署ElasticSearch

官网
官方github地址
下载地址
HDSS7-12.host.com上：

安装

复制/opt/src 
1
2
3
4
 
[root@hdss7-12 src]# ls -l|grep elasticsearch-5.6.15.tar.gz
-rw-r--r-- 1 root root  72262257 Jan 30 15:18 elasticsearch-5.6.15.tar.gz
[root@hdss7-12 src]# tar xf elasticsearch-5.6.15.tar.gz -C /opt
[root@hdss7-12 src]# ln -s /opt/elasticsearch-5.6.15/ /opt/elasticsearch

配置

elasticsearch.yml

复制 
1
2
3
4
5
6
7
8
9
 
[root@hdss7-12 src]# mkdir -p /data/elasticsearch/{data,logs}
[root@hdss7-12 elasticsearch]# vi config/elasticsearch.yml
cluster.name: es.od.com
node.name: hdss7-12.host.com
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/logs
bootstrap.memory_lock: true
network.host: 10.4.7.12
http.port: 9200

jvm.options

复制 
1
2
3
 
[root@hdss7-12 elasticsearch]# vi config/jvm.options
-Xms512m
-Xmx512m

创建普通用户

复制 
1
2
3
 
[root@hdss7-12 elasticsearch]# useradd -s /bin/bash -M es
[root@hdss7-12 elasticsearch]# chown -R es.es /opt/elasticsearch-5.6.15
[root@hdss7-12 elasticsearch]# chown -R es.es /data/elasticsearch

文件描述符

复制/etc/security/limits.d/es.conf 
1
2
3
4
 
es hard nofile 65536
es soft fsize unlimited
es hard memlock unlimited
es soft memlock unlimited

调整内核参数

复制 
1
2
3
4
 
[root@hdss7-12 elasticsearch]# sysctl -w vm.max_map_count=262144
or
[root@hdss7-12 elasticsearch]# echo "vm.max_map_count=262144" > /etc/sysctl.conf
[root@hdss7-12 elasticsearch]# sysctl -p

启动

复制 
1
2
3
4
5
6
 
[root@hdss7-12 elasticsearch]# su - es
su: warning: cannot change directory to /home/es: No such file or directory
-bash-4.2$ /opt/elasticsearch/bin/elasticsearch &
[1] 8714
[root@hdss7-12 elasticsearch]# ps -ef|grep elastic|grep -v grep
es 8714 1 58 10:29 pts/0 00:00:19 /usr/java/jdk/bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/elasticsearch -cp /opt/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch

调整ES日志模板

复制 
1
2
3
4
5
6
7
8
 
[root@hdss7-12 elasticsearch]# curl -XPUT http://10.4.7.12:9200/_template/k8s -d '{
  "template" : "k8s*",
  "index_patterns": ["k8s*"],  
  "settings": {
    "number_of_shards": 5,
 "number_of_replicas": 0
 }
}'

部署kafka

官网
官方github地址
下载地址
HDSS7-11.host.com上：

安装

复制/opt/src 
1
2
3
4
 
[root@hdss7-11 src]# ls -l|grep kafka
-rw-r--r-- 1 root root  57028557 Mar 23 08:57 kafka_2.12-2.2.0.tgz
[root@hdss7-11 src]# tar xf kafka_2.12-2.1.1.tgz -C /opt
[root@hdss7-11 src]# ln -s /opt/kafka_2.12-2.1.1/ /opt/kafka

配置

复制/opt/kafka/config/server.properties 
1
2
3
4
5
6
 
log.dirs=/data/kafka/logs
zookeeper.connect=localhost:2181
log.flush.interval.messages=10000
log.flush.interval.ms=1000
delete.topic.enable=true
host.name=hdss7-11.host.com

启动

复制/opt/kafka 
1
2
3
 
[root@hdss7-11 kafka]# bin/kafka-server-start.sh -daemon config/server.properties
[root@hdss7-11 kafka]# netstat -luntp|grep 9092
tcp6       0      0 10.4.7.12:9092         :::*                    LISTEN      17543/java

部署kafka-manager

官方github地址
源码下载地址
运维主机HDSS7-200.host.com上：

方法一：1、准备Dockerfile

复制/data/dockerfile/kafka-manager/Dockerfile 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
FROM hseeberger/scala-sbt

ENV ZK_HOSTS=10.4.7.11:2181 \
     KM_VERSION=2.0.0.2

RUN mkdir -p /tmp && \
 cd /tmp && \
 wget https://github.com/yahoo/kafka-manager/archive/${KM_VERSION}.tar.gz && \
 tar xxf ${KM_VERSION}.tar.gz && \
 cd /tmp/kafka-manager-${KM_VERSION} && \
 sbt clean dist && \
 unzip -d / ./target/universal/kafka-manager-${KM_VERSION}.zip && \
 rm -fr /tmp/${KM_VERSION} /tmp/kafka-manager-${KM_VERSION}

WORKDIR /kafka-manager-${KM_VERSION}

EXPOSE 9000
ENTRYPOINT ["./bin/kafka-manager","-Dconfig.file=conf/application.conf"]

方法一：2、制作docker镜像

复制/data/dockerfile/kafka-manager 
1
2
 
[root@hdss7-200 kafka-manager]# docker build . -t harbor.od.com/infra/kafka-manager:v2.0.0.2
(漫长的过程)

方法二：直接下载docker镜像

镜像下载地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
[root@hdss7-200 ~]# docker pull sheepkiller/kafka-manager:stable
[root@hdss7-200 ~]# docker tag 34627743836f harbor.od.com/infra/kafka-manager:stable
[root@hdss7-200 ~]# docker push harbor.od.com/infra/kafka-manager:stable
docker push harbor.od.com/infra/kafka-manager:stable
The push refers to a repository [harbor.od.com/infra/kafka]
ef97dbc2670b: Pushed 
ec01aa005e59: Pushed 
de05a1bdf878: Pushed 
9c553f3feafd: Pushed 
581533427a4f: Pushed 
f6b229974fdd: Pushed 
ae150883d6e2: Pushed 
3df7be729841: Pushed 
f231cc200afe: Pushed 
9752c15164a8: Pushed 
9ab7eda5c826: Pushed 
402964b3d72e: Pushed 
6b3f8ebf864c: Pushed 
stable: digest: sha256:57fd46a3751284818f1bc6c0fdf097250bc0feed03e77135fb8b0a93aa8c6cc7 size: 3056

 

准备资源配置清单

  • Deployment
  • Service
  • Ingress

vi /data/k8s-yaml/kafka-manager/deployment.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-manager
  namespace: infra
 labels: 
 name: kafka-manager
spec:
 replicas: 1
 selector:
 matchLabels: 
 name: kafka-manager
 template:
 metadata:
 labels: 
 app: kafka-manager
 name: kafka-manager
 spec:
 containers:
 - name: kafka-manager
 image: harbor.od.com/infra/kafka-manager:stable
 ports:
 - containerPort: 9000
 protocol: TCP
 env:
 - name: ZK_HOSTS
 value: zk1.od.com:2181
 - name: APPLICATION_SECRET
 value: letmein
 imagePullPolicy: IfNotPresent
 imagePullSecrets:
 - name: harbor
 restartPolicy: Always
 terminationGracePeriodSeconds: 30
 securityContext: 
 runAsUser: 0
 schedulerName: default-scheduler
 strategy:
 type: RollingUpdate
 rollingUpdate: 
 maxUnavailable: 1
 maxSurge: 1
 revisionHistoryLimit: 7
 progressDeadlineSeconds: 600

vi /data/k8s-yaml/kafka-manager/svc.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
kind: Service
apiVersion: v1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
 ports:
 - protocol: TCP
 port: 9000
 targetPort: 9000
 selector: 
 app: kafka-manager
 clusterIP: None
 type: ClusterIP
 sessionAffinity: None

vi /data/k8s-yaml/kafka-manager/ingress.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kafka-manager
  namespace: infra
spec:
 rules:
 - host: km.od.com
 http:
 paths:
 - path: /
 backend: 
 serviceName: kafka-manager
 servicePort: 9000

应用资源配置清单

任意一台运算节点上：

复制 
1
2
3
4
5
6
 
[root@hdss7-21 kafka-manager]# kubectl apply -f http://k8s-yaml.od.com/kafka-manager/deployment.yaml 
deployment.extensions/kafka-manager created
[root@hdss7-21 kafka-manager]# kubectl apply -f http://k8s-yaml.od.com/kafka-manager/svc.yaml 
service/kafka-manager created
[root@hdss7-21 kafka-manager]# kubectl apply -f http://k8s-yaml.od.com/kafka-manager/ingress.yaml 
ingress.extensions/kafka-manager created

 

解析域名

HDSS7-11.host.com上

复制/var/named/od.com.zone 
1
 
km	60 IN A 10.4.7.10

 

浏览器访问

http://km.od.com

实验文档4：kubernetes集群的监控和日志分析_第7张图片

部署filebeat

官方下载地址
运维主机HDSS7-200.host.com上：

制作docker镜像

准备Dockerfile

  • Dockerfile
  • Entrypoint

vi /data/dockerfile/filebeat/Dockerfile

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
FROM debian:jessie

ENV FILEBEAT_VERSION=7.0.1 \
    FILEBEAT_SHA1=fdddfa32a7d9db5ac4504b34499e6259e09b86205bac842f78fddd45e8ee00c3cb76419af2313659fd23db4fcbcc29f7568a3663c5d3c52ac0fc9e641d0ae8b1

RUN set -x && \
 apt-get update && \
 apt-get install -y wget && \
 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -O /opt/filebeat.tar.gz && \
 cd /opt && \
 echo "${FILEBEAT_SHA1} filebeat.tar.gz" | sha512sum -c - && \
 tar xzvf filebeat.tar.gz && \
 cd filebeat-* && \
 cp filebeat /bin && \
 cd /opt && \
 rm -rf filebeat* && \
 apt-get purge -y wget && \
 apt-get autoremove -y && \
 apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]

vi /data/dockerfile/filebeat/docker-entrypoint.sh

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
#!/bin/bash

ENV=${ENV:-"test"}
PROJ_NAME=${PROJ_NAME:-"no-define"}
MULTILINE=${MULTILINE:-"^\d{2}"}

cat > /etc/filebeat.yaml << EOF
filebeat.inputs:
- type: log
 fields_under_root: true
 fields:
 topic: logm-${PROJ_NAME}
 paths:
 - /logm/*.log
 - /logm/*/*.log
 - /logm/*/*/*.log
 - /logm/*/*/*/*.log
 - /logm/*/*/*/*/*.log
 scan_frequency: 120s
 max_bytes: 10485760
 multiline.pattern: '$MULTILINE'
 multiline.negate: true
 multiline.match: after
 multiline.max_lines: 100
- type: log
 fields_under_root: true
 fields:
 topic: logu-${PROJ_NAME}
 paths:
 - /logu/*.log
 - /logu/*/*.log
 - /logu/*/*/*.log
 - /logu/*/*/*/*.log
 - /logu/*/*/*/*/*.log
 - /logu/*/*/*/*/*/*.log
output.kafka:
 hosts: ["10.4.7.11:9092"]
 topic: k8s-fb-$ENV-%{[topic]}
 version: 2.0.0
 required_acks: 0
 max_message_bytes: 10485760
EOF

set -xe

# If user don't provide any command
# Run filebeat
if [[ "$1" == "" ]]; then
 exec filebeat -c /etc/filebeat.yaml 
else
 # Else allow the user to run arbitrarily commands like bash
 exec "$@"
fi

制作镜像

复制/data/dockerfile/filebeat 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
 
[root@hdss7-200 filebeat]# docker build . -t harbor.od.com/infra/filebeat:v7.0.1
...
+ apt-get autoremove -y
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
 ca-certificates libicu52 libidn11 libpsl0 libssl1.0.0 openssl
0 upgraded, 0 newly installed, 6 to remove and 6 not upgraded.
After this operation, 33.6 MB disk space will be freed.
(Reading database ... 7962 files and directories currently installed.)
Removing ca-certificates (20141019+deb8u4) ...
Removing dangling symlinks from /etc/ssl/certs... done.
Removing libpsl0:amd64 (0.5.1-1) ...
Removing libicu52:amd64 (52.1-8+deb8u7) ...
Removing libidn11:amd64 (1.29-1+deb8u3) ...
Removing openssl (1.0.1t-1+deb8u11) ...
Removing libssl1.0.0:amd64 (1.0.1t-1+deb8u11) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
+ apt-get clean
+ rm -rf /var/lib/apt/lists/deb.debian.org_debian_dists_jessie_Release /var/lib/apt/lists/deb.debian.org_debian_dists_jessie_Release.gpg /var/lib/apt/lists/deb.debian.org_debian_dists_jessie_main_binary-amd64_Packages.gz /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/security.debian.org_debian-security_dists_jessie_updates_InRelease /var/lib/apt/lists/security.debian.org_debian-security_dists_jessie_updates_main_binary-amd64_Packages.gz /tmp/* /var/tmp/*
 ---> a78678659f2c
Removing intermediate container 2cfff130c15c
Step 4 : COPY docker-entrypoint.sh /
 ---> 62dc7fe5a98f
Removing intermediate container 87e271482593
Step 5 : ENTRYPOINT /docker-entrypoint.sh
 ---> Running in d367b6e3bb5a
 ---> 23c8fbdc088a
Removing intermediate container d367b6e3bb5a
Successfully built 23c8fbdc088a
[root@hdss7-200 filebeat]# docker tag 23c8fbdc088a harbor.od.com/infra/filebeat:v7.0.1
[root@hdss7-200 filebeat]# docker push !$
docker push harbor.od.com/infra/filebeat:v7.0.1
The push refers to a repository [harbor.od.com/infra/filebeat]
6a765e653161: Pushed 
8e89ae5c6fc2: Pushed 
9abb3997a540: Pushed 
v7.0.1: digest: sha256:c35d7cdba29d8555388ad41ac2fc1b063ed9ec488082e33b5d0e91864b3bb35c size: 948

修改资源配置清单

使用dubbo-demo-consumer的Tomcat版镜像

复制/data/k8s-yaml/dubbo-demo-consumer/deployment.yaml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
 labels: 
 name: dubbo-demo-consumer
spec:
 replicas: 1
 selector:
 matchLabels: 
 name: dubbo-demo-consumer
 template:
 metadata:
 labels: 
 app: dubbo-demo-consumer
 name: dubbo-demo-consumer
 spec:
 containers:
 - name: dubbo-demo-consumer
 image: harbor.od.com/app/dubbo-demo-web:tomcat
 ports:
 - containerPort: 8080
 protocol: TCP
 env:
 - name: C_OPTS
 value: -Denv=dev -Dapollo.meta=config.od.com
 imagePullPolicy: IfNotPresent
 volumeMounts:
 - mountPath: /opt/tomcat/logs
 name: logm
 - name: filebeat
 image: harbor.od.com/infra/filebeat:v7.0.1
 env:
 - name: ENV
 value: test
 - name: PROJ_NAME
 value: dubbo-demo-web
 imagePullPolicy: IfNotPresent
 volumeMounts:
 - mountPath: /logm
 name: logm
 volumes:
 - emptyDir: {}
 name: logm
 imagePullSecrets:
 - name: harbor
 restartPolicy: Always
 terminationGracePeriodSeconds: 30
 securityContext: 
 runAsUser: 0
 schedulerName: default-scheduler
 strategy:
 type: RollingUpdate
 rollingUpdate: 
 maxUnavailable: 1
 maxSurge: 1
 revisionHistoryLimit: 7
 progressDeadlineSeconds: 600

 

浏览器访问http://km.od.com

看到kafaka-manager里，topic打进来，即为成功。
实验文档4：kubernetes集群的监控和日志分析_第8张图片

验证数据

复制/opt/kafka/bin 
1
 
./kafka-console-consumer.sh --bootstrap-server 10.9.6.200:9092 --topic k8s-fb-test-logm-dubbo-demo-web --from-beginning

部署logstash

运维主机HDSS7-200.host.com上：

选版本

logstash选型
实验文档4：kubernetes集群的监控和日志分析_第9张图片

准备docker镜像

  • 下载官方镜像
复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 
[root@hdss7-200 ~]# docker pull logstash:6.7.2
6.7.2: Pulling from library/logstash
8ba884070f61: Pull complete 
063405b57b96: Pull complete 
0a0115b8825c: Pull complete 
825b124900cb: Pull complete 
ed736d9e3c41: Pull complete 
6efb3934697a: Pull complete 
c0a3a0c8ebc2: Pull complete 
f55d8adfbc3d: Pull complete 
3a6c56fbbef8: Pull complete 
ca45dc8946a2: Pull complete 
8b852a079ea9: Pull complete 
Digest: sha256:46eaff19af5e14edd9b78e1d5bf16f6abcd9ad50e0338cbaf3024f2aadb2903b
Status: Downloaded newer image for logstash:6.7.2

[root@hdss7-200 ~]# docker tag 857d9a1f8221 harbor.od.com/public/logstash:v6.7.2

[root@hdss7-200 ~]# docker push harbor.od.com/public/logstash:v6.7.2
docker push harbor.od.com/public/logstash:v6.7.2
The push refers to a repository [harbor.od.com/public/logstash]
c2b00f70cade: Mounted from public/filebeat 
9a2c2851596d: Mounted from public/filebeat 
86564f3ca0e2: Mounted from public/filebeat 
e9bc8faf823a: Mounted from public/filebeat 
3f3bcfa85067: Mounted from public/filebeat 
65d3b56913bd: Mounted from public/filebeat 
9b48a60607ee: Mounted from public/filebeat 
df859db41dd0: Mounted from public/filebeat 
1cbe912d7c00: Mounted from public/filebeat 
ab5fbaca7e70: Mounted from public/filebeat 
d69483a6face: Mounted from public/filebeat 
v6.7.2: digest: sha256:6aacf97dfbcc5c64c2f1a12f43ee48a8dadb98657b9b8d4149d0fee0ec18be81 size: 2823
  • 自定义Dockerfile
  • Dockerfile
复制 
1
2
 
From harbor.od.com/public/logstash:v6.7.2
ADD logstash.yml /usr/share/logstash/config
  • logstash.yml
复制 
 

 

1
2
3
 
http.host: "0.0.0.0"
path.config: /etc/logstash
xpack.monitoring.enabled: false

 

  • 创建自定义镜像
复制/data/dockerfile/logstash 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
[root@hdss7-200 logstash]# docker build . -t harbor.od.com/infra/logstash:v6.7.2
Sending build context to Docker daemon 249.3 MB
Step 1 : FROM harbor.od.com/public/logstash:v6.7.2
v6.7.2: Pulling from public/logstash
e60be144a6a5: Pull complete 
6292c2b22d35: Pull complete 
1bb4586d90e7: Pull complete 
3f11f6d21132: Pull complete 
f0aaeeafebd3: Pull complete 
38c751a91f0f: Pull complete 
b80e2bad9681: Pull complete 
e3c97754ddde: Pull complete 
840f2d82a9fb: Pull complete 
32063a9aaefb: Pull complete 
e87b22bf50f5: Pull complete 
Digest: sha256:6aacf97dfbcc5c64c2f1a12f43ee48a8dadb98657b9b8d4149d0fee0ec18be81
Status: Downloaded newer image for harbor.od.com/public/logstash:v6.7.2
 ---> 857d9a1f8221
Step 2 : ADD logstash.yml /usr/share/logstash/config
 ---> 2ad32d3f5fef
Removing intermediate container 1d3a1488c1b7
Successfully built 2ad32d3f5fef
[root@hdss7-200 logstash]# docker push harbor.od.com/infra/logstash:v6.7.2

启动docker镜像

  • 创建配置
复制/etc/logstash/logstash-test.conf 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
input {
  kafka {
    bootstrap_servers => "10.4.7.11:9092"
    client_id => "10.4.7.200"
    consumer_threads => 4
 group_id => "k8s_test"
 topics_pattern => "k8s-fb-test-.*"
 }
}

filter {
 json {
 source => "message"
 }
}

output {
 elasticsearch {
 hosts => ["10.4.7.12:9200"]
 index => "k8s-test-%{+YYYY.MM}"
 }
}
  • 启动logstash镜像
复制 
1
2
3
 
[root@hdss7-200 ~]# docker run -d --name logstash-test -v /etc/logstash:/etc/logstash harbor.od.com/infra/logstash:v6.7.2 -f /etc/logstash/logstash-test.conf
[root@hdss7-200 ~]# docker ps -a|grep logstash
a5dcf56faa9a        harbor.od.com/infra/logstash:v6.7.2                                                                                "/usr/local/bin/docke"   8 seconds ago       Up 6 seconds                5044/tcp, 9600/tcp           jovial_swanson
  • 验证ElasticSearch里的索引
复制 
1
2
3
 
[root@hdss7-200 ~]# curl http://10.4.7.12:9200/_cat/indices?v
health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   k8s-test-2019.04 H3MY9d8WSbqQ6uL0DFhenQ   5   0         55            0    249.9kb        249.9kb

部署Kibana

运维主机HDSS7-200.host.com上：

准备docker镜像

kibana官方镜像下载地址

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
 
[root@hdss7-200 ~]# docker pull kibana:5.6.16
5.6.16: Pulling from library/kibana

8014725ed6f5: Pull complete 
19b590251e94: Pull complete 
7fa3e54c0b5a: Pull complete 
60ee9811b356: Pull complete 
d18bafa420f4: Pull complete 
8cee55751899: Pull complete 
c395be470eb2: Pull complete 
Digest: sha256:71f776596244877597fd679b2fa6fb0f1fa9c5b11388199997781d1ce77b73b1
Status: Downloaded newer image for kibana:5.6.16
[root@hdss7-200 ~]# docker tag 62079cf74c23 harbor.od.com/infra/kibana:v5.6.16
[root@hdss7-200 ~]# docker push harbor.od.com/infra/kibana:v5.6.16
docker push harbor.od.com/infra/kibana:v5.6.16
The push refers to a repository [harbor.od.com/infra/kibana]
be94745c5390: Pushed 
652dcbd52cdd: Pushed 
a3508a095ca7: Pushed 
56d52080e2fe: Pushed 
dbce28d91bf0: Pushed 
dcddc432ebdf: Pushed 
3e466685bf43: Pushed 
d69483a6face: Mounted from infra/logstash 
v5.6.16: digest: sha256:17dd243d6cc4e572f74f3de83eafc981e54c1710f8fe2d0bf74357b28bddaf08 size: 1991

 

解析域名

HDSS7-11.host.com上

复制/var/named/od.com.zone 
1
 
kibana	60 IN A 10.4.7.10

 

准备资源配置清单

  • Deployment
  • Service
  • Ingress

vi /data/k8s-yaml/kibana/deployment.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kibana
  namespace: infra
 labels: 
 name: kibana
spec:
 replicas: 1
 selector:
 matchLabels: 
 name: kibana
 template:
 metadata:
 labels: 
 app: kibana
 name: kibana
 spec:
 containers:
 - name: kibana
 image: harbor.od.com/infra/kibana:v5.6.16
 ports:
 - containerPort: 5601
 protocol: TCP
 env:
 - name: ELASTICSEARCH_URL
 value: http://10.4.7.12:9200
 imagePullPolicy: IfNotPresent
 imagePullSecrets:
 - name: harbor
 restartPolicy: Always
 terminationGracePeriodSeconds: 30
 securityContext: 
 runAsUser: 0
 schedulerName: default-scheduler
 strategy:
 type: RollingUpdate
 rollingUpdate: 
 maxUnavailable: 1
 maxSurge: 1
 revisionHistoryLimit: 7
 progressDeadlineSeconds: 600

vi /data/k8s-yaml/kibana/svc.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
kind: Service
apiVersion: v1
metadata: 
  name: kibana
  namespace: infra
spec:
 ports:
 - protocol: TCP
 port: 5601
 targetPort: 5601
 selector: 
 app: kibana
 clusterIP: None
 type: ClusterIP
 sessionAffinity: None

vi /data/k8s-yaml/kibana/ingress.yaml

复制 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: kibana
  namespace: infra
spec:
 rules:
 - host: kibana.od.com
 http:
 paths:
 - path: /
 backend: 
 serviceName: kibana
 servicePort: 5601

 

 

应用资源配置清单

任意运算节点上：

复制 
1
2
3
4
5
6
 
[root@hdss7-21 ~]# kubectl apply -f deployment.yaml 
deployment.extensions/kibana created
[root@hdss7-21 ~]# kubectl apply -f svc.yaml 
service/kibana created
[root@hdss7-21 ~]# kubectl apply -f ingress.yaml 
ingress.extensions/kibana created

 

浏览器访问

http://kibana.od.com
实验文档4：kubernetes集群的监控和日志分析_第10张图片

kibana的使用

实验文档4：kubernetes集群的监控和日志分析_第11张图片

选择区域

  • @timestamp

    对应日志的时间戳

  • log.file.path

    对应日志文件名

  • message

    对应日志内容

时间选择器

  • 选择日志时间

    快速时间
    绝对时间
    相对时间

环境选择器

  • 选择对应环境的日志

    k8s-test-
    k8s-prod-

项目选择器

  • 对应filebeat的PROJ_NAME值
  • Add a fillter
  • topic is ${PROJ_NAME}

    dubbo-demo-service
    dubbo-demo-web

关键字选择器

  • exception
  • error
 

你可能感兴趣的:(实验文档4：kubernetes集群的监控和日志分析)

  • 斤斤计较的婚姻到底有多难？ 白心之岂必有为
    很多人私聊我会问到在哪个人群当中斤斤计较的人最多？我都会回答他，一般婚姻出现问题的斤斤计较的人士会非常多，以我多年经验，在婚姻落的一塌糊涂的人当中，斤斤计较的人数占比在20～30%以上，也就是说10个婚姻出现问题的斤斤计较的人有2-3个有多不减。在婚姻出问题当中，有大量的心理不平衡的、尖酸刻薄的怨妇。在婚姻中仅斤斤计较有两种类型：第一种是物质上的，另一种是精神上的。在物质与精神上抠门已经严重的影响
  • 情绪觉察日记第37天 露露_e800
    今天是家庭关系规划师的第二阶最后一天，慧萍老师帮我做了个案，帮我处理了埋在心底好多年的一份恐惧，并给了我深深的力量！这几天出来学习，爸妈过来婆家帮我带小孩，妈妈出于爱帮我收拾东西，并跟我先生和婆婆产生矛盾，妈妈觉得他们没有照顾好我…。今晚回家见到妈妈，我很欣赏她并赞扬她，妈妈说今晚要跟我睡我说好，当我们俩躺在床上准备睡觉的时候，我握着妈妈的手对她说:妈妈这几天辛苦你了，你看你多利害把我们的家收拾得
  • 芦花鞋一四 许叶晗
    又是在一个寒冷的夏日里，青铜和葵花决定今天一起去卖芦花鞋，奶奶亲手给他们做了一碗热乎乎的粥对他们说:“就靠你们两挣生活费了这碗粥赶紧趁热喝了吧！”于是青铜和葵花喝完了奶奶给她们做的粥，就准备去镇上卖卢花鞋，这回青铜和葵花穿着新的芦花鞋来到了镇上。青铜这回看到了很多人都在卖，用手势表达对葵花说:“这回有好多人在抢我们生意呢！我们必须得吆喝起来。”葵花点了点头。可是谁知他们也大声的叫，卖芦花喽！卖芦花
  • QQ群采集助手，精准引流必备神器 2401_87347160 其他经验分享
    功能概述微信群查找与筛选工具是一款专为微信用户设计的辅助工具，它通过关键词搜索功能，帮助用户快速找到相关的微信群，并提供筛选是否需要验证的群组的功能。主要功能关键词搜索：用户可以输入关键词，工具将自动查找包含该关键词的微信群。筛选功能：工具提供筛选机制，用户可以选择是否只显示需要验证或不需要验证的群组。精准引流：通过上述功能，用户可以更精准地找到目标群组，进行有效的引流操作。3.设备需求该工具可以
  • 关于沟通这件事，项目经理不需要每次都面对面进行 流程大师兄
    很多项目经理都会遇到这样的问题，项目中由于事情太多，根本没有足够的时间去召开会议，那在这种情况下如何去有效地管理项目中的利益相关者？当然，不建议电子邮件也不需要开会的话，建议可以采取下面几种方式来形成有效的沟通，这几种方式可以帮助你努力的通过各种办法来保持和各方面的联系。项目经理首先要问自己几个问题，项目中哪些利益相关者是必须要进行沟通的？可以列出项目中所有的利益相关者清单，同时也整理出项目中哪些
  • 机器学习与深度学习间关系与区别 ℒℴѵℯ心·动ꦿ໊ོ꫞ 人工智能学习深度学习python
    一、机器学习概述定义机器学习（MachineLearning,ML）是一种通过数据驱动的方法，利用统计学和计算算法来训练模型，使计算机能够从数据中学习并自动进行预测或决策。机器学习通过分析大量数据样本，识别其中的模式和规律，从而对新的数据进行判断。其核心在于通过训练过程，让模型不断优化和提升其预测准确性。主要类型1.监督学习（SupervisedLearning）监督学习是指在训练数据集中包含输入
  • 铭刻于星（四十二） 随风至
    69夜晚，绍敏同学做完功课后，看了眼房外，没听到动静才敢从书包的夹层里拿出那个心形纸团。折痕压得很深，都有些旧了，想来是已经写好很久了。绍敏同学慢慢地、轻轻地捏开折叠处，待到全部拆开后，又反复抚平纸张，然后仔细地一字字默看。只是开头的三个字是第一次看到，让她心漏跳了几拍。“亲爱的绍敏：从四年级的时候，我就喜欢你了，但是我一直不敢说，怕影响你学习。六年级的时候听说有人跟你表白，你接受了，我很难过，但
  • 底层逆袭到底有多难，不甘平凡的你准备好了吗？让吴起给你说说 造命者说
    底层逆袭到底有多难，不甘平凡的你准备好了吗？让吴起给你说说我叫吴起，生于公元前440年的战国初期，正是群雄并起、天下纷争不断的时候。后人说我是军事家、政治家、改革家，是兵家代表人物。评价我一生历仕鲁、魏、楚三国，通晓兵家、法家、儒家三家思想，在内政军事上都有极高的成就。周安王二十一年（公元前381年），因变法得罪守旧贵族，被人乱箭射死。我出生在卫国一个“家累万金”的富有家庭，从年轻时候起就不甘平凡
  • 2020-01-25 晴岚85
    郑海燕坚持分享590天2020.1.24在生活中只存在两个问题。一个问题是：你知道想要达成的目标是什么，但却不知道如何才能达成；另一个问题是：你不知道你的目标是什么。前一个是行动的问题，后一个是结果的问题。通过制定具体的下一步行动，可以解决不知道如何开始行动的问题。而通过去想象结果，对结果做预估，可以解决找不着目标的问题。对于所有吸引我们注意力，想要完成的任务，你可以先想象一下，预期的结果究竟是什
  • 随笔 | 仙一般的灵气 海思沧海
    仙岛今天，我看了你全部，似乎已经进入你的世界我不知道，这是否是梦幻，还是你仙一般的灵气吸引了我也许每一个人都要有一份属于自己的追求，这样才能够符合人生的梦想，生活才能够充满着阳光与快乐我不知道，我为什么会这样的感叹，是在感叹自己的人生，还是感叹自己一直没有孜孜不倦的追求只感觉虚度了光阴，每天活在自己的梦中，活在一个不真实的世界是在逃避自己，还是在逃避周围的一切有时候我嘲笑自己，嘲笑自己如此的虚无，
  • 想家 爆米花机
    也许不同于大家对家乡的思念，我对家乡甚至是疯狂的不舍。还未踏出车站就感觉到幸福，我享受这里的夕阳、这里的浓烈柴火味、这里每一口家常菜。我是宅女，我贪恋家的安逸。刚刚踏出大学校门，初出茅庐，无法适应每年只能国庆和春节回家。我焦虑、失眠、无端发脾气，是无法适应工作的节奏，是无法接受我将一步步离开家乡的事实。我不想承认自己胸无大志，选择再次踏上征程。图片发自App
  • 【iOS】MVC设计模式 Magnetic_h iosmvc设计模式objective-c学习ui
    MVC前言如何设计一个程序的结构，这是一门专门的学问，叫做"架构模式"（architecturalpattern），属于编程的方法论。MVC模式就是架构模式的一种。它是Apple官方推荐的App开发架构，也是一般开发者最先遇到、最经典的架构。MVC各层controller层Controller/ViewController/VC（控制器）负责协调Model和View，处理大部分逻辑它将数据从Mod
  • OC语言多界面传值五大方式 Magnetic_h iosui学习objective-c开发语言
    前言在完成暑假仿写项目时，遇到了许多需要用到多界面传值的地方，这篇博客来总结一下比较常用的五种多界面传值的方式。属性传值属性传值一般用前一个界面向后一个界面传值，简单地说就是通过访问后一个视图控制器的属性来为它赋值，通过这个属性来做到从前一个界面向后一个界面传值。首先在后一个界面中定义属性@interfaceBViewController:UIViewController@propertyNSSt
  • 一百九十四章. 自相矛盾 巨木擎天
    唉！就这么一夜，林子感觉就像过了很多天似的，先是回了阳间家里，遇到了那么多不可思议的事情儿。特别是小伙伴们，第二次与自己见面时，僵硬的表情和恐怖的气氛，让自己如坐针毡，打从心眼里难受！还有东子，他现在还好吗？有没有被人欺负？护城河里的小鱼小虾们，还都在吗？水不会真的干枯了吧？那对相亲相爱漂亮的太平鸟儿，还好吧！春天了，到了做窝、下蛋、喂养小鸟宝宝的时候了，希望它们都能够平安啊！虽然没有看见家人，也
  • UI学习——cell的复用和自定义cell Magnetic_h ui学习
    目录cell的复用手动（非注册）自动（注册）自定义cellcell的复用在iOS开发中，单元格复用是一种提高表格（UITableView）和集合视图（UICollectionView）滚动性能的技术。当一个UITableViewCell或UICollectionViewCell首次需要显示时，如果没有可复用的单元格，则视图会创建一个新的单元格。一旦这个单元格滚动出屏幕，它就不会被销毁。相反，它被添
  • element实现动态路由+面包屑 软件技术NINI vue案例vue.js前端
    el-breadcrumb是ElementUI组件库中的一个面包屑导航组件，它用于显示当前页面的路径，帮助用户快速理解和导航到应用的各个部分。在Vue.js项目中，如果你已经安装了ElementUI，就可以很方便地使用el-breadcrumb组件。以下是一个基本的使用示例：安装ElementUI（如果你还没有安装的话）:你可以通过npm或yarn来安装ElementUI。bash复制代码npmi
  • 10月|愿你的青春不负梦想-读书笔记-01 Tracy的小书斋
    本书的作者是俞敏洪，大家都很熟悉他了吧。俞敏洪老师是我行业的领头羊吧，也是我事业上的偶像。本日摘录他书中第一章中的金句：『一个人如果什么目标都没有，就会浑浑噩噩，感觉生命中缺少能量。能给我们能量的，是对未来的期待。第一件事，我始终为了进步而努力。与其追寻全世界的骏马，不如种植丰美的草原，到时骏马自然会来。第二件事，我始终有阶段性的目标。什么东西能给我能量？答案是对未来的期待。』读到这里的时候，我便
  • C语言宏函数 南林yan C语言c语言
    一、什么是宏函数？通过宏定义的函数是宏函数。如下，编译器在预处理阶段会将Add(x,y)替换为((x)*(y))#defineAdd(x,y)((x)*(y))#defineAdd(x,y)((x)*(y))intmain(){inta=10;intb=20;intd=10;intc=Add(a+d,b)*2;cout<
  • 地推话术，如何应对地推过程中家长的拒绝 校师学
    相信校长们在做地推的时候经常遇到这种情况：市场专员反馈家长不接单，咨询师反馈难以邀约这些家长上门，校区地推疲软，招生难。为什么？仅从地推层面分析，一方面因为家长受到的信息轰炸越来越多，对信息越来越“免疫”；而另一方面地推人员的专业能力和营销话术没有提高，无法应对家长的拒绝，对有意向的家长也不知如何跟进，眼睁睁看着家长走远；对于家长的疑问，更不知道如何有技巧地回答，机会白白流失。由于回答没技巧和专业
  • 谢谢你们，爱你们！ 鹿游儿
    昨天家人去泡温泉，二个孩子也带着去，出发前一晚，匆匆下班，赶回家和孩子一起收拾。饭后，我拿出笔和本子（上次去澳门时做手帐的本子）写下了1\2\3\4\5\6\7\8\9,让后让小壹去思考，带什么出发去旅游呢？她在对应的数字旁边画上了，泳衣、泳圈、肖恩、内衣内裤、tapuy、拖鞋……画完后，就让她自己对着这个本子，将要带的，一一带上，没想到这次带的书还是这本《便便工厂》(晚上姑婆发照片过来，妹妹累得
  • C语言如何定义宏函数？ 小九格物 c语言
    在C语言中，宏函数是通过预处理器定义的，它在编译之前替换代码中的宏调用。宏函数可以模拟函数的行为，但它们不是真正的函数，因为它们在编译时不会进行类型检查，也不会分配存储空间。宏函数的定义通常使用#define指令，后面跟着宏的名称和参数列表，以及宏展开后的代码。宏函数的定义方式：1.基本宏函数：这是最简单的宏函数形式，它直接定义一个表达式。#defineSQUARE(x)((x)*(x))2.带参
  • 微服务下功能权限与数据权限的设计与实现 nbsaas-boot 微服务java架构
    在微服务架构下，系统的功能权限和数据权限控制显得尤为重要。随着系统规模的扩大和微服务数量的增加，如何保证不同用户和服务之间的访问权限准确、细粒度地控制，成为设计安全策略的关键。本文将讨论如何在微服务体系中设计和实现功能权限与数据权限控制。1.功能权限与数据权限的定义功能权限：指用户或系统角色对特定功能的访问权限。通常是某个用户角色能否执行某个操作，比如查看订单、创建订单、修改用户资料等。数据权限：
  • 理解Gunicorn：Python WSGI服务器的基石 范范0825 ipythonlinux运维
    理解Gunicorn：PythonWSGI服务器的基石介绍Gunicorn，全称GreenUnicorn，是一个为PythonWSGI（WebServerGatewayInterface）应用设计的高效、轻量级HTTP服务器。作为PythonWeb应用部署的常用工具，Gunicorn以其高性能和易用性著称。本文将介绍Gunicorn的基本概念、安装和配置，帮助初学者快速上手。1.什么是Gunico
  • 小丽成长记（四十三） 玲玲54321
    小丽发现，即使她好不容易调整好自己的心态下一秒总会有不确定的伤脑筋的事出现，一个接一个的问题，人生就没有停下的时候，小问题不断出现。不过她今天看的书，她接受了人生就是不确定的，厉害的人就是不断创造确定性，在Ta的领域比别人多的确定性就能让自己脱颖而出，显示价值从而获得的比别人多的利益。正是这样的原因，因为从前修炼自己太少，使得她现在在人生道路上打怪起来困难重重，她似乎永远摆脱不了那种无力感，有种习
  • 学点心理知识，呵护孩子健康 静候花开_7090
    昨天听了华中师范大学教育管理学系副教授张玲老师的《哪里才是学生心理健康的最后庇护所，超越教育与技术的思考》的讲座。今天又重新学习了一遍，收获匪浅。张玲博士也注意到了当今社会上的孩子由于心理问题导致的自残、自杀及伤害他人等恶性事件。她向我们普及了一个重要的命题，她说心理健康的一些基本命题，我们与我们通常的一些教育命题是不同的，她还举了几个例子，让我们明白我们原来以为的健康并非心理学上的健康。比如如果
  • 2021年12月19日，春蕾教育集团团建活动感受——黄晓丹 黄错错加油
    感受:1.从陌生到熟悉的过程。游戏环节让我们在轻松的氛围中得到了锻炼，也增长了不少知识。2.游戏过程中，我们贡献的是个人力量，展现的是团队的力量。它磨合的往往不止是工作的熟悉，更是观念上契合度的贴近。3.这和工作是一样的道理。在各自的岗位上，每个人摆正自己的位置、各司其职充分发挥才能，并团结一致劲往一处使，才能实现最大的成功。新知:1.团队精神需要不断地创新。过去，人们把创新看作是冒风险，现在人们
  • Cell Insight | 单细胞测序技术又一新发现，可用于HIV-1和Mtb共感染个体诊断 尐尐呅
    结核病是艾滋病合并其他疾病中导致患者死亡的主要原因。其中结核病由结核分枝杆菌（Mycobacteriumtuberculosis,Mtb）感染引起，获得性免疫缺陷综合症（艾滋病）由人免疫缺陷病毒（Humanimmunodeficiencyvirustype1,HIV-1）感染引起。国家感染性疾病临床医学研究中心/深圳市第三人民医院张国良团队携手深圳华大生命科学研究院吴靓团队，共同研究得出单细胞测序
  • c++ 的iostream 和 c++的stdio的区别和联系 黄卷青灯77 c++算法开发语言iostreamstdio
    在C++中，iostream和C语言的stdio.h都是用于处理输入输出的库，但它们在设计、用法和功能上有许多不同。以下是两者的区别和联系：区别1.编程风格iostream（C++风格）：C++标准库中的输入输出流类库，支持面向对象的输入输出操作。典型用法是cin（输入）和cout（输出），使用>操作符来处理数据。更加类型安全，支持用户自定义类型的输入输出。#includeintmain(){in
  • 瑶池防线 谜影梦蝶
    冥华虽然逃过了影梦的军队，但他是一个忠臣，他选择上报战况。败给影梦后成逃兵，高层亡尔还活着，七重天失守......随便一条，即可处死冥华。冥华自然是知道以仙界高层的习性此信一发自己必死无疑，但他还选择上报实情，因为责任。同样此信送到仙宫后，知道此事的人，大多数人都认定冥华要完了，所以上到仙界高层，下到扫大街的，包括冥华自己，全都准备好迎接冥华之死。如果仙界现在还属于两方之争的话，冥华必死无疑。然而
  • 爬山后遗症 璃绛
    爬山，攀登，一步一步走向制高点，是一种挑战。成功抵达是一种无法言语的快乐，在山顶吹吹风，看看风景，这是从未有过的体验。然而，爬山一时爽，下山腿打颤，颠簸的路，一路向下走，腿部力量不够，走起来抖到不行，停不下来了！第二天必定腿疼，浑身酸痛，坐立难安！
  • Algorithm 香水浓 javaAlgorithm
    冒泡排序 public static void sort(Integer[] param) { for (int i = param.length - 1; i > 0; i--) { for (int j = 0; j < i; j++) { int current = param[j]; int next = param[j + 1];
  • mongoDB 复杂查询表达式 开窍的石头 mongodb
    1:count    Pg: db.user.find().count();    统计多少条数据 2:不等于$ne    Pg: db.user.find({_id:{$ne:3}},{name:1,sex:1,_id:0});    查询id不等于3的数据。 3：大于$gt $gte(大于等于) &n
  • Jboss Java heap space异常解决方法, jboss OutOfMemoryError : PermGen space 0624chenhong jvmjboss
    转自 http://blog.csdn.net/zou274/article/details/5552630 解决办法： window->preferences->java->installed jres->edit jre 把default vm arguments 的参数设为-Xms64m -Xmx512m ----------------
  • 文件上传 下载 解析 相对路径 不懂事的小屁孩 文件上传
    有点坑吧，弄这么一个简单的东西弄了一天多，身边还有大神指导着，网上各种百度着。 下面总结一下遇到的问题： 文件上传，在页面上传的时候，不要想着去操作绝对路径，浏览器会对客户端的信息进行保护，避免用户信息收到攻击。 在上传图片，或者文件时，使用form表单来操作。 前台通过form表单传输一个流到后台，而不是ajax传递参数到后台，代码如下: <form action=&
  • 怎么实现qq空间批量点赞 换个号韩国红果果 qq
    纯粹为了好玩！！ 逻辑很简单 1 打开浏览器console；输入以下代码。 先上添加赞的代码 var tools={}; //添加所有赞 function init(){ document.body.scrollTop=10000; setTimeout(function(){document.body.scrollTop=0;},2000);//加
  • 判断是否为中文 灵静志远 中文
    方法一： public class Zhidao { public static void main(String args[]) { String s = "sdf灭礌 kjl d{';\fdsjlk是"; int n=0; for(int i=0; i<s.length(); i++) { n = (int)s.charAt(i); if((
  • 一个电话面试后总结 a-john 面试
    今天，接了一个电话面试，对于还是初学者的我来说，紧张了半天。 面试的问题分了层次，对于一类问题，由简到难。自己觉得回答不好的地方作了一下总结：   在谈到集合类的时候，举几个常用的集合类，想都没想，直接说了list,map。   然后对list和map分别举几个类型：   list方面：ArrayList,LinkedList。在谈到他们的区别时，愣住了
  • MSSQL中Escape转义的使用 aijuans MSSQL
    IF OBJECT_ID('tempdb..#ABC') is not null drop table tempdb..#ABC create table #ABC ( PATHNAME NVARCHAR(50) ) insert into #ABC SELECT N'/ABCDEFGHI' UNION ALL SELECT N'/ABCDGAFGASASSDFA' UNION ALL
  • 一个简单的存储过程 asialee mysql存储过程构造数据批量插入
               今天要批量的生成一批测试数据，其中中间有部分数据是变化的，本来想写个程序来生成的，后来想到存储过程就可以搞定，所以随手写了一个，记录在此：            DELIMITER $$ DROP PROCEDURE IF EXISTS inse
  • annot convert from HomeFragment_1 to Fragment 百合不是茶 android导包错误
    创建了几个类继承Fragment, 需要将创建的类存储在ArrayList<Fragment>中; 出现不能将new 出来的对象放到队列中,原因很简单;     创建类时引入包是:import android.app.Fragment;      创建队列和对象时使用的包是:import android.support.v4.ap
  • Weblogic10两种修改端口的方法 bijian1013 weblogic端口号配置管理config.xml
    一.进入控制台进行修改    1.进入控制台:  http://127.0.0.1:7001/console     2.展开左边树菜单         域结构->环境->服务器-->点击AdminServer(管理) &
  • mysql 操作指令 征客丶 mysql
    一、连接mysql 进入 mysql 的安装目录； $ bin/mysql -p [host IP 如果是登录本地的mysql 可以不写 -p 直接 -u] -u [userName] -p 输入密码，回车，接连； 二、权限操作［如果你很了解mysql数据库后，你可以直接去修改系统表，然后用 mysql> flush privileges; 指令让权限生效］ 1、赋权 mys
  • 【Hive一】Hive入门 bit1129 hive
    Hive安装与配置 Hive的运行需要依赖于Hadoop，因此需要首先安装Hadoop2.5.2，并且Hive的启动前需要首先启动Hadoop。   Hive安装和配置的步骤   1. 从如下地址下载Hive0.14.0   http://mirror.bit.edu.cn/apache/hive/    2.解压hive，在系统变
  • ajax 三种提交请求的方法 BlueSkator Ajaxjqery
    1、ajax 提交请求 $.ajax({ type:"post", url : "${ctx}/front/Hotel/getAllHotelByAjax.do", dataType : "json", success : function(result) { try { for(v
  • mongodb开发环境下的搭建入门 braveCS 运维
      linux下安装mongodb 1）官网下载mongodb-linux-x86_64-rhel62-3.0.4.gz 2）linux 解压  gzip -d mongodb-linux-x86_64-rhel62-3.0.4.gz; mv mongodb-linux-x86_64-rhel62-3.0.4 mongodb-linux-x86_64-rhel62-
  • 编程之美-最短摘要的生成 bylijinnan java数据结构算法编程之美
    import java.util.HashMap; import java.util.Map; import java.util.Map.Entry; public class ShortestAbstract { /** * 编程之美 最短摘要的生成 * 扫描过程始终保持一个[pBegin,pEnd]的range,初始化确保[pBegin,pEnd]的ran
  • json数据解析及typeof chengxuyuancsdn jstypeofjson解析
    // json格式 var people='{"authors": [{"firstName": "AAA","lastName": "BBB"},' +' {"firstName": "CCC&
  • 流程系统设计的层次和目标 comsci 设计模式数据结构sql框架脚本
                                  流程系统设计的层次和目标  
  • RMAN List和report 命令 daizj oraclelistreportrman
    LIST 命令 使用RMAN LIST 命令显示有关资料档案库中记录的备份集、代理副本和映像副本的 信息。使用此命令可列出： • RMAN 资料档案库中状态不是AVAILABLE 的备份和副本 • 可用的且可以用于还原操作的数据文件备份和副本 • 备份集和副本，其中包含指定数据文件列表或指定表空间的备份 • 包含指定名称或范围的所有归档日志备份的备份集和副本 • 由标记、完成时间、可
  • 二叉树:红黑树 dieslrae 二叉树
        红黑树是一种自平衡的二叉树,它的查找,插入,删除操作时间复杂度皆为O(logN),不会出现普通二叉搜索树在最差情况时时间复杂度会变为O(N)的问题.     红黑树必须遵循红黑规则,规则如下     1、每个节点不是红就是黑。     2、根总是黑的  &
  • C语言homework3，7个小题目的代码 dcj3sjt126com c
    1、打印100以内的所有奇数。 # include <stdio.h> int main(void) { int i; for (i=1; i<=100; i++) { if (i%2 != 0) printf("%d ", i); } return 0; }  2、从键盘上输入10个整数，
  • 自定义按钮, 图片在上, 文字在下, 居中显示 dcj3sjt126com 自定义
    #import <UIKit/UIKit.h> @interface MyButton : UIButton -(void)setFrame:(CGRect)frame ImageName:(NSString*)imageName Target:(id)target Action:(SEL)action Title:(NSString*)title Font:(CGFloa
  • MySQL查询语句练习题，测试足够用了 flyvszhb sqlmysql
    http://blog.sina.com.cn/s/blog_767d65530101861c.html 1.创建student和score表 CREATE  TABLE  student ( id  INT(10)  NOT NULL  UNIQUE  PRIMARY KEY  , name  VARCHAR
  • 转：MyBatis Generator 详解 happyqing mybatis
      MyBatis Generator 详解 http://blog.csdn.net/isea533/article/details/42102297   MyBatis Generator详解 http://git.oschina.net/free/Mybatis_Utils/blob/master/MybatisGeneator/MybatisGeneator.
  • 让程序员少走弯路的14个忠告 jingjing0907 工作计划学习
      无论是谁，在刚进入某个领域之时，有再大的雄心壮志也敌不过眼前的迷茫：不知道应该怎么做，不知道应该做什么。下面是一名软件开发人员所学到的经验，希望能对大家有所帮助   1.不要害怕在工作中学习。 只要有电脑，就可以通过电子阅读器阅读报纸和大多数书籍。如果你只是做好自己的本职工作以及分配的任务，那是学不到很多东西的。如果你盲目地要求更多的工作，也是不可能提升自己的。放
  • nginx和NetScaler区别 流浪鱼 nginx
    NetScaler是一个完整的包含操作系统和应用交付功能的产品，Nginx并不包含操作系统，在处理连接方面，需要依赖于操作系统，所以在并发连接数方面和防DoS攻击方面，Nginx不具备优势。 2.易用性方面差别也比较大。Nginx对管理员的水平要求比较高，参数比较多，不确定性给运营带来隐患。在NetScaler常见的配置如健康检查，HA等，在Nginx上的配置的实现相对复杂。 3.策略灵活度方
  • 第11章 动画效果（下） onestopweb 动画
    index.html <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/
  • FAQ - SAP BW BO roadmap blueoxygen BOBW
    http://www.sdn.sap.com/irj/boc/business-objects-for-sap-faq   Besides, I care that how to integrate tightly.   By the way, for BW consultants, please just focus on Query Designer which i
  • 关于java堆内存溢出的几种情况 tomcat_oracle javajvmjdkthread
    【情况一】： 　　 java.lang.OutOfMemoryError: Java heap space：这种是java堆内存不够，一个原因是真不够，另一个原因是程序中有死循环； 　　如果是java堆内存不够的话，可以通过调整JVM下面的配置来解决： 　　<jvm-arg>-Xms3062m</jvm-arg> 　　<jvm-arg>-Xmx
  • Manifest.permission_group权限组 阿尔萨斯 Permission
    结构 继承关系 public static final class Manifest.permission_group extends Object java.lang.Object android. Manifest.permission_group 常量 ACCOUNTS 直接通过统计管理器访问管理的统计 COST_MONEY可以用来让用户花钱但不需要通过与他们直接牵涉的权限 D
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他