Kubernetes 访问外部服务
文章目录
-
- 一、概述
-
- 1.K8S访问外部服务
-
- 1.1.ExternalName方式
- 1.2.Endpoint方式
- 2.ExternalIP
- 二、基础中间件
-
- 1.PostgreSQL
-
- 1.1.创建Endpoint
- 1.2.创建镜像
- 1.3.创建服务
- 1.4.测试
- 1.5.搭建Postgresql数据库
- 2.MySQL
-
- 2.1.创建Endpoint
- 2.2.创建镜像
- 2.3.创建服务
- 2.4.测试
- 2.5.搭建MySQL数据库
- 3.RabbitMQ
-
- 3.1.创建Endpoint
- 3.2.测试
- 4.Redis
-
- 4.1.创建Endpoint
- 4.2.创建镜像
- 4.3.创建服务
- 4.4.测试
- 4.5.搭建Redis
- 5.Elasticsearch
-
- 5.1.创建Endpoint
- 5.2.测试
- 三、系统中间件
-
- 1.调用链监控（Zipkin）
-
- 1.1.Zipkin
- 1.2.部署Elasticsearch
- 1.3.部署Kibana
- 1.4.部署RabbitMQ
- 1.5.Docker方式部署Zipkin服务
- 1.6.创建外部服务Endpoint
-
- 1.6.1.RabbitMQ
- 1.6.2.Elasticsearch
- 1.7.K8S部署Zipkin服务
- 1.8.HTTP方式部署Zipkin生产者（msa-zipkin-producer）
-
- 1.8.1.配置文件
- 1.8.2.创建镜像
- 1.8.3.创建服务
- 1.8.4.测试
- 1.9.RabbitMQ方式部署Zipkin生产者（msa-zipkin-producer）
-
- 1.9.1.配置文件
- 1.9.2.删除msa-zipkin-producer服务
- 1.9.3.重新创建服务
- 1.9.4.测试
- 2.流量控制（Sentinel）
-
- 2.1.Sentinel
- 2.2.Docker部署Sentinel服务
- 2.3.K8S部署Sentinel服务
- 2.4.创建Endpoint
- 2.5.部署sentinel生产者（msa-sentinel-producer）
-
- 2.5.1.配置文件
- 2.5.2.创建镜像
- 2.5.3.创建服务
- 2.5.4.测试
- 2.6.部署sentinel消费者（msa-sentinel-consumer）
-
- 2.6.1.配置文件
- 2.6.2.创建镜像
- 2.6.3.创建服务
- 2.6.4.测试
- 2.7.K8S访问外部Sentinel
- 2.8.K8S访问内部Sentinel
- 2.9.总结
- 3.分布式任务调度平台（XXL-JOB）
-
- 3.1.XXL-JOB
- 3.2.Docker部署XXL-JOB服务
- 3.3.K8S部署XXL-JOB服务
- 3.4.配置XXL-JOB
- 3.5.部署XXL-JOB执行器（msa-ext-job）
-
- 3.5.1.配置文件
- 3.5.2.创建镜像
- 3.5.3.创建服务
- 3.5.4.测试
- 3.6.测试XXL-JOB集群
- 4.日志集合（ELK）
-
- 4.1.ELASTIC STACK
- 4.2.环境搭建
- 4.3.配置RabbitMQ
- 4.4.创建Endpoint
- 4.5.创建镜像
- 4.6.创建服务
- 4.4.测试
- 5.系统监控（Prometheus）
-
- 5.1.Prometheus
- 5.2.环境搭建
- 5.3.部署Spring Boot程序（msa-ext-prometheus）
-
- 2.5.1.配置文件
- 2.5.2.创建镜像
- 2.5.3.创建服务
- 2.5.4.测试
- 5.4.Grafana配置
- 5.5.Ingress
- 四、总结
-
- 1.基础中间件
- 2.系统中间件
-
- 2.1.调用链监控（Zipkin）
- 2.2.流量控制（Sentinel）
- 2.3.分布式任务调度平台（XXL-JOB）
- 2.4.日志集合（ELK）
- 2.5.系统监控（Prometheus）
- 2.6.分布式配置管理中心（Apollo-阿波罗）
- Kubernetes
# Kubernetes
https://iothub.org.cn/docs/kubernetes/
https://iothub.org.cn/docs/kubernetes/microservice/k8s-ext-deploy/
一、概述
kubernetes部署无状态服务，有状态服务部署到k8s外部。
1.微服务基本都是无状态服务，部署到k8s内部
2.中间件基本都是有状态服务，部署到k8s外部
1.K8S访问外部服务
k8s集群内的pod需要访问mysql，由于mysql的性质，不适合部署在k8s集群内，故k8s集群内的应用需要链接mysql时，需要配置链接外网的mysql,本次测试 k8s集群ip段为192.168.23.xx。以下提供两种方式，Endpoint和ExternalName方式。
Service可以抽象访问Pod集群，同时 Service也可以抽象其他后端
- 在生产环境中使用外部数据库，在测试环境中使用自己的数据库
- 将自己的Service指向其他集群或者其他命名空间的Service
- 迁移应用到k8s，但是还是有些应用运行在k8s之
1.1.ExternalName方式
通过完全限定域名（FQDN）访问外部服务——创建ExternalName类型的服务。
ExternalName类型的服务创建后，pod可以通过external-service.default.svc.cluster.local域名连接到外部服务，或者通过externale-service。当需要指向其他外部服务时，只需要修改spec.externalName的值即可。
说明： ExternalName 接受 IPv4 地址字符串，但作为包含数字的 DNS 名称，而不是 IP 地址。 类似于 IPv4 地址的外部名称不能由 CoreDNS 或 ingress-nginx 解析，因为外部名称旨在指定规范的 DNS 名称。 要对 IP 地址进行硬编码，请考虑使用 [headless Services](#headless-services)。
参考1：
> mkdir -p ~/mysql-endpoint
> cd ~/mysql-endpoin
> cat <<EOF > my-mysql-external.yaml
apiVersion: v1
kind: Service
metadata:
name: my-mysql-external #此名字随便起
namespace: my-first-app #在固定的命名空间下
spec:
type: ExternalName
externalName: www.baidu.com ##提供方的服务完全限定域名,如rds域名等。
ports:
- port: 80
# ExternalName类型的服务创建后，pod可以通过my-mysql-external.default.svc.cluster.local域名连接到外部服务，
# 或者通过my-mysql-external。当需要指向其他外部服务时，只需要修改spec.externalName的值即可。
EOF
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: prod
spec:
type: ExternalName
externalName: my.database.example.com
1.2.Endpoint方式
endpoint介绍
服务和pod不是直接连接，而是通过Endpoint资源进行连通。endpoint资源是暴露一个服务的ip地址和port的列表。
选择器用于构建ip和port列表，然后存储在endpoint资源中。当客户端连接到服务时，服务代理选择这些列表中的ip和port对中的一个，并将传入连接重定向到在该位置监听的服务器。
endpoint是一个单独的资源并不是服务的属性，endpoint的名称必须和服务的名称相匹配。
创建
为没有选择器的服务创建endpoint资源：$ kubectl create -f endpoint.yml
endpoint对象需要与服务相同的名称，并包含该服务的目标ip和port列表，服务和endpoint资源都发布到服务器后，这样服务就可以像具有pod选择器那样的服务正常使用。
# 查看服务的Endpoint
$ kubectl describe svc service_name
# 通过kubectl describe可以看到Endpoints字段。
# 查看endpoints信息
$ kubectl get endpoints endpoint_name
通过endpoint配置外部服务
将pod关联到外部具有两个endpoint的服务上。
实例1：
创建 endpoints
创建my-mysql-endpoints.yaml
> mkdir -p ~/mysql-endpoint
> cd ~/mysql-endpoint
> cat <<EOF > my-mysql-endpoints.yaml
apiVersion: v1
kind: Endpoints
apiVersion: v1
metadata:
name: my-mysql-endpoint #此名字需与 my-mysql-service.yaml文件中的 metadata.name 的值一致
namespace: my-first-app #在固定的命名空间下
subsets:
- addresses:
- ip: 192.168.23.1 ## 宿主机，由于我的虚拟机ping不通我本机的mysql（安装mysql时候禁用了未打开）
- ip: 220.181.38.148 ## 随便取的一个 公网Ip
ports:
- port: 3306
EOF
创建service
创建my-mysql-service.yaml
> cat <<EOF > my-mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
name: my-mysql-endpoint #此名字需与 my-mysql-endpoints.yaml文件中的 metadata.name 的值一致
namespace: my-first-app #在固定的命名空间下
spec:
ports:
- port: 3306
### 验证，进入，ping一下配置的Ip地址。
# kubectl exec -it my-first-springcloud-94cdd7487-xxxxx -n my-first-app -- /bin/sh
# ping 220.181.38.148
EOF
部署pod 进行验证
部署自己的服务到 my-first-app命名空间下，并且进入容器，进行ping测试
> kubectl exec -it my-first-springcloud-94cdd7487-xxxxx -n my-first-app -- /bin/sh
> ping 220.181.38.148 ## 也可以在容器中直接调用对应的端口测试，此处只演示ping通，就代表能访问了
PING 220.181.38.148 (220.181.38.148): 56 data bytes
64 bytes from 220.181.38.148: seq=0 ttl=127 time=5.356 ms
64 bytes from 220.181.38.148: seq=1 ttl=127 time=4.946 ms
64 bytes from 220.181.38.148: seq=2 ttl=127 time=18.165 ms
。。。。
> 也可以查看service的 Endpoints 信息
> kubectl describe svc my-mysql-endpoint -n my-first-app
Name: my-mysql-endpoint
Namespace: my-first-app
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: 10.102.160.141
Port: <unset> 3306/TCP
TargetPort: 3306/TCP
Endpoints: 192.168.23.1:3306,220.181.38.148:3306
Session Affinity: None
Events: <none>
注意
此种方式只适合Ip访问，对于像阿里云rds等数据库的。需要用域名。则需要用ExternalName方式不而不是Endpoints方式。
实例2：
1.创建mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql-production
namespace: ms
spec:
ports:
- port: 3306
targetPort: 3306
protocol: TCP
2.创建mysql-endpoints.yaml
kind: Endpoints
apiVersion: v1
metadata:
name: mysql-production
namespace: ms
subsets:
- addresses:
- ip: 10.255.20.176
ports:
- port: 3306
实例3：
访问 ES
外部的 es ip为 192.168.0.200 端口为 9400；这里创建一个 Endpoints 和 Service。
cat es.yaml
apiVersion: v1
kind: Service
metadata:
name: es-svc
namespace: klvchen
spec:
ports:
- port: 9400
targetPort: 9400
protocol: TCP
name: tcp
---
apiVersion: v1
kind: Endpoints
metadata:
name: es-svc
namespace: klvchen
subsets:
- addresses:
- ip: 192.168.0.200
ports:
- port: 9400
name: tcp
# 在 K8S 中的容器使用 es-svc.klvchen.svc.cluster.local:9400 就可以访问到 es 了。
访问 Kibana
通过 ingress 管理 Kibana，让用户通过 kibana.klvchen.com 这个域名来访问 192.168.0.200 上的 kibana。
注意：kibana.klvchen.com 域名需要解析到 ingress 上
cat kibana.yaml
apiVersion: v1
kind: Service
metadata:
name: kibana-svc
namespace: klvchen
spec:
ports:
- port: 5601
targetPort: 5601
protocol: TCP
name: http
---
apiVersion: v1
kind: Endpoints
metadata:
name: kibana-svc
namespace: klvchen
subsets:
- addresses:
- ip: 192.168.0.200
ports:
- port: 5601
name: http
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana
namespace: klvchen
spec:
rules:
- host: kibana.klvchen.com
http:
paths:
- path: /
backend:
serviceName: kibana-svc
servicePort: http
2.ExternalIP
如果外部的 IP 路由到集群中一个或多个 Node 上，Kubernetes Service 会被暴露给这些 externalIPs。通过外部 IP（作为目的 IP 地址）进入到集群，打到 Service 端口上的流量，将会被路由到 Service 的 Endpoint 上。
externalIPs 不会被 Kubernetes 管理，它属于集群管理员的职责范畴。
根据 Service 的规定，externalIPs 可以同任意的 ServiceType 来一起指定。在下面的例子中，my-service 可以在【模拟外网IP】“10.0.0.240”(externalIP:port) 上被客户端访问。
二、基础中间件
1.PostgreSQL
1.1.创建Endpoint
postgresql.yml
# postgresql.yml
apiVersion: v1
kind: Service
metadata:
name: postgresql-svc
namespace: dev
spec:
ports:
- port: 5432
targetPort: 5432
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: postgresql-svc
namespace: dev
subsets:
- addresses:
- ip: 182.92.210.65
ports:
- port: 5432
[root@k8s-master postgresql]# vim postgresql.yml
[root@k8s-master postgresql]# kubectl apply -f postgresql.yml
service/postgresql-svc created
endpoints/postgresql-svc created
[root@k8s-master postgresql]# kubectl describe ep postgresql-svc -n dev
Name: postgresql-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 182.92.210.65
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5432 TCP
Events: <none>
[root@k8s-master postgresql]# kubectl describe svc postgresql-svc -n dev
Name: postgresql-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.98.186.25
IPs: 10.98.186.25
Port: <unset> 5432/TCP
TargetPort: 5432/TCP
Endpoints: 182.92.210.65:5432
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
postgresql-svc.dev.svc.cluster.local:5432
1.2.创建镜像
Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-postgresql:1.0.0 .
# 1.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-postgresql:1.0.0 .
[root@k8s-master postgresql]# docker build -t 172.51.216.85:8888/springcloud/msa-ext-postgresql:1.0.0 .
Sending build context to Docker daemon 38.65MB
Step 1/9 : FROM openjdk:8-jdk-alpine
---> a3562aa0b991
Step 2/9 : VOLUME /temp
---> Using cache
---> ac1307e171b0
Step 3/9 : ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
---> Using cache
---> 26a5ca6d54c4
Step 4/9 : ENV JAVA_POS=""
---> Using cache
---> bf77f79bc401
Step 5/9 : ENV ACTIVE="-Dspring.profiles.active=test"
---> Using cache
---> 994f1ecfad1f
Step 6/9 : ADD *.jar app.jar
---> 1d58ae821a58
Step 7/9 : ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
---> Running in c522080a58f8
Removing intermediate container c522080a58f8
---> 5a108972ce58
Step 8/9 : ENV TZ=Asia/Shanghai
---> Running in b67877aab643
Removing intermediate container b67877aab643
---> 2e81b08253e8
Step 9/9 : RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
---> Running in 3d8cfa22bd5b
Removing intermediate container 3d8cfa22bd5b
---> 83d3a622e5c1
Successfully built 83d3a622e5c1
Successfully tagged 172.51.216.85:8888/springcloud/msa-ext-postgresql:1.0.0
# 8-jdk-alpine 镜像小了很多
[root@k8s-master postgresql]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-ext-postgresql 1.0.0 83d3a622e5c1 34 seconds ago 143MB
...
# 2.推送镜像
[root@k8s-master postgresql]# docker push 172.51.216.85:8888/springcloud/msa-ext-postgresql:1.0.0
The push refers to repository [172.51.216.85:8888/springcloud/msa-ext-postgresql]
3a38d43ebd51: Pushed
249d30d27f82: Pushed
ceaf9e1ebef5: Mounted from springcloud/msa-zipkin-producer
9b9b7f3d56a0: Mounted from springcloud/msa-zipkin-producer
f1b5933fe4b5: Mounted from springcloud/msa-zipkin-producer
1.0.0: digest: sha256:4df6d2fe5f8ac8aaac7c06023c18522a509eefbe8be990870d0279b4b8a681f6 size: 1366
1.3.创建服务
msa-ext-postgresql.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-ext-postgresql
labels:
app: msa-ext-postgresql
spec:
type: ClusterIP
ports:
- port: 8111
targetPort: 8111
selector:
app: msa-ext-postgresql
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-ext-postgresql
spec:
replicas: 2
selector:
matchLabels:
app: msa-ext-postgresql
template:
metadata:
labels:
app: msa-ext-postgresql
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-ext-postgresql
image: 172.51.216.85:8888/springcloud/msa-ext-postgresql:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8111
env:
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
[root@k8s-master postgresql]# vim msa-ext-postgresql.yaml
# 创建
[root@k8s-master postgresql]# kubectl apply -f msa-ext-postgresql.yaml
service/msa-ext-postgresql created
deployment.apps/msa-ext-postgresql created
# 查看
[root@k8s-master postgresql]# kubectl get all -n dev | grep msa-ext-postgresql
pod/msa-ext-postgresql-75f6bf657d-mngkd 1/1 Running 0 9m53s
pod/msa-ext-postgresql-75f6bf657d-tzf5x 1/1 Running 0 9m53s
service/msa-ext-postgresql ClusterIP 10.106.82.105 <none> 8111/TCP 9m53s
deployment.apps/msa-ext-postgresql 2/2 2 2 9m53s
replicaset.apps/msa-ext-postgresql-75f6bf657d 2 2 2 9m53s
[root@k8s-master postgresql]# curl http://10.106.82.105:8111/hello
[UserInfo(id=3, name=CCC, age=30), UserInfo(id=4, name=DDD, age=40), UserInfo(id=5, name=EEE, age=50), UserInfo(id=6, name=FFF, age=60), UserInfo(id=7, name=GGG, age=70), UserInfo(id=8, name=HHH, age=80), UserInfo(id=9, name=III, age=90), UserInfo(id=10, name=JJJ, age=100), UserInfo(id=2, name=BBB, age=88), UserInfo(id=1, name=AAA, age=33)]
1.4.测试
访问地址：
curl http://10.106.82.105:8111/hello
注意：经过测试直接使用IP地址也可以访问，不用配置端点
1.5.搭建Postgresql数据库
docker run -d --name timescaledb --network host --restart=always \
-e LANG="C.UTF-8" \
-e 'TZ=Asia/Shanghai' \
-e "POSTGRES_DB=postgres" \
-e "POSTGRES_USER=postgres" \
-e "POSTGRES_PASSWORD=postgres" \
-v /k8s/pg/data/psql:/var/lib/postgresql/data \
timescale/timescaledb:2.1.0-pg12
182.92.210.65
5432
postgres/postgres
2.MySQL
2.1.创建Endpoint
mysql.yml
# mysql.yml
apiVersion: v1
kind: Service
metadata:
name: mysql-svc
namespace: dev
spec:
ports:
- port: 3306
targetPort: 3306
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: mysql-svc
namespace: dev
subsets:
- addresses:
- ip: 182.92.210.65
ports:
- port: 3306
[root@k8s-master mysql]# vim mysql.yml
[root@k8s-master mysql]# kubectl apply -f mysql.yml
service/mysql-svc created
endpoints/mysql-svc created
[root@k8s-master mysql]# kubectl describe ep mysql-svc -n dev
Name: mysql-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 182.92.210.65
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 3306 TCP
Events: <none>
[root@k8s-master mysql]# kubectl describe svc mysql-svc -n dev
Name: mysql-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.99.194.252
IPs: 10.99.194.252
Port: <unset> 3306/TCP
TargetPort: 3306/TCP
Endpoints: 182.92.210.65:3306
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
mysql-svc.dev.svc.cluster.local:3306
2.2.创建镜像
Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-mysql:1.0.0 .
# 1.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-mysql:1.0.0 .
[root@k8s-master mysql]# docker build -t 172.51.216.85:8888/springcloud/msa-ext-mysql:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master mysql]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-ext-mysql 1.0.0 0619f0e88e25 28 seconds ago 145MB
...
# 2.推送镜像
[root@k8s-master mysql]# docker push 172.51.216.85:8888/springcloud/msa-ext-mysql:1.0.0
2.3.创建服务
msa-ext-mysql.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-ext-mysql
labels:
app: msa-ext-mysql
spec:
type: ClusterIP
ports:
- port: 8112
targetPort: 8112
selector:
app: msa-ext-mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-ext-mysql
spec:
replicas: 2
selector:
matchLabels:
app: msa-ext-mysql
template:
metadata:
labels:
app: msa-ext-mysql
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-ext-mysql
image: 172.51.216.85:8888/springcloud/msa-ext-mysql:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8112
env:
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
[root@k8s-master mysql]# vim msa-ext-mysql.yaml
# 创建
[root@k8s-master mysql]# kubectl apply -f msa-ext-mysql.yaml
service/msa-ext-mysql created
deployment.apps/msa-ext-mysql created
# 查看
[root@k8s-master mysql]# kubectl get all -n dev | grep msa-ext-mysql
pod/msa-ext-mysql-79f5594b6b-2xq7r 1/1 Running 0 48s
pod/msa-ext-mysql-79f5594b6b-jcvwx 1/1 Running 0 48s
service/msa-ext-mysql ClusterIP 10.111.50.177 <none> 8112/TCP 48s
deployment.apps/msa-ext-mysql 2/2 2 2 48s
replicaset.apps/msa-ext-mysql-79f5594b6b 2 2 2 48s
[root@k8s-master mysql]# curl http://10.111.50.177:8112/hello
[UserInfo(id=1, name=AAA, age=10), UserInfo(id=2, name=BBB, age=20), UserInfo(id=3, name=CCC, age=30), UserInfo(id=4, name=DDD, age=40), UserInfo(id=5, name=EEE, age=50), UserInfo(id=6, name=FFF, age=60), UserInfo(id=7, name=GGG, age=70), UserInfo(id=8, name=HHH, age=80), UserInfo(id=9, name=III, age=90), UserInfo(id=10, name=JJJ, age=100)]
2.4.测试
访问地址：
curl http://10.111.50.177:8112/hello
注意：经过测试直接使用IP地址也可以访问，不用配置端点
2.5.搭建MySQL数据库
#创建目录
mkdir -p /k8s/mysql/conf
mkdir -p /k8s/mysql/data
chmod 777 * -R /k8s/mysql/conf
chmod 777 * -R /k8s/mysql/data
#创建配置文件
vim /k8s/mysql/conf/my.cnf
#输入一下内容
[mysqld]
log-bin=mysql-bin #开启二进制日志
server-id=119 #服务id，取本机IP最后三位
sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
#启动容器
docker run -d --restart=always \
-p 3306:3306 \
-v /k8s/mysql/data:/var/lib/mysql \
-v /k8s/mysql/conf:/etc/mysql/ \
-e MYSQL_ROOT_PASSWORD=root \
--name mysql \
percona:5.7.23
#进入mysql容器内部
docker exec -it mysql bash
#登陆mysql
mysql -u root -p
3.RabbitMQ
3.1.创建Endpoint
rabbitmq.yml
# rabbitmq.yml
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
namespace: dev
spec:
ports:
- port: 5672
targetPort: 5672
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: rabbitmq-svc
namespace: dev
subsets:
- addresses:
- ip: 172.51.216.98
ports:
- port: 5672
[root@k8s-master msa-zipkin-producer]# kubectl apply -f rabbitmq.yml
service/rabbitmq-svc unchanged
endpoints/rabbitmq-svc created
[root@k8s-master msa-zipkin-producer]# kubectl describe ep rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]# kubectl describe ep rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]#
[root@k8s-master msa-zipkin-producer]#
[root@k8s-master msa-zipkin-producer]# kubectl describe svc rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.101.38.222
IPs: 10.101.38.222
Port: <unset> 5672/TCP
TargetPort: 5672/TCP
Endpoints: 172.51.216.98:5672
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
rabbitmq-svc.dev.svc.cluster.local:5672
3.2.测试
在**调用链监控（Zipkin）**测试中已经测试通过，不再测试，原理与其他基础中间件相同。
注意：经过测试直接使用IP地址也可以访问，不用配置端点
4.Redis
4.1.创建Endpoint
redis.yml
# redis.yml
apiVersion: v1
kind: Service
metadata:
name: redis-svc
namespace: dev
spec:
ports:
- port: 26379
targetPort: 26379
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: redis-svc
namespace: dev
subsets:
- addresses:
- ip: 182.92.210.65
ports:
- port: 26379
[root@k8s-master redis]# vim redis.yml
[root@k8s-master redis]# kubectl apply -f redis.yml
service/redis-svc created
endpoints/redis-svc created
[root@k8s-master redis]# kubectl describe ep redis-svc -n dev
Name: redis-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 182.92.210.65
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 26379 TCP
Events: <none>
[root@k8s-master redis]# kubectl describe svc redis-svc -n dev
Name: redis-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.104.67.141
IPs: 10.104.67.141
Port: <unset> 26379/TCP
TargetPort: 26379/TCP
Endpoints: 182.92.210.65:26379
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
redis-svc.dev.svc.cluster.local:26379
4.2.创建镜像
Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-redis:1.0.0 .
# 1.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-redis:1.0.0 .
[root@k8s-master redis]# docker build -t 172.51.216.85:8888/springcloud/msa-ext-redis:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master redis]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-ext-redis 1.0.0 c5ba87a89010 24 seconds ago 132MB
...
# 2.推送镜像
[root@k8s-master mysql]# docker push 172.51.216.85:8888/springcloud/msa-ext-redis:1.0.0
4.3.创建服务
msa-ext-redis.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-ext-redis
labels:
app: msa-ext-redis
spec:
type: ClusterIP
ports:
- port: 8113
targetPort: 8113
selector:
app: msa-ext-redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-ext-redis
spec:
replicas: 2
selector:
matchLabels:
app: msa-ext-redis
template:
metadata:
labels:
app: msa-ext-redis
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-ext-redis
image: 172.51.216.85:8888/springcloud/msa-ext-redis:1.0.2
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8113
env:
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
[root@k8s-master redis]# vim msa-ext-redis.yaml
# 创建
[root@k8s-master redis]# kubectl apply -f msa-ext-redis.yaml
service/msa-ext-redis created
deployment.apps/msa-ext-redis created
# 查看
[root@k8s-master redis]# kubectl get all -n dev | grep msa-ext-redis
pod/msa-ext-redis-98c67b6b-kx8n8 1/1 Running 0 9s
pod/msa-ext-redis-98c67b6b-stsw6 1/1 Running 0 9s
service/msa-ext-redis ClusterIP 10.108.244.13 <none> 8113/TCP 9s
deployment.apps/msa-ext-redis 2/2 2 2 9s
replicaset.apps/msa-ext-redis-98c67b6b 2 2 2 9s
[root@k8s-master redis]# curl http://10.108.244.13:8113/hello
hello，this is redis messge! ### Mon Nov 01 18:57:00 CST 2021
4.4.测试
访问地址：
curl http://10.108.244.13:8113/hello
注意：经过测试直接使用IP地址也可以访问，不用配置端点
4.5.搭建Redis
# 创建目录
mkdir -p /k8s/redis/master/log
mkdir -p /k8s/redis/master/data
# 对log目录进行读写授权
chmod 777 /k8s/redis/master/log
# 获取配置文件
1.获取redis配置文件
redis官方提供了一个配置文件样例，通过wget工具下载下来。我用的root用户，就直接下载到/root目录里了
wget http://download.redis.io/redis-stable/redis.conf
注意：下载6.0的安装包获取配置文件
# 修改配置文件
# 注释这一行，表示Redis可以接受任意ip的连接
# bind 127.0.0.1
# 端口
port 6379
# 关闭保护模式
protected-mode no
# 让redis服务后台运行
daemonize no
# 开启数据持久化
appendonly yes
# 设定密码(可选，如果这里开启了密码要求，slave的配置里就要加这个密码. 只是练习配置，就不使用密码认证了)
# requirepass Mypwd@123456
# 从库时需要增加主库配置
# 主库密码（一个集群密码需要保持一致）
masterauth Mypwd@123456
# 配置日志路径，为了便于排查问题，指定redis的日志文件目录
logfile "/var/log/redis/redis.log"
***************************************************************
# 拉取镜像
docker pull redis:6.0
# 运行master容器
docker run -d --network host --restart=always \
--name redis \
--privileged=true \
-v /k8s/redis/master/redis.conf:/usr/local/etc/redis/redis.conf \
-v /k8s/redis/master/data:/data \
-v /k8s/redis/master/log:/var/log/redis \
redis:6.0 \
redis-server /usr/local/etc/redis/redis.conf
5.Elasticsearch
5.1.创建Endpoint
elasticsearch.yml
# elasticsearch.yml
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-svc
namespace: dev
spec:
ports:
- port: 9200
targetPort: 9200
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: elasticsearch-svc
namespace: dev
subsets:
- addresses:
- ip: 172.51.216.98
ports:
- port: 9200
[root@k8s-master msa-zipkin-producer]# kubectl apply -f elasticsearch.yml
service/elasticsearch-svc created
endpoints/elasticsearch-svc created
[root@k8s-master msa-zipkin-producer]# kubectl describe ep elasticsearch-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]# kubectl describe ep elasticsearch-svc -n dev
Name: elasticsearch-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 9200 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]# kubectl describe svc elasticsearch-svc -n dev
Name: elasticsearch-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.105.9.231
IPs: 10.105.9.231
Port: <unset> 9200/TCP
TargetPort: 9200/TCP
Endpoints: 172.51.216.98:9200
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
elasticsearch-svc.dev.svc.cluster.local:9200
5.2.测试
在**调用链监控（Zipkin）**测试中已经测试通过，不再测试，原理与其他基础中间件相同。
注意：经过测试直接使用IP地址也可以访问，不用配置端点。
三、系统中间件
1.调用链监控（Zipkin）
1.1.Zipkin
1.Zipkin 服务端
关于 Zipkin 的服务端，在使用 Spring Boot 2.x 版本后，官方就不推荐自行定制编译了，反而是直接提供了编译好的 jar 包来给我们使用，
详情请看 https://github.com/openzipkin/zipkin/issues/1962
并且以前的@EnableZipkinServer也已经被打上了@Deprecated
If you decide to make a custom server, you accept responsibility for troubleshooting your build or configuration problems, even if such problems are a reaction to a change made by the OpenZipkin maintainers. In other words, custom servers are possible, but not supported.
EnableZipkinServer.javagithub.com/openzipkin/zipkin/blob/master/zipkin-server/src/main/java/zipkin/server/EnableZipkinServer.java
简而言之就是：私自改包，后果自负。
所以官方提供了一键脚本
curl -sSL https://zipkin.io/quickstart.sh | bash -s
java -jar zipkin.jar
如果用 Docker 的话，直接
docker run -d -p 9411:9411 openzipkin/zipkin
访问 http://8.131.70.152:9411/zipkin/
2.通信方式
zipkin客户端与服务端通信方式：
HTTP方式
RabbitMQ方式
Rabbitmq方式
依赖加上rabbitmq的依赖，依赖如下：
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
</dependency>
在配置文件上需要配置rabbitmq的配置，配置信息如下：
spring:
rabbitmq:
host: localhost
username: guest
password: guest
port: 5672
3.存储方式
# zipkin是支持将链路数据存储在mysql、cassandra、elasticsearch中的。
https://github.com/openzipkin/zipkin/tree/master/zipkin-storage
# masql
https://github.com/openzipkin/zipkin/tree/master/zipkin-storage/mysql-v1
https://github.com/openzipkin/zipkin/blob/master/zipkin-storage/mysql-v1/src/main/resources/mysql.sql
STORAGE_TYPE=mysql MYSQL_HOST=localhost MYSQL_TCP_PORT=3306 MYSQL_USER=root MYSQL_PASS=123456 MYSQL_DB=zipkin java -jar zipkin.jar
java -jar zipkin.jar --zipkin.torage.type=mysql --zipkin.torage.mysql.host=localhost --zipkin.torage.mysql.port=3306 --zipkin.torage.mysql.username=root --zipkin.torage.mysql.password=123456
Elasticsearch
https://github.com/openzipkin/zipkin/tree/master/zipkin-storage/elasticsearch
zipkin-server支持将链路数据存储在ElasticSearch中。需要安装ElasticSearch和Kibana，
下载地址为https://www. elastic.co/products/elasticsearch。
安装完成后启动，其中ElasticSearch的默认端口号为9200，Kibana的默认端口号为5601。
同理，zipkin连接elasticsearch也是从环境变量中读取的，elasticsearch相关的环境变量和对应的属性如下：
STORAGE_TYPE=elasticsearch ES_HOSTS=http://localhost:9200 ES_INDEX=zipkin java -jar zipkin.jar
java -jar zipkin.jar --STORAGE_TYPE=elasticsearch --ES_HOSTS=http://localhost:9200 --ES_INDEX=zipkin
java -jar zipkin.jar --STORAGE_TYPE=elasticsearch --ES_HOSTS=http://localhost:9200 --ES_INDEX=zipkin
java -jar zipkin.jar --zipkin.torage.type=elasticsearch --zipkin.torage.elasticsearch.hosts=http://localhost:9200 --zipkin.torage.elasticsearch.index=zipkin
4.部署方式
部署方式有：MQ方式、HTTP方式。推荐MQ方式。
MQ方式
HTTP方式
1.2.部署Elasticsearch
# 下载镜像 查看镜像
docker pull elasticsearch:7.1.1
# 运行 elasticsearch
docker run -d --name elasticsearch --network host --restart=always \
-e "discovery.type=single-node" elasticsearch:7.1.1
# 检测 elasticsearch 是否启动成功
curl 127.0.0.1:9200
[root@localhost ~]# curl 127.0.0.1:9200
{
"name" : "localhost.localdomain",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "mXpk62HvT3mVlqEdFBEdQA",
"version" : {
"number" : "7.1.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "7a013de",
"build_date" : "2019-05-23T14:04:00.380842Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
# ES地址
http://172.51.216.98:9200
1.3.部署Kibana
# 下载镜像 查看镜像
docker pull kibana:7.1.1
# 运行 Kibana
docker run -d --name kibana --network host --restart=always kibana:7.1.1
# 修改配置文件
docker exec -it kibana /bin/bash
/usr/share/kibana/config/kibana.yml
server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://172.51.216.98:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
#Kibana地址：
http://172.51.216.98:5601
1.4.部署RabbitMQ
***************************************************************
# 安装准备工作
# 拉取镜像
# docker pull rabbitmq:management
# 运行容器
docker run -d --network host --restart=always --name rabbitmq rabbitmq:management
# 容器要挂载的目录
docker exec -it rabbitmq /bin/bash
配置文件目录：/etc/rabbitmq
数据存储目录：/var/lib/rabbitmq
日志目录：/var/log/rabbitmq
# 宿主机创建目录
mkdir -p /ntms/rabbitmq/lib/
mkdir -p /ntms/rabbitmq/etc/
mkdir -p /ntms/rabbitmq/log/
# 容器内容复制到宿主机
docker cp -a rabbitmq:/var/lib/rabbitmq /ntms/rabbitmq/lib/
docker cp -a rabbitmq:/etc/rabbitmq /ntms/rabbitmq/etc/
docker cp -a rabbitmq:/var/log/rabbitmq /ntms/rabbitmq/log/
# 删除容器
docker rm -f rabbitmq
***************************************************************
# 运行容器
docker run -d --network host --restart=always --name rabbitmq \
-v /ntms/rabbitmq/etc/rabbitmq:/etc/rabbitmq \
-v /ntms/rabbitmq/lib/rabbitmq:/var/lib/rabbitmq \
-v /ntms/rabbitmq/log/rabbitmq/:/var/log/rabbitmq \
rabbitmq:management
# 访问地址
http://172.51.216.98:15672/
账户/密码：
guest/guest
5672
1.5.Docker方式部署Zipkin服务
1.RabbitMQ通信方式
# 此种方式也支持HTTP方式
# 拉取镜像
docker pull openzipkin/zipkin:2
# 最新版本
docker pull openzipkin/zipkin:latest
# 运行容器
docker run -d --name zipkin -p 9411:9411 --restart=always \
-e RABBIT_ADDRESSES=172.51.216.98:5672 \
-e RABBIT_USER=guest \
-e RABBIT_PASSWORD=guest \
-e STORAGE_TYPE=elasticsearch \
-e ES_HOSTS=http://172.51.216.98:9200 \
openzipkin/zipkin:latest
2.HTTP通信方式
# 最新版本
docker pull openzipkin/zipkin:latest
# 运行容器
docker run -d --name zipkin -p 9411:9411 --restart=always \
-e STORAGE_TYPE=elasticsearch \
-e ES_HOSTS=http://172.17.88.25:9200 \
openzipkin/zipkin:latest
# zipkin地址
http://172.51.216.98:9411/zipkin/
1.6.创建外部服务Endpoint
1.6.1.RabbitMQ
# rabbitmq.yml
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
namespace: dev
spec:
ports:
- port: 5672
targetPort: 5672
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: rabbitmq-svc
namespace: dev
subsets:
- addresses:
- ip: 172.51.216.98
ports:
- port: 5672
[root@k8s-master msa-zipkin-producer]# kubectl apply -f rabbitmq.yml
service/rabbitmq-svc unchanged
endpoints/rabbitmq-svc created
[root@k8s-master msa-zipkin-producer]# kubectl describe ep rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]# kubectl describe svc rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.101.38.222
IPs: 10.101.38.222
Port: <unset> 5672/TCP
TargetPort: 5672/TCP
Endpoints: 172.51.216.98:5672
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
rabbitmq-svc.dev.svc.cluster.local:5672
1.6.2.Elasticsearch
# elasticsearch.yml
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-svc
namespace: dev
spec:
ports:
- port: 9200
targetPort: 9200
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: elasticsearch-svc
namespace: dev
subsets:
- addresses:
- ip: 172.51.216.98
ports:
- port: 9200
[root@k8s-master msa-zipkin-producer]# kubectl apply -f elasticsearch.yml
service/elasticsearch-svc created
endpoints/elasticsearch-svc created
[root@k8s-master msa-zipkin-producer]# kubectl describe ep elasticsearch-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]# kubectl describe ep elasticsearch-svc -n dev
Name: elasticsearch-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 9200 TCP
Events: <none>
[root@k8s-master msa-zipkin-producer]# kubectl describe svc elasticsearch-svc -n dev
Name: elasticsearch-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.105.9.231
IPs: 10.105.9.231
Port: <unset> 9200/TCP
TargetPort: 9200/TCP
Endpoints: 172.51.216.98:9200
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
elasticsearch-svc.dev.svc.cluster.local:9200
1.7.K8S部署Zipkin服务
zipkin-server.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: zipkin-server
labels:
app: zipkin-server
spec:
type: NodePort
ports:
- port: 9411
targetPort: 9411
nodePort: 30411 #对外暴露30411端口
selector:
app: zipkin-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: zipkin-server
spec:
replicas: 2
selector:
matchLabels:
app: zipkin-server
template:
metadata:
labels:
app: zipkin-server
spec:
containers:
- name: zipkin-server
image: openzipkin/zipkin:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9411
env:
- name: RABBIT_ADDRESSES
value: "rabbitmq-svc.dev.svc.cluster.local:5672"
- name: RABBIT_USER
value: "guest"
- name: RABBIT_PASSWORD
value: "guest"
- name: STORAGE_TYPE
value: "elasticsearch"
- name: ES_HOSTS
value: "elasticsearch-svc.dev.svc.cluster.local:9200"
# 创建
[root@k8s-master msa-zipkin-producer]# kubectl apply -f zipkin-server.yaml
service/zipkin-server created
deployment.apps/zipkin-server created
[root@k8s-master msa-zipkin-producer]# kubectl get all -n dev | grep zipkin-server
pod/zipkin-server-84c5fdcf5d-qf4zh 1/1 Running 0 2m51s
pod/zipkin-server-84c5fdcf5d-x57zf 1/1 Running 0 2m51s
service/zipkin-server NodePort 10.97.79.39 <none> 9411:30411/TCP 2m51s
deployment.apps/zipkin-server 2/2 2 2 2m51s
replicaset.apps/zipkin-server-84c5fdcf5d 2 2 2 2m51s
# 访问地址
http://172.51.216.81:30411/zipkin
1.8.HTTP方式部署Zipkin生产者（msa-zipkin-producer）
1.8.1.配置文件
1.微服务配置文件
# application-k8shttp.yml
server:
port: 8006
spring:
application:
name: msa-zipkin-producer
sleuth:
web:
client:
enabled: true
sampler:
probability: 1.0 # 将采样比例设置为 1.0，也就是全部都需要。默认是 0.1
zipkin:
base-url: http://zipkin-server.dev.svc.cluster.local:9411/ # 指定了 Zipkin 服务器的地址：svcname.namespace.svc.cluster.local:port
eureka:
client:
#此客户端是否获取eureka服务器注册表上的注册信息，默认为true
fetch-registry: true
#实例是否在eureka服务器上注册自己的信息以供其他服务发现，默认为true,即自己注册自己。
register-with-eureka: true
service-url:
defaultZone: ${EUREKA_SERVER:http://127.0.0.1:10001/eureka/}
#服务注册中心实例的主机名
instance:
hostname: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}
prefer-ip-address: true
instance-id: ${spring.cloud.client.ip-address}:${server.port}
ip-address: ${spring.cloud.client.ip-address}
lease-expiration-duration-in-seconds: 15
lease-renewal-interval-in-seconds: 3
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
zipkin-server.dev.svc.cluster.local:9411
2.Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0 .
3.k8s-yaml
msa-zipkin-producer.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-zipkin-producer
labels:
app: msa-zipkin-producer
spec:
type: ClusterIP
ports:
- port: 8006
targetPort: 8006
selector:
app: msa-zipkin-producer
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-zipkin-producer
spec:
replicas: 2
selector:
matchLabels:
app: msa-zipkin-producer
template:
metadata:
labels:
app: msa-zipkin-producer
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-zipkin-producer
image: 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8006
env:
- name: EUREKA_SERVER
value: "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
- name: ACTIVE
value: "-Dspring.profiles.active=k8shttp"
1.8.2.创建镜像
# 1.上传JAR
把 msa-zipkin-producer-1.0.0.jar 上传到服务器（172.51.216.81），位置（/k8s/springcloud/zipkin）
# 2.创建Dockerfile
[root@k8s-master zipkin]# vim Dockerfile
[root@k8s-master zipkin]# ll
total 52580
-rw-r--r-- 1 root root 396 Oct 28 16:29 Dockerfile
-rw-r--r-- 1 root root 53835607 Oct 28 16:27 msa-zipkin-producer-1.0.0.jar
# 3.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0 .
[root@k8s-master zipkin]# docker build -t 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0 .
Sending build context to Docker daemon 53.84MB
Step 1/9 : FROM openjdk:8-jdk-alpine
---> a3562aa0b991
Step 2/9 : VOLUME /temp
---> Using cache
---> ac1307e171b0
Step 3/9 : ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
---> Using cache
---> 26a5ca6d54c4
Step 4/9 : ENV JAVA_POS=""
---> Using cache
---> bf77f79bc401
Step 5/9 : ENV ACTIVE="-Dspring.profiles.active=test"
---> Using cache
---> 994f1ecfad1f
Step 6/9 : ADD *.jar app.jar
---> 8614329d8c8a
Step 7/9 : ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
---> Running in 40e580252308
Removing intermediate container 40e580252308
---> 4c47cc9a73a2
Step 8/9 : ENV TZ=Asia/Shanghai
---> Running in 54b1a5a7ffe7
Removing intermediate container 54b1a5a7ffe7
---> 0dfe3967db37
Step 9/9 : RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
---> Running in 745ce576d614
Removing intermediate container 745ce576d614
---> c21950e00550
Successfully built c21950e00550
Successfully tagged 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0
# 8-jdk-alpine 镜像小了很多
[root@k8s-master zipkin]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-zipkin-producer 1.0.0 c21950e00550 34 seconds ago 159MB
...
# 4.推送镜像
[root@k8s-master zipkin]# docker push 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0
The push refers to repository [172.51.216.85:8888/springcloud/msa-zipkin-producer]
8fa0f3371377: Pushed
d413b7bd33d8: Pushed
ceaf9e1ebef5: Mounted from springcloud/msa-admin-client
9b9b7f3d56a0: Mounted from springcloud/msa-admin-client
f1b5933fe4b5: Mounted from springcloud/msa-admin-client
1.0.0: digest: sha256:23704a6481b27ae2d4b78eea4974828764db5e94cf7ea0f41c08b33430e02f54 size: 1366
1.8.3.创建服务
[root@k8s-master zipkin]# vim msa-zipkin-producer.yaml
# 创建
[root@k8s-master zipkin]# kubectl apply -f msa-zipkin-producer.yaml
service/msa-zipkin-producer created
deployment.apps/msa-zipkin-producer created
# 查看
[root@k8s-master zipkin]# kubectl get all -n dev | grep msa-zipkin-producer
pod/msa-zipkin-producer-55bc6b6dcc-4mw6f 1/1 Running 0 43s
pod/msa-zipkin-producer-55bc6b6dcc-8jc7h 1/1 Running 0 43s
service/msa-zipkin-producer ClusterIP 10.104.207.255 <none> 8006/TCP 43s
deployment.apps/msa-zipkin-producer 2/2 2 2 43s
replicaset.apps/msa-zipkin-producer-55bc6b6dcc 2 2 2 43s
[root@k8s-master zipkin]# kubectl get pod -n dev -o wide | grep msa-zipkin-producer
msa-zipkin-producer-55bc6b6dcc-4mw6f 1/1 Running 0 101s 10.244.107.200 k8s-node3 <none> <none>
msa-zipkin-producer-55bc6b6dcc-8jc7h 1/1 Running 0 101s 10.244.107.223 k8s-node3 <none> <none>
[root@k8s-master zipkin]# curl 10.104.207.255:8006/hello
hello，this is first messge! ### Fri Oct 29 11:21:08 CST 2021
[root@k8s-master zipkin]# curl 10.104.207.255:8006/say
say，this is first messge！ $$$Fri Oct 29 11:21:30 CST 2021
1.8.4.测试
访问地址：
curl 10.104.207.255:8006/hello
curl 10.104.207.255:8006/say
zipkin地址：
http://172.51.216.81:30411/zipkin/
Kibana地址：
http://172.51.216.98:5601
1.9.RabbitMQ方式部署Zipkin生产者（msa-zipkin-producer）
1.9.1.配置文件
1.微服务配置文件
# application-k8smq.yml
server:
port: 8006
spring:
application:
name: msa-zipkin-producer
sleuth:
web:
client:
enabled: true
sampler:
probability: 1.0 # 将采样比例设置为 1.0，也就是全部都需要。默认是 0.1
zipkin:
# rabbitmq:
# queue: msa-zipkin
# discovery-client-enabled: false
sender:
type: rabbit
rabbitmq:
host: rabbitmq-svc.dev.svc.cluster.local
username: guest
password: guest
port: 5672
eureka:
client:
#此客户端是否获取eureka服务器注册表上的注册信息，默认为true
fetch-registry: true
#实例是否在eureka服务器上注册自己的信息以供其他服务发现，默认为true,即自己注册自己。
register-with-eureka: true
service-url:
defaultZone: ${EUREKA_SERVER:http://127.0.0.1:10001/eureka/}
#服务注册中心实例的主机名
instance:
hostname: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}
prefer-ip-address: true
instance-id: ${spring.cloud.client.ip-address}:${server.port}
ip-address: ${spring.cloud.client.ip-address}
lease-expiration-duration-in-seconds: 15
lease-renewal-interval-in-seconds: 3
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
rabbitmq-svc.dev.svc.cluster.local:9411
2.Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0 .
3.k8s-yaml
msa-zipkin-producer.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-zipkin-producer
labels:
app: msa-zipkin-producer
spec:
type: ClusterIP
ports:
- port: 8006
targetPort: 8006
selector:
app: msa-zipkin-producer
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-zipkin-producer
spec:
replicas: 2
selector:
matchLabels:
app: msa-zipkin-producer
template:
metadata:
labels:
app: msa-zipkin-producer
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-zipkin-producer
image: 172.51.216.85:8888/springcloud/msa-zipkin-producer:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8006
env:
- name: EUREKA_SERVER
value: "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
- name: ACTIVE
value: "-Dspring.profiles.active=k8smq"
1.9.2.删除msa-zipkin-producer服务
[root@k8s-master msa-zipkin-producer]# ll
total 52584
-rw-r--r-- 1 root root 396 Oct 28 16:29 Dockerfile
-rw-r--r-- 1 root root 53835607 Oct 28 16:27 msa-zipkin-producer-1.0.0.jar
-rw-r--r-- 1 root root 1155 Oct 28 16:33 msa-zipkin-producer.yaml
# 1.删除服务
[root@k8s-master msa-zipkin-producer]# kubectl delete -f msa-zipkin-producer.yaml
service "msa-zipkin-producer" deleted
deployment.apps "msa-zipkin-producer" deleted
[root@k8s-master msa-zipkin-producer]# kubectl get all -n dev | grep msa-zipkin-producer
# 2.修改 msa-zipkin-producer.yaml
- name: ACTIVE
value: "-Dspring.profiles.active=k8smq"
1.9.3.重新创建服务
# 创建
[root@k8s-master msa-zipkin-producer]# kubectl apply -f msa-zipkin-producer.yaml
service/msa-zipkin-producer created
deployment.apps/msa-zipkin-producer created
# 查看
[root@k8s-master zipkin]# kubectl get all -n dev | grep msa-zipkin-producer
pod/msa-zipkin-producer-7d5485b79-p8w4c 1/1 Running 0 12s
pod/msa-zipkin-producer-7d5485b79-vj7gj 1/1 Running 0 12s
service/msa-zipkin-producer ClusterIP 10.111.116.227 <none> 8006/TCP 12s
deployment.apps/msa-zipkin-producer 2/2 2 2 12s
replicaset.apps/msa-zipkin-producer-7d5485b79 2 2 2 12s
[root@k8s-master zipkin]# curl 10.111.116.227:8006/hello
hello，this is first messge! ### Fri Oct 29 11:35:33 CST 2021
[root@k8s-master zipkin]# curl 10.111.116.227:8006/say
say，this is first messge！ $$$Fri Oct 29 11:36:16 CST 2021
1.9.4.测试
# 访问地址
curl 10.111.116.227:8006/hello
curl 10.111.116.227:8006/hello
curl 10.111.116.227:8006/say
# zipkin地址
http://172.51.216.81:30411/zipkin/
# Kibana地址
http://172.51.216.98:5601
# 参考
https://hub.docker.com/r/openzipkin/zipkin
https://zipkin.io/
https://github.com/openzipkin/zipkin
kubectl exec -it msa-zipkin-producer-777975c454-mhp7x -n dev -- /bin/sh
kubectl logs -f msa-zipkin-producer-777975c454-mhp7x -n dev
2.流量控制（Sentinel）
2.1.Sentinel
Sentinel: 分布式系统的流量防卫兵
Sentinel 是什么？
随着微服务的流行，服务和服务之间的稳定性变得越来越重要。Sentinel 以流量为切入点，从流量控制、熔断降级、系统负载保护等多个维度保护服务的稳定性。
Sentinel 具有以下特征:
- 丰富的应用场景：Sentinel 承接了阿里巴巴近 10 年的双十一大促流量的核心场景，例如秒杀（即突发流量控制在系统容量可以承受的范围）、消息削峰填谷、集群流量控制、实时熔断下游不可用应用等
- 完备的实时监控：Sentinel 同时提供实时的监控功能。您可以在控制台中看到接入应用的单台机器秒级数据，甚至 500 台以下规模的集群的汇总运行情况
- 广泛的开源生态：Sentinel 提供开箱即用的与其它开源框架/库的整合模块，例如与 Spring Cloud、Apache Dubbo、gRPC、Quarkus 的整合。您只需要引入相应的依赖并进行简单的配置即可快速地接入 Sentinel。同时 Sentinel 提供 Java/Go/C++ 等多语言的原生实现
- 完善的 SPI 扩展机制：Sentinel 提供简单易用、完善的 SPI 扩展接口。您可以通过实现扩展接口来快速地定制逻辑。例如定制规则管理、适配动态数据源等
Sentinel 的主要特性：
Sentinel 的开源生态：
Sentinel 分为两个部分:
- 核心库（Java 客户端）不依赖任何框架/库，能够运行于所有 Java 运行时环境，同时对 Dubbo / Spring Cloud 等框架也有较好的支持
- 控制台（Dashboard）基于 Spring Boot 开发，打包后可以直接运行，不需要额外的 Tomcat 等应用容器
2.2.Docker部署Sentinel服务
# 拉取镜像
docker pull bladex/sentinel-dashboard
# 运行镜像
docker run --name sentinel --restart=always -d --network host bladex/sentinel-dashboard
# 运行镜像
docker run --name sentinel --restart=always -d -p 8858:8858 -d bladex/sentinel-dashboard
# 访问dashboard
访问地址：http://172.51.216.98:8858/#/login
账户密码：sentinel/sentinel
# Linux运行
nohup java -Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8718 -Dproject.name=sentinel-dashboard -jar /usr/local/sentinel-dashboard-1.8.0.jar &
2.3.K8S部署Sentinel服务
sentinel-server.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: sentinel-server
labels:
app: sentinel-server
spec:
type: NodePort
ports:
- port: 8858
targetPort: 8858
nodePort: 30858 #对外暴露30858端口
selector:
app: sentinel-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: sentinel-server
spec:
replicas: 1
selector:
matchLabels:
app: sentinel-server
template:
metadata:
labels:
app: sentinel-server
spec:
containers:
- name: sentinel-server
image: bladex/sentinel-dashboard:1.7.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8858
# 参考容器配置
docker run --name sentinel --restart=always -d -p 8858:8858 -d bladex/sentinel-dashboard
# 创建
[root@k8s-master sentinel]# kubectl apply -f sentinel-server.yaml
service/sentinel-server unchanged
deployment.apps/sentinel-server created
[root@k8s-master sentinel]# kubectl get all -n dev | grep sentinel-server
pod/sentinel-server-5697bdc87b-qfc2b 0/1 ContainerCreating 0 59s
service/sentinel-server NodePort 10.96.6.193 <none> 8858:30858/TCP 2m42s
deployment.apps/sentinel-server 0/1 1 0 59s
replicaset.apps/sentinel-server-5697bdc87b 1 1 0 59s
# 访问dashboard
访问地址：http://172.51.216.81:30858/#/login
账户密码：sentinel/sentinel
2.4.创建Endpoint
sentinel.yml
# rabbitmq.yml
apiVersion: v1
kind: Service
metadata:
name: sentinel-svc
namespace: dev
spec:
ports:
- port: 8858
targetPort: 8858
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: sentinel-svc
namespace: dev
subsets:
- addresses:
- ip: 172.51.216.98
ports:
- port: 8858
[root@k8s-master sentinel]# kubectl apply -f sentinel.yml
service/sentinel-svc created
endpoints/sentinel-svc created
[root@k8s-master sentinel]# kubectl describe ep sentinel-svc -n dev
Name: sentinel-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 8858 TCP
Events: <none>
[root@k8s-master sentinel]# kubectl describe svc sentinel-svc -n dev
Name: sentinel-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.96.106.50
IPs: 10.96.106.50
Port: <unset> 8858/TCP
TargetPort: 8858/TCP
Endpoints: 172.51.216.98:8858
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
sentinel-svc.dev.svc.cluster.local:8858
2.5.部署sentinel生产者（msa-sentinel-producer）
2.5.1.配置文件
1.微服务配置文件
# application-k8s.yml
server:
port: 8211
spring:
application:
name: msa-sentinel-producer
cloud:
sentinel:
transport:
port: 8719
dashboard: ${DASHBOARD:172.51.216.98:8858}
eureka:
client:
#此客户端是否获取eureka服务器注册表上的注册信息，默认为true
fetch-registry: true
#实例是否在eureka服务器上注册自己的信息以供其他服务发现，默认为true,即自己注册自己。
register-with-eureka: true
service-url:
defaultZone: ${EUREKA_SERVER:http://127.0.0.1:10001/eureka/}
#服务注册中心实例的主机名
instance:
hostname: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}
prefer-ip-address: true
instance-id: ${spring.cloud.client.ip-address}:${server.port}
ip-address: ${spring.cloud.client.ip-address}
lease-expiration-duration-in-seconds: 15
lease-renewal-interval-in-seconds: 3
2.Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-sentinel-producer:1.0.0 .
3.k8s-yaml
msa-sentinel-producer.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-sentinel-producer
labels:
app: msa-sentinel-producer
spec:
type: ClusterIP
ports:
- port: 8211
targetPort: 8211
selector:
app: msa-sentinel-producer
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-sentinel-producer
spec:
replicas: 2
selector:
matchLabels:
app: msa-sentinel-producer
template:
metadata:
labels:
app: msa-sentinel-producer
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-sentinel-producer
image: 172.51.216.85:8888/springcloud/msa-sentinel-producer:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8211
env:
- name: EUREKA_SERVER
value: "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
- name: DASHBOARD
value: "sentinel-server.dev.svc.cluster.local:8858"
2.5.2.创建镜像
# 1.上传JAR
把 msa-sentinel-producer-1.0.0.jar 上传到服务器（172.51.216.81），位置（/k8s/springcloud/sentinel/producer）
# 2.创建Dockerfile
[root@k8s-master producer]# vim Dockerfile
[root@k8s-master producer]# ll
total 49500
-rw-r--r-- 1 root root 396 Nov 3 12:17 Dockerfile
-rw-r--r-- 1 root root 50683517 Nov 3 11:43 msa-sentinel-producer-1.0.0.jar
# 3.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-sentinel-producer:1.0.0 .
[root@k8s-master producer]# docker build -t 172.51.216.85:8888/springcloud/msa-sentinel-producer:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master producer]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-sentinel-producer 1.0.0 726635d821a9 29 seconds ago 156MB
...
# 4.推送镜像
[root@k8s-master producer]# docker push 172.51.216.85:8888/springcloud/msa-sentinel-producer:1.0.0
2.5.3.创建服务
[root@k8s-master producer]# vim msa-sentinel-producer.yaml
# 创建
[root@k8s-master producer]# kubectl apply -f msa-sentinel-producer.yaml
service/msa-sentinel-producer created
deployment.apps/msa-sentinel-producer created
# 查看
[root@k8s-master producer]# kubectl get all -n dev | grep msa-sentinel-producer
pod/msa-sentinel-producer-7748d85459-28k82 1/1 Running 0 62s
pod/msa-sentinel-producer-7748d85459-pmh2w 1/1 Running 0 62s
service/msa-sentinel-producer ClusterIP 10.96.36.253 <none> 8211/TCP 62s
deployment.apps/msa-sentinel-producer 2/2 2 2 62s
replicaset.apps/msa-sentinel-producer-7748d85459 2 2 2 62s
[root@k8s-master producer]# curl 10.96.36.253:8211/hello
hello，this is first messge! ### Wed Nov 03 13:01:55 CST 2021
[root@k8s-master producer]# curl 10.96.36.253:8211/say
say，this is first messge！ $$$Wed Nov 03 13:02:04 CST 2021
2.5.4.测试
# 访问地址
curl 10.96.36.253:8211/hello
curl 10.96.36.253:8211/say
2.6.部署sentinel消费者（msa-sentinel-consumer）
2.6.1.配置文件
1.微服务配置文件
# application-k8s.yml
server:
port: 8212
spring:
application:
name: msa-sentinel-consumer
cloud:
sentinel:
transport:
port: 8719
dashboard: ${DASHBOARD:172.51.216.98:8858}
eureka:
client:
#此客户端是否获取eureka服务器注册表上的注册信息，默认为true
fetch-registry: true
#实例是否在eureka服务器上注册自己的信息以供其他服务发现，默认为true,即自己注册自己。
register-with-eureka: true
service-url:
defaultZone: ${EUREKA_SERVER:http://127.0.0.1:10001/eureka/}
#服务注册中心实例的主机名
instance:
hostname: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}
prefer-ip-address: true
instance-id: ${spring.cloud.client.ip-address}:${server.port}
ip-address: ${spring.cloud.client.ip-address}
lease-expiration-duration-in-seconds: 15
lease-renewal-interval-in-seconds: 3
2.Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-sentinel-consumer:1.0.0 .
3.k8s-yaml
msa-sentinel-consumer.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-sentinel-consumer
labels:
app: msa-sentinel-consumer
spec:
type: ClusterIP
ports:
- port: 8212
targetPort: 8212
selector:
app: msa-sentinel-consumer
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-sentinel-consumer
spec:
replicas: 2
selector:
matchLabels:
app: msa-sentinel-consumer
template:
metadata:
labels:
app: msa-sentinel-consumer
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-sentinel-consumer
image: 172.51.216.85:8888/springcloud/msa-sentinel-consumer:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8212
env:
- name: EUREKA_SERVER
value: "http://msa-eureka-0.msa-eureka.dev:10001/eureka/,http://msa-eureka-1.msa-eureka.dev:10001/eureka/,http://msa-eureka-2.msa-eureka.dev:10001/eureka/"
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
- name: DASHBOARD
value: "sentinel-server.dev.svc.cluster.local:8858"
2.6.2.创建镜像
# 1.上传JAR
把 msa-sentinel-consumer-1.0.0.jar 上传到服务器（172.51.216.81），位置（/k8s/springcloud/sentinel/consumer）
# 2.创建Dockerfile
[root@k8s-master consumer]# vim Dockerfile
[root@k8s-master consumer]# ll
total 49504
-rw-r--r-- 1 root root 396 Nov 3 12:36 Dockerfile
-rw-r--r-- 1 root root 50686107 Nov 3 11:43 msa-sentinel-consumer-1.0.0.jar
# 3.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-sentinel-consumer:1.0.0 .
[root@k8s-master consumer]# docker build -t 172.51.216.85:8888/springcloud/msa-sentinel-consumer:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master consumer]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-sentinel-consumer 1.0.0 1a2dbd89bad1 22 seconds ago 156MB
...
# 4.推送镜像
[root@k8s-master consumer]# docker push 172.51.216.85:8888/springcloud/msa-sentinel-consumer:1.0.0
2.6.3.创建服务
[root@k8s-master consumer]# vim msa-sentinel-consumer.yaml
# 创建
[root@k8s-master consumer]# kubectl apply -f msa-sentinel-consumer.yaml
service/msa-sentinel-consumer created
deployment.apps/msa-sentinel-consumer created
# 查看
[root@k8s-master consumer]# kubectl get all -n dev | grep msa-sentinel-consumer
pod/msa-sentinel-consumer-79b4ff65df-7ttnj 1/1 Running 0 22s
pod/msa-sentinel-consumer-79b4ff65df-ps9s9 1/1 Running 0 22s
service/msa-sentinel-consumer ClusterIP 10.99.163.236 <none> 8212/TCP 22s
deployment.apps/msa-sentinel-consumer 2/2 2 2 22s
replicaset.apps/msa-sentinel-consumer-79b4ff65df 2 2 2 22s
[root@k8s-master producer]# curl 10.99.163.236:8212/hello
hello，this is first messge! ### Wed Nov 03 13:04:10 CST 2021 openfeign + sentinel!!!
[root@k8s-master producer]# curl 10.99.163.236:8212/say
say，this is first messge！ $$$Wed Nov 03 13:04:15 CST 2021 openfeign + sentinel!!!
2.6.4.测试
# 访问地址
curl 10.99.163.236:8212/hello
curl 10.99.163.236:8212/say
2.7.K8S访问外部Sentinel
1.Sentinel服务
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
sentinel-svc.dev.svc.cluster.local:8858
2.修改配置，重新部署
msa-sentinel-producer.yaml、msa-sentinel-producer.yaml
...
env:
...
- name: DASHBOARD
value: "sentinel-svc.dev.svc.cluster.local:8858"
3.测试
[root@k8s-master consumer]# kubectl get all -n dev | grep msa-sentinel-consumer
pod/msa-sentinel-consumer-75f855f78-bmzpm 1/1 Running 0 23s
pod/msa-sentinel-consumer-75f855f78-xlnpn 1/1 Running 0 23s
service/msa-sentinel-consumer ClusterIP 10.108.135.162 <none> 8212/TCP 23s
deployment.apps/msa-sentinel-consumer 2/2 2 2 23s
replicaset.apps/msa-sentinel-consumer-75f855f78 2 2 2 23s
[root@k8s-master consumer]# curl 10.108.135.162:8212/hello
hello，this is first messge! ### Wed Nov 03 13:32:42 CST 2021 openfeign + sentinel!!!
[root@k8s-master consumer]# curl 10.108.135.162:8212/say
say，this is first messge！ $$$Wed Nov 03 13:32:49 CST 2021 openfeign + sentinel!!!
测试失败！看不到调用结果。
2.8.K8S访问内部Sentinel
1.创建Endpoint(参照2.4.创建Endpoint)
[root@k8s-master sentinel]# kubectl get all -n dev | grep sentinel-server
pod/sentinel-server-5697bdc87b-qfc2b 0/1 ContainerCreating 0 59s
service/sentinel-server NodePort 10.96.6.193 <none> 8858:30858/TCP 2m42s
deployment.apps/sentinel-server 0/1 1 0 59s
replicaset.apps/sentinel-server-5697bdc87b 1 1 0 59s
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
sentinel-server.dev.svc.cluster.local:8858
2.修改配置，重新部署
msa-sentinel-producer.yaml、msa-sentinel-producer.yaml
...
env:
...
- name: DASHBOARD
value: "sentinel-server.dev.svc.cluster.local:8858"
3.测试
[root@k8s-master producer]# kubectl get all -n dev | grep msa-sentinel-consumer
pod/msa-sentinel-consumer-9888fd67c-4sp9f 1/1 Running 0 57s
pod/msa-sentinel-consumer-9888fd67c-wjgdv 1/1 Running 0 57s
service/msa-sentinel-consumer ClusterIP 10.111.103.164 <none> 8212/TCP 57s
deployment.apps/msa-sentinel-consumer 2/2 2 2 57s
replicaset.apps/msa-sentinel-consumer-9888fd67c 2 2 2 57s
[root@k8s-master consumer]# curl 10.111.103.164:8212/hello
hello，this is first messge! ### Wed Nov 03 13:32:42 CST 2021 openfeign + sentinel!!!
[root@k8s-master consumer]# curl 10.111.103.164:8212/say
say，this is first messge！ $$$Wed Nov 03 13:32:49 CST 2021 openfeign + sentinel!!!
# 单实例测试成功
4.运行多个sentinel服务，页面无法正常登陆。
[root@k8s-master sentinel]# kubectl get all -n dev | grep sentinel-server
pod/sentinel-server-5697bdc87b-6c5jj 1/1 Running 0 73s
pod/sentinel-server-5697bdc87b-7czz6 1/1 Running 0 73s
pod/sentinel-server-5697bdc87b-vv45z 1/1 Running 0 73s
service/sentinel-server NodePort 10.99.15.237 <none> 8858:30858/TCP 73s
deployment.apps/sentinel-server 3/3 3 3 73s
replicaset.apps/sentinel-server-5697bdc87b 3 3 3 73s
# 多实例页面无法正常访问
2.9.总结
Sentinel在K8S中以Deployment部署单实例，可以正常使用。多实例页面不能正常访问。
3.分布式任务调度平台（XXL-JOB）
3.1.XXL-JOB
XXL-JOB是一个分布式任务调度平台，其核心设计目标是开发迅速、学习简单、轻量级、易扩展。现已开放源代码并接入多家公司线上产品线，开箱即用。
# 参考
https://github.com/xuxueli/xxl-job
https://www.xuxueli.com/xxl-job/#%E3%80%8A%E5%88%86%E5%B8%83%E5%BC%8F%E4%BB%BB%E5%8A%A1%E8%B0%83%E5%BA%A6%E5%B9%B3%E5%8F%B0XXL-JOB%E3%80%8B
3.2.Docker部署XXL-JOB服务
# 下载镜像
docker pull xuxueli/xxl-job-admin:2.3.0
docker pull xuxueli/xxl-job-admin:2.2.0
# 创建数据库
xxl-job-2.3.0\doc\db\tables_xxl_job.sql
root
root
# 运行容器
docker run --network host -d --restart=always --name xxl-job-admin \
-e PARAMS="--spring.datasource.username=root \
--spring.datasource.password=root \
--spring.datasource.url=jdbc:mysql://172.51.216.98:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai" \
-v /tmp:/data/applogs \
xuxueli/xxl-job-admin:2.3.0
3.3.K8S部署XXL-JOB服务
xxl-job.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: xxl-job
labels:
app: xxl-job
spec:
type: NodePort
ports:
- port: 8080
targetPort: 8080
nodePort: 30880 #对外暴露30880端口
selector:
app: xxl-job
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: xxl-job
spec:
replicas: 1
selector:
matchLabels:
app: xxl-job
template:
metadata:
labels:
app: xxl-job
spec:
containers:
- name: xxl-job
image: xuxueli/xxl-job-admin:2.3.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
env:
- name: PARAMS
value: "--spring.datasource.username=root --spring.datasource.password=root --spring.datasource.url=jdbc:mysql://172.51.216.98:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai"
# 参考容器配置
docker run --network host -d --restart=always --name xxl-job-admin \
-e PARAMS="--spring.datasource.username=root \
--spring.datasource.password=root \
--spring.datasource.url=jdbc:mysql://172.51.216.98:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai" \
-v /tmp:/data/applogs \
xuxueli/xxl-job-admin:2.3.0
# 创建
[root@k8s-master xxljob]# kubectl apply -f xxl-job.yaml
service/xxl-job created
deployment.apps/xxl-job unchanged
[root@k8s-master xxljob]# kubectl get all -n dev | grep xxl-job
pod/xxl-job-dbc6f75f5-62z4j 1/1 Running 0 2m11s
service/xxl-job NodePort 10.105.29.133 <none> 8080:30880/TCP 64s
deployment.apps/xxl-job 1/1 1 1 2m11s
replicaset.apps/xxl-job-dbc6f75f5 1 1 1 2m11s
# 访问dashboard
访问地址：http://172.51.216.81:30880/xxl-job-admin
账户密码：admin/123456
3.4.配置XXL-JOB
1.配置执行器
2.配置任务
3.5.部署XXL-JOB执行器（msa-ext-job）
3.5.1.配置文件
1.微服务配置文件
# application-k8s.yml
# web port
server.port=8116
# no web
#spring.main.web-environment=false
# log config
#logging.config=classpath:logback.xml
### xxl-job admin address list, such as "http://address" or "http://address01,http://address02"
xxl.job.admin.addresses=http://xxl-job.dev.svc.cluster.local:8080/xxl-job-admin
### xxl-job, access token
xxl.job.accessToken=
### xxl-job executor appname
xxl.job.executor.appname=msa-ext-job
### xxl-job executor registry-address: default use address to registry , otherwise use ip:port if address is null
xxl.job.executor.address=
### xxl-job executor server-info
xxl.job.executor.ip=
xxl.job.executor.port=9999
### xxl-job executor log-path
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
### xxl-job executor log-retention-days
xxl.job.executor.logretentiondays=30
# K8S 中的容器访问xxl-job
svcname.namespace.svc.cluster.local:port
xxl-job.dev.svc.cluster.local:8080
2.Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-job:1.0.0 .
3.k8s-yaml
msa-ext-job.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-ext-job
labels:
app: msa-ext-job
spec:
type: ClusterIP
ports:
- port: 8116
targetPort: 8116
selector:
app: msa-ext-job
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-ext-job
spec:
replicas: 3
selector:
matchLabels:
app: msa-ext-job
template:
metadata:
labels:
app: msa-ext-job
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-ext-job
image: 172.51.216.85:8888/springcloud/msa-ext-job:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8116
env:
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
3.5.2.创建镜像
# 1.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-job:1.0.0 .
[root@k8s-master xxljob]# docker build -t 172.51.216.85:8888/springcloud/msa-ext-job:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master xxljob]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-ext-job 1.0.0 5c0c46f8bda5 23 seconds ago 133MB
...
# 4.推送镜像
[root@k8s-master xxljob]# docker push 172.51.216.85:8888/springcloud/msa-ext-job:1.0.0
3.5.3.创建服务
# 创建
[root@k8s-master xxljob]# kubectl apply -f msa-ext-job.yaml
service/msa-ext-job created
deployment.apps/msa-ext-job created
# 查看
[root@k8s-master xxljob]# kubectl get all -n dev | grep msa-ext-job
pod/msa-ext-job-5879fc4844-77p62 1/1 Running 0 45s
pod/msa-ext-job-5879fc4844-h2g52 1/1 Running 0 45s
pod/msa-ext-job-5879fc4844-hwhts 1/1 Running 0 45s
service/msa-ext-job ClusterIP 10.101.225.7 <none> 8116/TCP 45s
deployment.apps/msa-ext-job 3/3 3 3 45s
replicaset.apps/msa-ext-job-5879fc4844 3 3 3 45s
3.5.4.测试
访问dashboard
访问地址：http://172.51.216.81:30880/xxl-job-admin
账户密码：admin/123456
1.查看管理界面
2.查看Pod日志
# 查看pod
[root@k8s-master xxljob]# kubectl get pod -n dev -o wide | grep msa-ext-job
msa-ext-job-5879fc4844-77p62 1/1 Running 0 10m 10.244.107.248 k8s-node3 <none> <none>
msa-ext-job-5879fc4844-h2g52 1/1 Running 0 10m 10.244.169.185 k8s-node2 <none> <none>
msa-ext-job-5879fc4844-hwhts 1/1 Running 0 10m 10.244.107.250 k8s-node3 <none> <none>
[root@k8s-master xxljob]# kubectl logs --tail=10 -f msa-ext-job-5879fc4844-77p62 -n dev
----job one 1---Thu Nov 04 13:44:05 CST 2021
----job one 1---Thu Nov 04 13:44:08 CST 2021
----job one 1---Thu Nov 04 13:44:11 CST 2021
----job one 1---Thu Nov 04 13:44:14 CST 2021
----job one 1---Thu Nov 04 13:44:17 CST 2021
----job one 1---Thu Nov 04 13:44:20 CST 2021
----job one 1---Thu Nov 04 13:44:23 CST 2021
----job one 1---Thu Nov 04 13:44:26 CST 2021
----job one 1---Thu Nov 04 13:44:29 CST 2021
[root@k8s-master xxljob]# kubectl logs --tail=10 -f msa-ext-job-5879fc4844-h2g52 -n dev
----job one 1---Thu Nov 04 13:45:08 CST 2021
----job one 1---Thu Nov 04 13:45:11 CST 2021
----job one 1---Thu Nov 04 13:45:14 CST 2021
----job one 1---Thu Nov 04 13:45:17 CST 2021
----job one 1---Thu Nov 04 13:45:20 CST 2021
----job one 1---Thu Nov 04 13:45:23 CST 2021
----job one 1---Thu Nov 04 13:45:26 CST 2021
----job one 1---Thu Nov 04 13:45:29 CST 2021
----job one 1---Thu Nov 04 13:45:32 CST 2021
----job one 1---Thu Nov 04 13:45:35 CST 2021
[root@k8s-master xxljob]# kubectl logs --tail=10 -f msa-ext-job-5879fc4844-hwhts -n dev
----job one 1---Thu Nov 04 13:46:21 CST 2021
----job one 1---Thu Nov 04 13:46:24 CST 2021
----job one 1---Thu Nov 04 13:46:27 CST 2021
----job one 1---Thu Nov 04 13:46:30 CST 2021
----job one 1---Thu Nov 04 13:46:33 CST 2021
----job one 1---Thu Nov 04 13:46:36 CST 2021
----job one 1---Thu Nov 04 13:46:39 CST 2021
----job one 1---Thu Nov 04 13:46:42 CST 2021
----job one 1---Thu Nov 04 13:46:45 CST 2021
----job one 1---Thu Nov 04 13:46:48 CST 2021
3.6.测试XXL-JOB集群
1.修改配置创建服务
xxl-job.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: xxl-job
spec:
replicas: 3
selector:
matchLabels:
app: xxl-job
template:
...
# 创建
[root@k8s-master xxljob]# kubectl apply -f xxl-job.yaml
service/xxl-job created
deployment.apps/xxl-job unchanged
[root@k8s-master xxljob]# kubectl get all -n dev | grep xxl-job
pod/xxl-job-dbc6f75f5-9hfjt 1/1 Running 0 3m3s
pod/xxl-job-dbc6f75f5-lfftn 1/1 Running 0 3m3s
pod/xxl-job-dbc6f75f5-xvgx9 1/1 Running 0 3m3s
service/xxl-job NodePort 10.103.255.144 <none> 8080:30880/TCP 3m3s
deployment.apps/xxl-job 3/3 3 3 3m3s
replicaset.apps/xxl-job-dbc6f75f5 3 3 3 3m3s
# 访问dashboard
访问地址：http://172.51.216.81:30880/xxl-job-admin
账户密码：admin/123456
测试3个xxl-job实例，可以正常工作。
4.日志集合（ELK）
4.1.ELASTIC STACK
ELK是三个开源软件的缩写，分别表示：Elasticsearch , Logstash和Kibana。
ELK提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用，是目前主流的一种日志分析平台。
一个完整的集中式日志系统，需要包含以下几个主要特点：
收集－能够采集多种来源的日志数据
传输－能够稳定的把日志数据传输到中央系统
存储－如何存储日志数据
分析－可以支持 UI 分析
警告－能够提供错误报告，监控机制
Elasticsearch
Elasticsearch 基于java，是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索
引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash
Logstash 基于java，是一个开源的用于收集,分析和存储日志的工具。
Kibana
Kibana 基于nodejs，也是一个开源和免费的工具，Kibana可以为 Logstash 和ElasticSearch 提供的日志分析友好的 Web 界面，可以汇总、分析和搜索重要数据日志。
Beats
Beats是elastic公司开源的一款采集系统监控数据的代理agent，是在被监控服务器上以客户端形式运行的数据收集器的统称，可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch，然后进行后续的数据分析活动。
Beats由如下组成:
**Packetbeat：**是一个网络数据包分析器，用于监控、收集网络流量信息，Packetbeat嗅探服务器之间的流量，解析应用层协议，并关联到消息的处理，其支 持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、MongoDB、Memcache等协议；
**Filebeat：**用于监控、收集服务器日志文件，其已取代 logstash forwarder；
**Metricbeat：**可定期获取外部系统的监控指标信息，其可以监控、收集 Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、Redis、System、Zookeeper等服务；
**Winlogbeat：**用于监控、收集Windows系统的日志信息；
部署方式：
4.2.环境搭建
需要部署RabbitMQ、Elasticsearch、Kibana，参考调用链监控（Zipkin）。
Logstash先不部署，日志发送到队列就可以。
4.3.配置RabbitMQ
Rabbitmq配置
建queue：msa-ext-elk-queue
建Exchange: msa-ext-elk-exchange 类型：direct topic: log
Exchange binding queue
1.创建队列
2.创建交换机
3.绑定交换机
4.4.创建Endpoint
rabbitmq.yml
# rabbitmq.yml
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-svc
namespace: dev
spec:
ports:
- port: 5672
targetPort: 5672
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: rabbitmq-svc
namespace: dev
subsets:
- addresses:
- ip: 172.51.216.98
ports:
- port: 5672
[root@k8s-master elk]# kubectl apply -f rabbitmq.yml
service/rabbitmq-svc unchanged
endpoints/rabbitmq-svc created
[root@k8s-master elk]# kubectl describe ep rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master elk]# kubectl describe ep rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Subsets:
Addresses: 172.51.216.98
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 5672 TCP
Events: <none>
[root@k8s-master elk]# kubectl describe svc rabbitmq-svc -n dev
Name: rabbitmq-svc
Namespace: dev
Labels: <none>
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 10.101.38.222
IPs: 10.101.38.222
Port: <unset> 5672/TCP
TargetPort: 5672/TCP
Endpoints: 172.51.216.98:5672
Session Affinity: None
Events: <none>
# K8S 中的容器使用访问
svcname.namespace.svc.cluster.local:port
rabbitmq-svc.dev.svc.cluster.local:5672
4.5.创建镜像
Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-elk:1.0.0 .
# 1.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-elk:1.0.0 .
[root@k8s-master elk]# docker build -t 172.51.216.85:8888/springcloud/msa-ext-elk:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master elk]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-ext-elk 1.0.0 708012b77f65 19 seconds ago 126MB
...
# 2.推送镜像
[root@k8s-master elk]# docker push 172.51.216.85:8888/springcloud/msa-ext-elk:1.0.0
4.6.创建服务
msa-ext-elk.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-ext-elk
labels:
app: msa-ext-elk
spec:
type: ClusterIP
ports:
- port: 8114
targetPort: 8114
selector:
app: msa-ext-elk
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-ext-elk
spec:
replicas: 2
selector:
matchLabels:
app: msa-ext-elk
template:
metadata:
labels:
app: msa-ext-elk
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-ext-elk
image: 172.51.216.85:8888/springcloud/msa-ext-elk:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8114
env:
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
[root@k8s-master elk]# vim msa-ext-elk.yaml
# 创建
[root@k8s-master elk]# kubectl apply -f msa-ext-elk.yaml
service/msa-ext-elk created
deployment.apps/msa-ext-elk created
# 查看
[root@k8s-master elk]# kubectl get all -n dev | grep msa-ext-elk
pod/msa-ext-elk-6974d77cb7-kzncm 1/1 Running 0 4s
pod/msa-ext-elk-6974d77cb7-w4dmw 1/1 Running 0 4s
service/msa-ext-elk ClusterIP 10.96.49.31 <none> 8114/TCP 4s
deployment.apps/msa-ext-elk 2/2 2 2 4s
replicaset.apps/msa-ext-elk-6974d77cb7 2 2 2 5s
[root@k8s-master elk]# curl http://10.96.49.31:8114/hello
hello，this is elk messge! ### Tue Nov 02 15:31:58 CST 2021
4.4.测试
访问地址：
curl http://10.96.49.31:8114/hello
注意：经过测试直接使用IP地址也可以访问，不用配置端点
5.系统监控（Prometheus）
5.1.Prometheus
Prometheus 是一套开源的系统监控报警框架。它启发于 Google 的 borgmon 监控系统，由工作在 SoundCloud 的 google 前员工在 2012 年创建，作为社区开源项目进行开发，并于 2015 年正式发布。2016 年，Prometheus 正式加入 Cloud Native Computing Foundation，成为受欢迎度仅次于 Kubernetes 的项目。
部署方式：
K8S内部微服务把Metrics端点暴露出来，Prometheus从端点直接拉取指标。
5.2.环境搭建
1.安装Prometheus
### 安装prometheus
# 创建目录
mkdir -p /ntms/prometheus
mkdir -p /ntms/prometheus/data
mkdir -p /ntms/prometheus/rules
chmod 777 -R /ntms/prometheus/data
# 创建配置文件
/ntms/prometheus/prometheus.yml
touch prometheus.yml
# 修改配置文件
/prometheus/prometheus/prometheus.yml
# prometheus.yml：
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
alerting:
alertmanagers:
- static_configs:
- targets:
- 172.17.88.22:9093
rule_files:
- "rules/*.yml"
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
labels:
instance: prometheus
- job_name: alertmanager
scrape_interval: 5s
static_configs:
- targets: ['172.17.88.22:9093']
labels:
instance: alert
- job_name: spring-boot-actuator
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['172.51.216.81:30158']
labels:
instance: spring-actuator
- job_name: msa-deploy-producer
metrics_path: '/dproducer/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['gateway.k8s.com:31208']
labels:
instance: producer
- job_name: msa-deploy-consumer
metrics_path: '/dconsumer/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['gateway.k8s.com:31208']
labels:
instance: consumer
***************************************************************
### 运行docker
docker run -d --network host --name prometheus --restart=always \
-v /ntms/prometheus:/etc/prometheus \
-v /ntms/prometheus/data:/prometheus \
-e TZ=Asia/Shanghai \
prom/prometheus
### 访问地址
http://172.51.216.98:9090
2.安装Grafana
# 创建目录
mkdir -p /ntms/grafana/data
chmod 777 -R /ntms/grafana/data
### 运行docker
docker run -d --network host --name=grafana --restart=always \
-v /ntms/grafana/data:/var/lib/grafana \
grafana/grafana
### 访问地址
# grafana
http://172.51.216.98:3000
# 默认账号密码
用户名：admin
密码： admin
3.配置数据源
5.3.部署Spring Boot程序（msa-ext-prometheus）
2.5.1.配置文件
1.微服务配置文件
# application-k8s.yml
management:
metrics:
tags:
application: ${spring.application.name}
region: iids
endpoints:
web:
base-path: /actuator
exposure:
include: "*"
endpoint:
health:
show-details: always
2.Dockerfile
FROM openjdk:8-jdk-alpine
VOLUME /temp
ENV JVM_OPS="-Xms256m -Xmx256m -XX:PermSize=512M -XX:MaxPermSize=512m"
ENV JAVA_POS=""
ENV ACTIVE="-Dspring.profiles.active=test"
ADD *.jar app.jar
ENTRYPOINT java -Djava.security.egd=file:/dev/./urandom -jar ${JVM_OPS} ${ACTIVE} app.jar ${JAVA_OPS}
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#编译命令
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-prometheus:1.0.0 .
3.k8s-yaml
msa-ext-prometheus.yaml
apiVersion: v1
kind: Service
metadata:
namespace: dev
name: msa-ext-prometheus
labels:
app: msa-ext-prometheus
spec:
type: NodePort
ports:
- port: 8158
name: msa-ext-prometheus
targetPort: 8158
nodePort: 30158 #对外暴露30158端口
selector:
app: msa-ext-prometheus
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dev
name: msa-ext-prometheus
spec:
replicas: 3
selector:
matchLabels:
app: msa-ext-prometheus
template:
metadata:
labels:
app: msa-ext-prometheus
spec:
imagePullSecrets:
- name: harborsecret #对应创建私有镜像密钥Secret
containers:
- name: msa-ext-prometheus
image: 172.51.216.85:8888/springcloud/msa-ext-prometheus:1.0.0
imagePullPolicy: Always #如果省略imagePullPolicy,策略为IfNotPresent
ports:
- containerPort: 8158
env:
- name: ACTIVE
value: "-Dspring.profiles.active=k8s"
2.5.2.创建镜像
# 1.创建镜像
# docker build -t 172.51.216.85:8888/springcloud/msa-ext-prometheus:1.0.0 .
[root@k8s-master prometheus]# docker build -t 172.51.216.85:8888/springcloud/msa-ext-prometheus:1.0.0 .
# 8-jdk-alpine 镜像小了很多
[root@k8s-master prometheus]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
172.51.216.85:8888/springcloud/msa-ext-prometheus 1.0.0 7107a79a6460 25 seconds ago 123MB
...
# 4.推送镜像
[root@k8s-master prometheus]# docker push 172.51.216.85:8888/springcloud/msa-ext-prometheus:1.0.0
2.5.3.创建服务
# 创建
[root@k8s-master prometheus]# kubectl apply -f msa-ext-prometheus.yaml
service/msa-ext-prometheus created
deployment.apps/msa-ext-prometheus created
# 查看
[root@k8s-master prometheus]# kubectl get all -n dev | grep msa-ext-prometheus
pod/msa-ext-prometheus-6599578495-cjqd9 1/1 Running 0 32s
pod/msa-ext-prometheus-6599578495-tvmdr 1/1 Running 0 32s
pod/msa-ext-prometheus-6599578495-wzlvb 1/1 Running 0 32s
service/msa-ext-prometheus NodePort 10.105.77.203 <none> 8158:30158/TCP 32s
deployment.apps/msa-ext-prometheus 3/3 3 3 32s
replicaset.apps/msa-ext-prometheus-6599578495 3 3 3 32s
# 访问 http://172.51.216.81:30158/actuator
{"_links":{"self":{"href":"http://172.51.216.81:30158/actuator","templated":false},"beans":{"href":"http://172.51.216.81:30158/actuator/beans","templated":false},"caches":{"href":"http://172.51.216.81:30158/actuator/caches","templated":false},"caches-cache":{"href":"http://172.51.216.81:30158/actuator/caches/{cache}","templated":true},"health-path":{"href":"http://172.51.216.81:30158/actuator/health/{*path}","templated":true},"health":{"href":"http://172.51.216.81:30158/actuator/health","templated":false},"info":{"href":"http://172.51.216.81:30158/actuator/info","templated":false},"conditions":{"href":"http://172.51.216.81:30158/actuator/conditions","templated":false},"configprops":{"href":"http://172.51.216.81:30158/actuator/configprops","templated":false},"env-toMatch":{"href":"http://172.51.216.81:30158/actuator/env/{toMatch}","templated":true},"env":{"href":"http://172.51.216.81:30158/actuator/env","templated":false},"loggers":{"href":"http://172.51.216.81:30158/actuator/loggers","templated":false},"loggers-name":{"href":"http://172.51.216.81:30158/actuator/loggers/{name}","templated":true},"heapdump":{"href":"http://172.51.216.81:30158/actuator/heapdump","templated":false},"threaddump":{"href":"http://172.51.216.81:30158/actuator/threaddump","templated":false},"prometheus":{"href":"http://172.51.216.81:30158/actuator/prometheus","templated":false},"metrics":{"href":"http://172.51.216.81:30158/actuator/metrics","templated":false},"metrics-requiredMetricName":{"href":"http://172.51.216.81:30158/actuator/metrics/{requiredMetricName}","templated":true},"scheduledtasks":{"href":"http://172.51.216.81:30158/actuator/scheduledtasks","templated":false},"mappings":{"href":"http://172.51.216.81:30158/actuator/mappings","templated":false}}}
# 访问http://172.51.216.81:30158/actuator/prometheus
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads{application="springboot-prometheus",region="iids",} 16.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{application="springboot-prometheus",id="mapped",region="iids",} 0.0
jvm_buffer_memory_used_bytes{application="springboot-prometheus",id="direct",region="iids",} 8192.0
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads{application="springboot-prometheus",region="iids",} 20.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage{application="springboot-prometheus",region="iids",} 0.08191068754564687
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of major GC",application="springboot-prometheus",cause="Metadata GC Threshold",region="iids",} 1.0
jvm_gc_pause_seconds_sum{action="end of major GC",application="springboot-prometheus",cause="Metadata GC Threshold",region="iids",} 0.081
jvm_gc_pause_seconds_count{action="end of minor GC",application="springboot-prometheus",cause="Allocation Failure",region="iids",} 2.0
jvm_gc_pause_seconds_sum{action="end of minor GC",application="springboot-prometheus",cause="Allocation Failure",region="iids",} 0.055
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of major GC",application="springboot-prometheus",cause="Metadata GC Threshold",region="iids",} 0.0
jvm_gc_pause_seconds_max{action="end of minor GC",application="springboot-prometheus",cause="Allocation Failure",region="iids",} 0.0
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{application="springboot-prometheus",level="error",region="iids",} 0.0
logback_events_total{application="springboot-prometheus",level="debug",region="iids",} 0.0
logback_events_total{application="springboot-prometheus",level="info",region="iids",} 7.0
logback_events_total{application="springboot-prometheus",level="trace",region="iids",} 0.0
logback_events_total{application="springboot-prometheus",level="warn",region="iids",} 0.0
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{application="springboot-prometheus",area="heap",id="Eden Space",region="iids",} 7.1630848E7
jvm_memory_committed_bytes{application="springboot-prometheus",area="heap",id="Tenured Gen",region="iids",} 1.78978816E8
jvm_memory_committed_bytes{application="springboot-prometheus",area="nonheap",id="Metaspace",region="iids",} 3.8535168E7
jvm_memory_committed_bytes{application="springboot-prometheus",area="nonheap",id="Compressed Class Space",region="iids",} 4980736.0
jvm_memory_committed_bytes{application="springboot-prometheus",area="heap",id="Survivor Space",region="iids",} 8912896.0
jvm_memory_committed_bytes{application="springboot-prometheus",area="nonheap",id="Code Cache",region="iids",} 1.1010048E7
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{application="springboot-prometheus",id="mapped",region="iids",} 0.0
jvm_buffer_count_buffers{application="springboot-prometheus",id="direct",region="iids",} 1.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds{application="springboot-prometheus",region="iids",} 1.636345459824E9
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files{application="springboot-prometheus",region="iids",} 26.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes{application="springboot-prometheus",region="iids",} 1.78978816E8
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total{application="springboot-prometheus",region="iids",} 1.77294488E8
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total{application="springboot-prometheus",region="iids",} 0.0
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{application="springboot-prometheus",area="heap",id="Eden Space",region="iids",} 7.1630848E7
jvm_memory_max_bytes{application="springboot-prometheus",area="heap",id="Tenured Gen",region="iids",} 1.78978816E8
jvm_memory_max_bytes{application="springboot-prometheus",area="nonheap",id="Metaspace",region="iids",} -1.0
jvm_memory_max_bytes{application="springboot-prometheus",area="nonheap",id="Compressed Class Space",region="iids",} 1.073741824E9
jvm_memory_max_bytes{application="springboot-prometheus",area="heap",id="Survivor Space",region="iids",} 8912896.0
jvm_memory_max_bytes{application="springboot-prometheus",area="nonheap",id="Code Cache",region="iids",} 2.5165824E8
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{application="springboot-prometheus",region="iids",state="blocked",} 0.0
jvm_threads_states_threads{application="springboot-prometheus",region="iids",state="terminated",} 0.0
jvm_threads_states_threads{application="springboot-prometheus",region="iids",state="waiting",} 12.0
jvm_threads_states_threads{application="springboot-prometheus",region="iids",state="timed-waiting",} 2.0
jvm_threads_states_threads{application="springboot-prometheus",region="iids",state="new",} 0.0
jvm_threads_states_threads{application="springboot-prometheus",region="iids",state="runnable",} 6.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count{application="springboot-prometheus",region="iids",} 1.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{application="springboot-prometheus",id="mapped",region="iids",} 0.0
jvm_buffer_total_capacity_bytes{application="springboot-prometheus",id="direct",region="iids",} 8192.0
# HELP tomcat_sessions_created_sessions_total
# TYPE tomcat_sessions_created_sessions_total counter
tomcat_sessions_created_sessions_total{application="springboot-prometheus",region="iids",} 0.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{application="springboot-prometheus",area="heap",id="Eden Space",region="iids",} 2.2972656E7
jvm_memory_used_bytes{application="springboot-prometheus",area="heap",id="Tenured Gen",region="iids",} 1.4420504E7
jvm_memory_used_bytes{application="springboot-prometheus",area="nonheap",id="Metaspace",region="iids",} 3.5914072E7
jvm_memory_used_bytes{application="springboot-prometheus",area="nonheap",id="Compressed Class Space",region="iids",} 4457376.0
jvm_memory_used_bytes{application="springboot-prometheus",area="heap",id="Survivor Space",region="iids",} 0.0
jvm_memory_used_bytes{application="springboot-prometheus",area="nonheap",id="Code Cache",region="iids",} 1.0979584E7
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage{application="springboot-prometheus",region="iids",} 3.5283607087356907E-7
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads{application="springboot-prometheus",region="iids",} 20.0
# HELP tomcat_sessions_expired_sessions_total
# TYPE tomcat_sessions_expired_sessions_total counter
tomcat_sessions_expired_sessions_total{application="springboot-prometheus",region="iids",} 0.0
# HELP tomcat_sessions_alive_max_seconds
# TYPE tomcat_sessions_alive_max_seconds gauge
tomcat_sessions_alive_max_seconds{application="springboot-prometheus",region="iids",} 0.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files{application="springboot-prometheus",region="iids",} 1048576.0
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes{application="springboot-prometheus",region="iids",} 1.4420504E7
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m{application="springboot-prometheus",region="iids",} 0.3134765625
# HELP tomcat_sessions_active_current_sessions
# TYPE tomcat_sessions_active_current_sessions gauge
tomcat_sessions_active_current_sessions{application="springboot-prometheus",region="iids",} 0.0
# HELP tomcat_sessions_active_max_sessions
# TYPE tomcat_sessions_active_max_sessions gauge
tomcat_sessions_active_max_sessions{application="springboot-prometheus",region="iids",} 0.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds{application="springboot-prometheus",region="iids",} 567.748
# HELP tomcat_sessions_rejected_sessions_total
# TYPE tomcat_sessions_rejected_sessions_total counter
tomcat_sessions_rejected_sessions_total{application="springboot-prometheus",region="iids",} 0.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes{application="springboot-prometheus",region="iids",} 6770.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total{application="springboot-prometheus",region="iids",} 6692952.0
### 查看指标
http://localhost:8080/actuator
http://localhost:8080/actuator/prometheus
http://localhost:8080/actuator/metrics
http://localhost:8080/actuator/metrics/jvm.memory.max
http://localhost:8080/actuator/loggers
http://localhost:8080/actuator/prometheus
2.5.4.测试
访问地址：http://172.51.216.81:30158/actuator
Prometheus地址：http://172.51.216.98:9090
5.4.Grafana配置
### 配置Grafana模板
# 搜索官网模板
https://grafana.com/
https://grafana.com/grafana/dashboards
# 根据模板ID查找模板
ID: XXXX
https://grafana.com/grafana/dashboards/XXXX
### 选择模板
# 4701
https://grafana.com/grafana/dashboards/4701
在Grafana中配置4701如下
5.5.Ingress
# 配置方式，注意配置本地hosts
http://gateway.k8s.com:31208/dconsumer/actuator/prometheus
http://gateway.k8s.com:31208/dproducer/actuator/prometheus
- job_name: msa-deploy-producer
metrics_path: '/dproducer/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['gateway.k8s.com:31208']
labels:
instance: producer
- job_name: msa-deploy-consumer
metrics_path: '/dconsumer/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['gateway.k8s.com:31208']
labels:
instance: consumer
# 配置本地hosts
[root@hadoop101 mort]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.51.216.81 gateway.k8s.com
四、总结
1.基础中间件
K8S内服服务访问外部服务可以配置Endpoint，也可以不配置。例如
# postgresql.yml
apiVersion: v1
kind: Service
metadata:
name: postgresql-svc
namespace: dev
spec:
ports:
- port: 5432
targetPort: 5432
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: postgresql-svc
namespace: dev
subsets:
- addresses:
- ip: 182.92.210.65
ports:
- port: 5432
2.系统中间件
2.1.调用链监控（Zipkin）
Zipkin-Server部署在K8S内部，Deployment方式部署，可以部署多个实例，通过消息队列收集监控数据。
MQ方式
2.2.流量控制（Sentinel）
Sentinel在K8S中以Deployment部署单实例，可以正常使用。多实例页面不能正常访问。
2.3.分布式任务调度平台（XXL-JOB）
XXL-JOB部署在K8S内部，Deployment方式部署，可以部署多个实例。
2.4.日志集合（ELK）
ELK部署在集群外部，K8S内部微服务通过消息队列发送日志。
2.5.系统监控（Prometheus）
Prometheus部署在集群外部，K8S内部微服务把Metrics端点暴露出来，Prometheus从端点直接拉取指标。
通过Ingress暴露Metrics端点。
# Ingress方式配置
- job_name: msa-deploy-producer
metrics_path: '/dproducer/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['gateway.k8s.com:31208']
labels:
instance: producer
- job_name: msa-deploy-consumer
metrics_path: '/dconsumer/actuator/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['gateway.k8s.com:31208']
labels:
instance: consumer
# 实际地址
http://gateway.k8s.com:31208/dconsumer/actuator/prometheus
http://gateway.k8s.com:31208/dproducer/actuator/prometheus
问题1：微服务多实例无法区分，后期解决。
2.6.分布式配置管理中心（Apollo-阿波罗）
后期测试。
- Kubernetes
# Kubernetes
https://iothub.org.cn/docs/kubernetes/
https://iothub.org.cn/docs/kubernetes/microservice/k8s-ext-deploy/