kubernetes环境部署Spring Cloud
目标
将sc的所有组件和微服务部署到k8s环境上。
部署环境
k8s环境
| Host Name | Role | IP |
|---|---|---|
| master1 | k8s-master01/etcd | 192.168.200.87 |
| master2 | k8s-master02/etcd | 192.168.200.206 |
| master3 | k8s-master03/etcd | 192.168.200.209 |
| node1 | k8s-node01 | 192.168.200.11 |
| node2 | k8s-node02 | 192.168.200.12 |
| node3 | k8s-node03 | 192.168.200.205 |
Spring Cloud部署架构
在这里插入图片描述
- apollo部署在物理机上；
- MySQL集群部署成headless service；
- Eureka采用hostNetwork=true的方式，固定宿主机IP和端口;
- 业务系统的前端部署在Ngnix上，访问gateway后端时，需要配置gateway后端的service name+port。
- Gateway而管理前端需要配置gateway后端的URL，所以后端采用Ingress的方式暴露服务，在gateway前端的/etc/hosts上配置后端域名和主机的映射关系；同时Gateway前端也配置成ingress。最后在用户访问的主机上配置Gateway前后段的域名和ip的映射关系；
- 开发环境可以在/etc/hosts中配置域名的映射关系或者开发环境采用Nodeport的形式暴露服务，但需要维护port。
配置动态PVC
-
Step 1: Get connection information for your NFS server. Make sure your NFS server is accessible from your Kubernetes cluster and get the information you need to connect to it. At a minimum you will need its hostname.
-
Step 2: Get the NFS-Client Provisioner files. To setup the provisioner you will download a set of YAML files, edit them to add your NFS server’s connection information and then apply each with the kubectl / oc command.
Get all of the files in the deploy directory of this repository. These instructions assume that you have cloned the external-storage repository and have a bash-shell open in the nfs-client directory.
- Step 3: Setup authorization. If your cluster has RBAC enabled or you are running OpenShift you must authorize the provisioner. If you are in a namespace/project other than “default” edit deploy/rbac.yaml.
Kubernetes:
# Set the subject of the RBAC objects to the current namespace where the provisioner is being deployed
$ NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
$ NAMESPACE=${NS:-default}
$ sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml
$ kubectl create -f deploy/rbac.yaml
- Step 4: Configure the NFS-Client provisioner
Note: To deploy to an ARM-based environment, use: deploy/deployment-arm.yaml instead, otherwise use deploy/deployment.yaml.
Next you must edit the provisioner’s deployment file to add connection information for your NFS server. Edit deploy/deployment.yaml and replace the two occurences of with your server’s hostname.
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/nfs
- name: NFS_SERVER
value:
- name: NFS_PATH
value: /var/nfs
volumes:
- name: nfs-client-root
nfs:
server:
path: /var/nfs
可以修改PROVISIONER_NAME为能够描述nfs存储的名字，比如nfs-storage。设置NFS服务器为192.168.200.13,路径为/nfs_data. 其中PROVISIONER_NAME要和deploy/class.yaml的Storage Class定义保持一致:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/nfs # or choose another name, must match deployment's env PROVISIONER_NAME' 注意不要和静态nfs的重名，可以通过kubectl get storageclass查看。
parameters:
archiveOnDelete: "false" # When set to "false" your PVs will not be archived
# by the provisioner upon deletion of the PVC.
[root@k8s-master01 nfs-client]# kubectl apply -f ./deploy/deployment.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
serviceaccount/nfs-client-provisioner configured
deployment.extensions/nfs-client-provisioner created
[root@k8s-master01 nfs-client]# kubectl apply -f ./deploy/class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
部署步骤
MySQL高可用集群
参考：https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/
拉取镜像：
docker pull ist0ne/xtrabackup
docker tag ist0ne/xtrabackup:latest gcr.io/google-samples/xtrabackup:1.0
[root@k8s-master01 mysql]# vim mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql
labels:
app: mysql
data:
master.cnf: |
[mysqld]
log-bin
log_bin_trust_function_creators=1
lower_case_table_names=1
slave.cnf: |
[mysqld]
super-read-only
log_bin_trust_function_creators=1
[root@k8s-master01 mysql]# kubectl apply -f mysql-configmap.yaml
configmap/mysql created
创建mysql service:
# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
clusterIP: None
selector:
app: mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the master: mysql-0.mysql.
apiVersion: v1
kind: Service
metadata:
name: mysql-read
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
因为后续的statefulset中的 mysql-1/2的headless service pod需要从mysql-0的pod同步数据。
[root@k8s-master01 mysql]# vim mysql-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
selector:
matchLabels:
app: mysql
serviceName: mysql
replicas: 2
template:
metadata:
labels:
app: mysql
spec:
initContainers:
- name: init-mysql
image: mysql:5.7
command:
- bash
- "-c"
- |
set -ex
# Generate mysql server-id from pod ordinal index.
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
# Add an offset to avoid reserved server-id=0 value.
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
# Copy appropriate conf.d files from config-map to emptyDir.
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d/
else
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
- name: conf
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
- name: clone-mysql
image: gcr.io/google-samples/xtrabackup:1.0
command:
- bash
- "-c"
- |
set -ex
# Skip the clone if data already exists.
[[ -d /var/lib/mysql/mysql ]] && exit 0
# Skip the clone on master (ordinal index 0).
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal -eq 0 ]] && exit 0
# Clone data from previous peer.
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# Prepare the backup.
xtrabackup --prepare --target-dir=/var/lib/mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_ALLOW_EMPTY_PASSWORD
value: "1"
ports:
- name: mysql
containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 500m
memory: 1Gi
livenessProbe:
exec:
command: ["mysqladmin", "ping"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
# Check we can execute queries over TCP (skip-networking is off).
command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
- name: xtrabackup
image: gcr.io/google-samples/xtrabackup:1.0
ports:
- name: xtrabackup
containerPort: 3307
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
# Determine binlog position of cloned data, if any.
if [[ -f xtrabackup_slave_info ]]; then
# XtraBackup already generated a partial "CHANGE MASTER TO" query
# because we're cloning from an existing slave.
mv xtrabackup_slave_info change_master_to.sql.in
# Ignore xtrabackup_binlog_info in this case (it's useless).
rm -f xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
# We're cloning directly from master. Parse binlog position.
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm xtrabackup_binlog_info
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
fi
# Check if we need to complete a clone by starting replication.
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
echo "Initializing replication from clone position"
# In case of container restart, attempt this at-most-once.
mv change_master_to.sql.in change_master_to.sql.orig
mysql -h 127.0.0.1 <参考:
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/exposing-tcp-udp-services.md
# 部署nginx-ingress-controller相关的服务帐户、集群角色、集群角色绑定、Deployment、ConfigMap
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
# 暴露某些端口
$ cat nginx-ingress-service.yml
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app: ingress-nginx
ports:
- name: mysql
port: 3306
targetPort: 3306
# 用上述描述文件部署
$ kubectl apply -f nginx-ingress-service.yml
# 等一会儿后，重启Docker for macOS后，应该有进程监听3306端口了
$ lsof -i :3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
com.docke 36484 jeremy 37u IPv4 0xe746861636421a57 0t0 TCP *:mysql (LISTEN)
com.docke 36484 jeremy 39u IPv6 0xe7468616205d110f 0t0 TCP localhost:mysql (LISTEN)
# 然后创建tcp服务相关的ConfigMap，其中mysql是mysql服务的名称，如要反向代理其它tcp服务，相应地修改data里的定义
$ cat nginx-tcp-configmap.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-tcp-configmap
namespace: kube-system
data:
"3306": default/mysql:3306
# 最后修改nginx-ingress-controller运行时的参数，指定tcp服务反向代理的configmap，添加--tcp-services-configmap=kube-system/nginx-tcp-configmap启动参数
kubectl edit deployment nginx-ingress-controller
这时在本机就可以访问mysql服务了：
mysql -uroot -p -h127.0.0.1 -P3306
至此，无论是http协议还是tcp协议的服务，都可以很方便地暴露给外部使用了。
2019-06-13T02:02:13.801814Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=mysql-0-bin' to avoid this problem.
2019-06-13T02:02:13.806673Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-06-13T02:02:13.806698Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-06-13T02:02:13.806703Z 0 [Note] InnoDB: Uses event mutexes
2019-06-13T02:02:13.806712Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-06-13T02:02:13.806716Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-06-13T02:02:13.806720Z 0 [Note] InnoDB: Using Linux native AIO
2019-06-13T02:02:13.807148Z 0 [Note] InnoDB: Number of pools: 1
2019-06-13T02:02:13.807345Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-06-13T02:02:13.810214Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-06-13T02:02:13.823604Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-06-13T02:02:13.827015Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-06-13T02:02:13.840419Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2019-06-13T02:02:13.840449Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2019-06-13T02:02:13.840459Z 0 [Note] InnoDB: Retrying to lock the first data file
2019-06-13T02:02:14.841482Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2019-06-13T02:02:14.841529Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2019-06-13T02:02:15.842949Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
2019-06-13T02:02:15.842987Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2019-06-13T02:02:16.844989Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
[root@k8s-master01 mysql]# df -h|grep mysql
df: ‘/nfs_data’: Stale file handle
192.168.200.13:/nfs_data/default-data-mysql-0-pvc-21eea4dc-867c-11e9-8646-fa163ea2df31 99G 12G 83G 12% /var/lib/kubelet/pods/1880cca9-8d80-11e9-8646-fa163ea2df31/volumes/kubernetes.io~nfs/pvc-21eea4dc-867c-11e9-8646-fa163ea2df31
192.168.200.13:/nfs_data/default-data-mysql-0-pvc-21eea4dc-867c-11e9-8646-fa163ea2df31/mysql 99G 12G 83G 12% /var/lib/kubelet/pods/1880cca9-8d80-11e9-8646-fa163ea2df31/volume-subpaths/pvc-21eea4dc-867c-11e9-8646-fa163ea2df31/mysql/0
Apollo
采集非容器方案.
- portal地址为：http://192.168.200.196:8070
- meta地址为：http://192.168.200.196:8080
Eureka集群
三个eureka指定node启动,并且使用network=host。
- pom.xml文件中，需要增加：
org.springframework.boot
spring-boot-maven-plugin
否则会报no main manifest attribute.
mvn clean package -Pprod
- application.xml
spring:
application:
name: register
security:
user:
name: admin
password: admin
role: USER,ADMIN
devtools:
restart:
enabled: true
livereload:
enabled: false
server:
port: 8761
servlet:
context-path: /
info:
project:
version: v2.0.0
eureka:
instance:
# hostname: localhost
appname: ${spring.application.name}
instanceId: exampleRegistry:${spring.application.instance_id:${random.value}}
prefer-ip-address: true
# enable-self-preservation: false
lease-renewal-interval-in-seconds: 50
lease-expiration-duration-in-seconds: 90
metadata-map:
profile: ${spring.profiles.active}
version: ${info.project.version}
contextPath: ${server.servlet.context-path}
client:
enabled: true
fetch-registry: true
register-with-eureka: true
instance-info-replication-interval-seconds: 10
registry-fetch-interval-seconds: 10
service-url:
defaultZone: http://admin:${security.user.password:admin}@${eureka-rs0.ip}:${eureka-rs0.port}/eureka/,
http://admin:${security.user.password:admin}@${eureka-rs1.ip}:${eureka-rs1.port}/eureka/,
http://admin:${security.user.password:admin}@${eureka-rs2.ip}:${eureka-rs2.port}/eureka/
server:
enable-self-preservation: false
#5秒清理一次
eviction-interval-timer-in-ms: 5000
example:
http:
version: V_1_1 # To use HTTP/2 you will need SSL support (see above the "server.ssl" configuration)
# CORS is only enabled by default with the "dev" profile, so BrowserSync can access the API
cors:
allowed-origins: "*"
allowed-methods: GET, PUT, POST, DELETE, OPTIONS
allowed-headers: "*"
exposed-headers:
allow-credentials: true
max-age: 1800
security:
authentication:
jwt:
# The secret key should come from a Spring Cloud Config server, and not from this file
# If using the "native" profile, it should be located in the "central-config" directory
# If using the "git" profile, it should come from an external Git repository
secret: 123456
# Token is valid 24 hours
token-validity-in-seconds: 86400
token-validity-in-seconds-for-remember-me: 2592000
- 生成镜像
Dockerfile:
FROM openjdk:8-jre
# label for the image
LABEL Description="Eureka Server" Version="2.0.0-SNAPSHOT"
# the version of the archive
ARG VERSION=2.0.0-SNAPSHOT
# mount the temp volume
VOLUME /tmp
ARG JAR_FILE=target/sc-ms-register-${VERSION}.jar
ADD ${JAR_FILE} app.jar
#启动命令在k8s的deployment中指定
#ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
构建镜像：
bash-3.2$ docker build -t sc-ms-register:2.0.0-SNAPSHOT .
Sending build context to Docker daemon 54.54MB
Step 1/6 : FROM openjdk:8-jre
---> 8227a97e3736
Step 2/6 : LABEL Description="Eureka Server" Version="2.0.0-SNAPSHOT"
---> Using cache
---> 088688d8a491
Step 3/6 : ARG VERSION=2.0.0-SNAPSHOT
---> Using cache
---> 31f1ba9edbab
Step 4/6 : VOLUME /tmp
---> Using cache
---> 6616906c090f
Step 5/6 : ARG JAR_FILE=target/sc-ms-register-${VERSION}.jar
---> Using cache
---> 3c2958385876
Step 6/6 : ADD ${JAR_FILE} app.jar
---> cb455140ec1d
Successfully built cb455140ec1d
Successfully tagged sc-ms-register:2.0.0-SNAPSHOT
打上私服的tag:
bash-3.2$ docker tag sc-ms-register:2.0.0-SNAPSHOT 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT
推送到私服：
bash-3.2$ docker push 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT
The push refers to repository [192.168.200.197:80/spring-ms/sc-ms-register]
aeca00338f0a: Pushed
31c25f68bea2: Layer already exists
1e8e4eaf7e3f: Layer already exists
3954384c6668: Layer already exists
2c719774c1e1: Layer already exists
ec62f19bb3aa: Layer already exists
f94641f1fe1f: Layer already exists
latest: digest: sha256:a95264c093cfdd189d5e3d0631d158b7383f4dd7f193f0f2f9583563eb618296 size: 1794
- eurake-01.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: eureka-0
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: eureka-0
template:
metadata:
labels:
app: eureka-0
spec:
nodeName: k8s-master01
terminationGracePeriodSeconds: 60
hostNetwork: true
containers:
- name: eureka
image: 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT
command: ["java"]
args: ["-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar","--server.port=8761","--spring.application.name=eureka","--eureka-rs0.ip=192.168.200.87","--eureka-rs0.port=8761","--eureka-rs1.ip=192.168.200.11","--eureka-rs1.port=8762","--eureka-rs2.ip=192.168.200.12","--eureka-rs2.port=8763"]
ports:
- name: http
containerPort: 8761
hostPort: 8761
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8761
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8761
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: eureka-0
namespace: default
labels:
app: eureka-0
spec:
ports:
- port: 8761
name: eureka-0
targetPort: 8761
selector:
app: eureka-0
eureka-1.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: eureka-1
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: eureka-1
template:
metadata:
labels:
app: eureka-1
spec:
nodeName: k8s-node01
terminationGracePeriodSeconds: 60
hostNetwork: true
containers:
- name: eureka
image: 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT
command: ["java"]
args: ["-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar","--server.port=8762","--spring.application.name=eureka","--eureka-rs0.ip=192.168.200.87","--eureka-rs0.port=8761","--eureka-rs1.ip=192.168.200.11","--eureka-rs1.port=8762","--eureka-rs2.ip=192.168.200.12","--eureka-rs2.port=8763"]
ports:
- name: http
containerPort: 8762
hostPort: 8762
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8762
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8762
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: eureka-1
namespace: default
labels:
app: eureka-1
spec:
ports:
- port: 8762
name: eureka-1
targetPort: 8762
selector:
app: eureka-1
eureka-2.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: eureka-2
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: eureka-2
template:
metadata:
labels:
app: eureka-2
spec:
nodeName: k8s-node02
terminationGracePeriodSeconds: 60
hostNetwork: true
containers:
- name: eureka
image: 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT
command: ["java"]
args: ["-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar","--server.port=8763","--spring.application.name=eureka","--eureka-rs0.ip=192.168.200.87","--eureka-rs0.port=8761","--eureka-rs1.ip=192.168.200.11","--eureka-rs1.port=8762","--eureka-rs2.ip=192.168.200.12","--eureka-rs2.port=8763"]
ports:
- name: http
containerPort: 8763
hostPort: 8763
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8763
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8763
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: eureka-2
namespace: default
labels:
app: eureka-2
spec:
ports:
- port: 8763
name: eureka-2
targetPort: 8763
selector:
app: eureka-2
[root@k8s-master01 register]# kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/eureka-0-594df67869-g4m46 1/1 Running 0 50m 192.168.200.87 k8s-master01
pod/eureka-1-5dbb8d44b7-wvf6g 1/1 Running 0 50m 192.168.200.11 k8s-node01
pod/eureka-2-5df95bfd86-4jjlt 1/1 Running 2 25m 192.168.200.12 k8s-node02
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/eureka-0 ClusterIP 10.104.94.195 8761/TCP 65m app=eureka-0
service/eureka-1 ClusterIP 10.109.14.113 8762/TCP 65m app=eureka-1
service/eureka-2 ClusterIP 10.107.96.28 8763/TCP 65m app=eureka-2
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/eureka-0 1/1 1 1 50m eureka 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT app=eureka-0
deployment.apps/eureka-1 1/1 1 1 50m eureka 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT app=eureka-1
deployment.apps/eureka-2 1/1 1 1 25m eureka 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT app=eureka-2
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/eureka-0-594df67869 1 1 1 50m eureka 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT app=eureka-0,pod-template-hash=594df67869
replicaset.apps/eureka-1-5dbb8d44b7 1 1 1 50m eureka 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT app=eureka-1,pod-template-hash=5dbb8d44b7
replicaset.apps/eureka-2-5df95bfd86 1 1 1 25m eureka 192.168.200.197:80/spring-ms/sc-ms-register:2.0.0-SNAPSHOT app=eureka-2,pod-template-hash=5df95bfd86
Gateway集群
- pom.xml文件中，需要增加：
org.springframework.boot
spring-boot-maven-plugin
否则会报no main manifest attribute.
mvn clean install -Pprod -Dmaven.test.skip=true
- application.yml
注意mysql的URL。
spring:
application:
name: gateway
boot:
admin:
context-path: /admin
probed-endpoints: health,env, metrics, httptrace, threaddump, jolokia, info, logfile, refresh, heapdump, loggers, auditevents
ui:
title: 监控中心
datasource:
url: jdbc:mysql://mysql-0.mysql/ms2?useUnicode=true&characterEncoding=utf-8
password:
username: root
devtools:
livereload:
enabled: true
restart:
enabled: false
jackson:
serialization:
indent_output: true
jpa:
database: MYSQL
database-platform: org.hibernate.dialect.MySQL5InnoDBDialect
properties:
hibernate:
cache:
use_second_level_cache: false
current_session_context_class: org.springframework.orm.hibernate5.SpringSessionContext
dialect: org.hibernate.dialect.MySQL5InnoDBDialect
format_sql: false
id:
new_generator_mappings: true
jdbc:
batch_size: 10
show_sql: true
show-sql: true
messages:
cache-duration: 1
profiles:
active: dev
include: swagger
eureka:
client:
enabled: true
fetch-registry: true
healthcheck:
enabled: true
register-with-eureka: true
service-url:
defaultZone: http://admin:[email protected]:8761/eureka/,http://admin:[email protected]:8762/eureka/,http://admin:[email protected]:8763/eureka/
instance:
instanceId: ${spring.application.name}:${random.value}
appname: ${spring.application.name}
prefer-ip-address: true
lease-renewal-interval-in-seconds: 50
lease-expiration-duration-in-seconds: 90
hystrix:
command:
default:
execution:
isolation:
strategy: SEMAPHORE
semaphore:
maxConcurrentRequests: 300
zuul:
ratelimit:
key-prefix: prefix
repository: IN_MEMORY
enabled: true
default-policy-list:
- limit: 1
quota: 1
refresh-interval: 10
type: #optional
- user
- origin
- url
policy-list::
ms-application:
- limit: 10
quota: 1
refresh-interval: 60
type:
- url
routes:
ms-application:
path: /ms-application/**
service-id: ms-application
strip-prefix: true
test163:
path: /test163/**
url: http://www.163.com
ribbon-isolation-strategy: semaphore
semaphore:
max-semaphores: 300
example:
swagger:
default-include-pattern: /api/.*
basePackage: com.sc
contact-email: [email protected]
contact-name: example
contact- url: http://1xxxx/portal/example
description: 网关Restful APIs
license: Apache 2.0
licenseUrl: http://apache.org
termsOfServiceUrl: http://xxxx/portal/example
title: 网关Restful APIs
version: 1.0.0
cache:
hazelcast:
backup-count: 1
time-to-live-seconds: 3600
cors:
allow-credentials: true
allowed-headers: '*'
allowed-methods: GET, PUT, POST, DELETE, OPTIONS
allowed-origins: '*'
exposed-headers: ''
max-age: 1800
http:
version: V_1_1
logging:
logstash:
enabled: false
host: localhost
port: 5000
queue-size: 512
spectator-metrics:
enabled: false
security:
authentication:
jwt:
secret: ms-key
token-validity-in-seconds: 86400
token-validity-in-seconds-for-remember-me: 2592000
management:
endpoint:
health:
show-details: ALWAYS
endpoints:
web:
exposure:
include: '*'
ribbon:
eureka:
enabled: true
httpclient:
enabled: false
okhttp:
enabled: true
readTimeout: 90000
connectTimeout: 90000
server:
port: 8082
servlet:
context-path: /
session:
cookie:
http-only: true
logging:
file: ./gateway_log.log
- 生成镜像
Dockerfile：
FROM openjdk:8-jre
# label for the image
LABEL Description="GATEWAY Server" Version="2.0.0-SNAPSHOT"
# the version of the archive
ARG VERSION=2.0.0-SNAPSHOT
# mount the temp volume
VOLUME /tmp
ARG JAR_FILE=target/sc-ms-gateway-${VERSION}.jar
ADD ${JAR_FILE} app.jar
#启动命令在k8s的deployment中指定
#ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
docker build -t sc-ms-gateway:latest .
docker tag sc-ms-gateway:latest 192.168.200.197:80/spring-ms/sc-ms-gateway:latest
docker push 192.168.200.197:80/spring-ms/sc-ms-gateway:latest
- MySQL集群导入数据
[root@k8s-master01 gateway]# kubectl run mysql-client --image=mysql:5.7
root@k8s-master01 gateway]# kubectl exec -it mysql-client-56f68f8c7b-klvb9 /bin/sh
# mysql -h mysql-0.mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 16565
Server version: 5.7.26-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>create database ms;
mysql>use ms;
执行数据库脚本。
- k8s部署
gateway-deployment.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gateway
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: gateway
template:
metadata:
labels:
app: gateway
spec:
terminationGracePeriodSeconds: 60
containers:
- name: gateway
image: 192.168.200.197:80/spring-ms/sc-ms-gateway:2.0.0-SNAPSHOT
command: ["java"]
args: ["-jar", "/app.jar"]
ports:
- name: http
containerPort: 8082
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8082
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8082
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: gateway
namespace: default
labels:
app: gateway
spec:
selector:
app: gateway
ports:
- name: http
port: 8082
protocol: TCP
gateway-ingress.yaml：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-ingress
spec:
rules:
- host: gateway.io
http:
paths:
- path: /
backend:
serviceName: gateway
servicePort: 8082
通过trafik-ui查看gateway部署成功：
注意：作为后端的服务API的统一入口，需要部署成gateway-ingress的形式，所有k8s集群内的所有主机都需要在/etc/hosts文件中配置类似：
192.168.200.87 gateway.io的配置内容。
WEB UI
- 构建源码工程
npm install
npm run build
- pom.xml文件中，需要增加：
org.springframework.boot
spring-boot-maven-plugin
否则会报no main manifest attribute.
mvn clean install -Pprod -Dmaven.test.skip=true
- 生成镜像
Dockerfile:
FROM openjdk:8-jre
# label for the image
LABEL Description="gateway web" Version="2.0.0-SNAPSHOT"
# the version of the archive
ARG VERSION=2.0.0-SNAPSHOT
# mount the temp volume
VOLUME /tmp
ARG JAR_FILE=target/sc-ms-web-${VERSION}.jar
ADD ${JAR_FILE} app.jar
docker build -t sc-ms-web:2.0.0-SNAPSHOT .
docker tag sc-ms-web:2.0.0-SNAPSHOT 192.168.200.197:80/spring-ms/sc-ms-web:2.0.0-SNAPSHOT
docker push 192.168.200.197:80/spring-ms/sc-ms-web:2.0.0-SNAPSHOT
- k8s部署
gateway-web-deployment.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gateway-web
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: gateway-web
template:
metadata:
labels:
app: gateway-web
spec:
terminationGracePeriodSeconds: 60
containers:
- name: gateway-web
image: 192.168.200.197:80/spring-ms/sc-ms-web:2.0.0-SNAPSHOT
imagePullPolicy: Always
command: ["java"]
args: ["-jar", "/app.jar"]
ports:
- name: http
containerPort: 8081
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8081
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8081
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: gateway-web
namespace: default
labels:
app: gateway-web
spec:
selector:
app: gateway-web
ports:
- name: http
port: 8081
protocol: TCP
gateway-web-ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway-web-ingress
spec:
rules:
- host: ms.example.com.io
http:
paths:
- path: /
backend:
serviceName: gateway-web
servicePort: 8081
- 在本地的/etc/hosts配置：
192.168.200.87 ms.example.com.io
后台微服务
配置在apollo管理。
-
源码工程
-
生成镜像
Dockerfile:
FROM openjdk:8-jre
# label for the image
LABEL Description="ms-application" Version="2.0.0-SNAPSHOT"
# the version of the archive
ARG VERSION=2.0.0-SNAPSHOT
# mount the temp volume
VOLUME /tmp
ARG JAR_FILE=target/sc-ms-application-${VERSION}.jar
ADD ${JAR_FILE} app.jar
#启动命令在k8s的deployment中指定
#ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-Denv=DEV","-Dapollo.meta=http://192.168.200.196:8080","-jar","/app.jar"]
bash-3.2$ mvn clean package -Dmaven.test.skip=true
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.sc.example:sc-ms-application:jar:2.0.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.springframework.boot:spring-boot-starter:jar -> duplicate declaration of version (?) @ com.sc.example:sc-ms:2.0.0-SNAPSHOT, /Users/lxz/workstation/java/git/example/example-ms-platform-src/pom.xml, line 139, column 16
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.springframework.cloud:spring-cloud-starter-netflix-eureka-server:jar -> duplicate declaration of version 2.0.1.RELEASE @ com.sc.example:sc-ms:2.0.0-SNAPSHOT, /Users/lxz/workstation/java/git/example/example-ms-platform-src/pom.xml, line 173, column 16
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.springframework.boot:spring-boot-starter-test:jar -> duplicate declaration of version 2.0.2.RELEASE @ com.sc.example:sc-ms:2.0.0-SNAPSHOT, /Users/lxz/workstation/java/git/example/example-ms-platform-src/pom.xml, line 203, column 16
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ---------< com.sc.example:sc-ms-application >---------
[INFO] Building sc-ms-application 2.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from thirdparty: http://132.126.2.216:8080/nexus/content/repositories/thirdparty/com/sc/example/sc-common/2.0.0-SNAPSHOT/maven-metadata.xml
Downloading from sc: http://132.126.2.216:8080/nexus/content/repositories/sc/com/sc/example/sc-common/2.0.0-SNAPSHOT/maven-metadata.xml
[WARNING] Could not transfer metadata com.sc.example:sc-common:2.0.0-SNAPSHOT/maven-metadata.xml from/to sc (http://132.126.2.216:8080/nexus/content/repositories/sc): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Could not transfer metadata com.sc.example:sc-common:2.0.0-SNAPSHOT/maven-metadata.xml from/to thirdparty (http://132.126.2.216:8080/nexus/content/repositories/thirdparty): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Failure to transfer com.sc.example:sc-common:2.0.0-SNAPSHOT/maven-metadata.xml from http://132.126.2.216:8080/nexus/content/repositories/sc was cached in the local repository, resolution will not be reattempted until the update interval of sc has elapsed or updates are forced. Original error: Could not transfer metadata com.sc.example:sc-common:2.0.0-SNAPSHOT/maven-metadata.xml from/to sc (http://132.126.2.216:8080/nexus/content/repositories/sc): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Failure to transfer com.sc.example:sc-common:2.0.0-SNAPSHOT/maven-metadata.xml from http://132.126.2.216:8080/nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of thirdparty has elapsed or updates are forced. Original error: Could not transfer metadata com.sc.example:sc-common:2.0.0-SNAPSHOT/maven-metadata.xml from/to thirdparty (http://132.126.2.216:8080/nexus/content/repositories/thirdparty): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
Downloading from sc: http://132.126.2.216:8080/nexus/content/repositories/sc/com/sc/example/sc-core-data-api/2.0.0-SNAPSHOT/maven-metadata.xml
Downloading from thirdparty: http://132.126.2.216:8080/nexus/content/repositories/thirdparty/com/sc/example/sc-core-data-api/2.0.0-SNAPSHOT/maven-metadata.xml
[WARNING] Could not transfer metadata com.sc.example:sc-core-data-api:2.0.0-SNAPSHOT/maven-metadata.xml from/to sc (http://132.126.2.216:8080/nexus/content/repositories/sc): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Could not transfer metadata com.sc.example:sc-core-data-api:2.0.0-SNAPSHOT/maven-metadata.xml from/to thirdparty (http://132.126.2.216:8080/nexus/content/repositories/thirdparty): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Failure to transfer com.sc.example:sc-core-data-api:2.0.0-SNAPSHOT/maven-metadata.xml from http://132.126.2.216:8080/nexus/content/repositories/sc was cached in the local repository, resolution will not be reattempted until the update interval of sc has elapsed or updates are forced. Original error: Could not transfer metadata com.sc.example:sc-core-data-api:2.0.0-SNAPSHOT/maven-metadata.xml from/to sc (http://132.126.2.216:8080/nexus/content/repositories/sc): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Failure to transfer com.sc.example:sc-core-data-api:2.0.0-SNAPSHOT/maven-metadata.xml from http://132.126.2.216:8080/nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of thirdparty has elapsed or updates are forced. Original error: Could not transfer metadata com.sc.example:sc-core-data-api:2.0.0-SNAPSHOT/maven-metadata.xml from/to thirdparty (http://132.126.2.216:8080/nexus/content/repositories/thirdparty): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
Downloading from sc: http://132.126.2.216:8080/nexus/content/repositories/sc/com/sc/example/sc-core-data/2.0.0-SNAPSHOT/maven-metadata.xml
Downloading from thirdparty: http://132.126.2.216:8080/nexus/content/repositories/thirdparty/com/sc/example/sc-core-data/2.0.0-SNAPSHOT/maven-metadata.xml
[WARNING] Could not transfer metadata com.sc.example:sc-core-data:2.0.0-SNAPSHOT/maven-metadata.xml from/to sc (http://132.126.2.216:8080/nexus/content/repositories/sc): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[WARNING] Could not transfer metadata com.sc.example:sc-core-data:2.0.0-SNAPSHOT/maven-metadata.xml from/to thirdparty (http://132.126.2.216:8080/nexus/content/repositories/thirdparty): Connect to 132.126.2.216:8080 [/132.126.2.216] failed: Operation timed out (Connection timed out)
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ sc-ms-application ---
[INFO] Deleting /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/target
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (default-resources) @ sc-ms-application ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ sc-ms-application ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ sc-ms-application ---
[INFO]
[INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ sc-ms-application ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 20 source files to /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/target/classes
[INFO] /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/src/main/java/com/sc/example/ms/application/repository/CategoryDaoImpl.java: 某些输入文件使用了未经检查或不安全的操作。
[INFO] /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/src/main/java/com/sc/example/ms/application/repository/CategoryDaoImpl.java: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ sc-ms-application ---
[INFO] Not copying test resources
[INFO]
[INFO] --- maven-compiler-plugin:3.6.0:testCompile (default-testCompile) @ sc-ms-application ---
[INFO] Not compiling test sources
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ sc-ms-application ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ sc-ms-application ---
[INFO] Building jar: /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/target/sc-ms-application-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- dockerfile-maven-plugin:1.4.0:build (build-image) @ sc-ms-application ---
[INFO] Building Docker context /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application
[INFO]
[INFO] Image will be built as 192.168.200.197:80/spring-ms/sc-ms-application:2.0.0-SNAPSHOT
[INFO]
[INFO] Step 1/6 : FROM openjdk:8-jre
[INFO]
[INFO] Pulling from library/openjdk
[INFO] Digest: sha256:ac846ab1c19f6b57038c01cc566c1df762577e5102ad44e491ad93aeadd0140e
[INFO] Status: Image is up to date for openjdk:8-jre
[INFO] ---> 06a3f089f899
[INFO] Step 2/6 : LABEL Description="ms-application" Version="2.0.0-SNAPSHOT"
[INFO]
[INFO] ---> Running in aea2918da031
[INFO] Removing intermediate container aea2918da031
[INFO] ---> 1129d0209444
[INFO] Step 3/6 : ARG VERSION=2.0.0-SNAPSHOT
[INFO]
[INFO] ---> Running in b082c564f49b
[INFO] Removing intermediate container b082c564f49b
[INFO] ---> 67f67e1632a0
[INFO] Step 4/6 : VOLUME /tmp
[INFO]
[INFO] ---> Running in 36f6b201920f
[INFO] Removing intermediate container 36f6b201920f
[INFO] ---> a0c5e4c81630
[INFO] Step 5/6 : ARG JAR_FILE=target/sc-ms-application-${VERSION}.jar
[INFO]
[INFO] ---> Running in 42935412305b
[INFO] Removing intermediate container 42935412305b
[INFO] ---> c4e1079acbbe
[INFO] Step 6/6 : ADD ${JAR_FILE} app.jar
[INFO]
[INFO] ---> 4f4749899fe3
[INFO] Successfully built 4f4749899fe3
[INFO] Successfully tagged 192.168.200.197:80/spring-ms/sc-ms-application:2.0.0-SNAPSHOT
[INFO]
[INFO] Detected build of image with id 4f4749899fe3
[INFO] Building jar: /Users/lxz/workstation/java/git/example/example-ms-platform-src/sc-ms-application/target/sc-ms-application-2.0.0-SNAPSHOT-docker-info.jar
[INFO] Successfully built 192.168.200.197:80/spring-ms/sc-ms-application:2.0.0-SNAPSHOT
[INFO]
[INFO] --- spring-boot-maven-plugin:2.0.4.RELEASE:repackage (default) @ sc-ms-application ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:13 min
[INFO] Finished at: 2019-06-11T10:10:02+08:00
[INFO] ------------------------------------------------------------------------
由于采用了docker plugin,所以会看上生成镜像的输出日志.
手动构建镜像：
bash-3.2$ docker build -t sc-ms-application:2.0.0-SNAPSHOT .
Sending build context to Docker daemon 81.32MB
Step 1/6 : FROM openjdk:8-jre
---> 06a3f089f899
Step 2/6 : LABEL Description="ms-application" Version="2.0.0-SNAPSHOT"
---> Using cache
---> 1129d0209444
Step 3/6 : ARG VERSION=2.0.0-SNAPSHOT
---> Using cache
---> 67f67e1632a0
Step 4/6 : VOLUME /tmp
---> Using cache
---> a0c5e4c81630
Step 5/6 : ARG JAR_FILE=target/sc-ms-application-${VERSION}.jar
---> Using cache
---> c4e1079acbbe
Step 6/6 : ADD ${JAR_FILE} app.jar
---> d8e2421758df
Successfully built d8e2421758df
Successfully tagged sc-ms-application:2.0.0-SNAPSHOT
bash-3.2$ docker tag sc-ms-application:2.0.0-SNAPSHOT 192.168.200.197:80/spring-ms/sc-ms-application:2.0.0-SNAPSHOT
bash-3.2$ docker push 192.168.200.197:80/spring-ms/sc-ms-application:2.0.0-SNAPSHOT
The push refers to repository [192.168.200.197:80/spring-ms/sc-ms-application]
b346d3e74f5a: Pushed
64c0e2bac644: Mounted from spring-ms/sc-ms-web
1e8e4eaf7e3f: Mounted from spring-ms/sc-ms-web
3954384c6668: Mounted from spring-ms/sc-ms-web
2c719774c1e1: Mounted from spring-ms/sc-ms-web
ec62f19bb3aa: Mounted from spring-ms/sc-ms-web
f94641f1fe1f: Mounted from spring-ms/sc-ms-web
2.0.0-SNAPSHOT: digest: sha256:d8464c7d7a6b6cb361f0d5a82fd48784a8538e6d918aa36e68e3931abf903773 size: 1794
- apollo配置
META-INF/app.properties:
app.id=ms-application-k8s
example.swagger.default-include-pattern = /api/product/.*
example.swagger.basePackage = com.sc
example.swagger.contact-email = [email protected]
example.swagger.contact-name = example
example.swagger.contact- url = http://132.122.1.227/portal/example
example.swagger.description = DEMO RESTful APIs
example.swagger.license = Apache 2.0
example.swagger.licenseUrl = http://apache.org
example.swagger.termsOfServiceUrl = http://132.122.1.227/portal/example
example.swagger.title = DEMO RESTful APIs
example.swagger.version = 2.0.0
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds = 10000
eureka.instance.appname = ${spring.application.name}
eureka.instance.instanceId = ms-application-k8s:${random.value}
eureka.instance.prefer-ip-address = true
eureka.instance.leaseRenewalIntervalInSeconds = 10
eureka.instance.health-check-url-path = ${server.servlet.context-path}/actuator/health
eureka.instance.metadata-map.contextPath = ${server.servlet.context-path}
eureka.client.enabled = true
eureka.client.fetch-registry = false
eureka.client.register-with-eureka = true
eureka.client.registryFetchIntervalSeconds = 5
eureka.client.service-url.defaultZone = http://admin:[email protected]:8761/eureka/,http://admin:[email protected]:8762/eureka/,http://admin:[email protected]:8763/eureka/
spring.application.name = ms-application
spring.profiles.active = dev
spring.profiles.include = swagger
spring.devtools.restart.enabled = true
spring.devtools.livereload.enabled = false
spring.datasource.druid.filter.stat.log-slow-sql = true
spring.datasource.druid.filter.stat.slow-sql-millis = 2000
spring.datasource.druid.filters = stat,wall
spring.datasource.druid.initial-size = 5
spring.datasource.druid.max-active = 20
spring.datasource.druid.max-wait = 10
spring.datasource.druid.min-idle = 10
spring.datasource.druid.password =
spring.datasource.druid.stat-view-servlet.allow =
spring.datasource.druid.stat-view-servlet.deny =
spring.datasource.druid.stat-view-servlet.enabled = true
spring.datasource.druid.stat-view-servlet.login-password = admin
spring.datasource.druid.stat-view-servlet.login-username = admin
spring.datasource.druid.stat-view-servlet.reset-enable = true
spring.datasource.druid.stat-view-servlet.url-pattern = /druid/*
spring.datasource.druid.url = jdbc:mysql://mysql-0.mysql/demo?useUnicode=true&characterEncoding=utf-8
spring.datasource.druid.username = root
spring.datasource.druid.web-stat-filter.enabled = true
spring.datasource.druid.web-stat-filter.exclusions = *.gif,*.png,*.jpg,*.html,*.js,*.css,*.ico,/druid/*
spring.datasource.druid.web-stat-filter.url-pattern = /*
server.port = 8181
server.servlet.context-path = /
management.endpoints.web.exposure.include = *
management.endpoint.health.show-details = ALWAYS
timeout = 10000
batch = 200
dateFormat = yyyy-MM-dd HH:mm:ss.SSS
dateProperty = 2018-09-20 10:01:02.123
- MySQL集群导入数据
[root@k8s-master01 applicaion]# kubectl exec -it mysql-client-56f68f8c7b-s9rvr /bin/sh
# mysql -h mysql-0.mysql -u root
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 16565
Server version: 5.7.26-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>create database demo；
mysql>use demo;
执行数据库脚本。
- k8s配置
application-deployment.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: application-k8s
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: application-k8s
template:
metadata:
labels:
app: application-k8s
spec:
terminationGracePeriodSeconds: 60
containers:
- name: ms-application-k8s
image: 192.168.200.197:80/spring-ms/sc-ms-application:2.0.0-SNAPSHOT
command: ["java"]
args: ["-Denv=DEV","-Dapollo.meta=http://192.168.200.196:8080", "-jar","/app.jar"]
ports:
- name: http
containerPort: 8181
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8181
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8181
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: ms-application-k8s
namespace: default
labels:
app: application-k8s
spec:
selector:
app: application-k8s
ports:
- name: http
port: 8181
protocol: TCP
优化
configMap加载配置文件
目前sc的配置方式，有以下几种：
- 环境变量注入：这种方式把配置参数直接注入系统环境变量，应用直接从环境变量中获取配置信息，配置项较少的情况还是可以用用的，多了麻烦不说，配置的变更审计也是问题，configmap即是这种方式，好处是系统级，外部依赖较少。
- maven profile方式：执行类似：mvn clean package -Pprod命令，在编译打包阶段根将不同环境，加载不同的配置。
- spring boot：Spring Boot中提供了多环境配置功能，可以设置application-{env}.properties区分环境信息，启动时通过
spring.profiles.active=dev参数指定，在启动时将配置信息注入。 - 配置中心：这是微服务架构中比较主流的解决方案，有代表性的有携程的apollo、百度disconf以及spring cloud config。
由于网关的zuul配置项需要经后台服务配置内容，所以将网关gateway的配置分离出来，采用k8s的configMap方式.
-
删除代码工程中的application.yml和apollo的META-INF/app.properties。
-
gateway-config.yaml:
kind: ConfigMap
apiVersion: v1
metadata:
name: gateway-configmap
data:
application.yml: |-
spring:
application:
name: gateway
boot:
admin:
context-path: /admin
probed-endpoints: health,env, metrics, httptrace, threaddump, jolokia, info, logfile, refresh, heapdump, loggers, auditevents
ui:
title: 监控中心
datasource:
url: jdbc:mysql://mysql-0.mysql/ms2?useUnicode=true&characterEncoding=utf-8
password:
username: root
devtools:
livereload:
enabled: true
restart:
enabled: false
jackson:
serialization:
indent_output: true
jpa:
database: MYSQL
database-platform: org.hibernate.dialect.MySQL5InnoDBDialect
properties:
hibernate:
cache:
use_second_level_cache: false
current_session_context_class: org.springframework.orm.hibernate5.SpringSessionContext
dialect: org.hibernate.dialect.MySQL5InnoDBDialect
format_sql: false
id:
new_generator_mappings: true
jdbc:
batch_size: 10
show_sql: true
show-sql: true
messages:
cache-duration: 1
profiles:
active: dev
include: swagger
eureka:
client:
enabled: true
fetch-registry: true
healthcheck:
enabled: true
register-with-eureka: true
service-url:
defaultZone: http://admin:[email protected]:8761/eureka/,http://admin:[email protected]:8762/eureka/,http://admin:[email protected]:8763/eureka/
instance:
instanceId: ${spring.application.name}:${random.value}
appname: ${spring.application.name}
prefer-ip-address: true
lease-renewal-interval-in-seconds: 50
lease-expiration-duration-in-seconds: 90
hystrix:
command:
default:
execution:
isolation:
strategy: SEMAPHORE
semaphore:
maxConcurrentRequests: 300
zuul:
ratelimit:
key-prefix: prefix
repository: IN_MEMORY
enabled: true
default-policy-list:
- limit: 1
quota: 1
refresh-interval: 10
type: #optional
- user
- origin
- url
policy-list:
ms-application:
- limit: 10
quota: 1
refresh-interval: 60
type:
- url
routes:
ms-application:
path: /ms-application/**
service-id: ms-application
strip-prefix: true
ribbon-isolation-strategy: semaphore
semaphore:
max-semaphores: 300
example:
swagger:
default-include-pattern: /api/.*
basePackage: com.sc
contact-email: [email protected]
contact-name: example
contact- url: http://132.122.1.227/portal/example
description: 网关Restful APIs
license: Apache 2.0
licenseUrl: http://apache.org
termsOfServiceUrl: http://132.122.1.227/portal/example
title: 网关Restful APIs
version: 1.0.0
cache:
hazelcast:
backup-count: 1
time-to-live-seconds: 3600
cors:
allow-credentials: true
allowed-headers: '*'
allowed-methods: GET, PUT, POST, DELETE, OPTIONS
allowed-origins: '*'
exposed-headers: ''
max-age: 1800
http:
version: V_1_1
logging:
logstash:
enabled: false
host: localhost
port: 5000
queue-size: 512
spectator-metrics:
enabled: false
security:
authentication:
jwt:
secret: ms-key
token-validity-in-seconds: 86400
token-validity-in-seconds-for-remember-me: 2592000
management:
endpoint:
health:
show-details: ALWAYS
endpoints:
web:
exposure:
include: '*'
ribbon:
eureka:
enabled: true
httpclient:
enabled: false
okhttp:
enabled: true
readTimeout: 90000
connectTimeout: 90000
server:
port: 8082
servlet:
context-path: /
session:
cookie:
http-only: true
logging:
file: ./gateway_log.log
- gateway-deployment-config.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gateway
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: gateway
template:
metadata:
labels:
app: gateway
spec:
terminationGracePeriodSeconds: 60
containers:
- name: gateway
image: 192.168.200.197:80/spring-ms/sc-ms-gateway:2.0.0-SNAPSHOT
imagePullPolicy: Always
command: ["java"]
args: ["-jar", "/app.jar"]
ports:
- name: http
containerPort: 8082
volumeMounts:
- name: application-config
mountPath: "/config"
readOnly: true
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8082
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8082
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumes:
- name: application-config
configMap:
name: gateway-configmap
items:
- key: application.yml
path: application.yml
---
apiVersion: v1
kind: Service
metadata:
name: gateway
namespace: default
labels:
app: gateway
spec:
selector:
app: gateway
ports:
- name: http
port: 8082
protocol: TCP
其中mountPath为spring boot加载的路径，具体参考：https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-application-property-files。也可以通过指定 --spring.config.location来设置配置加载的路径。
运维
扩缩容
[root@k8s-master01 gateway-web]# kubectl scale --replicas=2 deployment.apps/gateway
[root@k8s-master01 gateway-web]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/apm-nfs-client-provisioner-84456ff76-z75qv 1/1 Running 0 26h
pod/application-k8s-5dd5687b5d-bdfw7 1/1 Running 0 54m
pod/curl-66bdcf564-4vg7d 1/1 Running 0 26h
pod/eureka-0-594df67869-qw569 1/1 Running 0 26h
pod/eureka-1-5dbb8d44b7-6gpb9 1/1 Running 0 26h
pod/gateway-77789f5f68-c76ft 1/1 Running 1 26h
pod/gateway-77789f5f68-gjwk6 1/1 Running 0 119s
pod/gateway-web-8d57df899-2v842 1/1 Running 0 160m
pod/jack-app-68c9ff9485-pkvtt 1/1 Running 0 26h
pod/jack-app-68c9ff9485-wwbt5 1/1 Running 0 26h
pod/mysql-0 2/2 Running 0 26h
pod/mysql-1 2/2 Running 0 26h
pod/mysql-client-56f68f8c7b-s9rvr 1/1 Running 4 26h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/eureka-0 ClusterIP 10.104.94.195 8761/TCP 6d1h
service/eureka-1 ClusterIP 10.109.14.113 8762/TCP 6d1h
service/gateway ClusterIP 10.97.29.218 8082/TCP 5d21h
service/gateway-web ClusterIP 10.97.81.26 8081/TCP 170m
service/kubernetes ClusterIP 10.96.0.1 443/TCP 14d
service/ms-application-k8s ClusterIP 10.107.7.171 8181/TCP 108m
service/mysql ClusterIP None 3306/TCP 7d1h
service/mysql-port NodePort 10.108.144.202 3306:30006/TCP 18h
service/mysql-read ClusterIP 10.106.147.56 3306/TCP 7d1h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/apm-nfs-client-provisioner 1/1 1 1 6d20h
deployment.apps/application-k8s 1/1 1 1 108m
deployment.apps/curl 1/1 1 1 11d
deployment.apps/eureka-0 1/1 1 1 5d20h
deployment.apps/eureka-1 1/1 1 1 5d20h
deployment.apps/gateway 2/2 2 2 5d17h
deployment.apps/gateway-web 1/1 1 1 160m
deployment.apps/jack-app 2/2 2 2 6d
deployment.apps/mysql-client 1/1 1 1 5d18h
NAME DESIRED CURRENT READY AGE
replicaset.apps/apm-nfs-client-provisioner-84456ff76 1 1 1 6d20h
replicaset.apps/application-k8s-5dd5687b5d 1 1 1 75m
replicaset.apps/application-k8s-7dc597469 0 0 0 80m
replicaset.apps/curl-66bdcf564 1 1 1 11d
replicaset.apps/eureka-0-594df67869 1 1 1 5d20h
replicaset.apps/eureka-1-5dbb8d44b7 1 1 1 5d20h
replicaset.apps/gateway-77789f5f68 2 2 2 5d17h
replicaset.apps/gateway-web-8d57df899 1 1 1 160m
replicaset.apps/jack-app-68c9ff9485 2 2 2 6d
replicaset.apps/mysql-client-56f68f8c7b 1 1 1 5d18h