微服务部署方案-v3

搭建

安装指定版本的docker

# 查看系统版本
 cat /etc/redhat-release

# 移除掉原来安装的docker
yum -y remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

# 卸载
yum remove -y docker-ce docker-ce-cli containerd.io

# 启用仓库的文件
yum-config-manager --enable docker-ce-nightly

# 配置yum aliyun源
## 备份
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
## 下载
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
## 生成缓存
yum makecache

# 配置centos docker-ce 源
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 更新yum
yum update -y

# 安装依赖软件
yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

# 查看所有版本
yum list docker-ce --showduplicates | sort -r

# 选择 18.06.3.ce-3.el7
yum install -y docker-ce-18.06.3.ce-3.el7 docker-ce-cli-18.06.3.ce-3.el7 containerd.io
# 查看所有版本
yum list docker-ce --showduplicates | sort -r

# 选择 18.06.3.ce-3.el7
yum install -y docker-ce-18.06.3.ce-3.el7 docker-ce-cli-18.06.3.ce-3.el7 containerd.io

# 启动docker
systemctl start docker

# 允许开机自启
systemctl enable docker

# 配置镜像源加速
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://q4jtpmzm.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

# 运行一个demo镜像
docker run hello-world

gitlab

https://docs.gitlab.com/omnibus/docker/

# 创建数据挂载点
rm -rf /data/tristan/gitlab/config /data/tristan/gitlab/logs /data/tristan/gitlab/data
mkdir -p  /data/tristan/gitlab/config /data/tristan/gitlab/logs /data/tristan/gitlab/data

# 运行镜像
docker run --detach \
  --hostname 192.168.71.220 \
  --publish 80:80 --publish 2289:22 \
  --name gitlab \
  --restart always \
  --volume /data/tristan/gitlab/config:/etc/gitlab \
  --volume /data/tristan/gitlab/logs:/var/log/gitlab \
  --volume /data/tristan/gitlab/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest
  
#限制gitlab的内存
docker stop gitlab
docker update --memory-swap 15360M gitlab
docker update --memory 15360M gitlab
docker start gitlab

#查看镜像运行情况
docker logs -f gitlab

http://192.168.71.220:8929 root/tanshilin

为root账号配置密码 root/tanshilin

jenkins

# 创建数据挂载点
mkdir -p /data/tristan/jenkins

# 运行镜像
docker run \
  -u root \
  -d --restart always \
  -p 8080:8080 \
  -p 50000:50000 \
  -v /data/tristan/jenkins:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --name jenkins \
  jenkinsci/blueocean
  
 # 查看镜像运行情况
 docker logs -f jenkins

访问地址:

http://192.168.71.220:8080 访问密码需要从docker的日志控制台拷贝出来

设置admin账号的密码 admin/admin

nexus

# 创建数据挂载点

# 创建本地文件夹
mkdir -p /data/tristan/nexus && chmod 777 /data/tristan/nexus

# 启动容器
docker run -d --restart always -p 8081:8081 --name nexus -v /data/tristan/nexus:/nexus-data sonatype/nexus3

# 查看镜像启动情况
docker logs -f nexus

访问:

​ http://192.168.71.220:8081 admin/admin123

k8s

搭建k8s

清理rancher

#删除所有容器
 docker rm -f $(sudo docker ps -qa)

#删除/var/etcd目录
 rm -rf /var/etcd

#删除/var/lib/kubelet/目录，删除前先卸载
for m in $(sudo tac /proc/mounts | sudo awk '{print $2}'|sudo grep /var/lib/kubelet);do
	umount $m||true
done

umount -f /var/lib/kubelet/pods/2aaf3247-61e1-11e9-a738-020860040104/volume-subpaths/filebeat-config/filebeat/1


umount -f /var/lib/kubelet/pods/55b15b49-61e5-11e9-9b17-020860040104/volume-subpaths/filebeat-config/filebeat/1

umount -f /var/lib/kubelet/

 rm -rf /var/lib/kubelet/

#删除/var/lib/rancher/目录，删除前先卸载
for m in $(sudo tac /proc/mounts | sudo awk '{print $2}'|sudo grep /var/lib/rancher);do
  umount $m||true
done

umount -f /var/lib/rancher/volumes

 rm -rf /var/lib/rancher/

#删除/run/kubernetes/ 目录
 rm -rf /run/kubernetes/

#删除所有的数据卷
 docker volume rm $(sudo docker volume ls -q)

#再次显示所有的容器和数据卷，确保没有残留
 docker ps -a
 docker volume ls

安装rancher

# 创建数据卷
rm -rf /data/tristan/rancher/
mkdir -p /data/tristan/rancher/mysql

# 运行rancher镜像
docker run -d --name rancher -v /data/tristan/rancher/mysql:/var/lib/mysql --restart=unless-stopped -p 8765:8080 rancher/server

#查看运行情况
docker logs -f rancher

# 加入主节点
sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.11 http://192.168.71.220:8765/v1/scripts/3277D056DBE1B20BDB55:1546214400000:Ovc5YLAsfQfvrZoGW2Es7U9w

# 修改密码
/usr/bin/mysqladmin -u root password 'new-password'

# 修改 访问控制 为 local 

访问地址:

http://192.168.71.220:8765 rancher/rancher

注意

如果第一次安装会出问题请重置服务器后再试一次

因为rancher版本比较快,请不要使用国内的镜像配置,尽量配置不要改配置

搭建之后如果性能/稳定性不够请搭建多节点(后面补充多节点部署)

停止并删除所有容器

docker stop $(docker ps -aq) 
docker rm $(docker ps -aq)

docker rmi $(docker images -aq)

一些指令

# 清理内存
echo 2 > /proc/sys/vm/drop_caches

echo 3 > /proc/sys/vm/drop_caches

https://my.oschina.net/wenzhenxi/blog/1824274

nfs

本地搭建方式

安装nfs服务

1）通过yum目录安装nfs服务和rpcbind服务：

$ yum -y install nfs-utils rpcbind

2）检查nfs服务是否正常安装

$ rpcinfo -p localhost

创建用户

为NFS服务其添加用户，并创建共享目录，以及设置用户设置共享目录的访问权限：

$ useradd -u nfs
$ mkdir -p /data/tristan/nfs
$ chmod a+w /data/tristan/nfs

配置共享目录

在nfs服务器中为客户端配置共享目录：

$ echo "/data/tristan/nfs *(rw,async,no_root_squash)" >> /etc/exports

通过执行如下命令是配置生效：

$exportfs -r

启动服务

1）由于必须先启动rpcbind服务，再启动nfs服务，这样才能让nfs服务在rpcbind服务上注册成功：

$ systemctl start rpcbind

2）启动nfs服务：

$ systemctl start nfs-server

3）设置rpcbind和nfs-server开机启动：

$ systemctl enable rpcbind

$ systemctl enable nfs-server

检查nfs服务是否正常启动

$ showmount -e localhost

$ mount -t nfs 127.0.0.1:/data/tristan/nfs /mnt

mkdir -p /data/tristan/nfs/es/0

mkdir -p /data/tristan/nfs/es/1

在 192.168.71.221上

mount -t nfs 192.168.71.220:/data/tristan/nfs /data

项目改造

服务器构建配置

settings.xml

  
  
  
    
    
  
    
      	
      nexus-snapshots
      ebuy-cloud-test
      ebuy-cloud-test
    
	
	
		registry.cn-shenzhen.aliyuncs.com
		xxx@xxx
		xxx
	
    
  
     
       
      nexus-releases   
      *   
      http://192.168.71.220:8081/repository/maven-public/   
        
       
      nexus-snapshots   
      *   
      http://192.168.71.220:8081/repository/maven-snapshots/   
    
  
   
  
	 
	
		nexus-snapshots
		
			nexus-snapshots
			nexus-snapshots
			http://192.168.71.220:8081/repository/maven-snapshots/
			
			registry.cn-shenzhen.aliyuncs.com
		
	
    
  
    
      nexus-snapshots  
    
   
  

目前项目结构

tree /F

ebuy-cloud
	springcloud
		cloud-eureka
		cloud-config
		cloud-oauth2
		cloud-gateway
	service-common
        commons-base
        commons-dto
        commons-http
        commons-task
        commons-utils
	service-modules
        dynamic-datasource
        service-amazon
        service-crm
        service-erp
        service-erp-crm
        service-kafka-demo
        service-logistics
        service-logistics-old
        service-mrp
        service-plan
        service-procurement
        service-procurement-old
        service-product
        service-warehouse
        service-workflow
        service-workflowengine
	cloud-basics
        service-alibaba-order
        service-aliyun-image-search
        service-basics-data
        service-batchtask
        service-file
        service-mail
        service-util
	cloud-datacenter
        service-elasticsearch
        service-oa
        service-system

父类工程

# pom.xml
    
    
        
            ${repository.id}
            ${repository.name}
            ${repository.url}
            default
        
    

# Jenkinsfile
pipeline {
    agent {
        docker {
            image 'maven:3-alpine'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Deliver') {
            steps {
                sh 'mvn clean deploy'
            }
        }
    }
}

特殊工程

eureka

# pom.xml
	
        
            
                org.springframework.boot
                spring-boot-maven-plugin
                
                    true
                
            
            
            
                com.spotify
                dockerfile-maven-plugin
                1.4.10
                
                    true
                    ${dockerRegistry.url}/ebuy-cloud/${project.artifactId}
                    ${project.version}.${maven.build.timestamp}
                    true
                    false
                    0
                    
                        target/${project.build.finalName}.jar
                    
                
                
                    
                        default
                        package
                    
                
            
        
    

# Jenkinsfile
pipeline {
    agent {
        docker {
            image 'maven:3-alpine'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Deliver') {
            steps {
                sh 'mvn clean package -DskipTests dockerfile:build dockerfile:tag dockerfile:push'
            }
        }
    }
}

# Dockerfile
FROM anapsix/alpine-java:latest
ARG EUREKA_USER_NAME
ARG EUREKA_USER_PASSWORD
ARG CLOUD_EUREKA_DEFAULTZONE_OTHER
ARG CLOUD_ZIPKIN_BASE_URL
ARG JAR_FILE
ADD ${JAR_FILE} app.jar
RUN echo 'Asia/Shanghai' > /etc/timezone
RUN bash -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar","${EUREKA_USER_NAME}","${EUREKA_USER_PASSWORD}","${CLOUD_EUREKA_DEFAULTZONE_OTHER}","${CLOUD_ZIPKIN_BASE_URL}"]

config

# pom.xml
	
        
            
                org.springframework.boot
                spring-boot-maven-plugin
                
                    true
                
            
            
            
                com.spotify
                dockerfile-maven-plugin
                1.4.10
                
                    true
                    ${dockerRegistry.url}/ebuy-cloud/${project.artifactId}
                    ${project.version}.${maven.build.timestamp}
                    true
                    false
                    0
                    
                        target/${project.build.finalName}.jar
                    
                
                
                    
                        default
                        package
                    
                
            
        
    

# Jenkinsfile
pipeline {
    agent {
        docker {
            image 'maven:3-alpine'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Deliver') {
            steps {
                sh 'mvn clean package -DskipTests dockerfile:build dockerfile:tag dockerfile:push'
            }
        }
    }
}

# Dockerfile
FROM anapsix/alpine-java:latest
ARG CLOUD_EUREKA_DEFAULTZONE
ARG CLOUD_ZIPKIN_BASE_URL
ARG CONFIG_GIT_URI
ARG CONFIG_GIT_USERNAME
ARG CONFIG_GIT_PASSWORD
ARG CONFIG_KAFKA_HOST
ARG CONFIG_KAFKA_PORT
ARG CONFIG_ZK_PORT
ARG JAR_FILE
ADD ${JAR_FILE} app.jar
RUN echo 'Asia/Shanghai' > /etc/timezone
RUN bash -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar","${CLOUD_EUREKA_DEFAULTZONE}","${CLOUD_ZIPKIN_BASE_URL}","${CONFIG_GIT_URI}","${CONFIG_GIT_USERNAME}","${CONFIG_GIT_PASSWORD}","${CONFIG_KAFKA_HOST}","${CONFIG_KAFKA_PORT}","${CONFIG_ZK_PORT}"]

标准工程

oauth2

# pom.xml
	
        
            
                org.springframework.boot
                spring-boot-maven-plugin
                
                    true
                
            
            
            
                com.spotify
                dockerfile-maven-plugin
                1.4.10
                
                    true
                    ${dockerRegistry.url}/ebuy-cloud/${project.artifactId}
                    ${project.version}.${maven.build.timestamp}
                    true
                    false
                    0
                    
                        target/${project.build.finalName}.jar
                    
                
                
                    
                        default
                        package
                    
                
            
        
    

# Jenkinsfile
pipeline {
    agent {
        docker {
            image 'maven:3-alpine'
            args '-v /root/.m2:/root/.m2'
        }
    }
    stages {
        stage('Deliver') {
            steps {
                sh 'mvn clean package -DskipTests dockerfile:build dockerfile:tag dockerfile:push'
            }
        }
    }
}

# Dockerfile
FROM anapsix/alpine-java:latest
ARG CLOUD_EUREKA_DEFAULTZONE
ARG JASYPT_ENCRYPTOR_PASSWORD
ARG JAR_FILE
ADD ${JAR_FILE} app.jar
RUN echo 'Asia/Shanghai' > /etc/timezone
RUN bash -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar","${CLOUD_EUREKA_DEFAULTZONE}","${JASYPT_ENCRYPTOR_PASSWORD}"]

# bootstrap.yml
eureka:
  client:
    serviceUrl:
      defaultZone: ${cloud_eureka_defaultzone:http://root:123456@localhost:9000/eureka/}
  instance:
    prefer-ip-address: true
spring:
  application:
    name: service-crm
  cloud:
    config:
      discovery:
        service-id: cloud-config
        enabled: true
logging:
  config: classpath:logback.xml
jasypt:
  encryptor:
    password: ${jasypt_encryptor_password:l&id81!lw}

使用搭建

gitlab

ebuy-cloud-config-dev
	ebuy-cloud-dev
		ebuy-cloud-dev-system/ebuy-cloud-dev-system
	
ebuy-cloud-config-test
	ebuy-cloud-test
		ebuy-cloud-test-system/ebuy-cloud-test-system
		
ebuy-cloud-config-product
	ebuy-cloud-product-system/

使用钉钉名称注册人员账号

创建ebuy-cloud-dev-system/ebuy-cloud-test-system/ebuy-cloud-product-system三个系统用户
创建ebuy-cloud-dev-report/ebuy-cloud-test-report/ebuy-cloud-product-report三个只读用户
创建ebuy-cloud-dev/ebuy-cloud-test/ebuy-cloud-product三个用户组
依次添加用户到用户组并设置权限 系统用户为 Maintainer权限，只读用户为 Report权限

root账号创建ebuy-cloud-config配置项目

从高向低fork项目
fork root/ebuy-cloud-config					-> ebuy-cloud-product/ebuy-cloud-config
fork ebuy-cloud-product/ebuy-cloud-config 	-> ebuy-cloud-test/ebuy-cloud-config
fork ebuy-cloud-test/ebuy-cloud-config 		-> ebuy-cloud-dev/ebuy-cloud-config

从低给高授权（Developer权限）
grant	ebuy-cloud-dev/ebuy-cloud-config	->	ebuy-cloud-test
grant	ebuy-cloud-test/ebuy-cloud-config	->	ebuy-cloud-product

低修改之后查看自己的项目中  高(项目名称一致且所属者后缀不同)的项目的成员,在钉钉找到该成员之后发送消息 合并配置文件
高登录之后向低发起合并请求然后去合并代码

nexus

创建snapshot库

aliyun docker registry

创建阿里云子账号

创建命名空间

创建仓库

jenkins

创建流水线

主要参数为 svn地址

rancher

http://192.168.71.220:8765

修改系统语言为中文

在商店中搜索kubernetes

查看该rancher模板中k8s的版本为 12 -> 对应docker 的版本为 18.06.3.ce-3.el7

修改dashboard的内存(有利于查看内存)

加入主机,根据提示的加入主机的指令去其他主机上去执行(三台即可)

等待初始化完成

…

k8s

http://192.168.71.220:8765 KUBERNETES 仪表板

http://192.168.71.220:8765/r/projects/1a5/kubernetes-dashboard:9090/#!/overview?namespace=default

确定主节点所在主机

# 需要将对外的那台主机进行标记标签
kubectl label nodes 192.168.71.221 type=outer

# 如果该主机的位置发生了变化需要删除原来的主机的标签
kubectl label nodes 192.168.71.104 type-

创建拉取镜像的凭证字符串

kubectl create secret docker-registry regcred --docker-server=registry.cn-shenzhen.aliyuncs.com --docker-username=xxx@xxx --docker-password=xxx [email protected]

一些指令:

# 查看所有版本
kubectl rollout history deployment servicemrp

# 回滚到指定版本
kubectl rollout undo deployment servicemrp --to-revision=1

编写部署文件

└─deploy
    │  README.MD
    │
    ├─middleware
    │      elasticsearch-router.yml
    │      elasticsearch-single.yml
    │      filebeat.yml
    │      kafka.yml
    │      xxljob-router.yml
    │      xxljob.yml
    │      zipkin-router.yml
    │      zipkin-single.yml
    │      zookeeper.yml
    │
    ├─service
    │  ├─cloud-basics
    │  │      servicealibabaorder.yml
    │  │      servicemail.yml
    │  │      serviceutil.yml
    │  │
    │  ├─cloud-datacenter
    │  │      serviceelasticsearch.yml
    │  │      serviceoa.yml
    │  │
    │  └─cloud-modules
    │          servicecrm.yml
    │          serviceerp.yml
    │          serviceerpcrm.yml
    │          servicelogistics.yml
    │          servicemrp.yml
    │          serviceplan.yml
    │          serviceprocurement.yml
    │          serviceproduct.yml
    │
    └─springcloud
            cloudconfig.yml
            cloudeureka.yml
            cloudgateway.yml
            router.yml
            serveroauth2.yml

middleware

filebeat-demo.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serviceerp2filebeat2
  labels:
    app: serviceerp2filebeat2
spec:
  replicas: 1
  template:
    metadata:
      name: serviceerp2filebeat2
      labels:
        app: serviceerp2filebeat2
    spec:
      containers:
        - name: serviceerp
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: CLOUD_EUREKA_DEFAULTZONE
              value: "http://root:[email protected]/eureka/,http://root:[email protected]/eureka/"
            - name: JASYPT_ENCRYPTOR_PASSWORD
              value: "l&id81!lw"
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
            - name: app-logs
              mountPath: /logs
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:5.4.0
          volumeMounts:
            - name: app-logs
              mountPath: /logs
            - name: filebeat-config
              mountPath: /usr/share/filebeat/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
        - name: app-logs
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: serviceerp2filebeat2



---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: default
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.prospectors:
    - input_type: log
      paths:
        - "/logs/*/*"
    output.elasticsearch:
      hosts: ["192.168.71.223:9200"]
    username: ""
    password: ""
    index: "filebeat-log"

xxljob.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xxljob
  labels:
    app: xxljob
spec:
  replicas: 1
  template:
    metadata:
      name: xxljob
      labels:
        app: xxljob
    spec:
      nodeSelector:
        type: outer
      containers:
        - name: xxljob
          image: xuxueli/xxl-job-admin:2.0.1
          imagePullPolicy: IfNotPresent
          env:
            - name: PARAMS
              value: "--spring.datasource.url=jdbc:mysql://192.168.71.219:3306/xxl-job?Unicode=true&characterEncoding=UTF-8 --spring.datasource.username=javaappuser --spring.datasource.password=javaappuser --spring.datasource.driver-class-name=com.mysql.jdbc.Driver"
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
          ports:
            - containerPort: 8080
              hostPort: 9988
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
  selector:
    matchLabels:
      app: xxljob

xxljob-router.yml

---
apiVersion: v1
kind: Service
metadata:
  name: xxljob
spec:
  selector:
    app: xxljob
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

zipkin-single.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  replicas: 1
  template:
    metadata:
      name: zipkin
      labels:
        app: zipkin
    spec:
      nodeSelector:
        type: outer
      containers:
        - name: zipkin
          image: openzipkin/zipkin
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
          ports:
            - containerPort: 9411
              hostPort: 9411
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
  selector:
    matchLabels:
      app: zipkin

zipkin-router.yml

---
apiVersion: v1
kind: Service
metadata:
  name: zipkin
spec:
  selector:
    app: zipkin
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9411

springcloud

cloudeureka.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudeureka
  labels:
    app: cloudeureka
spec:
  replicas: 1
  template:
    metadata:
      name: cloudeureka
      labels:
        app: cloudeureka
    spec:
      nodeSelector:
        type: outer
      containers:
        - name: cloudeureka
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: EUREKA_USER_NAME
              value: "root"
            - name: EUREKA_USER_PASSWORD
              value: "123456"
            - name: CLOUD_EUREKA_DEFAULTZONE_OTHER
              value: "http://root:[email protected]/eureka/"
            - name: CLOUD_ZIPKIN_BASE_URL
              value: ""
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
          ports:
            - containerPort: 9000
              hostPort: 9000
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: cloudeureka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudeureka2
  labels:
    app: cloudeureka2
spec:
  replicas: 1
  template:
    metadata:
      name: cloudeureka2
      labels:
        app: cloudeureka2
    spec:
      containers:
        - name: cloudeureka2
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: EUREKA_USER_NAME
              value: "root"
            - name: EUREKA_USER_PASSWORD
              value: "123456"
            - name: CLOUD_EUREKA_DEFAULTZONE_OTHER
              value: "http://root:[email protected]/eureka/"
            - name: CLOUD_ZIPKIN_BASE_URL
              value: ""
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
          ports:
            - containerPort: 9000
              hostPort: 9000
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: cloudeureka2

cloudeureka-router.yml

---
apiVersion: v1
kind: Service
metadata:
  name: cloudeureka
spec:
  selector:
    app: cloudeureka
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9000
---
apiVersion: v1
kind: Service
metadata:
  name: cloudeureka2
spec:
  selector:
    app: cloudeureka2
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9000

cloudconfig.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudconfig
  labels:
    app: cloudconfig
spec:
  replicas: 1
  template:
    metadata:
      name: cloudconfig
      labels:
        app: cloudconfig
    spec:
      containers:
        - name: cloudconfig
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: CLOUD_EUREKA_DEFAULTZONE
              value: "http://root:[email protected]/eureka/,http://root:[email protected]/eureka/"
            - name: CLOUD_ZIPKIN_BASE_URL
              value: ""
            - name: CONFIG_GIT_URI
              value: "http://192.168.71.220/ebuy-cloud-test/ebuy-cloud-config.git"
            - name: CONFIG_GIT_USERNAME
              value: "ebuy-cloud-test-report"
            - name: CONFIG_GIT_PASSWORD
              value: "ebuy-cloud-test-report"
            - name: CONFIG_KAFKA_HOST
              value: "192.168.71.223"
            - name: CONFIG_KAFKA_PORT
              value: "9092"
            - name: CONFIG_ZK_PORT
              value: "2181"
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: cloudconfig

serveroauth2.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: serveroauth2
  labels:
    app: serveroauth2
spec:
  replicas: 1
  template:
    metadata:
      name: serveroauth2
      labels:
        app: serveroauth2
    spec:
      nodeSelector:
        type: outer
      containers:
        - name: serveroauth2
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: CLOUD_EUREKA_DEFAULTZONE
              value: "http://root:[email protected]/eureka/,http://root:[email protected]/eureka/"
            - name: JASYPT_ENCRYPTOR_PASSWORD
              value: "l&id81!lw"
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
          ports:
            - containerPort: 9403
              hostPort: 9403
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: serveroauth2

serveroauth2-router.yml

---
apiVersion: v1
kind: Service
metadata:
  name: serveroauth2
spec:
  selector:
    app: serveroauth2
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9403

cloudgateway.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudgateway
  labels:
    app: cloudgateway
spec:
  replicas: 1
  template:
    metadata:
      name: cloudgateway
      labels:
        app: cloudgateway
    spec:
      nodeSelector:
        type: outer
      containers:
        - name: cloudgateway
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: CLOUD_EUREKA_DEFAULTZONE
              value: "http://root:[email protected]/eureka/,http://root:[email protected]/eureka/"
            - name: JASYPT_ENCRYPTOR_PASSWORD
              value: "l&id81!lw"
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
          ports:
            - containerPort: 5000
              hostPort: 5000
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: cloudgateway

service

案例

servicealibabaorder.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: servicealibabaorder
  labels:
    app: servicealibabaorder
spec:
  replicas: 1
  template:
    metadata:
      name: servicealibabaorder
      labels:
        app: servicealibabaorder
    spec:
      containers:
        - name: servicealibabaorder
          image: 
          imagePullPolicy: IfNotPresent
          env:
            - name: CLOUD_EUREKA_DEFAULTZONE
              value: "http://root:[email protected]/eureka/,http://root:[email protected]/eureka/"
            - name: JASYPT_ENCRYPTOR_PASSWORD
              value: "l&id81!lw"
          volumeMounts:
            - mountPath: /etc/localtime
              name: localtime
            - name: app-logs
              mountPath: /logs
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:5.4.0
          volumeMounts:
            - name: app-logs
              mountPath: /logs
            - name: filebeat-config
              mountPath: /usr/share/filebeat/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
        - name: app-logs
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config
      restartPolicy: Always
      imagePullSecrets:
        - name: regcred
  selector:
    matchLabels:
      app: servicealibabaorder

当需要更新更新镜像的时候需要直接编辑对应工程的部署文件的镜像版本即可

项目模块规划升级-v3

升级之后的结构:

ebuy-cloud
	springcloud
		cloud-eureka
		cloud-config
		cloud-oauth2
		cloud-gateway
	service-common
        commons-base
        commons-dto
        commons-http
        commons-task
        commons-utils
	service-modules
        canal.sample
        dynamic-datasource
        service-amazon
        service-crm
        service-erp
        service-erp-crm
        service-kafka-demo
        service-logistics
        service-logistics-old
        service-mrp
        service-plan
        service-procurement
        service-procurement-old
        service-product
        service-warehouse
        service-workflow
        service-workflowengine
        service-xxljob-demo
	cloud-basics
        service-alibaba-order
        service-aliyun-image-search
        service-basics-data
        service-batchtask
        service-file
        service-mail
        service-util
	cloud-datacenter
        service-elasticsearch
        service-oa
        service-system

将原来的spring cloud的组件工程放到单独的文件夹中

关于镜像命名和版本的问题

原来的做法

所有的镜像在同一个命名空间、同一个仓库、使用标签后缀实现镜像名称和版本的区分

遇到什么问题

k8s 使用 kubectl rollout status deployment service-xxx 实现镜像升级

这里有一个前提

原来的镜像的版本必须与当前的版本不同

jenkins -> k8s 的webhook不可用

现在的做法

镜像源

对每个服务的tag单独出来做版本划分

需要变成 四级目录

# 镜像源地址
registry.cn-shenzhen.aliyuncs.com	
# 命名空间
ebuy-cloud
# 服务名
service-xxx
# 递增版本号 (大版本号+时间戳版本号)
1.0.1

所以需要为每一个服务创建一个仓库

并且不允许 push 覆盖版本

– 生产 和 开发/测试 环境的镜像库隔离是否有必要

– 提供页面去执行kubectl rollout status deployment service-xxx 是否有必要

构建

修改pom.xml中tag标签中的值

去掉后面的环境区分

加上当前时间戳

部署

修改 k8s-deploy-service-xxx.yml 文件

​ 镜像名

​ 前缀 -> registry.cn-shenzhen.aliyuncs.com/ebuy-cloud

​ 仓库名 -> service-xxx

​ 版本号 -> lastest

关于仓库的想法

划分仓库有哪些好处?
可以更加清晰的看到很多版本

划分仓库有哪些坏处?
每加一个服务或者修改服务名称 就要加一个仓库

日志管理

搭建ek

# 创建保存elk文件的文件夹
mkdir -p /data/tristan/elk  && cd /data/tristan/elk

# 下载docker stack 文件
 wget https://github.com/deviantony/docker-elk/archive/master.zip

#解压并进入该docker stack 文件位置
unzip master.zip &&	cd docker-elk-master

# 修改es的内存
vi docker-stack.yml 
# 将256 改为 2048

# 初始化docker swarm集群
docker swarm init

# 运行容器
docker stack deploy -c docker-stack.yml elk

在上一个版本中(v2)

使用docker stack创建elk(elasticsearch/logstash/kibana)

配置容器的日志驱动为syslog 并且配置syslog-address地址为logstash的地址

在使用过程发现几个问题:

1、由于日志内容中没有zipkin的trace信息(spanId)导致并发调用的时候获取整个调用链日志不直观

2、 日志过于呈现形式为日志行,导致分析定位问题不直观

3、由于syslog的方式与elk耦合过重导致业务系统的启动会受到elk的影响

4、syslog会影响原来docker 日志的显示,降低docker 容器的性能

有几个好处:

配置简单，维护简单

设计思考

elk是否需要放到k8s容器中？

非常有必要

​ 放到k8s容器之后,再通过dnsServer去固定内部访问地址可以简化配置,使得开发/测试/生产环境的配置一致

​ 后期动态扩缩节点更方便(相比docker stack)

带来的坏处?

​ 搭建需要时间

实践

通过k8s 部署文件 搭建elk

将kibana固定到指定标签的主机上,提供外部访问端口

通过service固定elk的内部访问地址

修改Deployment的配置指向 elk

      - image: sz-pg-oam-docker-hub-001.tendcloud.com/library/filebeat:5.4.0
        name: filebeat
        volumeMounts:
        - name: app-logs
          mountPath: /log
        - name: filebeat-config
          mountPath: /etc/filebeat/

配置使用filebeat / fluent

参考资料

https://www.kubernetes.org.cn/4022.html nfs

https://www.kubernetes.org.cn/2011.html ek + filebeat

总结

第二个版本主要做了什么?
优化项目实现可配置式
基于gitlab中实现开发、测试、生产环境的切换、隔离和权限控制
jenkins构建项目打包镜像并推送到阿里云的docker镜像仓库同时触发docker service的重新拉取镜像并运行,单独开放只读用户给测试人员
基于docker swarm集群创建overlay网络、registry、docker service,注入参数到容器、挂载数据卷、开放全局端口、任意服务的高可用和高拓展已经容器的性能监控
基于elk+syslog+zipkin server实现分布式调用链及日志的聚合分析

在第三个部署版本中:
我会着重解决几个问题:
使用k8s替换docker swarm
使用fannel替换overlay
弱化spring cloud

遇到的问题?
服务器资源问题 – 目前已经有很多资源了,所以资源应该不是问题
因为一开始没有锁定代码版本导致现在的镜像的版本无法和原来的代码的版本保持一致 --从线上根据原来部署文件直接构建镜像
mysql 根据ip授权的问题,是应该给宿主机ip授权还是容器的ip授权 – 根据宿主机ip授权即可

v3(k8s)解决了什么问题?
原始发包容易出错
每个工程里面取改配置文件容易出错且效率低
发包的版本无法管理
单jar包部署的性能不够,稳定性不够，主机拓展性不够
日志问题、调用链问题