k8s结合jenkins部署docker项目

开启ssh服务

apt-get install openssh-server
sudo /etc/init.d/ssh start

kubernetes

swapoff -a关闭虚拟硬盘

apt-get install -y apt-transport-https ca-certificates curl software-properties-common

#docker源:

curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

#kubeadm源:

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat </etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

#科学上网:

apt-get update && apt-get install -y apt-transport-https curl
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat </etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF  
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

初始化master

执行：

kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=NumCPU

如果失败：
kubeadm config images list列出需要下载的依赖包

新建pullkub.sh文件:

#!/bin/bash
# 下面的镜像应该去除"k8s.gcr.io/"的前缀，版本换成kubeadm config images list命令获取到的版本
images=(
    kube-apiserver:v1.17.0
    kube-controller-manager:v1.17.0
    kube-scheduler:v1.17.0
    kube-proxy:v1.17.0
    pause:3.1
    etcd:3.4.3-0
    coredns:1.6.5
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
    docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done

chmod +x pullk8s.sh
bash pullk8s.sh  (或 ./pullk8s.sh)

执行完之后master机器重新执行:

kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=NumCPU

执行之后得到最后一条打印语句:

kubeadm join 192.168.0.102:6443 --token wcl2eh.dxn6t9w8kgiy6h9n --discovery-token-ca-cert-hash sha256:025fbd1d141b275af0f14283aea35e0c37493a11d679089f4cb9c346f5233566
注：每次执行init命令得到的token都不一样

初始化master第四步：

1、mkdir -p $HOME/.kube

2、cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

3、chown $(id -u):$(id -g) $HOME/.kube/config

安装网络插件：

sysctl net.bridge.bridge-nf-call-iptables=1

去https://github.com/xxz199539/k8sBeegoDemo/blob/master/flannel.yml获取flannel.yml到本地

执行：kubectl apply -f kube-flannel.yml

添加node

1.将master上的kubeadm join命令执行:

kubeadm join 192.168.0.102:6443 --token wcl2eh.dxn6t9w8kgiy6h9n --discovery-token-ca-cert-hash sha256:025fbd1d141b275af0f14283aea35e0c37493a11d679089f4cb9c346f5233566

2.关闭虚拟硬盘

swapoff -a

3.sysctl net.bridge.bridge-nf-call-iptables=1

查看node是否ready:kubectl get node
查看所有pod是否正常：kubectl get pod --all-namespaces -o wide
如果某个pod不正常：kubectl describe pod [podName]

master和node重启之后：
1.swapoff -a
2.systemctl daemon-reload
3.systemctl restart kubelet

jenkins及tomcat安装

1.安装tomcat

wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.30/bin/apache-tomcat-9.0.30.tar.gz

tar -xvf apache-tomcat-9.0.30.tar.g

mv apache-tomcat-9.0.30/ tomcat9

2.安装jdk

1.下载jdk并解压

2.配置环境变量

export JAVA_HOME=~/jdk1.8.0_221
export JRE_HOME=~/jdk1.8.0_221/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

3.下载Jenkins

wget http://mirrors.jenkins.io/war/latest/jenkins.war

mv jenkins.war tomcat9/webapps/

此时执行sh tomcat9/bin/startup.sh，启动tocat，访问"http://ip:8080/jenkins/"即可确认Jenkins是否启动成功

安装如下插件：

ssh/publish over ssh/git parameter/Maven Intergration plugin

jenkins + k8s发布实例

1.编写项目的Dockerfile

前提工作

注：可以采用github每次push执行构建或者在jenkins上手动点击构建
如果采用github的方式必须有一个公网ip。
这里的rigistry是镜像仓库，可以通过在本地讲代码打包之后推送到镜像仓库，在别的机器上通过登录docker就可以拉去镜像并执行。

1.拉取registry并运行

docker pull registry
docker run -p 5000:5000 -v /home/registry_images:/var/lib/registry -d registry

2.修改docker配置文件并重启docker

vim /etc/docker/daemon.json
加上一行"insecure-registries": ["xxxxxx:5000"],

systemctl restart docker

测试registry是否成功：

docker pull nginx
docker tag nginx 192.168.0.102:5000/nginx:test
docker push 192.168.0.102:5000/nginx:test

设置Jenkins服务器免密登录master：

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected] 

配置jenkins

因为我这里只搭一个简单的Demo，所以选择第一个。

echo "Stopping old contain"
for contain in `docker ps -a | grep ./main |grep -v grep|awk '{print $1}'`
do 
    docker stop $contain && docker rm $contain
done
for mysqlContain in `docker ps -a | grep mysql:5.7 |grep -v grep|awk '{print $1}'`
do 
    docker start $mysqlContain
done
for noneContain in `docker images | grep none |grep -v grep|awk '{print $3}'`
do 
    if [ ! $(docker ps -a | grep $noneContain |grep -v grep|awk '{print $1}') ];then
       echo "no contain need remove"
    else
        docker rm $(docker ps -a | grep $noneContain |grep -v grep|awk '{print $1}')
    fi
    docker rmi -f $noneContain
done  # 停止并删除正在运行的dokcer容器
export JAVA_HOME="/root/Downloads/jdk8"
export JRE_HOME="/root/Downloads/jdk8/jre"
export CLASSPATH="$JAVA_HOME/lib:$JRE_HOME/lib"
export GOROOT="/opt/go"
export GOTOOLS="$GOROOT/pkg/tool"
export GOARCH=386
export GOOS="linux"
export GOPATH="/root/Desktop/code"
export PATH="$PATH:$JAVA_HOME/bin:$GOROOT/bin"

cd /root/Desktop/code/src/k8sBeegoDemo #切换到项目目录下
time=$(date "+%m.%d")
docker build -t beego:v$time . && docker tag beego:v$time xxz199439/beegodemo:latest
 && docker push xxz199439/beegodemo:latest && docker rmi -f $(docker images | grep beego |grep -v grep|awk '{print $3}') 
 # 根据Dockerfile打包镜像，打tag再push到镜像仓库，这样在别的机器节能拉取这个镜像
eval "$(ssh-agent -s)"  # 有时ssh会拒绝连接
ssh-add
ssh [email protected] bash /root/Desktop/test.sh # 执行master的k8s部署脚本

master上的脚本内容如下：

#! /bin/bash

mkdir -p $HOME/.kube
rm $HOME/.kube/config
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
sysctl net.bridge.bridge-nf-call-iptables=1
docker pull xxz199439/beegodemo:latest & #从镜像仓库拉镜像
kubectl delete deployment.apps/k8sbeegodemo-deploy service/k8sbeegodemo-service ingress/goapp-ingress && echo 'delete success' 
/usr/bin/kubectl apply -f /root/Desktop/k8s.yml && echo 'namesapce/pod/deploy/server/ingress created success!'

这里部署k8s需要生成Namespace，Deployment，Service及Namespace，其配置文件在https://github.com/xxz199539/k8sBeegoDemo/blob/master/k8s.yml,这是新版本的配置文件。
脚本执行完之后可以查看集群状态

t@ubuntu:/root/Desktop# kubectl get deployments -n kube-apps |grep beego
k8sbeegodemo-deploy   2/2     2            2           154m
root@ubuntu:/root/Desktop# get svc -n kube-apps |grep beego
No command 'get' found, but there are 18 similar ones
get: command not found
root@ubuntu:/root/Desktop# kubectl get deployments -n kube-apps |grep beego
k8sbeegodemo-deploy   2/2     2            2           154m
root@ubuntu:/root/Desktop# kubectl get svc -n kube-apps |grep beego
k8sbeegodemo-service   NodePort   10.96.213.107           8081:31000/TCP   154m
root@ubuntu:/root/Desktop# kubectl get ingress -n kube-apps |grep goapp-ingress
goapp-ingress   k8s.local             80      155m
root@ubuntu:/root/Desktop# kubectl get pods -n kube-apps |grep beego
k8sbeegodemo-deploy-7df9fb84df-98phb   1/1     Running   4          125m
k8sbeegodemo-deploy-7df9fb84df-wdjzh   1/1     Running   5          125m

这样在集群内部就可以通过10.96.213.107这个集群IP来访问服务，在外部可以通过访问本来的服务地址来访问服务。

现在每次点击构建，就会在node上自动打包镜像并上传到仓库，再k8s部署服务。