k8s day10

JENKINS集成K8S项目实战
部署Jenkins环境:
    1.下载Jenkins软件包
curl -o jenkins-k8s.zip  http://192.168.17.253/Kubernetes/day10-/softwares/jenkins-k8s.zip

    2.解压软件包
yum -y install unzip
unzip jenkins-k8s.zip 

    3.安装JDK环境，如上图所示
cd jenkins-k8s && rpm -ivh jdk-8u102-linux-x64.rpm && java -version

    4.解压tomcat软件包
mkdir -pv /oldboyedu/softwares && tar xf apache-tomcat-8.0.27.tar.gz -C /oldboyedu/softwares

    5.删除tomcat的初始数据
rm -rf /oldboyedu/softwares/apache-tomcat-8.0.27/webapps/*

    6.将Jenkins的war包放入到tomcat的应用目录，注意，复制该名称
cp jenkins.war /oldboyedu/softwares/apache-tomcat-8.0.27/webapps/ROOT.war

    7.解压jenkins数据到"/root"目下，会创建一个隐藏目录(.jenkins)
tar xf jenkin-data.tar.gz -C /root/

    8.启动tomcat服务
/oldboyedu/softwares/apache-tomcat-8.0.27/bin/startup.sh 

    9.检查端口是否存在
ss -ntl | grep 8080

    10.验证Jenkins的WebUI是否可以正常访问
http://10.0.0.250:8080/

    11.在jenkins服务器上部署git软件
yum -y install git

将代码推送到Gitee仓库:
    1.注册账号并登录
https://gitee.com/

    2.创建Gitee项目
略。

    3.将本地代码推送到远程仓库
git config --global user.name "jasonyin2020"
git config --global user.email "[email protected]"
mkdir oldboyedu-linux82-yiliao
cd oldboyedu-linux82-yiliao
git init 
wget http://192.168.17.253/Kubernetes/day10-/softwares/jenkins-k8s/jenkins-k8s/yiliaoqixie.zip
unzip yiliaoqixie.zip 
rm -f yiliaoqixie.zip 
mv yiliaoqixie/* ./
rm -rf yiliaoqixie/
git add .
git commit -m 'first commit'
git remote add oldboyedu-linux82 https://gitee.com/jasonyin2020/oldboyedu-linux82-yiliao.git
git push -u oldboyedu-linux82 "master"

Jenkins编译镜像并推送到私有仓库:
    1.指定私有镜像仓库
cat > /etc/docker/daemon.json < {
  "insecure-registries": ["k8s151.oldboyedu.com:5000","10.0.0.7"],
  "registry-mirrors": ["https://tuv7rqqq.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl restart docker
docker info

    2.删除远程仓库的文件
git rm --cached -r *
git commit -m "new v0.2"
git push -u oldboyedu-linux82 "master"

    3.将本地文件推送到远程仓库
[[email protected] oldboyedu-linux82-yiliao]# ll -R
.:
total 4
drwxr-xr-x 2 root root  37 Sep 19 10:47 code
drwxr-xr-x 2 root root  24 Sep 19 10:47 conf
-rw-r--r-- 1 root root 437 Sep 19 10:47 Dockerfile

./code:
total 1516
-rw-r--r-- 1 root root 1552313 Sep 19 10:47 oldboyedu-yiliao.tar.gz

./conf:
total 4
-rw-r--r-- 1 root root 335 Sep 19 10:47 nginx.conf
[[email protected] oldboyedu-linux82-yiliao]# 
[[email protected] oldboyedu-linux82-yiliao]# cat conf/nginx.conf 
worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        root        /usr/local/nginx/html/yiliao/;
        server_name   yiliao.oldboyedu.com;
    }
}
[[email protected] oldboyedu-linux82-yiliao]# 
[[email protected] oldboyedu-linux82-yiliao]# cat Dockerfile 
FROM alpine

ONBUILD LABEL school=oldboyedu \
      class=linux82 \
      auther=JasonYin

EXPOSE 80

RUN sed -i 's#dl-cdn.alpinelinux.org#mirrors.aliyun.com#' /etc/apk/repositories && \
    apk update && apk  add nginx && \
    mkdir -p /usr/local/nginx/html/yiliao && \
    rm -rf /var/cache/

ADD code/oldboyedu-yiliao.tar.gz /usr/local/nginx/html/yiliao/

COPY conf/nginx.conf /etc/nginx/nginx.conf

CMD ["nginx","-g","daemon off;"]
[[email protected] oldboyedu-linux82-yiliao]# 
[[email protected] oldboyedu-linux82-yiliao]# git add *
[[email protected] oldboyedu-linux82-yiliao]# git commit -m 'new v0.3'
[[email protected] oldboyedu-linux82-yiliao]# git push -u oldboyedu-linux82 "master"

    4.将代码编译成镜像并推送到远程仓库
docker build -t k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1 .
docker push k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1

    5.测试镜像是否能正常使用
docker run -d k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1

Jenkins集成K8S实战:
    1.项目第一次部署采用资源清单的方式部署
[[email protected] jenkins]# cat 01-deploy-yiliao.yaml 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: oldboyedu-linux82-yiliao
spec:
  replicas: 3
  selector:
    matchLabels:
      apps: yiliao
  template:
    metadata:
      labels:
        apps: yiliao
    spec:
      tolerations:
      - operator: Exists
      containers:
      - image: k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1
        name: yiliao
        ports:
        - name: http
          containerPort: 80
[[email protected] jenkins]# 
[[email protected] jenkins]# cat 02-svc-yiliao.yaml 
apiVersion: v1
kind: Service
metadata:
  name: linux82-yiliao
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
     apps: yiliao
[[email protected] jenkins]# 
[[email protected] jenkins]# 
[[email protected] jenkins]# cat 03-ing-yiliao.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
   name: oldboyedu-linux82-yiliao
spec:
   rules:
   - host: yiliao.oldboyedu.com
     http:
       paths:
       - backend:
           serviceName: linux82-yiliao
           servicePort: 80
[[email protected] jenkins]# 
[[email protected] jenkins]# kubectl apply -f 01-deploy-yiliao.yaml 

    2.Jenkins节点部署kubectl工具
[[email protected] ~]# cat > /etc/yum.repos.d/kubernetes.repo < [kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
[[email protected] ~]# 
[[email protected] ~]# yum -y install kubectl

    3.拷贝kubeconfig文件到Jenkins节点
[[email protected] ~]# scp -r /root/.kube/ 10.0.0.250:~

    4.修改Jenkins的配置，更新K8S的镜像
docker build -t k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1 .
docker push k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1
kubectl set image  deployments oldboyedu-linux82-yiliao yiliao=k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:v0.1

    
    
基于Jenkins的参数化构建:
    1.添加Jenkins的变量
略。图片在群里，假设变量名称为"version"

    2.修改脚本
docker build -t k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:$version .
docker push k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:$version
kubectl set image  deployments oldboyedu-linux82-yiliao yiliao=k8s151.oldboyedu.com:5000/oldboyedu-linux82-yiliao:$version

    3.完善项目
(1)开发人员推送代码后，运维人员无需手动接入;
(2)代码发布上线后，钉钉|企业微信|邮箱|短信告警;

    
    
    
ElasticStack收集K8S日志：
[[email protected] elasticstack]# cat 01-deployment-es.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: oldboyedu-elk

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: oldboyedu-elk
  labels:
    k8s-app: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: elasticsearch
  template:
    metadata:
      labels:
        k8s-app: elasticsearch
    spec:
      tolerations:
      - operator: Exists
      containers:
      - image: k8s151.oldboyedu.com:5000/oldboyedu-elk/elasticsearch:7.17.5
        name: elasticsearch
        resources:
          limits:
            cpu: 2
            memory: 3Gi
          requests:
            cpu: 0.5 
            memory: 500Mi
        env:
          # 配置集群部署模式，此处我由于是实验，配置的是单点
          - name: "discovery.type"
            value: "single-node"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m" 
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-data
        persistentVolumeClaim:
          claimName: es-pvc

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: es-pvc
  namespace: oldboyedu-elk
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

---

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: oldboyedu-elk
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: 9200
  selector:
    k8s-app: elasticsearch
[[email protected] elasticstack]# 
[[email protected] elasticstack]# cat 02-deployment-kibana.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: oldboyedu-elk
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana
  template:
    metadata:
      labels:
        k8s-app: kibana
    spec:
      tolerations:
      - operator: Exists
      containers:
      - name: kibana
        image: k8s151.oldboyedu.com:5000/oldboyedu-elk/kibana:7.17.5
        resources:
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 0.5 
            memory: 500Mi
        env:
          - name: ELASTICSEARCH_HOSTS
            value: http://elasticsearch.oldboyedu-elk:9200
          - name: I18N_LOCALE
            value: zh-CN
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: oldboyedu-elk
spec:
  # type: NodePort
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
    # nodePort: 35601
  selector:
    k8s-app: kibana

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
   name: oldboyedu-linux82-kibana
   namespace: oldboyedu-elk
spec:
   rules:
   - host: kibana.oldboyedu.com
     http:
       paths:
       - backend:
           serviceName: kibana
           servicePort: 5601

[[email protected] elasticstack]# 
[[email protected] elasticstack]# cat 03-deployment-filebeat.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: oldboyedu-elk
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    output.elasticsearch:
      hosts: ['elasticsearch.oldboyedu-elk:9200']
      # 不建议修改索引，因为索引名称该成功后，pod的数据也将收集不到啦!
      # 除非你明确知道自己不收集Pod日志且需要自定义索引名称的情况下，可以打开下面的注释哟~
      # index: 'oldboyedu-linux-elk-%{+yyyy.MM.dd}'
    
    # 配置索引模板
    # setup.ilm.enabled: false
    # setup.template.name: "oldboyedu-linux-elk"
    # setup.template.pattern: "oldboyedu-linux-elk*"
    # setup.template.overwrite: true
    # setup.template.settings:
    #   index.number_of_shards: 3
    #   index.number_of_replicas: 0

---

# 注意，官方在filebeat 7.2就已经废弃docker类型，建议后期更换为container.
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: oldboyedu-elk
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |
    - type: docker
      containers.ids:
       - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true

---

apiVersion: apps/v1 
kind: DaemonSet
metadata:
  name: filebeat
  namespace: oldboyedu-elk
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      tolerations:
      - operator: Exists
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        # 注意官方的filebeat版本推荐使用"elastic/filebeat:7.10.2",
        # 如果高于该版本("elastic/filebeat:7.10.2")可能收集不到K8s集群的Pod相关日志指标哟~
        # 经过我测试，直到2022-04-01开源的7.12.2版本依旧没有解决该问题! 
        # filebeat和ES版本可以不一致哈，因为我测试ES的版本是7.17.2
        #
        # 待完成: 后续可以尝试更新最新的镜像，并将输入的类型更换为container，因为docker输入类型官方在filebeat 7.2已废弃！
        image: k8s151.oldboyedu.com:5000/oldboyedu-elk/elastic/filebeat:7.10.2
        # image: k8s151.oldboyedu.com:5000/oldboyedu-elk/elastic/filebeat:7.17.5
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        # 出问题后可以用作临时调试，注意需要将args注释哟
        # command: ["sleep","3600"]
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: oldboyedu-elk
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: oldboyedu-elk
  labels:
    k8s-app: filebeat
[[email protected] elasticstack]# 
[[email protected] elasticstack]# kubectl apply -f .

删除缓存:
    rm -rf /var/lib/filebeat-data/*
    
    
    
今日内容回顾:
    - JENKINS集成K8S项目实战
    - ELASTICSTACK收集K8S日志
    - Prometheus监控K8S数据