OpenShift 4 之 GitOps（3）用Helm+ArgoCD部署应用，并保持配置同步
文章目录
- 运行环境
- 用Helm创建样例Chart
- 根据Helm Chart安装OpenShift应用
- 从Helm Chat导出要部署的应用对象
- 将应用资源配置文件推送至Github的Repo
- 根据Github的配置创建OpenShift的应用资源
- 自动调整OpenShift的配置，以保持和Github中的配置同步
- 将Github中的新版配置同步更新至OpenShift
运行环境
- 安装ArgoCD服务器环境。
- 根据《OpenShift 4.3 之新特性 - 使用Helm部署OpenShift应用》安装helm客户端即可。
用Helm创建样例Chart
- 创建名为myapp的helm chart，然后可以查看chart中的deployment.yaml。
$ mkdir helmstuff
$ cd helmstuff/
$ helm create myapp
Creating myapp
$ find myapp/
myapp/
myapp/Chart.yaml
myapp/values.yaml
myapp/.helmignore
myapp/templates
myapp/templates/ingress.yaml
myapp/templates/deployment.yaml
myapp/templates/service.yaml
myapp/templates/serviceaccount.yaml
myapp/templates/NOTES.txt
myapp/templates/_helpers.tpl
myapp/templates/tests
myapp/templates/tests/test-connection.yaml
myapp/charts
根据Helm Chart安装OpenShift应用
- 创建OpenShift的helmstuff项目，然后通过helm安装应用。
$ oc new-project helmstuff
$ helm install adventure1 myapp/ -n helmstuff
NAME: adventure1
LAST DEPLOYED: Wed Mar 4 10:37:09 2020
NAMESPACE: helmstuff
STATUS: deployed
REVISION: 1
NOTES:
(1). Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace helmstuff -l "app.kubernetes.io/name=myapp,app.kubernetes.io/instance=adventure1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helmstuff port-forward $POD_NAME 8080:80
- 查看Helm列表。
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
adventure1 helmstuff 1 2020-03-04 10:37:09.669437379 +0000 UTC deployed myapp-0.1.0 1.16.0
- 查看pod状态为CrashLoopBackOff，然后helm中的adventure1，确认结果报错“Error: pod
adventure1-myapp-test-connection failed”。
$ oc get pod
NAME READY STATUS RESTARTS AGE
adventure1-myapp-5b64cf64cb-r65fk 0/1 CrashLoopBackOff 1 32s
$ helm test adventure1
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection running
NAME: adventure1
LAST DEPLOYED: Wed Mar 4 10:37:09 2020
NAMESPACE: helmstuff
STATUS: deployed
REVISION: 1
TEST SUITE: adventure1-myapp-test-connection
Last Started: Wed Mar 4 10:38:18 2020
Last Completed: Wed Mar 4 10:38:29 2020
Phase: Failed
NOTES:
(1). Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace helmstuff -l "app.kubernetes.io/name=myapp,app.kubernetes.io/instance=adventure1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helmstuff port-forward $POD_NAME 8080:80
Error: pod adventure1-myapp-test-connection failed
- 上面错误是由于权限问题。执行以下命令为容器中的操作用户提权。
$ oc adm policy add-scc-to-user anyuid -z adventure1-myapp
securitycontextconstraints.security.openshift.io/anyuid added to: ["system:serviceaccount:helmstuff:adventure1-myapp"]
- 先从helm中删除adventure1，然后重新创建helm的adventure1，最后再用helm测试adventure1，确认这次可测通过。
$ helm uninstall adventure1
release "adventure1" uninstalled
$ helm install adventure1 myapp/
NAME: adventure1
LAST DEPLOYED: Wed Mar 4 10:45:21 2020
NAMESPACE: helmstuff
STATUS: deployed
REVISION: 1
NOTES:
(1). Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace helmstuff -l "app.kubernetes.io/name=myapp,app.kubernetes.io/instance=adventure1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helmstuff port-forward $POD_NAME 8080:80
$ helm test adventure1
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection pending
Pod adventure1-myapp-test-connection succeeded
NAME: adventure1
LAST DEPLOYED: Wed Mar 4 10:45:21 2020
NAMESPACE: helmstuff
STATUS: deployed
REVISION: 1
TEST SUITE: adventure1-myapp-test-connection
Last Started: Wed Mar 4 10:45:40 2020
Last Completed: Wed Mar 4 10:45:50 2020
Phase: Succeeded
NOTES:
(1). Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace helmstuff -l "app.kubernetes.io/name=myapp,app.kubernetes.io/instance=adventure1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helmstuff port-forward $POD_NAME 8080:80
从Helm Chat导出要部署的应用对象
- 创建manifest目录，让后将helm中名为adventure1的manifest导出到manifest/adventure1.yaml。
$ mkdir manifest
$ helm get manifest adventure1 > manifest/adventure1.yaml
- 查看导出的manifest/adventure1.yaml文件内容。
$ cat manifest/adventure1.yaml
---
# Source: myapp/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: adventure1-myapp
labels:
helm.sh/chart: myapp-0.1.0
app.kubernetes.io/name: myapp
app.kubernetes.io/instance: adventure1
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
---
# Source: myapp/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: adventure1-myapp
labels:
helm.sh/chart: myapp-0.1.0
app.kubernetes.io/name: myapp
app.kubernetes.io/instance: adventure1
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: myapp
app.kubernetes.io/instance: adventure1
---
# Source: myapp/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: adventure1-myapp
labels:
helm.sh/chart: myapp-0.1.0
app.kubernetes.io/name: myapp
app.kubernetes.io/instance: adventure1
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: myapp
app.kubernetes.io/instance: adventure1
template:
metadata:
labels:
app.kubernetes.io/name: myapp
app.kubernetes.io/instance: adventure1
spec:
serviceAccountName: adventure1-myapp
securityContext:
{}
containers:
- name: myapp
securityContext:
{}
image: "nginx:1.16.0"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
protocol: TCP
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
resources:
{}
将应用资源配置文件推送至Github的Repo
- 依次执行以下命令，将myapp应用资源推送的自己的Github账户中。
$ git init
Initialized empty Git repository in /home/xiaoyliu-redhat.com/helmstuff/.git/
$ git add *
$ git commit -m "initial commit of helm chart and working manifest"
[master (root-commit) c03cd60] initial commit of helm chart and working manifest
Committer: GTPE Student <[email protected]>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:
git config --global user.name "Your Name"
git config --global user.email [email protected]
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author
11 files changed, 416 insertions(+)
create mode 100644 manifest/adventure1.yaml
create mode 100644 myapp/.helmignore
create mode 100644 myapp/Chart.yaml
create mode 100644 myapp/templates/NOTES.txt
create mode 100644 myapp/templates/_helpers.tpl
create mode 100644 myapp/templates/deployment.yaml
create mode 100644 myapp/templates/ingress.yaml
create mode 100644 myapp/templates/service.yaml
create mode 100644 myapp/templates/serviceaccount.yaml
create mode 100644 myapp/templates/tests/test-connection.yaml
create mode 100644 myapp/values.yaml
$ git remote add origin https://github.com/YOUR-GITHUB/gitops-helm-argocd.git
$ git push -u origin master
Username for 'https://github.com': liuxiaoyu-git
Password for 'https://[email protected]':
Counting objects: 17, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (15/15), done.
Writing objects: 100% (17/17), 5.30 KiB | 0 bytes/s, done.
Total 17 (delta 0), reused 0 (delta 0)
To https://github.com/liuxiaoyu-git/gitops-helm-argocd.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.
根据Github的配置创建OpenShift的应用资源
- 将Github资源加到ArgoCD中的Repo。
$ argocd repo add https://github.com/YOUR-GITHUB/gitops-helm-argocd.git
repository 'https://github.com/liuxiaoyu-git/gitops-helm-argocd.git' added
$ argocd repo list
TYPE NAME REPO INSECURE LFS CREDS STATUS MESSAGE
git https://github.com/liuxiaoyu-git/gitops-helm-argocd.git false false false Successful
- 新建一个名为adventure1的ArgoCD应用，用它在github中的配置资源与OpenShift中的helmstuff项目建立关联。
$ argocd app create --project default --name adventure1 --repo https://github.com/liuxiaoyu-git/gitops-helm-argocd.git --path manifest --dest-server https://kubernetes.default.svc --dest-namespace helmstuff --revision master --sync-policy none
application 'adventure1' created
- 进入ArgoCD的控制台，查看adventure1应用。确认当前OpenShift的用资源和Github Repo中的资源是“Synced”的。
自动调整OpenShift的配置，以保持和Github中的配置同步
- 根据名为adventure1-myapp的OpenShift Service对象市场一个新的对象：adventure1-mybad。
$ oc get svc adventure1-myapp -o json | jq 'del(.spec.clusterIP)' | sed "s/adventure1-myapp/adventure1-mybad/g" | oc create -f -
service/adventure1-mybad created
$ oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
adventure1-myapp ClusterIP 172.30.12.170 <none> 80/TCP 3h38m
adventure1-mybad ClusterIP 172.30.225.179 <none> 80/TCP 10s
- 此时可在ArgoCD控制台中的adventure1应用界面中看到已经是“OutOfSync”状态，且在名为adventure1-mybad的Service下方显示了黄色标记。
- 修改ArgoCD中adventure1应用的配置，然后通过ArgoCD控制台确认名为adventure1-mybad的Service已经被删除。
$ argocd app set adventure1 --sync-policy automated --auto-prune --self-heal
$ oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
adventure1-myapp ClusterIP 172.30.12.170 <none> 80/TCP 3h59m
- 删除本项目中OpenShift的Deployment对象，确认ArgoCD会自动根据Github的配置重新创建一个新的Deployment对象。
$ oc delete deployment adventure1-myapp && oc get pods -w
deployment.extensions "adventure1-myapp" deleted
NAME READY STATUS RESTARTS AGE
adventure1-myapp-5b64cf64cb-l9p5g 1/1 Terminating 1 3h47m
adventure1-myapp-test-connection 0/1 Completed 0 3h47m
NAME READY STATUS RESTARTS AGE
adventure1-myapp-5b64cf64cb-l9p5g 1/1 Terminating 1 3h47m
adventure1-myapp-5b64cf64cb-l9p5g 0/1 Terminating 1 3h47m
adventure1-myapp-5b64cf64cb-l9p5g 0/1 Terminating 1 3h47m
adventure1-myapp-5b64cf64cb-2kf2n 0/1 ContainerCreating 0 10s
adventure1-myapp-5b64cf64cb-2kf2n 0/1 ContainerCreating 0 18s
adventure1-myapp-5b64cf64cb-2kf2n 0/1 Running 0 18s
adventure1-myapp-5b64cf64cb-2kf2n 1/1 Running 0 19s
将Github中的新版配置同步更新至OpenShift
- 先设置ArogCD，关闭Github和OpenShift自动同步配置的功能。
$ argocd app set adventure1 --sync-policy none
- 修改现有Helm Chart，将version从“0.1.0”改为“0.1.1”，将appVersion从“1.16.0”改为“1.16.1”.
$ sed -i 's/1.16.0/1.16.1/g' myapp/Chart.yaml
$ sed -i 's/0.1.0/0.1.1/g' myapp/Chart.yaml
$ cat myapp/Chart.yaml
apiVersion: v2
name: myapp
description: A Helm chart for Kubernetes
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.1.1
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: 1.16.1
- 更新Helm中的adventure1配置，并查看改配置修改前后的变化。
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
adventure1 helmstuff 1 2020-03-04 10:45:21.981629148 +0000 UTC deployed myapp-0.1.0 1.16.0
$ helm upgrade adventure1 myapp/
Release "adventure1" has been upgraded. Happy Helming!
NAME: adventure1
LAST DEPLOYED: Wed Mar 4 14:49:11 2020
NAMESPACE: helmstuff
STATUS: deployed
REVISION: 2
NOTES:
(1). Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace helmstuff -l "app.kubernetes.io/name=myapp,app.kubernetes.io/instance=adventure1" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace helmstuff port-forward $POD_NAME 8080:80
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
adventure1 helmstuff 2 2020-03-04 14:49:11.760250994 +0000 UTC deployed myapp-0.1.1 1.16.1
- 通过命令和控制台查看ArgoCD的adventure1应用的同步状态，发现此时对于变化的配置，OpenShift和Github是没有同步的。
$ argocd app get adventure1
Name: adventure1
Project: default
Server: https://kubernetes.default.svc
Namespace: helmstuff
URL: https://argocd-server-argocd.apps.cluster-beijing-b510.beijing-b510.example.opentlc.com/applications/adventure1
Repo: https://github.com/liuxiaoyu-git/gitops-helm-argocd.git
Target: master
Path: manifest
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: OutOfSync from master (c03cd60)
Health Status: Healthy
GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service helmstuff adventure1-mybad Succeeded Pruned pruned
ServiceAccount helmstuff adventure1-myapp OutOfSync serviceaccount/adventure1-myapp configured. Warning: apply should be used on resource created by either create --save-config or apply
Service helmstuff adventure1-myapp OutOfSync Healthy service/adventure1-myapp configured. Warning: apply should be used on resource created by either create --save-config or apply
apps Deployment helmstuff adventure1-myapp OutOfSync Healthy deployment.apps/adventure1-myapp created
Pod helmstuff adventure1-myapp-test-connection Healthy
5. 在ArdoCD控制台中查看Service的详细配置，其中在DIFF中显示了这个服务在OpenShift和Github的配置差异。
6. 用helm重新生成manifest/adventure1.yaml文件
$ helm get manifest adventure1 > manifest/adventure1.yaml
- 比较新旧版adventure1.yaml文件后，将本地变化的配置文件提交到Github。
$ git diff manifest/adventure1.yaml
$ git add *
$ git commit -m "updated app version and manifests"
[master b844df1] updated app version and manifests
Committer: GTPE Student <[email protected]>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:
git config --global user.name "Your Name"
git config --global user.email [email protected]
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author
2 files changed, 9 insertions(+), 9 deletions(-)
$ git push origin master
Username for 'https://github.com': liuxiaoyu-git
Password for 'https://[email protected]':
Counting objects: 11, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 534 bytes | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/liuxiaoyu-git/gitops-helm-argocd.git
c03cd60..b844df1 master -> master
- 再次打开ArgoCD的同步选项，然后在ArgoCD控制台中确认应用的状态应为“Synced”状态。
$ argocd app set adventure1 --sync-policy automated --auto-prune --self-heal
