简介
Helm是一个由CNCF孵化和管理的项目，用于对需要在Kubernetes 上部署的复杂应用进行定义、安装和更新。Helm以Chart的方式对应用软件进行描述，可以方便地创建、版本化、共享和发布复杂的应用软件。
helm3架构
image-20211008155057021.png
helm3安装
#github地址
#https://github.com/helm/helm
#本文示例使用的是v3.7.0版本
wget https://get.helm.sh/helm-v3.7.0-linux-amd64.tar.gz
#解压->helm放入PATH一个路径下
helm中三大概念
Chart：一个Helm包，其中包含运行一个应用所需要的工具和资源定义，还可能包含Kubernetes集群中的服务定义，类似于Homebrew 中的formula、APT中的dpkg或者Yum中的RPM文件。
Release：在Kubernetes集群上运行的一个Chart实例。在同一个 集群上，一个Chart可以被安装多次。例如有一个MySQL Chart，如果想在服务器上运行两个MySQL数据库，就可以基于这个Chart安装两次。 每次安装都会生成新的Release，会有独立的Release名称。
Repository：用于存放和共享Chart仓库。 简单来说，Helm整个系统的主要任务就是，在仓库中查找需要的 Chart，然后将Chart以Release的形式安装到Kubernetes集群中。
Helm Chart的使用
下面将使用一个例子展示helm chart的使用。
创建
$ helm create nginx
该命令会创建一个nginx文件目录，tree查看目录结构
$ tree
.
├── charts #包含chart依赖的其他chart
├── Chart.yaml #包含了chart信息的YAML文件
├── templates #模板目录， 当和values 结合时，可生成有效的Kubernetes manifest文件
│ ├── deployment.yaml
│ ├── _helpers.tpl
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests #测试
│ └── test-connection.yaml
└── values.yaml #chart 默认的配置值
Chart.yaml
$ cat Chart.yaml
apiVersion: v2 #在heml3中apiVersion必须是v2
name: nginx #chart名字
description: A Helm chart for Kubernetes #chart描述
type: application #chart类型 application（默认）、library
version: 0.1.0 #chart的版本
appVersion: "1.16.0" #应用的版本
values.yaml
$ cat values.yaml
# Default values for nginx.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: nginx
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: ClusterIP
port: 80
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
nodeSelector: {}
tolerations: []
affinity: {}
templates目录下存放了应用编排文件。
-
（_）开头的文件用来存储局部和辅助对象，供其他chart模板使用。模板命令都是嵌入在
{{和}}之间的。cat _helpers.tpl {{/* Expand the name of the chart. */}} {{- define "nginx.name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} {{- end }} {{/* Create a default fully qualified app name. We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). If release name contains chart name it will be used as a full name. */}} {{- define "nginx.fullname" -}} {{- if .Values.fullnameOverride }} {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} {{- else }} {{- $name := default .Chart.Name .Values.nameOverride }} {{- if contains $name .Release.Name }} {{- .Release.Name | trunc 63 | trimSuffix "-" }} {{- else }} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} {{- end }} {{- end }} {{- end }} {{/* Create chart name and version as used by the chart label. */}} {{- define "nginx.chart" -}} {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} {{- end }} {{/* Common labels */}} {{- define "nginx.labels" -}} helm.sh/chart: {{ include "nginx.chart" . }} {{ include "nginx.selectorLabels" . }} {{- if .Chart.AppVersion }} app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end }} {{/* Selector labels */}} {{- define "nginx.selectorLabels" -}} app.kubernetes.io/name: {{ include "nginx.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} {{/* Create the name of the service account to use */}} {{- define "nginx.serviceAccountName" -}} {{- if .Values.serviceAccount.create }} {{- default (include "nginx.fullname" .) .Values.serviceAccount.name }} {{- else }} {{- default "default" .Values.serviceAccount.name }} {{- end }} {{- end }} -
yaml格式的编排文件中将某些字段设置为“模板命令”，这些“模板命令”会在Helm部署应用时进行参数注入和模板的动态渲染。
$ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "nginx.fullname" . }} labels: {{- include "nginx.labels" . | nindent 4 }} spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} selector: matchLabels: {{- include "nginx.selectorLabels" . | nindent 6 }} template: metadata: {{- with .Values.podAnnotations }} annotations: {{- toYaml . | nindent 8 }} {{- end }} labels: {{- include "nginx.selectorLabels" . | nindent 8 }} spec: {{- with .Values.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} serviceAccountName: {{ include "nginx.serviceAccountName" . }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} containers: - name: {{ .Chart.Name }} securityContext: {{- toYaml .Values.securityContext | nindent 12 }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: - name: http containerPort: 80 protocol: TCP livenessProbe: httpGet: path: / port: http readinessProbe: httpGet: path: / port: http resources: {{- toYaml .Values.resources | nindent 12 }} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }}可是使用
helm template [NAME] [CHART] [flags]命令在本地渲染Helm Chart并打印最终的应用模板内容，values.yaml和（_）开头的文件中的值会被注入到yaml中，参数--set则会覆盖value.yaml文件中的值。$ helm template nginx ./nginx --set image.repository=docker.io/library/nginx,service.type=NodePort ... ... --- # Source: nginx/templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: helm.sh/chart: nginx-0.1.0 app.kubernetes.io/name: nginx app.kubernetes.io/instance: nginx app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: nginx app.kubernetes.io/instance: nginx template: metadata: labels: app.kubernetes.io/name: nginx app.kubernetes.io/instance: nginx spec: serviceAccountName: nginx securityContext: {} containers: - name: nginx securityContext: {} image: "docker.io/library/nginx:1.16.0" imagePullPolicy: IfNotPresent ports: - name: http containerPort: 80 protocol: TCP livenessProbe: httpGet: path: / port: http readinessProbe: httpGet: path: / port: http resources: {} --- ... ... -
NOTES.txt在
helm install或helm upgrade命令的最后，Helm会打印出对用户有用的信息，该文件是纯文本，但会像模板一样处理， 所有正常的模板函数和对象都是可用的。$ cat NOTES.txt 1\. Get the application URL by running these commands: {{- if .Values.ingress.enabled }} {{- range $host := .Values.ingress.hosts }} {{- range .paths }} http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }} {{- end }} {{- end }} {{- else if contains "NodePort" .Values.service.type }} export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "nginx.fullname" . }}) export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT {{- else if contains "LoadBalancer" .Values.service.type }} NOTE: It may take a few minutes for the LoadBalancer IP to be available. You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "nginx.fullname" . }}' export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "nginx.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") echo http://$SERVICE_IP:{{ .Values.service.port }} {{- else if contains "ClusterIP" .Values.service.type }} export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "nginx.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}") echo "Visit http://127.0.0.1:8080 to use your application" kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT {{- end }}
部署
helm install [NAME] [CHART] [flags]命令安装部署，同样values.yaml和（_）开头的文件中的值会被注入到yaml中，参数--set则会覆盖value.yaml文件中的值。
$ helm -n default install nginx ./nginx --set image.repository=docker.io/library/nginx,service.type=NodePort
NAME: nginx
LAST DEPLOYED: Sat Oct 9 16:26:50 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1\. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx)
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
-
查看部署的release
$ helm -n default ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx default 1 2021-10-09 16:26:50.820014356 +0800 CST deployed nginx-0.1.0 1.16.0 -
release版本信息存储
默认存储在release相应namespace下的secret（可配置configmap、sql），每一个REVISION都会生成一个secret。
$ kubectl -n default get secret -l "owner=helm" NAME TYPE DATA AGE sh.helm.release.v1.nginx.v1 helm.sh/release.v1 1 1m查看存储内容
kubectl -n default get secret sh.helm.release.v1.nginx.v1 -ojsonpath={.data.release} |base64 -d |base64 -d |gzip -d -
测试release是否符合预期（需等待所有部署的资源对象创建成功），原理：通过在集群中部署templates/tests/下的yaml，根据容器退出状态判断（exit 0：通过）是否测试通过。测试的yaml必须含有注解：helm.sh/hook: test。
$ helm test nginx NAME: nginx LAST DEPLOYED: Sat Oct 9 16:26:50 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: nginx-test-connection Last Started: Sat Oct 9 16:47:02 2021 Last Completed: Sat Oct 9 16:47:20 2021 Phase: Succeeded NOTES: 1\. Get the application URL by running these commands: export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx) export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT
升级、回滚、卸载
-
helm upgrade [RELEASE] [CHART] [flags]命令进行升级。升级之前查看nginx release当前版本：REVISION=1
helm -n default ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx default 1 2021-10-09 16:26:50.820014356 +0800 CST deployed nginx-0.1.0 1.16.0使用升级命令，将deployment中的镜像tag升级到1.17.0，可使用
--dry-run试运行测试是否可以升级成功，并打印升级清单。$ helm -n default upgrade nginx ./nginx/ --set image.repository=docker.io/library/nginx,service.type=NodePort,image.tag=1.17.0 Release "nginx" has been upgraded. Happy Helming! NAME: nginx LAST DEPLOYED: Sun Oct 10 13:41:17 2021 NAMESPACE: default STATUS: deployed REVISION: 2 NOTES: 1\. Get the application URL by running these commands: export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx) export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT再次查看nginx release当前版本：REVISION=2
$ helm -n default ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION nginx default 2 2021-10-10 13:41:17.486033411 +0800 CST deployed nginx-0.1.0 1.16.0 -
helm rollback命令进行回滚。[REVISION] [flags] 回滚之前可使用
helm history RELEASE_NAME [flags]查看nginx release历史版本。$ helm -n default history nginx REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Sat Oct 9 16:26:50 2021 superseded nginx-0.1.0 1.16.0 Install complete 2 Sun Oct 10 13:41:17 2021 deployed nginx-0.1.0 1.16.0 Upgrade complete将nginx release回滚到REVISION=1，回滚完成后可查看deployment image tag：1.16.0。
$ helm -n default rollback nginx 1 Rollback was a success! Happy Helming! -
helm uninstall RELEASE_NAME [...] [flags]命令卸载release，卸载完成后release、release secret、k8s manifest都被删除。$ helm -n default uninstall nginx release "nginx" uninstalled
仓库使用
Harbor
-
使用helm安装harbor
$ helm repo add harbor https://helm.goharbor.io "harbor" has been added to your repositories #查看本地repo列表 $ helm repo ls NAME URL harbor https://helm.goharbor.io #查看harbor仓库helm chart版本 $ helm search repo harbor NAME CHART VERSION APP VERSION DESCRIPTION harbor/harbor 1.7.3 2.3.3 An open source trusted cloud native registry th... #部署harbor release #persistence.enabled=false 关闭持久化存储 #expose.type=nodePort service类型设置为nodeport方便暴露服务 #expose.tls.enabled=false 关闭tls #--create-namespace 没有harbor命名空间，则会创建 #externalURL中的ip为本机网卡ip $ helm -n harbor install harbor harbor/harbor --set persistence.enabled=false,expose.type=nodePort,expose.tls.enabled=false,externalURL=http://10.23.18.211:30002 --create-namespace NAME: harbor LAST DEPLOYED: Mon Oct 11 18:38:47 2021 NAMESPACE: harbor STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Please wait for several minutes for Harbor deployment to complete. Then you should be able to visit the Harbor portal at http://10.23.18.211:30002 For more details, please visit https://github.com/goharbor/harbor #等待所有pod running $ kubectl -n harbor get pod NAME READY STATUS RESTARTS AGE harbor-chartmuseum-6c9f9c84f8-ztpgp 1/1 Running 0 3m20s harbor-core-c6dc8c895-vn5kx 1/1 Running 0 3m19s harbor-database-0 1/1 Running 0 3m20s harbor-jobservice-6c4d647bcb-nd64t 1/1 Running 0 3m20s harbor-nginx-7bc597c58b-67x6x 1/1 Running 0 3m19s harbor-notary-server-5f598b9555-4qjpm 1/1 Running 0 3m20s harbor-notary-signer-65b84d688d-g92tv 1/1 Running 0 3m19s harbor-portal-945d945f-ks6tx 1/1 Running 0 3m20s harbor-redis-0 1/1 Running 0 3m20s harbor-registry-6d976587dc-dbrnt 2/2 Running 0 3m20s harbor-trivy-0 1/1 Running 0 3m20s浏览器访问http://10.23.18.211:30002即可访问harbor页面，admin/Harbor12345为默认用户密码。
-
推送nginx chart到仓库
登录harbor页面，项目-->新建项目，创建一个名为helm的helm chart仓库。
添加helm仓库。
$ helm repo add my-harbor http://10.23.18.211:30002/chartrepo/helm --username=admin --password=Harbor12345 "my-harbor" has been added to your repositories #查看当前repo $ helm repo ls NAME URL harbor https://helm.goharbor.io my-harbor http://10.23.18.211:30002/chartrepo/helm由于原生helm没有推送命令，所以需要安装一个推送插件。
$ helm plugin install https://github.com/chartmuseum/helm-push.git Downloading and installing helm-push v0.10.0 ... https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz Installed plugin: cm-push #查看已安装的插件 $ helm plugin ls NAME VERSION DESCRIPTION cm-push 0.10.0 Push chart package to ChartMuseum（备选方案）如果服务器不能访问外网。可以从能够访问外网的机器clone插件代码和插件压缩包，然后放到此服务器上。
#在能够访问外网的机器上 $ git clone https://github.com/chartmuseum/helm-push.git $curl https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz #将插件代码上传到服务器/root/.cache/helm/plugins目录下，创建release目录 $ mkdiir /root/.cache/helm/plugins/helm-push/release #将插件压缩包上传到服务器/root/.cache/helm/plugins/helm-push/releases下 $ cd /root/.cache/helm/plugins #vim 编辑helm-push/scripts/install_plugin.sh，根据需要注释掉 `# Download with curl if possible.`下边代码，以注释掉curl为例： $ vim helm-push/scripts/install_plugin.sh $ cat helm-push/scripts/install_plugin.sh ... ... # Download with curl if possible. if [ -x "$(which curl 2>/dev/null)" ]; then # curl -sSL "${url}" -o "releases/v${version}.tar.gz" echo "jump curl" else wget -q "${url}" -O "releases/v${version}.tar.gz" fi ... ... #安装推送插件 $ helm plugin install ./helm-push/ Downloading and installing helm-push v0.10.0 ... https://github.com/chartmuseum/helm-push/releases/download/v0.10.0/helm-push_0.10.0_linux_amd64.tar.gz jump curl Installed plugin: cm-push [root@master01 plugins]# helm plugin ls NAME VERSION DESCRIPTION cm-push 0.10.0 Push chart package to ChartMuseum推送nginx chart。
#cd到nginx目录，推送 $ helm cm-push ./nginx my-harbor Pushing nginx-0.1.0.tgz to my-harbor... Done. #更新本地缓存的my-harbor仓库信息，可以确认nginx chart上传到harbor上，也可以在harbor页面上确认 $ helm repo update my-harbor Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "my-harbor" chart repository Update Complete. ⎈Happy Helming!⎈ $ helm search repo my-harbor NAME CHART VERSION APP VERSION DESCRIPTION my-harbor/nginx 0.1.0 1.16.0 A Helm chart for Kubernetes -
拉取、部署仓库中的chart
helm pull [chart URL | repo/chartname] [...] [flags]拉取仓库中chart，pull只会拉取到本地当前目录下的.tgz压缩包，可使用tar xzvf xxx.tgz解压。#更新本地helm repo 缓存 $ helm repo update #pull拉取nginx chart $ helm pull my-harbor/nginxhelm install [NAME] [CHART] [flags]部署仓库中的chart，和上边部署篇幅类似，把使用本地文件改成使用仓库中的chart即可。$ helm -n test install nginx my-harbor/nginx --set image.repository=docker.io/library/nginx,service.type=NodePort --create-namespace NAME: nginx LAST DEPLOYED: Mon Oct 11 19:47:36 2021 NAMESPACE: test STATUS: deployed REVISION: 1 NOTES: 1\. Get the application URL by running these commands: export NODE_PORT=$(kubectl get --namespace test -o jsonpath="{.spec.ports[0].nodePort}" services nginx) export NODE_IP=$(kubectl get nodes --namespace test -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT