kubernetes下搭建harbor企业级docker仓库
前言
harbor是由vmware开源的企业级docker repository，提供私有仓库，安全认证，权限管理，漏洞扫描及仓库复制等多种功能，因为之前搭建的harbor在线上环境存在各种各样的问题（之前搭建的方式目前发现已被官方宣布弃用https://github.com/vmware/harbor/blob/master/docs/kubernetes_deployment.md，如ui显示不正常，权限认证不正常），于是近期抽空用官方推荐的helm方式对harbor进行重新部署，并替换掉线上harbor
准备
- Kubernetes集群1.8+
- Kubernetes Ingress Controller已启用(这里我们选用的traefik-ingress)
- kubectl 客户端 1.8+
- 可选的持久化功能需要准备PV或者SC（我们这里使用nfs创建的PV和PVC，如果有条件使用SC的会更加方便）
部署过程
- 首先按照官方教程安装Helm，然后初始化Helm。
注意： 初始化需要使用下面命令使用canary镜像，否则无法正常安装，会报错helm部署文件的格式不正确，目前这是一个已知issue:https://github.com/vmware/harbor/issues/4484
helm init --canary-image
如果你和我一样，在这之前已经用官方教程安装好了helm，则可使用下面命令更新helm server:
helm init --canary-image --upgrade
- 下载helm部署代码并进入harbor helm目录。
git clone https://github.com/vmware/harbor
cd harbor/contrib/helm/harbor
推荐使用这个连接下载指定的文件夹，否则在网络不佳的情况下拉取全部代码会比较耗时:
https://minhaskamal.github.io/DownGit/#/home
- 更新helm dependency
harbor的helm部署依赖了postgresql的helm，在官方的安装文档没有明确说明，因为我也是第一次使用，直接按照官方文档说明安装，就会缺失postgresql的部署，导致整个服务无法启动
helm dependency update
- 安装harbor
这里官方提供了两种方式，Insecure和Secure，我这里选用的是Secure安全的部署方式，让harbor自己生成CA和SSL，简单方便。执行如下命令安装harbor:
helm install . --debug --name hub --set externalDomain=harbor.my.domain
externalDomain为外部能访问到harbor的域名，到目前为止，你可以在本地/etc/hosts中添加域名解析，从本地访问进行测试，待测试完成后再加入traefik-ingress中，当然你也可以直接在traefik-ingress中添加域名解析。 - 添加traefik-ingress的域名解析：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-default-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "traefik"
spec:
rules:
- host: harbor.my.domain
http:
paths:
- path: /
backend:
serviceName: my-release-harbor-ui
servicePort: 80
- path: /
- host: harbor.my.domain
helm harbor自定义配置
注意： 我这里把所有自定义配置放到后面的附录中，供大家参考,同时可以点击https://github.com/Anteoy/harbor-helm/commit/226b296d130b4f956f8463eecf2aa473bc1e844c,查看我上传到github的自定义配置，从github阅读更清晰。
- 重要：在生产环境中需要持久化数据存储，否则pod重启或重建会造成数据丢失。如果有创建好的storageClass可以直接在values.yaml配置，如果没有或暂时不能使用SC的，比如我这里只有nfs，则需要像我一样多修改一些配置。另外,nfs其实在社区也有storageClass的驱动库，不过我看了下使用起来较为繁琐，于是这里仍然修改为使用nfs。
- 修改templates默认的namespace，注意需要修改依赖的chart文件夹下的postgresql-0.9-1.tgz压缩文件中的部署yaml模板，否则harbor的ui,register,mysql和postgresql不在一个namespace下，不能正常安装
- 注意修改postgresql的持久化数据卷，可参考https://github.com/kubernetes/charts/tree/master/stable/postgresql
附录
templates/adminserver/adminserver-cm.yaml
@@ -1,6 +1,7 @@
apiVersion: v1
kind: ConfigMap
metadata:
+ namespace: class100-ops
name: "{{ template "harbor.fullname" . }}-adminserver"
labels:
{{ include "harbor.labels" . | indent 4 }}
1 templates/adminserver/adminserver-secrets.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: "{{ template "harbor.fullname" . }}-adminserver"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: adminserver
5 templates/adminserver/adminserver-ss.yaml
@@ -2,6 +2,7 @@ apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: "{{ template "harbor.fullname" . }}-adminserver"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: adminserver
@@ -42,13 +43,15 @@ spec:
volumeMounts:
- name: adminserver-config
mountPath: /etc/adminserver/config
+ subPath: harbor-v1/adminserver/
- name: adminserver-key
mountPath: /etc/adminserver/key
subPath: key
volumes:
{{- if not .Values.persistence.enabled }}
- name: adminserver-config
- emptyDir: {}
+ persistentVolumeClaim:
+ claimName: harbor-pvc
{{- end }}
- name: adminserver-key
secret:
1 templates/adminserver/adminserver-svc.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: "{{ template "harbor.fullname" . }}-adminserver"
+ namespace: class100-ops
spec:
ports:
- port: 80
1 templates/clair/clair-cm.yaml
@@ -3,6 +3,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "harbor.fullname" . }}-clair
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: clair
1 templates/clair/clair-dpl.yaml
@@ -3,6 +3,7 @@ apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "harbor.fullname" . }}-clair
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: clair
1 templates/clair/clair-svc.yaml
@@ -6,6 +6,7 @@ apiVersion: v1
kind: Service
metadata:
name: clair
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
1 templates/ingress/ingress.yaml
@@ -2,6 +2,7 @@ apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "{{ template "harbor.fullname" . }}-ingress"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
annotations:
1 templates/ingress/secret.yaml
@@ -5,6 +5,7 @@ apiVersion: v1
kind: Secret
metadata:
name: "{{ template "harbor.fullname" . }}-ingress"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
type: kubernetes.io/tls
1 templates/jobservice/jobservice-cm.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: "{{ template "harbor.fullname" . }}-jobservice"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
data:
1 templates/jobservice/jobservice-dpl.yaml
@@ -2,6 +2,7 @@ apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: "{{ template "harbor.fullname" . }}-jobservice"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: jobservice
1 templates/jobservice/jobservice-secrets.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: "{{ template "harbor.fullname" . }}-jobservice"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
type: Opaque
1 templates/jobservice/jobservice-svc.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: "{{ template "harbor.fullname" . }}-jobservice"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
1 templates/mysql/mysql-secret.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: "{{ template "harbor.fullname" . }}-mysql"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
type: Opaque
7 templates/mysql/mysql-ss.yaml
@@ -2,6 +2,7 @@ apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: "{{ template "harbor.fullname" . }}-mysql"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: mysql
@@ -33,10 +34,12 @@ spec:
volumeMounts:
- name: mysql-data
mountPath: /var/lib/mysql
+ subPath: harbor-v1/mysql-data/
{{- if not .Values.persistence.enabled }}
volumes:
- - name: "mysql-data"
- emptyDir: {}
+ - name: mysql-data
+ persistentVolumeClaim:
+ claimName: harbor-pvc
{{- end -}}
{{- if .Values.persistence.enabled }}
volumeClaimTemplates:
1 templates/mysql/mysql-svc.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: "{{ template "harbor.fullname" . }}-mysql"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
1 templates/registry/registry-cm.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: "{{ template "harbor.fullname" . }}-registry"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
data:
1 templates/registry/registry-secret.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: "{{ template "harbor.fullname" . }}-registry"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
type: Opaque
5 templates/registry/registry-ss.yaml
@@ -2,6 +2,7 @@ apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
name: "{{ template "harbor.fullname" . }}-registry"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: registry
@@ -37,6 +38,7 @@ spec:
volumeMounts:
- name: registry-data
mountPath: /var/lib/registry
+ subPath: harbor-v1/registry-data/
- name: registry-root-certificate
mountPath: /etc/registry/root.crt
subPath: root.crt
@@ -47,7 +49,8 @@ spec:
{{- if not .Values.registry.objectStorage }}
{{- if not .Values.persistence.enabled }}
- name: registry-data
- emptyDir: {}
+ persistentVolumeClaim:
+ claimName: harbor-pvc
{{- end }}
{{- end }}
- name: registry-root-certificate
1 templates/registry/registry-svc.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: "{{ template "harbor.fullname" . }}-registry"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
1 templates/ui/ui-cm.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: ConfigMap
metadata:
name: "{{ template "harbor.fullname" . }}-ui"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
data:
3 templates/ui/ui-dpl.yaml
@@ -2,6 +2,7 @@ apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: "{{ template "harbor.fullname" . }}-ui"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
component: ui
@@ -54,8 +55,10 @@ spec:
subPath: private_key.pem
- name: ca-download
mountPath: /etc/ui/ca
+ subPath: harbor-v1/ui-ca/
- name: psc
mountPath: /etc/ui/token
+ subPath: harbor-v1/ui-psc/
volumes:
- name: ui-config
configMap:
1 templates/ui/ui-secrets.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Secret
metadata:
name: "{{ template "harbor.fullname" . }}-ui"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
type: Opaque
1 templates/ui/ui-svc.yaml
@@ -2,6 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: "{{ template "harbor.fullname" . }}-ui"
+ namespace: class100-ops
labels:
{{ include "harbor.labels" . | indent 4 }}
spec:
3 values.yaml
@@ -287,4 +287,5 @@ postgresql:
postgresPassword: not-a-secure-password
postgresDatabase: clair
persistence:
- enabled: false
+ enabled: true
+ existingClaim: harbor-pvc 参考
- https://github.com/vmware/harbor/issues/4484
- https://github.com/vmware/harbor/tree/master/contrib/helm/harbor
- https://github.com/vmware/harbor/issues/4481
- https://kubernetes.io/docs/concepts/storage/storage-classes/#introduction