k8s(十三)、企业级docker仓库Harbor在kubernetes上搭建使用
前言
Harbor是Docker镜像仓库管理系统，用于企业级管理容器镜像，支持对接ldap进行权限管理,由VMVare中国团队开发。
项目地址:
https://github.com/goharbor/harbor/
安装方式:
1.Docker Compose方式部署:
https://github.com/goharbor/harbor/blob/master/docs/installation_guide.md
2.通过kubernetes直接部署:
新版本已废弃
3.通过helm/kubernetes方式部署:
https://github.com/goharbor/harbor-helm/blob/master/README.md
推荐使用,本文将采用此方式
Helm安装
helm简介
Helm是一个用于Kubernetes的包管理工具,每一个安装包构成一个chart基本单位.想象一下,一个完整的部署,可能包含前端\后端\中间件\DB的部署实例,以及k8s管理对应的secret/configmap/svc/ing/pv/pvc等众多资源类型,在部署管理或者卸载之前,如果逐一编辑管理这些资源的yaml文件,工作量比较庞大且容易疏漏.helm的出现很好的解决了这个问题,使用模板以及变量替换的方式,按需生成部署yaml文件包,并且向kubernetes api发送调度请求,虽然额外地引入了一些学习成本,但对于部署包的管理无疑是带来了很大的便利性的.
helm组件
Helm
Helm是一个cli客户端，可以完成如下内容：
1.创建/打包/调试Chart
2.创建本地Chart仓库,管理本地和远程Chart仓库
3.与Tiller交互并完成Chart的安装，升级，删除，回滚，查看等操作
Tiller
1.监听helm客户端的请求,根据chart生成相应的release
2.将release的资源信息发送给kubernetes api,实现资源增删改查.
3.追踪release的状态,实现资源的更新/回滚操作
安装:
安装较为简单,参考官方文档,不做复述:
https://docs.helm.sh/using_helm/#installing-helm
Harbor安装
这里采用次新版本的v1.5.3版本
root@h1:# wget https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# tar -xf https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# cd harbor-1.5.3/contrib/helm/harbor/
helm方式安装的harbor默认是https的,因此需要k8s集群ingress网关开启https协议,traefik开启https参考此前文章:
k8s（二）、对外服务
更新helm dependency:
harbor的helm部署依赖了postgresql的helm，在官方的安装文档没有明确说明，直接按照官方文档说明安装，就会缺失postgresql的部署，导致helm install failed
helm dependency update
修改value.yaml文件,添加pvc持久存储并将相应组件上下文中的volume部分注释打开:
mysql:
volumes:
data:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
adminserver:
volumes:
config:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 1G
registry:
volumes:
data:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 5G
完整value.yaml文件如下:
# Configure persisten Volumes per application
## Applications that require storage have a `volumes` definition which will be used
## when `persistence.enabled` is set to true.
## example
# mysql:
# volumes:
# data:
## Persistent Volume Storage Class
## If defined, storageClassName:
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: "-"
# accessMode: ReadWriteOnce
# size: 1Gi
mysql:
volumes:
data:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
adminserver:
volumes:
config:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 1G
registry:
volumes:
data:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 5G
## Configure resource requests and limits per application
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# mysql:
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
persistence:
enabled: true
# The tag for Harbor docker images.
harborImageTag: &harbor_image_tag v1.4.0
# The FQDN for Harbor service.
externalDomain: harbor.my.domain
# If set to true, you don't need to set tlsCrt/tlsKey/caCrt, but must add
# Harbor FQDN as insecure-registries for your docker client.
insecureRegistry: false
#insecureRegistry: true
# The TLS certificate for Harbor. The common name of tlsCrt must match the externalDomain above.
tlsCrt:
#tlsCrt: root/traefik/ssl/kokoerp.crt
tlsKey:
#tlsKey: root/traefik/ssl/kokoerp.key
caCrt:
# The secret key used for encryption. Must be a string of 16 chars.
secretKey: not-a-secure-key
# These annotations allow the registry to work behind the nginx
# ingress controller.
ingress:
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/body-size: "0"
ingress.kubernetes.io/proxy-body-size: "0"
adminserver:
image:
repository: vmware/harbor-adminserver
tag: *harbor_image_tag
pullPolicy: IfNotPresent
emailHost: "smtp.mydomain.com"
emailPort: "25"
emailUser: "[email protected]"
emailSsl: "false"
emailFrom: "admin "
emailIdentity: ""
emailInsecure: "False"
emailPwd: not-a-secure-password
adminPassword: Harbor12345
authenticationMode: "db_auth"
selfRegistration: "on"
ldap:
url: "ldaps://ldapserver"
searchDN: ""
baseDN: ""
filter: "(objectClass=person)"
uid: "uid"
scope: "2"
timeout: "5"
verifyCert: "True"
## Persist data to a persistent volume
volumes:
config:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 1Gi
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
## jobservice
#
jobservice:
image:
repository: vmware/harbor-jobservice
tag: *harbor_image_tag
pullPolicy: IfNotPresent
secret: not-a-secure-secret
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
## UI
#
ui:
image:
repository: vmware/harbor-ui
tag: *harbor_image_tag
pullPolicy: IfNotPresent
secret: not-a-secure-secret
privateKeyPem: |
-----BEGIN RSA PRIVATE KEY-----
MIIJKAIBAAKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
/zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
AQKCAgBEUigO8/4UJse6xKr3APHv7E94NjKtjMqPT8RhDCLhqAH/lRuClTVb8k0Y
RILi6oHggsKGDvkS1vJEESCU5LfYBjDAX/r/M0I7gp6TU1AukAXKMdETvkfoMbg/
9j7W/G152hF4KztvjwmcHyUd7aay+SDh0n1taPm/FzaXfgONwmQFmo40uQ2SfwhX
I3tD6iMWjASLV4eRfe5w88WpJQ3r5IGYMNuKFF1RcV7MNL3xMHBAwl1kudmRWY4w
p6+83Gc0m+2AQbY70TkQuRbeUFkIBsWn99yEqXC+7h2us+JLm57iGN1ByQvVnEwL
Zs7Pl0Hge4leSxeZWhv+aE1R/jm/VdG4dglInuhED0ug8WAJg58IkDYfMKOOALHx
+0CNHE02XqqUIFwboZJSYTjMYvFL1i14L30FWnqH/0kDs4whXHbnGWhVustsMSK9
iyIGepuGhMnvtUF1wa/SrBd12qfDj68QHDXsKKbs6eTNYHfn3QL9uisrfMIa5HAt
nX2YOsAVxg+yvxkWD6n1DU+a/+pAu6iAgiwyxSZiyn6vJUE2zO6pJNbk1kJW6jU3
A69srtbO4jQn4EM859XYSqdqwXgJL+XJEYNbBcHalmiIOvRg9CCvDSKS7M5rJ0M1
L7oCzl6EW+zUb4JHkSO7V5uxIZu2sEduw5gofQ3OT9L/qDhDIQKCAQEA8T/8okF2
Q7SOj3su6KKX6H/ab31SvHECf/oeJtH8ZfLBYL55Yof0pZwq8iXQ26d8cH7FPKBo
hz0RZ9i2S3bYkzEVCPv9ISFg1NACxL3dU0PMBnmbmg2vPhMzEuQI2JOUu6ILOXEN
mImvfjZXps/b8OjQgzicH0skBBcbUlXT3a4fF52ktC8FiXgBG9JYg5CsXmfPRxci
ITa4w4ZLEuECmtJieS0MdKXPLwUVv3e2BlNS6c1JzXyp6EyX/euJ8cCe3n/GHbTY
2j1OO+xTQfQJVf6S9f2mSzjdHe9KZwWKgyxQ9dZ9Qtho2z/gUN9/UkL52fdljjlw
++b/z9Ppcl9K0QKCAQEA7y4Fv8dPFLLnr0R/S7eoAKa0S95xVe97EJHVUhWyOI09
K9VdZHp6be8W0Yd9h/Ks8Zi4EPRiTTaF3yA3iADwdKFeZt49jGzeM+Gl7Q2Ll98W
I5gOdJkHSVAP2uK7qSjZ8lPCu4iUYRsae+Psam7Yd6X17RP0M966PlUFj1nnrJjQ
EN4zeh/m01q9vqebB9C1W/ZiJ6rpt6VVHAcOQQ69F/lKdTif4XCvbMIhIXTYNifk
1oIv2qTDnfzzv+bgrlvpBJPpPYR0Oc7WoEpyd1Y9IzienLZi8RnujV//FXEmJ45E
F9GE1HOmoERdEWA1bMYhOO5OfRY1HSMuFMA4+5ojSwKCAQEAmwubio/1uMemw3HQ
kPRGGsdolDR/4tniWGtfy2UzCDY+r7Vaf8eOpIy8UQmatEBsykO+8RrKcvf9Yrc1
WUSVJevqb+67HPq9p6fTz6uSPXwZ+KNZLGXVFVjzfxWM1dvrP7eB7TXKHhmG7t9v
76Yw3SBTObI9LCN3jyVmisDcO+E23E+VVbPOpC260K2b81ocXUPsQ+0LIztu/UIm
p4hyyxug6+3WznTttXNYKch+9IvCgr5Ly0NuUvw+xpMFAZjgwXBu3BKpN4Ek8YAN
dhqnkVveCTguErQF78IlGBbIkUr+8TAbKsW4hggEWxV4V17yAnJsEz65bTtldqTj
qHyzsQKCAQBGhv6g/2d9Rgf1cbBLpns+vel6Wbx3x6c1SptpmgY0kMlR7JeeclM5
qX/EBzzn4pJGp27XaQi3lfVBxyE41HYTHiZVFQF3L/8Rs18XGKBqBxljI4pXrWwt
nRMfyy3lAqvJvhM082A1hiV4FMx40fi4x1JON00SIoIusSlzjOI4zdLEtpDdWRza
g+5hktCvLEbeODfXVJmYUoNXQWldm7f8osDm8eyLMIw5+MCGOgsrZPYgnsD3qxAX
vSgvFSh5oZaDiA4F2tHe3fQBzhIUyHQ8t4xlz447ZBcozv7L1tKWZWgE0f5mGzgu
GBqNbh4y1fWj8Plp/ytoTSBgdBIZdukjAoIBAELJPSVFnlf/gv6OWRCHyKxquGjv
fEn/E8bw5WSqMcj/7wiSJozr0Y8oyWjtWXObliLRQXcEhC8w3lLMjNqnFzQOAI7s
Oa6BQPigqyXZPXG5GK+V0TlUYvZQn9sfCq4YCxUBNtQ4GHbKKl3FGQL3rJiuFr6G
fVcetuDFNCiIGYbUF+giJ2cEN3a/Q+7fR6V4xC7VDdL+BqM09wZ6R98G48XzCKKp
ekNpEfmvJiuk9tFFQwDPWcQ6uyHqesK/Wiweo5nh5y2ZPipwcb0uBoYOQH60NqEL
6MXRVNdtKujjl1XZkG053Nvcz/YfF6lFjDekwgfd9m49b/s0EGTrl7z9z8Y=
-----END RSA PRIVATE KEY-----
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
## MySQL Settings. Currently Harbor does not support an external
## MySQL server, only their own image. Until this is fixed, do not
## Change the settings below.
#
mysql:
image:
repository: vmware/harbor-db
tag: *harbor_image_tag
pullPolicy: IfNotPresent
# If left blank will use the included mysql service name.
host: ~
port: 3306
user: "root"
pass: "registry"
database: "registry"
volumes:
data:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 1Gi
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
registry:
image:
repository: vmware/registry-photon
tag: v2.6.2-v1.4.0
pullPolicy: IfNotPresent
httpSecret: not-a-secure-secret
logLevel:
# comment out one of the below to use your cloud's object storage.
# objectStorage:
# gcs:
# keyfile: ""
# bucket: ""
# chunksize: "5242880"
# s3:
# region: ""
# accesskey: ""
# secretkey: ""
# bucket: ""
# encrypt: "true"
# azure:
# accountname: ""
# accountkey: ""
# container: ""
rootCrt: |
-----BEGIN CERTIFICATE-----
MIIE0zCCArugAwIBAgIJAIgs3S+hsjhmMA0GCSqGSIb3DQEBCwUAMAAwHhcNMTcx
MTA5MTcyNzQ5WhcNMjcxMTA3MTcyNzQ5WjAAMIICIjANBgkqhkiG9w0BAQEFAAOC
Ag8AMIICCgKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
/zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
AaNQME4wHQYDVR0OBBYEFCMYYMOL0E/Uyj5wseDfIl7o4ELsMB8GA1UdIwQYMBaA
FCMYYMOL0E/Uyj5wseDfIl7o4ELsMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL
BQADggIBABG8fPvrrR+erpwQFuB/56j2i6sO+qoOJPpAMYwkzICrT0eerWAavwoy
f0UAKN7cUeEJXjIR7s7CogGFijWdaWaQsXUD0zJq5aotLYZLimEc1O0uAmJEsfYC
v7mG07eU6ge22sSo5hxhVplGt52hnXnT0DdgSRZpq2mvgd9lcopAidM+KHlaasXk
IecHKM99KX9D8smr0AcQ6M/Ygbf2qjO9YRmpBIjyQWEake4y/4LWm+3+v08ecg4B
g+iMC0Rw1QcPqgwaGaWu71RtYhyTg7SnAknb5nBcHIbLb0hdLgQTa3ZdtXgqchIi
GuFlEBmHFZP6bLJORRUQ0ari5wpXIsYfrB4T8PybTzva3OCMlEsMjuysFr9ewhzM
9UGLiSQNDyKA10J8WwlzbeD0AAW944hW4Dbg6SWv4gAo51T+6AukRdup5y6lfQ5a
h4Lbo6pzaA369IsJBntvKvia6hUf/SghnbG7pCHX/AEilcgTb13HndF/G+7aZgKR
mi9qvNRSDsE/BrgZawovp81+j6aL4y6UtXYspHr+SuWsKYsaH7pl5HspNCyJ5vV6
dpJAwosFBqSEnI333wAunpMYmi/jKHH/j4WqjLnCInp0/wouzYu42l8Pmz591BSp
Jag500bEBxqI2RLELgMt/bUdjp4N2M7mrxdrN+2579HTzb6Hviu9
-----END CERTIFICATE-----
## Persist data to a persistent volume
volumes:
data:
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 5Gi
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
clair:
enabled: true
image:
repository: vmware/clair-photon
tag: v2.0.1-v1.4.0
pullPolicy: IfNotPresent
## The following needs to match the credentials
## in the `postgresql` configuration under the
## `postgresql` namespace below.
postgresPassword: not-a-secure-password
postgresUser: clair
postgresDatabase: clair
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
# pgResources:
# requests:
# memory: 256Mi
# cpu: 100m
# volumes:
# pgData:
# storageClass: "cephrbd"
# accessMode: ReadWriteOnce
# size: 1Gi
# resources:
# requests:
# memory: 256Mi
# cpu: 100m
## Notary support is not yet fully implemented in the Helm Charts
## Enabling it will just break things.
#
notary:
enabled: false
## Settings for postgresql dependency.
## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
## for further configurables.
postgresql:
postgresUser: clair
postgresPassword: not-a-secure-password
postgresDatabase: clair
persistence:
# enabled: false
enabled: true
storageClass: "cephrbd"
accessMode: ReadWriteOnce
size: 8Gi
部署:
helm install . --debug --name hub --set externalDomain=hub.test.com
等待一段时间的镜像拉取后,查看pod:
hub-harbor-adminserver-0 1/1 Running 1 3h
hub-harbor-clair-6c7d9dcdb7-q4lv4 1/1 Running 2 3h
hub-harbor-jobservice-75f7fbcc9c-ggwp4 1/1 Running 2 3h
hub-harbor-mysql-0 1/1 Running 0 3h
hub-harbor-registry-0 1/1 Running 1 3h
hub-harbor-ui-57b4674ff9-kcfq6 1/1 Running 0 3h
hub-postgresql-ccf8d56d5-jg4wq 1/1 Running 1 3h
添加dns,指向ingress gateway(即traefik的node ip):
echo "192.168.1.238 hub.test.com" >> /etc/hosts
浏览器打开测试,默认登录口令为admin/Harbor12345
可以看到默认有一个公开项目library,尝试往这里推送镜像
Docker推送测试
找一台安装了docker的机器,修改docker服务脚本:
~# cat /lib/systemd/system/docker.service
#ExecStart=/usr/bin/dockerd -H fd://
ExecStart=/usr/bin/dockerd --insecure-registry hub.kokoerp.com
重启docker并登录测试:
~# systemctl daemon-reload
~# systemctl restart docker
root@h2:~# docker login hub.test.com
Username (admin): admin
Password:
Login Succeeded
推送镜像测试:
root@h2:~# docker tag busybox hub.test.com/library/busybox:test1
root@h2:~# docker push hub.test.com/library/busybox:test1
The push refers to a repository [hub.test.com/library/busybox]
8a788232037e: Pushed
test1: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527
可以看到推送成功
ldap配置:
harbor可以基于企业内部的ldap来获取人员信息,可以在ui上创建非公开项目,将ldap中获取的人员加入项目并赋予相应pull push权限,ui操作界面比较友好,这里就不展示了.