一、环境介绍
默认情况下，Harbor 不附带证书。可以在没有安全保护的情况下部署 Harbor，以便您可以通过 HTTP 连接到它。在生产环境中，推荐始终使用 HTTPS。要配置 HTTPS，必须创建 SSL 证书。可以使用由受信任的第三方 CA 签名的证书，也可以使用自签名证书。本文以自签名证书为例。
使用到的各个软件版本
操作系统版本：ubuntu 20.04
harbor版本：v2.5.3-797c3536
docker版本：20.10.8
kubernetets版本：1.22.2harbor地址：192.168.10.112 域名：harbor.snow.com
master01地址：192.168.10.100 二、仓库部署
配置主机名与hosts文件
root@harbor:~# cat /etc/hosts
127.0.0.1 localhost
192.168.10.112 harbor.snow.com
修改主机名
root@barbor:~# hostnamectl set-hostname harbor
root@harbor:~# bash
root@harbor:~# hostname
harbor下载harbor安装包
root@harbor:~#wget https://github.com/goharbor/harbor/releases/download/v2.5.3/harbor-offline-installer-v2.5.3.tgz
解压harbor安装包
root@harbor:~# tar xf harbor-offline-installer-v2.5.3.tgz -C /usr/local/src/
安装docker-compose
root@harbor:~# curl -SL https://github.com/docker/compose/releases/download/v2.7.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose生成证书颁发机构证书及私钥
root@harbor:/usr/local/src/harbor/certs# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
............................++++
............................................++++
e is 65537 (0x010001)
root@harbor:/usr/local/src/harbor/certs# openssl req -x509 -new -nodes -sha512 -days 3650 \
> -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.snow.com" \
> -key ca.key \
> -out ca.crt
root@harbor:/usr/local/src/harbor/certs# ls
ca.crt ca.key生成服务器私钥及证书签名请求（CSR）
root@harbor:/usr/local/src/harbor/certs# openssl genrsa -out harbor.snow.com.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
...........++++
.....................................................................++++
e is 65537 (0x010001)
生成证书签名请求
root@harbor:/usr/local/src/harbor/certs# openssl req -sha512 -new \
> -subj "/C=CN/ST=Shanghai/L=Shanghai/O=SmartX/OU=Lab/CN=harbor.snow.com" \
> -key harbor.snow.com.key \
> -out harbor.snow.com.csr
root@harbor:/usr/local/src/harbor/certs# ls
ca.crt ca.key harbor.snow.com.csr harbor.snow.com.key生成 x509 v3 扩展文件。
无论使用 FQDN 还是 IP 地址连接到Harbor服务器，都必须创建此文件，以便keyi 为的 Harbor 主机生成符合主题备用名称 (SAN) 和 x509 v3 的证书扩展要求。
root@harbor:/usr/local/src/harbor/certs# cat v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=harbor.snow.top
DNS.2=snow.top
DNS.3=harbor使用该v3.ext文件为 Harbor 服务器生成证书。
root@harbor:/usr/local/src/harbor/certs# openssl x509 -req -sha512 -days 3650 \
> -extfile v3.ext \
> -CA ca.crt -CAkey ca.key -CAcreateserial \
> -in harbor.snow.com.csr \
> -out harbor.snow.com.crt
Signature ok
subject=C = CN, ST = Shanghai, L = Shanghai, O = SmartX, OU = Lab, CN = harbor.snow.com
Getting CA Private Key将 harbor.snow.com.crt 转换为 harbor.snow.com.cert , 供 Docker 使用。Docker 守护进程将.crt文件解释为 CA 证书，.cert将文件解释为客户端证书。
root@harbor:/usr/local/src/harbor/certs# openssl x509 -inform PEM -in harbor.snow.com.crt -out harbor.snow.com.cert配置 harbor yml文件，注意需要修改hostname与指定证书位置。
root@harbor:/usr/local/src/harbor# cat harbor.yml |more
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.snow.com
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /usr/local/src/harbor/certs/harbor.snow.com.cert
private_key: /usr/local/src/harbor/certs/harbor.snow.com.key
安装harbor
root@harbor:/usr/local/src/harbor# ./install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 20.10.12
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.25.0
[Step 2]: loading Harbor images ...
Loaded image: goharbor/harbor-portal:v2.5.3
Loaded image: goharbor/harbor-core:v2.5.3
Loaded image: goharbor/redis-photon:v2.5.3
Loaded image: goharbor/prepare:v2.5.3
Loaded image: goharbor/harbor-db:v2.5.3
Loaded image: goharbor/chartmuseum-photon:v2.5.3
Loaded image: goharbor/harbor-jobservice:v2.5.3
Loaded image: goharbor/harbor-registryctl:v2.5.3
Loaded image: goharbor/nginx-photon:v2.5.3
Loaded image: goharbor/notary-signer-photon:v2.5.3
Loaded image: goharbor/harbor-log:v2.5.3
Loaded image: goharbor/harbor-exporter:v2.5.3
Loaded image: goharbor/registry-photon:v2.5.3
Loaded image: goharbor/notary-server-photon:v2.5.3
Loaded image: goharbor/trivy-adapter-photon:v2.5.3
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /usr/local/src/harbor
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/db/env
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating registryctl ... done
Creating registry ... done
Creating redis ... done
Creating harbor-db ... done
Creating harbor-core ... done
Creating nginx ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----
安装完harbor后验证
root@harbor:/usr/local/src/harbor# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0304ebf0cddc goharbor/nginx-photon:v2.5.3 "nginx -g 'daemon of…" 38 seconds ago Up 35 seconds (healthy) 0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp nginx
10d1ff19120b goharbor/harbor-jobservice:v2.5.3 "/harbor/entrypoint.…" 38 seconds ago Up 36 seconds (healthy) harbor-jobservice
2d5e3d8877f8 goharbor/harbor-core:v2.5.3 "/harbor/entrypoint.…" 39 seconds ago Up 38 seconds (healthy) harbor-core
ee6c12da89d9 goharbor/harbor-db:v2.5.3 "/docker-entrypoint.…" 43 seconds ago Up 39 seconds (healthy) harbor-db
5b7455e56733 goharbor/redis-photon:v2.5.3 "redis-server /etc/r…" 43 seconds ago Up 39 seconds (healthy) redis
a0c279ca0206 goharbor/harbor-registryctl:v2.5.3 "/home/harbor/start.…" 43 seconds ago Up 41 seconds (healthy) registryctl
478f35433f69 goharbor/registry-photon:v2.5.3 "/home/harbor/entryp…" 43 seconds ago Up 39 seconds (healthy) registry
eb33121f4126 goharbor/harbor-portal:v2.5.3 "nginx -g 'daemon of…" 43 seconds ago Up 39 seconds (healthy) harbor-portal
b0c7ab3a7133 goharbor/harbor-log:v2.5.3 "/bin/sh -c /usr/loc…" 44 seconds ago Up 43 seconds (healthy) 127.0.0.1:1514->10514/tcp harbor-log修改宿主机hosts文件，做好harbor主机的域名解析，在浏览器使用域名访问harbor服务器发现提示（忽略域名，域名是第一次测试的时候用的）
在当前页面用键盘输入thisisunsafe，不是在地址栏输入，就直接敲键盘就行了，页面即会自动刷新进入网页
使用默认用户名admin，密码Harbor12345即可登录。
三、配置docker客户端
将证书发送到master01服务器,注意在操作之前要先在master01创建好存放证书的文件夹/etc/docker/certs.d/harbor.snow.com
root@master01:~# mkdir -p /etc/docker/certs.d/harbor.snow.comroot@harbor:/usr/local/src/harbor/certs# scp harbor.snow.com.cert 192.168.10.100:/etc/docker/certs.d/harbor.snow.com/
[email protected]'s password:
harbor.snow.com.cert 100% 2102 2.3MB/s 00:00
root@harbor:/usr/local/src/harbor/certs# scp harbor.snow.com.key 192.168.10.100:/etc/docker/certs.d/harbor.snow.com/
[email protected]'s password:
harbor.snow.com.key 100% 2045 2.0MB/s 00:00
root@harbor:/usr/local/src/harbor/certs# scp ca.crt 192.168.10.100:/etc/docker/certs.d/harbor.snow.com
[email protected]'s password:
ca.crt 100% 2045 2.0MB/s 00:00
在master01节点看到证书已经成功发送过来
root@master01:/etc/docker/certs.d/harbor.snow.com# ls
ca.crt harbor.snow.com.cert harbor.snow.com.key修改docker的daemon.json文件，使其信任harbor.snow.com仓库
root@master01:/etc/docker# cat daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"insecure-registries":["harbor.snow.com"],
"registry-mirrors": [
"https://docker.mirrors.ustc.edu.cn",
"http://hub-mirror.c.163.com"
],
"max-concurrent-downloads": 10,
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"data-root": "/var/lib/docker"
}重启docker服务并修改其hosts文件使master01节点能够通过域名找到对应的harbor仓库
root@master01:~# systemctl restart docker
root@master01:~# cat /etc/hosts
127.0.0.1 localhost
192.168.10.112 harbor.snow.com登录仓库
root@master01:~# docker login harbor.snow.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded上传镜像进行测试，注意要提前在web页面创建好新的项目。
root@master01:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
calico/cni v3.19.2 05bf027c9836 17 months ago 146MB
为镜像改名字
root@master01:~# docker tag calico/cni:v3.19.2 harbor.snow.com/calico/cni:v3.19.2
推送到镜像仓库
root@master01:~# docker push harbor.snow.com/calico/cni:v3.19.2
The push refers to repository [harbor.snow.com/calico/cni]
758850cabe72: Pushed
e88fd392bb0c: Pushed
fb44224b4823: Pushed
v3.19.2: digest: sha256:00a619424a20d1b31c0f65d1194484aebe9f90dfc1d4d6396ce706941004860b size: 946
推送到镜像仓库后即可在web页面看到刚刚推送上去镜像
镜像拉取测试
root@master01:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kubernetesui/dashboard v2.3.1 e1482a24335a 18 months ago 220MB
root@master01:~# docker pull harbor.snow.com/calico/cni:v3.19.2
v3.19.2: Pulling from calico/cni
Digest: sha256:00a619424a20d1b31c0f65d1194484aebe9f90dfc1d4d6396ce706941004860b
Status: Downloaded newer image for harbor.snow.com/calico/cni:v3.19.2
harbor.snow.com/calico/cni:v3.19.2
root@master01:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kubernetesui/dashboard v2.3.1 e1482a24335a 18 months ago 220MB
harbor.snow.com/calico/cni v3.19.2 05bf027c9836 17 months ago 146MB
四、K8S集群配置使用私有镜像仓库Harbor
在配置k8s使用harbor之前需要先完成本文的第三步，先让docker可以正常使用harbor。
在docker成功登录harbor后会在用户家目录生成一个秘钥文件
root@master01:~# cat /root/.docker/config.json
{
"auths": {
"harbor.snow.com": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
}
}将秘钥进行base64加密
root@master01:~# cat /root/.docker/config.json | base64 -w 0
ewoJImF1dGhzIjogewoJCSJoYXJib3Iuc25vdy5jb20iOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=创建用户harbor认证的secret的yaml文件
root@master01:~/secret# cat registry-pull-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: registry-pull-secret
namespace: kube-system
data:
.dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoYXJib3Iuc25vdy5jb20iOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=
type: kubernetes.io/dockerconfigjson创建secret，注意secret为名称空间级别的资源，调用这个secret的yaml文件要与此secret在同一个名称空间下。
root@master01:~/secret# kubectl delete -f registry-pull-secret.yaml
secret "registry-pull-secret" deleted
root@master01:~/secret# kubectl apply -f registry-pull-secret.yaml
secret/registry-pull-secret created
root@master01:~/secret# kubectl get secret
NAME TYPE DATA AGE
default-token-rxd89 kubernetes.io/service-account-token 3 5h2m
registry-pull-secret kubernetes.io/dockerconfigjson 1 3h最后只需要在需要去harbor拉取镜像的pod的yaml文件中添加如下配置即可（只展示出了完成yaml文件的部分内容）注意其是与containers语句是平级的。
template:
metadata:
labels:
k8s-app: metrics-server
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
image: harbor.snow.com/base/metrics-server:v0.5.0
imagePullPolicy: Always配置完成后k8s集群即可成功拉取harbor中的镜像。如果在harbor中将要访问的镜像级别设置为公开，则不需要secret认证即可正常拉取。
ps：以上内容在本人实现环境中已试验成功，如发现有问题或表述不清的地方欢迎指正。