需求背景
基于TLS协议的harbor部署完成之后，我们可以正常通过https访问harbor并pull、push镜像一系列正常功能操作。但是我们需要发布到外网访问，对于很多机器没有外网IP来讲，我们需要一个全局的负载均衡通过虚拟主机形式暴露到外网访问。此文实现基于Traefik的方式通过TLS(letsencrypt)反向代理到后端Harbor(nginx TLS)。
架构图
整个操作流程看起来如下所示:
nginx (ssl,443) -> harbor(nginx,443,self-signed certs) -> harbor
因为我们ingress使用的Traefik，所以这里将Nginx替换，因为后端harbor-nginx我们使用的自签名证书，会校验ca签名证书，这里为了减少TLS的建立连接时间，所以在nginx或者traefik配置禁用对后端的证书检查即可。
Traefik
InsecureSkipVerify = true
Nginx
location /upstream {
....
proxy_ssl_verify off;
}
Harbor 自签名脚本
#!/bin/env bash
# 1. Create your own CA certificate
openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout ca.key \
-x509 -days 365 -out ca.crt
# 2. Generate a Certificate Signing Request:
# If you use FQDN like reg.yourdomain.com to connect your registry host, then you must use reg.yourdomain.com as CN (Common Name). Othe
rwise, if you use IP address to connect your registry host, CN can be anything like your name and so on:
openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout yourdomain.com.key \
-out yourdomain.com.csr
# 3.Generate the certificate of your registry host
# If you're using FQDN like reg.yourdomain.com to connect your registry host, then run this command to generate the certificate of your
registry host:
openssl x509 -req -days 3650 -in yourdomain.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out yourdomain.com.
crt
将生成的秘钥和证书移动至存放证书文件中，比如/data/apps/harbor/data/cert/
cp yourdomain.com.crt /data/apps/harbor/data/cert/
cp yourdomain.com.key /data/apps/harbor/data/cert/
接下来，编辑文件harbor.cfg，更新主机名和协议，并更新属性ssl_cert和ssl_cert_key：
#set hostname
hostname = yourdomain.com
#set ui_url_protocol
ui_url_protocol = https
......
#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert = /data/apps/harbor/data/cert/yourdomain.com.crt
ssl_cert_key = /data/apps/harbor/data/cert/yourdomain.com.key
如果是初次安装harbor，我们只需要执行如下脚本安装即可
./install.sh
如果已经安装harbor，我们只需要重新生成配置文件即可:
./prepare
如果Harbor已经运行我们停止并删除现有的容器。容器数据保留在文件系统中，所以不必担心数据会丢失
docker-compose down
然后我们启动harbor
docker-compose up -d
至此我们自签名完成了，Troubleshooting参考
配置Traefik
由于我们是基于Kubernetes Ingress对外提供服务，Harbor也没有部署在Kubernetes中，我们需要对Service和endpoint做一个关系映射绑定。
# vim harbor-ingress.yaml
---
kind: Service
apiVersion: v1
metadata:
name: harbor
namespace: kube-system
spec:
ports:
- protocol: TCP
port: 80
targetPort: 80
name: http
- protocol: TCP
port: 443
targetPort: 443
name: https
---
kind: Endpoints
apiVersion: v1
metadata:
name: harbor
namespace: kube-system
subsets:
- addresses:
# harbor机器IP
- ip: 192.168.100.2
ports:
- port: 80
name: http
- port: 443
name: https
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: harbor
namespace: kube-system
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/redirect-entry-point: https
# 这里不需要透传，设置之后traefik会返回404，有点问题
#traefik.ingress.kubernetes.io/pass-tls-cert: "true"
spec:
rules:
- host: yourdomain.com
http:
paths:
- backend:
serviceName: harbor
# proxy 到harbor nginx(tls)
servicePort: 443
如果基于Kuberntes部署的harbor不需要做endpoint绑定，直接通过Ingres proxy过去就行。
由于Traefik原生支持acme协议可以向letsencrypt服务器自动申请、续约证书请求，这里配置Traefik letsencrypt，以及配置禁用对后端的证书检查，kubernetes traefik ingress没有找到相关的annotation，只能对全局进行配置。
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-conf
namespace: kube-system
data:
traefik.toml: |
# traefik.toml
defaultEntryPoints = ["http","https"]
insecureSkipVerify = true
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[acme]
email = "[email protected]"
storageFile = "/acme/acme.json"
entryPoint = "https"
onDemand = true
onHostRule = true
[acme.httpChallenge]
entryPoint = "http"
[[acme.domains]]
main = "*.yourdomain.com"
因为letsencrypt泛域名支持需要DNS nameserver的认证并且只支持1级统配，所以这里为每个子域名申请一个私钥和证书，并且将私钥和证书保存至/acme/acme.json文件中，此文件我们通过volume的形式挂载出去。
如下是traefik以daemonset的形式部署，参考配置如下:
# vim traefik-ds.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: traefik-ingress-controller
namespace: kube-system
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: traefik-ingress-controller
namespace: kube-system
labels:
k8s-app: traefik-ingress-lb
spec:
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
nodeSelector:
edgenode: "true"
hostNetwork: true
containers:
- image: traefik
name: traefik-ingress-lb
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
- name: admin
containerPort: 8080
hostPort: 8080
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
args:
- --configfile=/config/traefik.toml
- --api
- --kubernetes
- --logLevel=INFO
volumeMounts:
- mountPath: "/config"
name: "config"
- mountPath: "/acme"
name: "acme"
volumes:
- name: config
configMap:
name: traefik-conf
- name: acme
hostPath:
path: /srv/configs/acme
---
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: http
- protocol: TCP
port: 443
name: https
- protocol: TCP
port: 8080
name: admin
type: NodePort
如上配置只是作为参考，生产环境我们可以基于etcd来存储我们的acme私钥证书，方便横向扩展我们的traefik。此时我们基于Traefik TLS 反向代理到 harbor(nginx，TLS)就完成了，docker login和docker pull,docker push测试没问题就完成。
参考文档:
github issue:
https://github.com/vmware/harbor/issues/3114
harbor自签名配置以及troubleshooting
https://github.com/vmware/harbor/blob/master/docs/configure_https.md#troubleshooting