通过helm在k8s上搭建Harbor

环境：
kuberneters版本：v1.22.3
helm版本：v3.7.1
helm chart版本：1.8.0

下文所需yaml文件在DeploymentFiles可下载

Harbor 是一个开源注册表，它通过策略和基于角色的访问控制来保护工件，确保镜像被扫描且没有漏洞，并将镜像签名为受信任的。

前期准备

1、安装helm

官网地址：【https://helm.sh/zh/docs/】
helm是k8s的包管理器，是查找、分享和使用软件构建k8s的最优方式。
charts代表着helm包，它包含在k8s集群内部运行应用程序，工具或服务所需的所有资源定义；
repository是用来存放和共享charts的地方；
release是运行在k8s集群中的chart的实例。

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

2、创建namespace

kubectl create namespace harbor

3、挂载NFS与创建目录

nfs服务的部署在另一篇文章，在此不赘述 （https://www.jianshu.com/p/2c20efbd5855）
①挂载nfs

$sudo vim /etc/exports
#增加以下内容
/hdd/nfs *(rw,sync,no_root_squash,no_subtree_check)

②在/hdd/nfs下创建所需要的目录

sudo mkdir -p /hdd/nfs/harbor/registry
sudo mkdir -p /hdd/nfs/harbor/chartmuseum
sudo mkdir -p /hdd/nfs/harbor/jobservice
sudo mkdir -p /hdd/nfs/harbor/database
sudo mkdir -p /hdd/nfs/harbor/redis
sudo mkdir -p /hdd/nfs/harbor/trivy

③修改文件目录权限
文件权限很重要，在这踩了很大的坑，Redis和database一直报权限不足
-R 代表harbor下的所有文件夹

sudo chmod -R 777 /hdd/nfs/harbor

如果以上权限还不够的话，将文件属主改为你当前用户

sudo chown -R 1000:1000 /hdd/nfs/

4、创建PV和PVC

①创建PV部署文件harbor-pv.yaml
spec.nfs.path和spec.nfs.server根据自己实际路径和IP填写；
spec.storageClassName与PVC中的storageClassName保持一致。
spec.capacity.storage可根据实际情况调整，PVC<=PV。

#registry-PV
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-registry
  labels:
    app: harbor-registry
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "harbor"
  mountOptions:
    - hard
  nfs:
    path: /hdd/nfs/harbor/registry
    server: 192.168.100.24
---
#harbor-chartmuseum-pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-chartmuseum
  labels:
    app: harbor-chartmuseum
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "harbor"
  mountOptions:
    - hard
  nfs:
    path: /hdd/nfs/harbor/chartmuseum
    server: 192.168.100.24
---
#harbor-jobservice-pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-jobservice
  labels:
    app: harbor-jobservice
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "harbor"
  mountOptions:
    - hard
  nfs:
    path: /hdd/nfs/harbor/jobservice
    server: 192.168.100.24
---
#harbor-database-pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-database
  labels:
    app: harbor-database
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "harbor"
  mountOptions:
    - hard
  nfs:
    path: /hdd/nfs/harbor/database
    server: 192.168.100.24
---
#harbor-redis-pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-redis
  labels:
    app: harbor-redis
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "harbor"
  mountOptions:
    - hard
  nfs:
    path: /hdd/nfs/harbor/redis
    server: 192.168.100.24
---
#harbor-trivy-pv
apiVersion: v1
kind: PersistentVolume
metadata:
  name: harbor-trivy
  labels:
    app: harbor-trivy
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "harbor"
  mountOptions:
    - hard
  nfs:
    path: /hdd/nfs/harbor/trivy
    server: 192.168.100.24

创建PV资源
-f 指定资源配置文件
PV相对集群而言，所以不需要指定命名空间

kubectl apply -f /etc/kubernetes/harbor/harbor-pv.yaml

②创建PVC部署文件harbor-pvc.yaml

#harbor-registry-pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-registry
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "harbor"
  resources:
    requests:
      storage: 20Gi
  selector:
    matchLabels:
      app: harbor-registry
---
#harbor-chartmuseum-pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-chartmuseum
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "harbor"
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: harbor-chartmuseum
---
#harbor-jobservice-pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-jobservice
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "harbor"
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: harbor-jobservice 
---
#harbor-database-pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-database
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "harbor"
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: harbor-database  
---
#harbor-redis-pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-redis
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "harbor"
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: harbor-redis
---
#harbor-trivy-pvc
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: harbor-trivy
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "harbor"
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      app: harbor-trivy

创建PVC资源
-n 指定命名空间

kubectl apply -f /etc/kubernetes/harbor/harbor-pvc.yaml -n harbor

创建自定义证书

默认情况下，harbor不附带证书。可以在没有安全性的情况下部署，通过HTTP连接。要配置HTTPS必须创建SSL证书。
创建/home/master/harbor_crt文件夹，cd进入harbor_crt文件夹内操作（可选，个人为了统一好管理）
①生成证书文件

## 获得证书
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 3650 -out ca.crt  -subj  "/C=CN/ST=Guangdong/L=Guangzhou/O=example/OU=example/CN=192.168.100.51"

## 生成证书签名请求
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout tls.key -out tls.csr  -subj  "/C=CN/ST=Guangdong/L=Guangzhou/O=example/OU=example/CN=192.168.100.51"

通过IP连接时，CN貌似是不生效的，会被忽略，因此需要创建一个配置文件来指定IP地址：

$vim extfile.cnf
#填入以下内容
subjectAltName = IP:192.168.100.51
## 生成证书
$ openssl x509 -req -days 3650 -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial  -extfile extfile.cnf -out tls.crt

②生成secret资源
创建 Kubernetes 的 Secret 资源，且将证书文件导入：

kubectl create secret generic harbor-tls --from-file=tls.crt --from-file=tls.key --from-file=ca.crt -n harbor

设置harbor配置清单

①从官网【https://github.com/goharbor/harbor-helm】下载v1.7.4Latest版本的values.yaml文件
②修改配置文件
我采用的是nodePort方式，修改expose.type为nodePort，按照别的方式的修改相应type即可。
externalURL，选择你任意可用的节点IP:port（注意协议与端口号匹配）；尽量别去修改默认密码，我第一次的时候是改成了别的密码，因为各种坑删除多次release然后pgdata没删干净，默认密码一直登不上去。
内容太多注释部分被我删除了，仔细对照下

expose:
  type: nodePort
  tls:
    enabled: true
    certSource: secret
    auto:
      commonName: ""
    secret:
      secretName: "harbor-tls"
      notarySecretName: "harbor-tls"
.（不变）
.
.
  nodePort:
    name: harbor
    ports:
      http:
        port: 80
        nodePort: 30002
      https:
        port: 443
        nodePort: 30003
      notary:
        port: 4443
        nodePort: 30004
  loadBalancer:
.（不变）
.
.
externalURL: https://192.168.100.51:30003

internalTLS:
. （不变）
.
.

persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      existingClaim: "harbor-registry"
      storageClass: "harbor"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 20Gi
    chartmuseum:
      existingClaim: "harbor-chartmuseum"
      storageClass: "harbor"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    jobservice:
      existingClaim: "harbor-jobservice"
      storageClass: "harbor"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    database:
      existingClaim: "harbor-database"
      storageClass: "harbor"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    redis:
      existingClaim: "harbor-redis"
      storageClass: "harbor"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    trivy:
      existingClaim: "harbor-trivy"
      storageClass: "harbor"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
.（不变）
.

安装harbor

①添加helm仓库

$ helm repo add harbor https://helm.goharbor.io

②部署harbor

helm install harbor harbor/harbor -f /etc/kubernetes/harbor/deployment_nodeport.yaml -n harbor

③查看是否部署完成

$ kubectl get deployment -n harbor

④访问harbor
浏览器输入地址（前面配置的externalURL）
默认用户：admin
默认密码：Harbor12345

服务器配置镜像仓库

在Ubuntu上通过docker login访问前面部署好的harbor时出错

图片.png

①因此要让docker信任我们的证书，为docker配置harbor证书
在/etc/docker目录下创建certs.d 文件夹，然后在 certs.d 文件夹下创建192.168.100.51:30003（IP:port）文件夹

$ mkdir -p /etc/docker/certs.d/192.168.100.51:30003

转换tls.crt为tls.cert，供docker使用，Docker 守护进程将.crt文件解释为 CA 证书，将.cert文件解释为客户端证书。

$cd harbor_tls/
$sudo openssl x509 -inform PEM -in tls.crt -out tls.cert

将前面创建了HTTPS的证书ca.crt、tls.cert、tls.key证书复制到192.168.100.51:30003文件夹内（每一台docker主机都需要）

$sudo cp harbor_tls/ca.crt /etc/docker/certs.d/192.168.100.51\:30003/
$sudo cp harbor_tls/tls.key /etc/docker/certs.d/192.168.100.51\:30003/
$sudo cp harbor_tls/tls.cert /etc/docker/certs.d/192.168.100.51\:30003/
#重启docker
$sudo systemctl daemon-reload
$sudo systemctl restart docker.service 

②让系统信任我们的根证书（可选）
update-ca-certificates命令将PEM格式的根证书内容附加到/etc/ssl/certs/ca-certificates.crt ，而/etc/ssl/certs/ca-certificates.crt 包含了系统自带的各种可信根证书.

$sudo cp harbor_tls/tls.crt /usr/local/share/ca-certificates
$sudo update-ca-certificates

再次访问harbor，成功登陆~快乐！

图片.png

参考文档：【http://www.mydlq.club/article/66/#documentTop】