要求:

要求
Kubernetes cluster 1.16+
Helm 2.10.0+

部署k8s

安装docker

$ cat /etc/docker/daemon.json

{
  "registry-mirrors": ["https://2a2urxbq.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries": ["10.20.8.133"],  #这里要为本地得地址,否则harbor部署完成后,docker login 会报错
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

安装k8s

部署horbor

安装helm

下载

官方链接: https://github.com/helm/helm/releases
$ wget https://get.helm.sh/helm-v3.4.0-linux-amd64.tar.gz

$ tar fx helm-v3.4.0-linux-amd64.tar.gz

$ cp linux-amd64/helm /usr/local/bin

$ helm version
version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

安装harbor

创建namespace

$ kubectl create namespace harbor

安装ingress controller

注意版本

$ kubectl apply -f https://kuboard.cn/install-script/v1.16.2/nginx-ingress.yaml

创建pv

生产环境使用ceph 的sc

# 试验使用本地目录文件
# 编辑yml文件,创建pv
$ kubectl apply -f .  -n harbor

下载源码

$ wget https://github.com/goharbor/harbor-helm/archive/v1.5.0.tar.gz
$ tar fx v1.5.0.tar.gz
$ cd harbor-helm-1.5.0/

修改ingress

默认的harbor ingress 证书为1年时间,这里可以使用secret的方式进行更改,参考附录1 创建证书和创建secret部分，使用方法1即可

$ vim values.yaml

expose:
  type: ingress
  tls:
    enabled: true
    certSource: secret    #修改为secret
    auto:
      commonName: ""
    secret:
      secretName: "harbor-self-tls"   #指定secret的名字
      notarySecretName: ""
   ingress:
    hosts:
      core: core.harbor.domain
      notary: notary.harbor.domain
    controller: default
    annotations:
      ingress.kubernetes.io/ssl-redirect: "true"
      ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-body-size: "0"

修改 StorageClass为ceph的sc

前提ceph已部署,ceph的sc已经创建，修改persistence配置块的storageClass为创建好的ceph sc

$ vim values.yaml
persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      existingClaim: ""
      storageClass: "rook-ceph-block" 
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    chartmuseum:
      existingClaim: ""
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
    jobservice:
      existingClaim: ""
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    database:
      existingClaim: ""
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    redis:
      existingClaim: ""
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
    trivy:
      existingClaim: ""
      storageClass: "rook-ceph-block"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi

安装

$ helm install harbor --debug --namespace harbor .

安装过程中harbor-harbor-database-0 一直无法Running的话，查看容器对应的volume,将dump.rdb删掉即可

卸载

$ helm uninstall harbor --debug --namespace harbor

查看pod

$ kubectl get pods -n harbor
NAME                                           READY   STATUS    RESTARTS   AGE
harbor-harbor-chartmuseum-75fcf4bccc-4dhlh     1/1     Running   0          88m
harbor-harbor-clair-d8f59f74-qzkrw             2/2     Running   9          88m
harbor-harbor-core-7fcbd6d86c-mqqwr            1/1     Running   8          88m
harbor-harbor-database-0                       1/1     Running   0          68m
harbor-harbor-jobservice-55f8b5f8bd-k8s9k      1/1     Running   0          62m
harbor-harbor-notary-server-597c779966-zhvz2   1/1     Running   8          88m
harbor-harbor-notary-signer-6c9fdc8655-qj9ss   1/1     Running   8          88m
harbor-harbor-portal-8456c5d77-j6ftj           1/1     Running   0          88m
harbor-harbor-redis-0                          1/1     Running   0          88m
harbor-harbor-registry-8456589f68-f5r2p        2/2     Running   0          88m
harbor-harbor-trivy-0                          1/1     Running   0          88m

查看service

$ kubectl get svc -n harbor
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
harbor-harbor-chartmuseum     ClusterIP   10.110.135.58            80/TCP              92m
harbor-harbor-clair           ClusterIP   10.103.49.219            8080/TCP            92m
harbor-harbor-core            ClusterIP   10.103.229.60            80/TCP              92m
harbor-harbor-database        ClusterIP   10.110.126.185           5432/TCP            92m
harbor-harbor-jobservice      ClusterIP   10.105.41.118            80/TCP              92m
harbor-harbor-notary-server   ClusterIP   10.101.181.231           4443/TCP            92m
harbor-harbor-notary-signer   ClusterIP   10.100.122.224           7899/TCP            92m
harbor-harbor-portal          ClusterIP   10.108.65.32             80/TCP              92m
harbor-harbor-redis           ClusterIP   10.109.188.51            6379/TCP            92m
harbor-harbor-registry        ClusterIP   10.107.14.62             5000/TCP,8080/TCP   92m
harbor-harbor-trivy           ClusterIP   10.107.82.132            8080/TCP            92m

查看pv和pvc

$ kubectl get pv,pvc -n harbor | grep harbor
persistentvolume/disk1.yml   5Gi        RWO            Recycle          Bound    harbor/harbor-harbor-chartmuseum                                        94m
persistentvolume/disk2.yml   5Gi        RWO            Recycle          Bound    harbor/data-harbor-harbor-trivy-0                                       94m
persistentvolume/disk3.yml   1Gi        RWO            Recycle          Bound    harbor/harbor-harbor-jobservice                                         94m
persistentvolume/disk4.yml   1Gi        RWO            Recycle          Bound    harbor/data-harbor-harbor-redis-0                                       94m
persistentvolume/disk5.yml   1Gi        RWO            Recycle          Bound    harbor/database-data-harbor-harbor-database-0                           94m
persistentvolume/disk6.yml   5Gi        RWO            Recycle          Bound    harbor/harbor-harbor-registry                                           94m
persistentvolumeclaim/data-harbor-harbor-redis-0               Bound    disk4.yml   1Gi        RWO                           93m
persistentvolumeclaim/data-harbor-harbor-trivy-0               Bound    disk2.yml   5Gi        RWO                           93m
persistentvolumeclaim/database-data-harbor-harbor-database-0   Bound    disk5.yml   1Gi        RWO                           93m
persistentvolumeclaim/harbor-harbor-chartmuseum                Bound    disk1.yml   5Gi        RWO                           93m
persistentvolumeclaim/harbor-harbor-jobservice                 Bound    disk3.yml   1Gi        RWO                           93m
persistentvolumeclaim/harbor-harbor-registry                   Bound    disk6.yml   5Gi        RWO                           93m

查看harbor ingress

$ kubectl get ingress -n harbor

查看ingress 配置

$ kubectl edit ingress harbor-harbor-ingress -n harbor

访问harbor

配置/etc/hosts

$ vim /etc/hosts
10.20.8.150 core.harbor.domain

访问

https://core.harbor.domain

默认用户名/密码 admin/Harbor12345

在k8s中使用harbor

$ kubectl get secrets/harbor-harbor-ingress -n harbor -o jsonpath="{.data.ca\.crt}" | base64 --decode
-----BEGIN CERTIFICATE-----
MIIC9DCCAdygAwIBAgIQS7zjdWHqFMC7RsAd3jYT/DANBgkqhkiG9w0BAQsFADAU
MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMjAxMDI4MDc1MDAwWhcNMjExMDI4MDc1
MDAwWjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCuu51lc9wZ2n1cXjn36cyNsUgZmDBJbJ9GMBjNHXa7ctEZQJdo
cMhFr315qxteeeZicNgnmlf00ETz6q9mRSXvfezY82yBdsYz2ZeLLcHfLdl5yu7J
RCSwTOAvOGPUvZIkkCU4L/2WCsfZrHLLMzQTnrT2LVYyzI9QkZMW2biafjfPdAC3
XVY+FPd6jUYpLXb5pEKuRusFTJWSP4Lu6Jw1f+ZwgMhyRWVmDVZlBPtcX8s108Fb
R097n3bvZ4YujM4HYTZ8dEKdj5jwxbNWntMLNYPkK7lgxWuwZJ0tzuWGZQI0qQzg
s9DFnetUeUnbMZ0F65iKaQW1CEBO+UajPyQDAgMBAAGjQjBAMA4GA1UdDwEB/wQE
AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw
AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAK6ii5qshCaUednjcvOieHoCu1CsCIURK
D6Zq5RoCdvU0v70s9fK5XohYMjw5InwYEYSTB05H2kbePjk+D4BgRmmg7Svc6Dr0
rsRfFCkG+cQFQ6/lf5flerAGCHGIsxJ2MLSG5P5s81wOPMRtwSF2bhZIcvjchPZE
1ExCQFeRkqgEP1zy3++XW5BL77k8pR0uK2NI16VM6GzT4IR7lEfrlYgjGEPjWma+
ZxL5UprZofuxkfiFzTcEUHqvmwMZo2Iwros5ARPtJikbeHn7BbSMBX2U4HOqHih9
6661XWyZAhdCC2NlPYeF+1Bd7rRLWyH7MU5BDVodvX7WesXmmbkFsg==
-----END CERTIFICATE-----

创建证书

$ mkdir -pv /etc/docker/certs.d/core.harbor.domain/

$ cat < /etc/docker/certs.d/core.harbor.domain/ca.crt
-----BEGIN CERTIFICATE-----
MIIC9DCCAdygAwIBAgIQS7zjdWHqFMC7RsAd3jYT/DANBgkqhkiG9w0BAQsFADAU
MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMjAxMDI4MDc1MDAwWhcNMjExMDI4MDc1
MDAwWjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCuu51lc9wZ2n1cXjn36cyNsUgZmDBJbJ9GMBjNHXa7ctEZQJdo
cMhFr315qxteeeZicNgnmlf00ETz6q9mRSXvfezY82yBdsYz2ZeLLcHfLdl5yu7J
RCSwTOAvOGPUvZIkkCU4L/2WCsfZrHLLMzQTnrT2LVYyzI9QkZMW2biafjfPdAC3
XVY+FPd6jUYpLXb5pEKuRusFTJWSP4Lu6Jw1f+ZwgMhyRWVmDVZlBPtcX8s108Fb
R097n3bvZ4YujM4HYTZ8dEKdj5jwxbNWntMLNYPkK7lgxWuwZJ0tzuWGZQI0qQzg
s9DFnetUeUnbMZ0F65iKaQW1CEBO+UajPyQDAgMBAAGjQjBAMA4GA1UdDwEB/wQE
AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw
AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAK6ii5qshCaUednjcvOieHoCu1CsCIURK
D6Zq5RoCdvU0v70s9fK5XohYMjw5InwYEYSTB05H2kbePjk+D4BgRmmg7Svc6Dr0
rsRfFCkG+cQFQ6/lf5flerAGCHGIsxJ2MLSG5P5s81wOPMRtwSF2bhZIcvjchPZE
1ExCQFeRkqgEP1zy3++XW5BL77k8pR0uK2NI16VM6GzT4IR7lEfrlYgjGEPjWma+
ZxL5UprZofuxkfiFzTcEUHqvmwMZo2Iwros5ARPtJikbeHn7BbSMBX2U4HOqHih9
6661XWyZAhdCC2NlPYeF+1Bd7rRLWyH7MU5BDVodvX7WesXmmbkFsg==
-----END CERTIFICATE-----
EOF

重启docker然后使用docker login 登录

用户名/密码: admin/Harbor12345

$ docker login core.harbor.domian
Username: admin
Password: 
Login Succeeded

上传镜像

$ docker tag nginx core.harbor.domain/library/nginx
$ docker push core.harbor.domain/library/nginx

报错1：push镜像提示文件大

rror parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "\r\n413 Request Entity Too Large\r\n\r\n413 Request Entity Too Large\r\n
nginx/1.17.3\r\n\r\n\r\n"

参考链接(https://blog.pragtechnologies.com/file-upload-limit-in-kubernetes/)

解决办法: 修改ingress

$ kubectl edit ingress harbor-harbor-ingress -n harbor

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-body-size: "0"
    ingress.kubernetes.io/ssl-redirect: "true"
    meta.helm.sh/release-name: harbor
    meta.helm.sh/release-namespace: harbor
    nginx.ingress.kubernetes.io/client-max-body-size: 10240m    #增加该字段
    nginx.ingress.kubernetes.io/proxy-body-size: 50m   # 修改该值
    nginx.ingress.kubernetes.io/ssl-redirect: "true"


修改configmap
$ kubectl edit configmap nginx-config -n nginx-ingress

apiVersion: v1
data:
  body-size: 50m                             #增加该字段
  client-max-body-size: 10240m                  #增加该字段
  server-names-hash-bucket-size: "1024"

查看log

$ cd /var/log/pods
$ ls

harbor_harbor-harbor-chartmuseum-75fcf4bccc-4dhlh_a10728ea-ab87-404b-a6eb-7191c2aa01ce
harbor_harbor-harbor-clair-d8f59f74-qzkrw_516a1ece-0625-45d4-ba86-167c5d27d34f
harbor_harbor-harbor-core-7fcbd6d86c-mqqwr_130f54e3-3f99-4f3d-a468-079033903915
harbor_harbor-harbor-database-0_b90fda78-0fd3-4230-8c0d-a680f1d00010
harbor_harbor-harbor-jobservice-55f8b5f8bd-k8s9k_eae58e10-6118-4827-b75a-23a1af5bf0b5
harbor_harbor-harbor-notary-server-597c779966-zhvz2_5fcb564b-242b-4429-9560-85c6c3a2d333
harbor_harbor-harbor-notary-signer-6c9fdc8655-qj9ss_0eeb3aab-227d-46fa-98fa-a443bbb22bb5
harbor_harbor-harbor-portal-8456c5d77-j6ftj_e582f16e-3269-42aa-b16a-ad4c2c9ec02f
harbor_harbor-harbor-redis-0_eee073c7-1a0e-43c5-b86a-ef64827d5d80
harbor_harbor-harbor-registry-8456589f68-f5r2p_5a203290-6a60-4119-afbc-dd8a4263173b
harbor_harbor-harbor-trivy-0_d2f15569-a856-4b10-92a4-f7f7dfe9e15d

附录1

创建证书

FQDN要为harbor使用的域名，vaules.yaml里面externalURL配置块的值

方法

## 获得证书
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 3650 -out ca.crt

## 生成证书签名请求
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout tls.key -out tls.csr

## 生成证书
$ openssl x509 -req -days 3650 -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt

创建secrect

$ cp core.harbor.domain.crt tls.crt
$ cp core.harbor.domain.key tls.key
$ kubectl create secret generic harbor-harbor-ingress --from-file=tls.crt --from-file=tls.key --from-file=ca.crt -n harbor

查看证书过期时间

$ openssl x509 -in ca.crt  -noout -dates

k8s 中部署harbor

要求:

部署k8s

部署horbor

安装helm

安装harbor

访问harbor

413 Request Entity Too Large

附录1

创建证书

你可能感兴趣的:(k8s 中部署harbor)