Kubernetes学习笔记-Part.07 Harbor搭建

目录
Part.01 Kubernets与docker
Part.02 Docker版本
Part.03 Kubernetes原理
Part.04 资源规划
Part.05 基础环境准备
Part.06 Docker安装
Part.07 Harbor搭建
Part.08 K8s环境安装
Part.09 K8s集群构建
Part.10 容器回退

第七章 Harbor搭建

Docker-Compose是用来管理容器的，类似用户容器管家，我们有N多台容器或者应用需要启动的时候，如果手动去操作，是非常耗费时间的，如果有了Docker-Compose只需要一个配置文件就可以帮我们搞定，但是Docker-Compose只能管理当前主机上的Docker，不能去管理其他服务器上的服务。与k8s的区别如下：

• compose是docker推出的（swarm也是，级别同k8s），k8s是CNCF推出的
• compose只能在一台宿主机上编排容器，而k8s可以在很多台机器上编排容器
  Docker-Compose由python实现，调用docker服务的API负责实现对docker容器集群的快速编排，即通过一个单独的yaml文件，来定义一组相关的容器来为一个项目服务。因此，harbor也是通过Docker-Compose来实现的。
  过程：harbor下有install.sh脚本，里面会调用docker-compose，通过配置文件harbor.yml来实现对harbor的安装。

7.1.安装dockers-compose

docker-compose软件是一个可执行的二进制文件，在harbor01上将二进制文件上传至/usr/local/bin后赋予执行权限。
下载链接：
https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64

cp /opt/harbor/docker-compose-linux-x86_64 /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

查看版本

[root@harbor01 ~]# docker-compose --version
Docker Compose version v2.16.0

7.2.安装harbor

7.2.1.安装

下载harbor安装包，下载页面：
https://github.com/goharbor/harbor/releases/tag/v2.7.2
上传后解压

tar -xvf /opt/harbor/harbor-offline-installer-v2.7.2.tgz -C /opt/harbor/

修改yaml配置文件

cp /opt/harbor/harbor/harbor.yml.tmpl /opt/harbor/harbor/harbor.yml

修改内容如下：

# 修改hostname
hostname: harbor01.k8s.local
# 不使用http协议，注释掉http和port选项
#http:
#  port: 80
# 启用https协议
https:
  port: 443
  # 证书位置
  certificate: /opt/harbor/harbor/certs/harbor.crt
  # 私钥位置
  private_key: /opt/harbor/harbor/certs/harbor.key
# 页面密码
harbor_admin_password: lnyd@LNsy115
database:
  # 数据库密码
  password: root123
# 存储位置
data_volume: /data

创建数据存储目录

mkdir /data

创建证书和私钥对应的路径

mkdir /opt/harbor/harbor/certs

7.2.2.生成自签证书

• 生成证书颁发机构证书
  生成CA证书私钥（ca.key）

[root@harbor01 harbor]# cd /opt/harbor/harbor/certs/
[root@harbor01 certs]# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
.........++
....................................................................................................................++
e is 65537 (0x10001)

生成CA证书（ca.crt）
调整-subj选项中的值以反映组织信息，如果使用FQDN连接Harbor主机，则必须将其指定为通用名称（CN）属性。

openssl req -x509 -new -nodes -sha512 -days 3650 \
  -subj "/C=CN/ST=Liaoning/L=Shenyang/O=kubernetes/OU=Personal/CN=harbor01.k8s.local" \
  -key ca.key \
  -out ca.crt

• 生成服务器证书
  证书通常包含一个.crt文件和一个.key文件
  生成私钥（harbor01.k8s.local.key）

[root@harbor01 certs]# openssl genrsa -out harbor01.k8s.local.key 4096
Generating RSA private key, 4096 bit long modulus
........................................................................................................................................++
.........................................................................................................++
e is 65537 (0x10001)

生成证书签名请求（harbor01.k8s.local.csr）

openssl req -sha512 -new \
     -subj "/C=CN/ST=Liaoning/L=Shenyang/O=kubernetes/OU=Personal/CN=harbor01.k8s.local" \
     -key harbor01.k8s.local.key \
     -out harbor01.k8s.local.csr

生成一个x509 v3扩展文件（v3.ext）
无论使用FQDN还是IP地址连接到Harbor主机，都必须创建此文件，以便可以为Harbor主机生成符合主题备用名称（SAN）和x509 v3的证书扩展要求。替换DNS条目以反映域。

cat > v3.ext <<-EOF
 authorityKeyIdentifier=keyid,issuer
 basicConstraints=CA:FALSE
 keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
 extendedKeyUsage = serverAuth
 subjectAltName = @alt_names

 [alt_names]
 DNS.1=harbor01.k8s.local
 DNS.2=harbor01.k8s.local
 DNS.3=harbor01.k8s.local
EOF

使用v3.ext文件生成Harbor服务器证书（harbor01.k8s.local.crt）

[root@harbor01 certs]# openssl x509 -req -sha512 -days 3650 \
>      -extfile v3.ext \
>      -CA ca.crt -CAkey ca.key -CAcreateserial \
>      -in harbor01.k8s.local.csr \
>      -out harbor01.k8s.local.crt
Signature ok
subject=/C=CN/ST=Liaoning/L=Shenyang/O=kubernetes/OU=Personal/CN=harbor01.k8s.local
Getting CA Private Key

7.2.3.配置daemon.json文件

在master01上配置镜像加速地址以及

{
  "registry-mirrors": ["https://harbor01.k8s.local"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "bip": "1.1.1.1/24"
}

将daemon.json文件分发至其他节点上

ansible all -m template -a 'src=/etc/docker/daemon.json dest=/etc/docker/'

注：
① docker的cgroup驱动程序默认设置为system，默认情况下Kubernetes cgroup为systemd，因此需要更改Docker cgroup驱动。否则会在后面的kubeadm init时报错；
② Docker从1.3.X之后，与docker registry交互默认使用的是https，http服务则需要增加insecure-registries配置。

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp: lookup localhost on [::1]:53: read udp [::1]:41922->[::1]:53: read: connection refused.

配置完成后，需要重启docker服务

ansible all -m systemd -a 'daemon_reload=yes'
ansible all -m service -a 'name=docker state=restarted'

7.2.4.启动harbor

在/opt/harbor下启动harbor

[root@harbor01 ~]# cd /opt/harbor/harbor
[root@harbor01 harbor]# ./install.sh

[Step 0]: checking if docker is installed ...

Note: docker version: 23.0.5

[Step 1]: checking docker-compose is installed ...

Note: Docker Compose version v2.17.3

[Step 2]: loading Harbor images ...
17d981d1fd47: Loading layer [==================================================>]  37.78MB/37.78MB
066f24b65b06: Loading layer [==================================================>]   8.91MB/8.91MB
f5c5b2da3f78: Loading layer [==================================================>]  3.584kB/3.584kB
4cd07c2f1254: Loading layer [==================================================>]   2.56kB/2.56kB
90b02d6624a2: Loading layer [==================================================>]  87.15MB/87.15MB
b1c452c676c1: Loading layer [==================================================>]  5.632kB/5.632kB
a07864b2e153: Loading layer [==================================================>]    108kB/108kB
26a29846faca: Loading layer [==================================================>]  44.03kB/44.03kB
15c5d56364b4: Loading layer [==================================================>]  88.09MB/88.09MB
07cc9a12826b: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.7.2
d381f65a97a8: Loading layer [==================================================>]   8.91MB/8.91MB
a5ba716047be: Loading layer [==================================================>]  25.63MB/25.63MB
8af720b31993: Loading layer [==================================================>]  4.608kB/4.608kB
cf85d4aafef0: Loading layer [==================================================>]  26.42MB/26.42MB
Loaded image: goharbor/harbor-exporter:v2.7.2
9090e472d914: Loading layer [==================================================>]  6.295MB/6.295MB
95706aae16e4: Loading layer [==================================================>]  4.096kB/4.096kB
1e59d3cfe0b1: Loading layer [==================================================>]  3.072kB/3.072kB
c15f397332af: Loading layer [==================================================>]  190.7MB/190.7MB
625812afd6af: Loading layer [==================================================>]  13.75MB/13.75MB
bc49c81af9a3: Loading layer [==================================================>]  205.2MB/205.2MB
Loaded image: goharbor/trivy-adapter-photon:v2.7.2
d632d8a25428: Loading layer [==================================================>]  91.15MB/91.15MB
cabcd0940bdc: Loading layer [==================================================>]  6.145MB/6.145MB
44ee4d8970ae: Loading layer [==================================================>]  1.249MB/1.249MB
2f6a0dd83f2a: Loading layer [==================================================>]  1.194MB/1.194MB
Loaded image: goharbor/harbor-portal:v2.7.2
1a216f8aa02a: Loading layer [==================================================>]  123.4MB/123.4MB
d089ab0054a9: Loading layer [==================================================>]  24.63MB/24.63MB
8f24b651395d: Loading layer [==================================================>]   5.12kB/5.12kB
f2d321b72ee5: Loading layer [==================================================>]  6.144kB/6.144kB
acee91b49dbe: Loading layer [==================================================>]  3.072kB/3.072kB
73f0a48672cf: Loading layer [==================================================>]  2.048kB/2.048kB
d1137d179e82: Loading layer [==================================================>]   2.56kB/2.56kB
93f0cd1915db: Loading layer [==================================================>]   2.56kB/2.56kB
9c825e10712c: Loading layer [==================================================>]   2.56kB/2.56kB
4cb9928e2724: Loading layer [==================================================>]  9.728kB/9.728kB
Loaded image: goharbor/harbor-db:v2.7.2
bef216058819: Loading layer [==================================================>]  5.767MB/5.767MB
8f27a70b8dba: Loading layer [==================================================>]  4.096kB/4.096kB
6b2d3322e8cd: Loading layer [==================================================>]  17.42MB/17.42MB
4bdfc014a9cd: Loading layer [==================================================>]  3.072kB/3.072kB
dc54a26bde1b: Loading layer [==================================================>]  30.78MB/30.78MB
f22d45960368: Loading layer [==================================================>]  48.99MB/48.99MB
Loaded image: goharbor/harbor-registryctl:v2.7.2
dfef2543aa70: Loading layer [==================================================>]  5.762MB/5.762MB
a68585f608e3: Loading layer [==================================================>]  8.999MB/8.999MB
295d31910dd4: Loading layer [==================================================>]  14.47MB/14.47MB
efd5b1579023: Loading layer [==================================================>]  29.29MB/29.29MB
7dfd2e3fc59e: Loading layer [==================================================>]  22.02kB/22.02kB
faa41d246ac8: Loading layer [==================================================>]  14.47MB/14.47MB
Loaded image: goharbor/notary-signer-photon:v2.7.2
17b21070628b: Loading layer [==================================================>]  5.767MB/5.767MB
65500e78d7c9: Loading layer [==================================================>]  91.76MB/91.76MB
42ee762ff7a8: Loading layer [==================================================>]  3.072kB/3.072kB
26fcbd0bc385: Loading layer [==================================================>]  4.096kB/4.096kB
dce96c29de1b: Loading layer [==================================================>]  92.56MB/92.56MB
Loaded image: goharbor/chartmuseum-photon:v2.7.2
5853ff7207cd: Loading layer [==================================================>]  44.11MB/44.11MB
93590529a39f: Loading layer [==================================================>]  65.93MB/65.93MB
45c0712d114a: Loading layer [==================================================>]  26.14MB/26.14MB
27d6fd7e5535: Loading layer [==================================================>]  65.54kB/65.54kB
b0c1525b1461: Loading layer [==================================================>]   2.56kB/2.56kB
b81d770e8744: Loading layer [==================================================>]  1.536kB/1.536kB
12bbb36d555f: Loading layer [==================================================>]  12.29kB/12.29kB
7a733d55d815: Loading layer [==================================================>]  2.621MB/2.621MB
e4007be64a14: Loading layer [==================================================>]    407kB/407kB
Loaded image: goharbor/prepare:v2.7.2
5bdb50147fe3: Loading layer [==================================================>]  8.909MB/8.909MB
7c7583a1eef8: Loading layer [==================================================>]  3.584kB/3.584kB
f5483be14faa: Loading layer [==================================================>]   2.56kB/2.56kB
9b67b6258fdf: Loading layer [==================================================>]  106.5MB/106.5MB
374df1d91d24: Loading layer [==================================================>]  107.3MB/107.3MB
Loaded image: goharbor/harbor-jobservice:v2.7.2
ec911fc21120: Loading layer [==================================================>]  91.15MB/91.15MB
Loaded image: goharbor/nginx-photon:v2.7.2
631cf08f9ff0: Loading layer [==================================================>]  5.767MB/5.767MB
db4216090ca5: Loading layer [==================================================>]  4.096kB/4.096kB
1f1103a3353e: Loading layer [==================================================>]  3.072kB/3.072kB
5e28d0ce371b: Loading layer [==================================================>]  17.42MB/17.42MB
bbbdbc284648: Loading layer [==================================================>]  18.21MB/18.21MB
Loaded image: goharbor/registry-photon:v2.7.2
3dc8df9174d5: Loading layer [==================================================>]  99.07MB/99.07MB
38e93b103e4f: Loading layer [==================================================>]  3.584kB/3.584kB
74b98ab194ce: Loading layer [==================================================>]  3.072kB/3.072kB
c203b688a2be: Loading layer [==================================================>]   2.56kB/2.56kB
525a15ff6933: Loading layer [==================================================>]  3.072kB/3.072kB
ea4e850eadfa: Loading layer [==================================================>]  3.584kB/3.584kB
5c345ac6af33: Loading layer [==================================================>]  20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.7.2
1c464948f4c8: Loading layer [==================================================>]  91.99MB/91.99MB
e23b5317ef75: Loading layer [==================================================>]  3.072kB/3.072kB
ad8e1bb2e672: Loading layer [==================================================>]   59.9kB/59.9kB
2eade6174326: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.7.2
dc782aa72031: Loading layer [==================================================>]  5.762MB/5.762MB
aead20724337: Loading layer [==================================================>]  8.999MB/8.999MB
22b6f665e30b: Loading layer [==================================================>]  15.88MB/15.88MB
4ded3a6c4ce0: Loading layer [==================================================>]  29.29MB/29.29MB
258a7b5fb17f: Loading layer [==================================================>]  22.02kB/22.02kB
be68b1b440c0: Loading layer [==================================================>]  15.88MB/15.88MB
Loaded image: goharbor/notary-server-photon:v2.7.2


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /opt/harbor/harbor
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir


Note: stopping existing Harbor instance ...


[Step 5]: starting Harbor ...
[+] Running 10/10
 ✔ Network harbor_harbor        Created                                                                                                                   0.1s
 ✔ Container harbor-log         Started                                                                                                                   0.7s
 ✔ Container redis              Started                                                                                                                   1.2s
 ✔ Container harbor-db          Started                                                                                                                   1.4s
 ✔ Container registry           Started                                                                                                                   1.5s
 ✔ Container registryctl        Started                                                                                                                   1.5s
 ✔ Container harbor-portal      Started                                                                                                                   1.5s
 ✔ Container harbor-core        Started                                                                                                                   1.8s
 ✔ Container harbor-jobservice  Started                                                                                                                   2.4s
 ✔ Container nginx              Started                                                                                                                   2.4s
✔ ----Harbor has been installed and started successfully.----

7.3.向docker主机上添加harbor证书

转换harbor01.k8s.local.crt为harbor01.k8s.local.cert，供Docker使用；Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书
在harbor01上，进行证书转换

cd /opt/harbor/harbor/certs/
openssl x509 -inform PEM -in harbor01.k8s.local.crt -out harbor01.k8s.local.cert

在master01上，直接登录harbor01，会提示证书问题的报错

[root@localhost ansible]# docker login https://harbor01.k8s.local -uadmin
Password:
Error response from daemon: Get "https://harbor01.k8s.local/v2/": x509: certificate signed by unknown authority

将harbor01上的服务器证书、密钥和CA文件复制到/etc/docker/certs.d/harbor01.k8s.local/目录下

ansible all -m file -a 'path=/etc/docker/certs.d/harbor01.k8s.local state=directory'
scp harbor01:/opt/harbor/harbor/certs/harbor01.k8s.local.cert /etc/docker/certs.d/harbor01.k8s.local/
scp harbor01:/opt/harbor/harbor/certs/harbor01.k8s.local.key /etc/docker/certs.d/harbor01.k8s.local/
scp harbor01:/opt/harbor/harbor/certs/ca.crt /etc/docker/certs.d/harbor01.k8s.local/

将harbor的证书复制到master01上，然后分发至所有其他节点上

ansible all -m template -a 'src=/etc/docker/certs.d/harbor01.k8s.local/harbor01.k8s.local.cert dest=/etc/docker/certs.d/harbor01.k8s.local/'
ansible all -m template -a 'src=/etc/docker/certs.d/harbor01.k8s.local/harbor01.k8s.local.key dest=/etc/docker/certs.d/harbor01.k8s.local/'
ansible all -m template -a 'src=/etc/docker/certs.d/harbor01.k8s.local/ca.crt dest=/etc/docker/certs.d/harbor01.k8s.local/'
ansible all -m systemd -a 'daemon_reload=yes'
ansible all -m service -a 'name=docker state=restarted'

重启docker后，需要重新启动harbor

cd /opt/harbor/harbor
./install.sh

登录到私有仓库上，显示“Login Succeeded”表示成功

[root@master01 ansible]# docker login https://harbor01.k8s.local -uadmin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

7.4.创建项目

访问https://192.168.111.20，用户名admin，密码lnyd@LNsy115

创建项目kubernetes，用于存放kubernetes集群组件的镜像