k8s学习笔记2-搭建harbor私有仓库
k8s学习笔记2-搭建harbor私有仓库
- 一.介绍
- 二.环境准备
-
- 1.docker-compose安装
- 2.下载和解压harbor
- 三.harbor部署
-
- 1.http方式部署
- 2.https方式部署
-
- a.自制证书部署
- b.第三方签名证书部署
- 3.配置开机自启动
- 四.访问harbor仓库（自签名的https仓库）
-
- 方式1：修改启动文件
- 方式2：分发ca.cert证书到其他docker引擎
- 方式3：k8s的pod访问
- 五.参考资料
一.介绍
本次安装时的机器系统为ubuntu 22.04,harbor的版本为v2.5.3，docker已经安装，docker的版本为v20.10.12，本次安装直接在上一篇博客的基础上做的，harbor直接安装到k8s-master1上。
二.环境准备
1.docker-compose安装
apt install pip -y
pip install docker-compose
检查安装情况
docker-compose --version
2.下载和解压harbor
官网地址：https://github.com/goharbor/harbor/releases
目前最新的版本为:v2.5.3
下载harbor版本
wget https://storage.googleapis.com/harbor-releases/release-2.5.0/harbor-online-installer-v2.5.3.tgz
解压harbor文件
root@k8s-master1:~# tar -xvf harbor-online-installer-v2.5.3.tgz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
三.harbor部署
1.http方式部署
修改harbor配置文件，如果使用非https方式部署，直接修改hostname字段，并注释https部分然后执行部署即可
root@k8s-master1:~# cd harbor/
root@k8s-master1:~/harbor# cp harbor.yml.tmpl harbor.yml
root@k8s-master1:~/harbor# nano harbor.yml
root@k8s-master1:~/harbor# more harbor.yml
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 192.168.100.240
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related confighttps
#https:
# https port for harbor, default is 443
# port: 443
# The path of cert and key files for nginx
# certificate: /your/certificate/path
# private_key: /your/private/key/path
.......
开始安装（下载镜像，估计需要6分钟左右，这个完全看网速）
./install.sh
安装好之后，登录界面（帐号默认为：admin，密码为：Harbor12345）
登录进去后
到目前为止，http方式部署已经完成
2.https方式部署
官方配置https文档：https://goharbor.io/docs/2.0.0/install-config/configure-https/
使用命令，将当前http部署的harbor删除，如果没有部署http，那就无需执行如下命令
root@k8s-master1:~/harbor#docker-compose down
root@k8s-master1:~/harbor#rm docker-compose.yml
a.自制证书部署
1.修改harbor.yml配置文件
root@k8s-master1:~/harbor# cat harbor.yml
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: registry.harbor.com
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
#特别说明：该harbor服务器的证书和私钥的路径，与自制证书或者第三方签名的证书的位置对应，目前我的路径就是这个位置
certificate: /data/cert/registry.harbor.com.crt
private_key: /data/cert/registry.harbor.com.key
......
2.生成自制证书
#!/bin/bash
############################生成证书颁发机构证书############################
#1、生成CA证书私钥
mkdir -p /root/harbor/ssl
cd /root/harbor/ssl
openssl genrsa -out ca.key 4096
#2、生成CA证书
openssl req -x509 -new -nodes -sha512 -days 3650 \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.harbor.com" \
-key ca.key \
-out ca.crt
############################生成服务器证书############################
#1、生成harbor服务器私钥
openssl genrsa -out registry.harbor.com.key 4096
#2、生成证书签名请求（CSR）
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.harbor.com" -key registry.harbor.com.key -out registry.harbor.com.csr
#3、生成一个x509 v3扩展文件
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=registry.harbor.com
DNS.2=registry.harbor
DNS.3=harbor
EOF
#4、使用该v3.ext文件为您的Harbor主机生成证书
openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in registry.harbor.com.csr \
-out registry.harbor.com.crt
############################提供证书给Harbor和Docker############################
#1、将服务器证书和密钥复制到Harbor主机上的certficates文件夹中
mkdir -p /data/cert
cp registry.harbor.com.crt /data/cert/
cp registry.harbor.com.key /data/cert/
#2、转换yourdomain.com.crt为yourdomain.com.cert，供Docker使用
openssl x509 -inform PEM -in registry.harbor.com.crt -out registry.harbor.com.cert
#3、将服务器证书，密钥和CA文件复制到Harbor主机上的Docker证书文件夹中
mkdir -p /etc/docker/certs.d/registry.harbor.com/
cp registry.harbor.com.cert /etc/docker/certs.d/registry.harbor.com/
cp registry.harbor.com.key /etc/docker/certs.d/registry.harbor.com/
cp ca.crt /etc/docker/certs.d/registry.harbor.com/
#4、重新启动Docker Engine
systemctl restart docker
3.开始安装
./install.sh
4.在k8s-master1机器，配置ip和域名对应
echo 192.168.100.240 registry.harbor.com >> /etc/hosts
5.安装成功后，使用https://registry.harbor.com进行登录，会提示有风险
提示有风险的原因，是因为这是我们自制的证书，系统中存在CA机构不能够对这个harbor的证书进行验证，这样的话，系统就认为这个网站是有风险的，不安全的。
我们直接在浏览器中，将我们生成的ca.crt安装到浏览器上，就可以啦，这样就可以进行正常访问了
在浏览器上，安装此ca.crt
先进入证书导入界面
点击import按钮导入ca.crt证书机构，全部勾选
6.导入成功后，进入https://registry.harbor.com,发现不会出现风险提示，并且小钥匙也正常了
b.第三方签名证书部署
部署第三放签名的证书，相对来说，比自制证书简单
比如说，我们在第三方机构申请到一个证书（registry.zhangsan.com）,这个证书的名字是registry.zhangsan.com.crt,私钥是registry.zhangsan.com.key，这两个文件，都存放在/data/zhangsan/这个文件夹下
1.修改harbor.yaml文件
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: registry.zhangsan.com
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /data/zhangsan/registry.zhangsan.com.crt
private_key: /data/zhangsan/registry.zhangsan.com.key
2.转换registry.zhangsan.com.crt为registry.zhangsan.com.cert，供Docker使用。
openssl x509 -inform PEM -in registry.zhangsan.com.crt -out registry.zhangsan.com.cert
3.直接安装即可
./install.sh
3.配置开机自启动
使用systemd来启动关闭harbor，下面这个地址Environment=harbor_install_path=/root，需要修改成自己的harbor的安装路径位置
cat > /usr/lib/systemd/system/harbor.service << 'EOF'
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=5
Environment=harbor_install_path=/root
ExecStart=/usr/local/bin/docker-compose -f ${harbor_install_path}/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f ${harbor_install_path}/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
EOF
配置开机自启动
systemctl enable --now harbor
四.访问harbor仓库（自签名的https仓库）
特别说明：该仓库是使用自签名的https的仓库，如果是使用第三方签名https的harbor仓库或者是http的方式，步骤会有些许的不一样
方式1：修改启动文件
1.修改启动文件之前登录，如下，会报错，因为系统无法认证自签名证书
(base) root@test:/etc/docker# docker login registry.harbor.com
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
Username (admin): admin
Password:
Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
(base) root@test:/etc/docker#
2.修改启动文件/lib/systemd/system/docker.service
最主要就是修改ExecStart后增加–insecure-registry=registry.harbor.com
就是启动的时候，把仓库地址给添加上去
GNU nano 4.8 /lib/systemd/system/docker.service Modified
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
Wants=containerd.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry=registry.harbor.com
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
3.重新启动docker
(base) root@test:/etc/docker# systemctl daemon-reload
(base) root@test:/etc/docker# systemctl restart docker
4.登录registry.harbor.com
(base) root@test:/etc/docker# docker login registry.harbor.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
5.向harbor服务器push镜像，push镜像前，现在本地随便给一个镜像tag一下，然后push上去
(base) root@test:/etc/docker# docker tag ubuntu:latest registry.harbor.com/library/ubuntu
(base) root@test:/etc/docker# docker push registry.harbor.com/library/ubuntu:v1
The push refers to repository [registry.harbor.com/library/ubuntu]
e59fc9495612: Layer already exists
v1: digest: sha256:aa6c2c047467afc828e77e306041b7fa4a65734fe3449a54aa9c280822b0d87d size: 529
(base) root@test:/etc/docker#
方式2：分发ca.cert证书到其他docker引擎
1.将方式1中的步骤2的修改删除，并且重新启动docker（主要为了恢复成最开始的时候），然后再重新登录registry.harbor.com，会出现无法登录的情况
(base) root@test:/etc/docker# docker login registry.harbor.com
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
Username (admin): admin
Password:
Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "registry.harbor.com")
(base) root@test:/etc/docker#
2.将在k8s-master1节点中，使用自签名的根证书分发给需要访问harbor服务的docker机器上，并且重新启动docker
(base) root@test:/etc/docker# mkdir -p /etc/docker/certs.d/registry.harbor.com/
(base) root@test:/etc/docker# scp 192.168.100.240:/root/harbor/ssl/ca.crt /etc/docker/certs.d/registry.harbor.com/
[email protected]'s password:
ca.crt 100% 2065 392.9KB/s 00:00
(base) root@test:/etc/docker# systemctl restart docker
(base) root@test:/etc/docker#
3.登录registry.harbor.com
(base) root@test:/etc/docker# docker login registry.harbor.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
(base) root@test:/etc/docker#
4.向harbor服务器push镜像，push镜像前，现在本地随便给一个镜像tag一下，然后push上去
(base) root@test:/etc/docker# docker tag ubuntu:latest registry.harbor.com/library/ubuntu
(base) root@test:/etc/docker# docker push registry.harbor.com/library/ubuntu:v1
The push refers to repository [registry.harbor.com/library/ubuntu]
e59fc9495612: Layer already exists
v1: digest: sha256:aa6c2c047467afc828e77e306041b7fa4a65734fe3449a54aa9c280822b0d87d size: 529
(base) root@test:/etc/docker#
方式3：k8s的pod访问
1.在k8s-master1节点，k8s集群使用类型为docker-registry的Secret来提供身份认证
root@k8s-master1:~# kubectl create secret docker-registry registry-key --docker-server registry.harbor.com --docker-username admin --docker-password 123456 --dry-run -o yaml > secret-regisry.yaml
root@k8s-master1:~# kubectl apply -f secret-regisry.yaml
secret/registry-key created
root@k8s-master1:~# kubectl get secrets
NAME TYPE DATA AGE
registry-key kubernetes.io/dockerconfigjson 1 5s
仓库的帐号密码是用base64进行加密，我们可以将其解密
root@k8s-master1:~# cat secret-regisry.yaml
apiVersion: v1
data:
.dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5oYXJib3IuY29tIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IjEyMzQ1NiIsImF1dGgiOiJZV1J0YVc0Nk1USXpORFUyIn19fQ==
kind: Secret
metadata:
creationTimestamp: null
name: registry-key
type: kubernetes.io/dockerconfigjson
root@k8s-master1:~# echo eyJhdXRocyI6eyJyZWdpc3RyeS5oYXJib3IuY29tIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IjEyMzQ1NiIsImF1dGgiOiJZV1J0YVc0Nk1USXpORFUyIn19fQ== |base64 -d
{"auths":{"registry.harbor.com":{"username":"admin","password":"123456","auth":"YWRtaW46MTIzNDU2"}}}root@k8s-master1:~#
2.新建一个名称为test.yaml文件的pod，见下面配置已经在registry.harbor.com网站，存放了busybox镜像。
apiVersion: batch/v1
kind: Job
metadata:
name: hello
spec:
template:
# 这里是 Pod 模板
spec:
imagePullSecrets:
- name: registry-key
containers:
- name: hello
image: registry.harbor.com/test/busybox:1.28
command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
restartPolicy: OnFailure
3.直接应用，会报错
root@k8s-master1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-whcs6 0/1 ErrImagePull 0 5s
root@k8s-master1:~# kubectl describe pods hello-whcs6
后会发现如下错误，无法拉取镜像
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15s default-scheduler Successfully assigned default/hello-whcs6 to k8s-node1
Normal Pulling 14s kubelet Pulling image "registry.harbor.com/test/busybox:1.28"
Warning Failed 13s kubelet Failed to pull image "registry.harbor.com/test/busybox:1.28": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry.harbor.com/v2/": x509: certificate has expired or is not yet valid: current time 2022-07-23T03:38:45Z is after 2021-04-28T12:00:00Z
Warning Failed 13s kubelet Error: ErrImagePull
4.在k8s-ndoe1和k8s-node2节点上，将该harbor网站的证书签发机构的证书放到，这个节点的docker上去
在k8s-node1上
root@k8s-node1:~# mkdir -p /etc/docker/certs.d/registry.harbor.com/
root@k8s-node1:~# scp 192.168.100.240:/root/harbor/ssl/ca.crt /etc/docker/certs.d/registry.harbor.com/
[email protected]'s password:
ca.crt 100% 2065 1.4MB/s 00:00
root@k8s-node1:~# systemctl restart docker
root@k8s-node1:~#
在k8s-node2上
root@k8s-node2:~# mkdir -p /etc/docker/certs.d/registry.harbor.com/
root@k8s-node2:~# scp 192.168.100.240:/root/harbor/ssl/ca.crt /etc/docker/certs.d/registry.harbor.com/
[email protected]'s password:
ca.crt 100% 2065 1.3MB/s 00:00
root@k8s-node2:~# systemctl restart docker
root@k8s-node2:~#
5.删除pod，重新部署
root@k8s-master1:~# kubectl delete -f test.yaml
job.batch "hello" deleted
root@k8s-master1:~# kubectl apply -f test.yaml
job.batch/hello created
root@k8s-master1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-t4tdq 1/1 Running 0 3s
root@k8s-master1:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-57ps9 1/1 Running 0 58s 10.244.1.12 k8s-node1 <none> <none>
五.参考资料
1.harbor安装并配置https
https://blog.csdn.net/networken/article/details/107502461
2.Ubuntu16 安装搭建Harbor
https://blog.csdn.net/qq_35720307/article/details/8669175