k8s 搭建Harbor私有仓库验证拉取镜像

环境准备

搭建Harbor之前已经搭建了k8s集群环境，Harbor服务器上已经安装了docker、docker-compose，并已启动docker

k8s集群master01：192.168.245.211
k8s集群master02：192.168.245.206

k8s集群node01：192.168.245.209
k8s集群node02：192.168.245.210

harbor服务器：192.168.245.205

安装harbor

1、将Harbor软件拷贝到虚拟机，解压到/usr/local/

[root@harbor ~]# tar zxvf harbor-offline-installer-v1.2.2.tgz -C /usr/local/

2、修改配置文件的hostname值，指定本harbor服务器ip

[root@harbor ~]# cd /usr/local/harbor/

[root@harbor harbor]# vim harbor.cfg 

hostname = 192.168.245.205   //hostname设置访问地址，可以使用ip、域名，不可以设置为127.0.0.1或localhost

3、启动harbor

修改完配置文件后，在当前目录执行./install.sh，Harbor服务就会根据当期目录下的docker-compose.yml开始下载依赖的镜像，检测并按照顺序依次启动各个服务

[root@harbor harbor]# sh install.sh

[Step 0]: checking installation environment ...

Note: docker version: 19.03.13

Note: docker-compose version: 1.21.1
……
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db          ... done
Creating registry           ... done
Creating harbor-adminserver ... done
Creating harbor-ui          ... done
Creating harbor-jobservice  ... done
Creating nginx              ... done

✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at http://192.168.245.209. 
For more details, please visit https://github.com/vmware/harbor .

验证访问

真机验证访问http://192.168.245.205，会出现如下登录界面就成功

由于Harbor的web服务使用了宿主机的80端口，所以在浏览器直接输入宿主机的IP地址即可访问Harbor的web管理页面

账号是admin，密码是前面设置的harbor_admin_password的值(默认是Harbor12345)

新建项目

创建一个私有项目project

上传镜像

上传镜像之前需要登陆，登陆之前需要修改以下文件加入"insecure-registries":[“192.168.245.205”]否则登陆会报错

[root@node1 ~]# vim /etc/docker/daemon.json

{
 "registry-mirrors": ["https://noj876oc.mirror.aliyuncs.com"],
 "insecure-registries":["192.168.245.205"]
}

修改完成后需要重启docker

[root@node1 ~]# systemctl restart docker

节点1登陆到harbor：

[root@node1 ~]# docker login 192.168.245.205
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

从公网拉取tomcat镜像

[root@node1 ~]# docker pull tomcat
Using default tag: latest
latest: Pulling from library/tomcat
57df1a1f1ad8: Pull complete 
71e126169501: Pull complete 
1af28a55c3f3: Pull complete 
03f1c9932170: Pull complete 
881ad7aafb13: Pull complete 
9c0ffd4062f3: Pull complete 
bd62e479351a: Pull complete 
48ee8bc64dbc: Pull complete 
07cb85cca4f0: Pull complete 
6a78fac8d191: Pull complete 
Digest: sha256:1bab37d5d97bd8c74a474b2c1a62bbf1f1b4b62f151c8dcc472c7d577eb3479d
Status: Downloaded newer image for tomcat:latest
docker.io/library/tomcat:latest

给tomcat镜像重新打tag

[root@node1 ~]# docker tag tomcat 192.168.245.205/project/tomcat
[root@node1 ~]# docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
nginx                                                             latest              992e3b7be046        6 days ago          133MB
192.168.245.205/project/tomcat                                    latest              f796d3d2c195        3 weeks ago         647MB
tomcat                                                            latest              f796d3d2c195        3 weeks ago         647MB

上传tomcat镜像到我们新建的project里

[root@node1 ~]# docker push 192.168.245.205/project/tomcat
The push refers to repository [192.168.245.205/project/tomcat]
b654a29de9ee: Pushed 
1485ce09f585: Pushed 
eb6e8fe5c6dc: Pushed 
8b185d674aef: Pushed 
4f17d163126f: Pushed 
df95ed2a791d: Pushed 
17bdf5e22660: Pushed 
d37096232ed8: Pushed 
6add0d2b5482: Pushed 
4ef54afed780: Pushed 
latest: digest: sha256:99c20ba4ab117d182a0aa2266123b2cfb425777495fd62e2ba37f489c3e2f808 size: 2421

登陆网页验证已上传

下载镜像

在node2上拉取私库的镜像，报错提示需要登陆

[root@node2 ~]# docker pull 192.168.245.205/project/tomcat
Using default tag: latest
Error response from daemon: pull access denied for 192.168.245.205/project/tomcat, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

node2登陆到harbor

[root@node2 ~]# docker login 192.168.245.205
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

拉取镜像

[root@node2 ~]# docker pull 192.168.245.205/project/tomcat
Using default tag: latest
latest: Pulling from project/tomcat
57df1a1f1ad8: Pull complete 
71e126169501: Pull complete 
1af28a55c3f3: Pull complete 
03f1c9932170: Pull complete 
881ad7aafb13: Pull complete 
9c0ffd4062f3: Pull complete 
bd62e479351a: Pull complete 
48ee8bc64dbc: Pull complete 
07cb85cca4f0: Pull complete 
6a78fac8d191: Pull complete 
Digest: sha256:99c20ba4ab117d182a0aa2266123b2cfb425777495fd62e2ba37f489c3e2f808
Status: Downloaded newer image for 192.168.245.205/project/tomcat:latest
192.168.245.205/project/tomcat:latest

查看本地镜像，node2成功拉取私库的镜像

[root@node2 ~]# docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
192.168.245.205/project/tomcat                                    latest              f796d3d2c195        3 weeks ago         647MB
nginx                                                             latest              7e4d58f0e5f3        4 weeks ago         133MB
kubernetesui/dashboard                                            v2.0.4              46d0a29c3f61        5 weeks ago         225MB

登陆网页查看下载次数

master上拉取镜像

因为node1和node2都已登陆到harbor，有一个属于自己的认证秘钥，在家目录下的.docker/config.json里面

[root@node1 ~]# vim .docker/config.json

{
        "auths": {
                "192.168.245.205": {
                        "auth": "YWRtaW46SGFyYm9yMTIzNDU="
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/19.03.13 (linux)"
        }
}

node1和node2分别查看秘钥，秘钥是一致的，因为我们要在k8s集群的master上面新建由私库拉取下来的镜像生成的pod，而登陆私库需要认证，所以需要获得秘钥认证才可以，这时可以直接用2个节点的秘钥，并生成可用的二进制秘钥，-w 0 表示生成秘钥不转行，默认转行不是正确的格式会出错

[root@node1 ~]# cat .docker/config.json |base64 -w 0
ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjI0NS4yMDUiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuMTMgKGxpbnV4KSIKCX0KfQ==[root@node1 ~]# 

[root@node2 ~]# cat .docker/config.json |base64 -w 0
ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjI0NS4yMDUiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuMTMgKGxpbnV4KSIKCX0KfQ==[root@node2 ~]# 

需要编辑secret文件，制作secret资源的yaml

[root@master01 ~]# vim registry-pull-secret.yaml

piVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjI0NS4yMDUiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuMTMgKGxpbnV4KSIKCX0KfQ==
type: kubernetes.io/dockerconfigjson

创建secret资源

[root@master01 ~]# kubectl create -f registry-pull-secret.yaml
secret/registry-pull-secret created

查看secret

[root@master01 ~]# kubectl get secret
NAME                   TYPE                                  DATA   AGE
default-token-m87v7    kubernetes.io/service-account-token   3      13d
registry-pull-secret   kubernetes.io/dockerconfigjson        1      8s

制作拉取tomcat镜像的deployment资源

[root@master01 ~]# vim tomcat-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-tomcat   //自定义名称
spec:
  replicas: 2   //创建2个副本
  template:
    metadata:
      labels:
        app: my-tomcat
    spec:
      imagePullSecrets:
      - name: registry-pull-secret   //指定secret资源的名称
      containers:
      - name: my-tomcat
        image: 192.168.245.205/project/tomcat    //指定私库的镜像
        ports:
        - containerPort: 80   //容器端口为80
---
apiVersion: v1
kind: Service
metadata:
  name: my-tomcat
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    nodePort: 31111    //对外暴露访问的端口为31111
  selector:
    app: my-tomcat

创建pod资源

[root@master01 ~]# kubectl create -f tomcat-deployment.yaml
deployment.extensions/my-tomcat created
service/my-tomcat created

查看pod

[root@master01 ~]# kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
my-tomcat-86c67d7bcf-879kh   1/1     Running   0          2m22s
my-tomcat-86c67d7bcf-tv75h   1/1     Running   0          2m22s

网页查看下载次数多了2次

查看pod具体信息

[root@master01 ~]# kubectl get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP            NODE              NOMINATED NODE
my-tomcat-86c67d7bcf-879kh   1/1     Running   0          3m5s    172.17.4.3    192.168.245.209   <none>
my-tomcat-86c67d7bcf-tv75h   1/1     Running   0          3m5s    172.17.75.3   192.168.245.210   <none>

网页访问tomcat首页成功：http://192.168.245.209:31111/