kubernetes学习：1.构建harbor镜像仓库

构建harbor镜像仓库

Harbor是一个用于存储和分发Docker镜像的企业级Registry服务器，可以在测试和生产环境中部署harbor对业务上的镜像进行统一的管理。harbor 的官方网站如下所示：
https://vmware.github.io/harbor/cn/

一般来讲我们在生产环境部署harbor的时候需要使用多节点方案。将数据库从配置项中抽取出来，后端的镜像资源存储使用统一的共享资源（ceph分配的磁盘挂载到指定目录中）。前端可以使用nginx或者haproxy对多个harbor服务进行负载均衡。

本文搭建一个单机的harbor镜像仓库，体会对镜像进行上传和下载的过程。上述的复杂架构就是在此基础上的不断拓展。

构建harbor

---

构建环境

具体的构建过程主要分为以下几个步骤：

• 操作系统： centos7.2
• docker版本： 1.12.6
• docker-compose： 1.9.0

首先安装docker服务：

[root@wecloud-test-harbor ~]# yum install docker -y

启动docker服务：

[root@wecloud-test-harbor ~]# systemctl start docker.service 
[root@wecloud-test-harbor ~]# systemctl enable docker.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

查看docker服务状态：

[root@wecloud-test-harbor ~]# systemctl status docker.service 
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2018-03-26 22:33:04 CST; 12s ago
     Docs: http://docs.docker.com
 Main PID: 2311 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─2311 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-run...
           └─2318 /usr/bin/docker-containerd-current -l 

---

安装docker-compose：

[root@wecloud-test-harbor ~]# yum install docker-compose -y

下载harbor的离线安装包（离线的安装速度会比较快一些）,下载地址如下：
https://storage.googleapis.com/harbor-releases/release-1.4.0/harbor-offline-installer-v1.4.0.tgz

解压harbor-offline-installer-v1.4.0.tgz：

[root@wecloud-test-harbor ~]# tar xvf harbor-offline-installer-v1.4.0.tgz

跳转到harbor目录，修改harbor.cfg配置文件：

[root@wecloud-test-harbor harbor]# grep "^[^#]" harbor.cfg 
# harbor的主机地址
hostname = 192.168.99.180
# 这里使用http协议，如果需要上传镜像，需要在docker.service文件中添加一个配置，后续会介绍到
ui_url_protocol = http
max_job_workers = 3 
customize_crt = on
ssl_cert = /data/cert/server.crt
ssl_cert_key = /data/cert/server.key
secretkey_path = /data
admiral_url = NA
log_rotate_count = 50
log_rotate_size = 200M
#××××××××××××××××××邮件的配置信息××××××××××××××××××
email_identity = 
email_server = smtp.mydomain.com
email_server_port = 25
email_username = [email protected]
email_password = abc
email_from = admin 
email_ssl = false
email_insecure = false
#××××××××××××××××××邮件的配置信息××××××××××××××××××
# 管理员密码，管理员帐号为：admin
harbor_admin_password = Harbor12345
# 使用数据库存储认证信息
auth_mode = db_auth
# 配置ldap信息
ldap_url = ldaps://ldap.mydomain.com
ldap_basedn = ou=people,dc=mydomain,dc=com
ldap_uid = uid 
ldap_scope = 2 
ldap_timeout = 5
ldap_verify_cert = true
self_registration = on
token_expiration = 30
project_creation_restriction = everyone
# 指定mysql数据库的信息
db_host = mysql
db_password = root123
db_port = 3306
db_user = root
# redis地址
redis_url =
clair_db_host = postgres
clair_db_password = password
clair_db_port = 5432
clair_db_username = postgres
clair_db = postgres
uaa_endpoint = uaa.mydomain.org
uaa_clientid = id
uaa_clientsecret = secret
uaa_verify_cert = true
uaa_ca_cert = /path/to/ca.pem
registry_storage_provider_name = filesystem
registry_storage_provider_config =

配置信息修改结束后，执行install.sh脚本执行具体的安装过程，安装完成后，harbor的容器都会启动：

安装过程主要分为以下几个步骤：

• 检测环境信息，docker的版本和docker-compose的版本信息;
• 导入harbor的镜像信息，这些镜像内容都已经包含在离线包中了，所以导入速度相对比较快;
• 检查Harbor的现有实例，如果现有实例存在则会删除;
• 启动harbor
  最后会提示我们启动成功，然后可以访问刚才配置的本地ip地址：
  Now you should be able to visit the admin portal at http://192.168.99.180.
  For more details, please visit https://github.com/vmware/harbor .

[root@wecloud-test-harbor harbor]# ./install.sh

查看harbor使用的镜像列表：

[root@wecloud-test-harbor harbor]# docker images 
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
vmware/clair-photon           v2.0.1-v1.4.0       a1df3526fe43        6 weeks ago         300.3 MB
vmware/notary-server-photon   v0.5.1-v1.4.0       3edfddb8ece2        6 weeks ago         211.2 MB
vmware/notary-signer-photon   v0.5.1-v1.4.0       cc70a05cdb6a        6 weeks ago         208.7 MB
vmware/registry-photon        v2.6.2-v1.4.0       8920f621ddd1        6 weeks ago         197.8 MB
vmware/nginx-photon           v1.4.0              20c8a01ac6ab        6 weeks ago         134.5 MB
vmware/harbor-log             v1.4.0              9e818c7a27ab        6 weeks ago         199.7 MB
vmware/harbor-jobservice      v1.4.0              29c14d91b043        6 weeks ago         190.6 MB
vmware/harbor-ui              v1.4.0              6cb4318eda6a        6 weeks ago         209.5 MB
vmware/harbor-adminserver     v1.4.0              8145970fa013        6 weeks ago         182.2 MB
vmware/harbor-db              v1.4.0              c38da34727f0        6 weeks ago         521 MB
vmware/mariadb-photon         v1.4.0              8457013cf6e3        6 weeks ago         521 MB
vmware/postgresql-photon      v1.4.0              59aa61520094        6 weeks ago         220.6 MB
vmware/harbor-db-migrator     1.4                 7a4d871b612e        9 weeks ago         1.146 GB
vmware/photon                 1.0                 9b411d78ad9e        10 weeks ago        129.7 MB

查看harbor的启动容器：

[root@wecloud-test-harbor harbor]# docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                   PORTS                                                              NAMES
6231d7bb19c3        vmware/nginx-photon:v1.4.0             "nginx -g 'daemon off"   3 minutes ago       Up 2 minutes             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp   nginx
23d83a8b15ce        vmware/harbor-jobservice:v1.4.0        "/harbor/start.sh"       3 minutes ago       Up 2 minutes (healthy)                                                                      harbor-jobservice
ddbd8ea06f59        vmware/harbor-ui:v1.4.0                "/harbor/start.sh"       3 minutes ago       Up 2 minutes (healthy)                                                                      harbor-ui
d8e70cc126b3        vmware/harbor-db:v1.4.0                "/usr/local/bin/docke"   3 minutes ago       Up 3 minutes (healthy)   3306/tcp                                                           harbor-db
f533c4ae2e5d        vmware/harbor-adminserver:v1.4.0       "/harbor/start.sh"       3 minutes ago       Up 2 minutes (healthy)                                                                      harbor-adminserver
68afc0b58219        vmware/registry-photon:v2.6.2-v1.4.0   "/entrypoint.sh serve"   3 minutes ago       Up 3 minutes (healthy)   5000/tcp                                                           registry
1aa9ed019604        vmware/harbor-log:v1.4.0               "/bin/sh -c /usr/loca"   3 minutes ago       Up 3 minutes (healthy)   127.0.0.1:1514->10514/tcp                                          harbor-log

直接访问harbor的网址：
输入的用户名为admin，密码为配置文件中harbor_admin_password的内容

默认有一个项目为library

我们可以点击新建创建额外的项目：

后续我们可以将自己的镜像推送到该项目中。

harbor的单机版已经搭建完成，后边我们需要介绍如何推送和拉取镜像信息。

---

推送镜像到harbor仓库

我们首先从其他的仓库获取一个测试镜像，如果是默认配置下，docker指定的仓库地址是docker hub，国内因为一些特殊原因拉取速度很慢，而且登录验证也无法通过。如下图所示：

[root@wecloud-test-harbor ~]# docker search centos
Error response from daemon: Get https://index.docker.io/v1/search?q=centos&n=25: x509: certificate signed by unknown authority
[root@wecloud-test-harbor ~]# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: zhangchiwd371
Password: 
Error response from daemon: Get https://index.docker.io/v1/users/: x509: certificate signed by unknown authority

所以可以配置docker的守护进程，修改镜像源为国内源。编辑/etc/docker/daemon.json文件（新版本的docker服务），设置镜像源的地址：

{
  "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}

修改完启动文件后重启docker服务：

[root@wecloud-test-harbor ~]# systemctl restart docker.service

再次搜索查看需要的docker镜像，速度飞快：

[root@wecloud-test-harbor ~]# docker search nginx
INDEX       NAME                                                             DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
docker.io   docker.io/nginx                                                  Official build of Nginx.                        8217      [OK]       
docker.io   docker.io/jwilder/nginx-proxy                                    Automated Nginx reverse proxy for docker c...   1299                 [OK]
docker.io   docker.io/richarvey/nginx-php-fpm                                Container running Nginx + PHP-FPM capable ...   536                  [OK]
docker.io   docker.io/jrcs/letsencrypt-nginx-proxy-companion                 LetsEncrypt container to use with nginx as...   333                  [OK]
docker.io   docker.io/kong                                                   Open-source Microservice & API Management ...   170       [OK]       
docker.io   docker.io/webdevops/php-nginx                                    Nginx with PHP-FPM                              97                   [OK]
docker.io   docker.io/kitematic/hello-world-nginx                            A light-weight nginx container that demons...   95                   
docker.io   docker.io/zabbix/zabbix-web-nginx-mysql                          Zabbix frontend based on Nginx web-server ...   48                   [OK]
docker.io   docker.io/bitnami/nginx                                          Bitnami nginx Docker Image                      45                   [OK]
docker.io   docker.io/linuxserver/nginx                                      An Nginx container, brought to you by Linu...   33                   
docker.io   docker.io/1and1internet/ubuntu-16-nginx-php-phpmyadmin-mysql-5   ubuntu-16-nginx-php-phpmyadmin-mysql-5          29                   [OK]
docker.io   docker.io/tobi312/rpi-nginx                                      NGINX on Raspberry Pi / armhf                   19                   [OK]
docker.io   docker.io/wodby/drupal-nginx                                     Nginx for Drupal container image                9                    [OK]
docker.io   docker.io/blacklabelops/nginx                                    Dockerized Nginx Reverse Proxy Server.          8                    [OK]
docker.io   docker.io/nginxdemos/nginx-ingress                               NGINX Ingress Controller for Kubernetes         8                    
docker.io   docker.io/webdevops/nginx                                        Nginx container                                 8                    [OK]
docker.io   docker.io/centos/nginx-18-centos7                                Platform for running nginx 1.8 or building...   6                    
docker.io   docker.io/1science/nginx                                         Nginx Docker images that include Consul Te...   4                    [OK]
docker.io   docker.io/nginxdemos/hello                                       NGINX webserver that serves a simple page ...   4                    [OK]
docker.io   docker.io/behance/docker-nginx                                   Provides base OS, patches and stable nginx...   2                    [OK]
docker.io   docker.io/pebbletech/nginx-proxy                                 nginx-proxy sets up a container running ng...   2                    [OK]
docker.io   docker.io/toccoag/openshift-nginx                                Nginx reverse proxy for Nice running on sa...   1                    [OK]
docker.io   docker.io/travix/nginx                                           NGinx reverse proxy                             1                    [OK]
docker.io   docker.io/goodguide/nginx-application-proxy                      No-configuration Nginx reverse proxy for a...   0                    [OK]
docker.io   docker.io/mailu/nginx                                            Mailu nginx frontend                            0                    [OK]
[root@wecloud-test-harbor ~]# 

为了测试效果，我们拉取nginx的官方镜像：

[root@wecloud-test-harbor ~]# docker pull docker.io/nginx
Using default tag: latest
Trying to pull repository docker.io/library/nginx ... 
latest: Pulling from docker.io/library/nginx
2a72cbf407d6: Pull complete 
fefa2faca81f: Pull complete 
080aeede8114: Pull complete 
Digest: sha256:c4ee0ecb376636258447e1d8effb56c09c75fe7acf756bf7c13efadf38aa0aca

可以看到已经镜像列表中已经多出了nginx 镜像：

[root@wecloud-test-harbor ~]# docker images 
REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE
docker.io/nginx                         latest              7f70b30f2cc6        4 days ago          108.7 MB

---

上传镜像

如果需要将这个镜像上传到我们自己的harbor仓库中，需要给该镜像重新生成一个tag标签。

[root@wecloud-test-harbor ~]# docker tag docker.io/nginx:latest 192.168.99.180/k8s/nginx:latest

注意这里的写法，新的标签是一个路径的格式：192.168.99.180/k8s/nginx:latest，其中192.168.99.180是我们的harbor仓库地址，k8s是项目名称，nginx是镜像名称，latest是镜像版本号。

查看镜像列表，可以看到我们多了一个新的镜像标签：

[root@wecloud-test-harbor ~]# docker images 
REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE
docker.io/nginx                         latest              7f70b30f2cc6        5 days ago          108.7 MB
192.168.99.180/k8s/nginx                latest              7f70b30f2cc6        5 days ago          108.7 MB

将该镜像推送到harbor仓库，这是一个写操作，需要事先登录harbor仓库：

[root@wecloud-test-harbor ~]# docker login 192.168.99.180
Username: admin
Password: 
Error response from daemon: Get https://192.168.99.180/v1/users/: dial tcp 192.168.99.180:443: getsockopt: connection refused

直接登录发现并不允许我们登录镜像，其实这是因为默认情况下是需要使用https协议的，而我们在harbor的配置中使用的是http协议。所以需要在docker.service启动脚本中指定非加密的登录方式。编辑/usr/lib/systemd/system/docker.service文件，在启动选项中追加如下配置：–insecure-registry=192.168.99.180：

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer

[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/run/containers/registries.conf
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/docker/daemon.json
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1
Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \
          --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
          --default-runtime=docker-runc \
          --exec-opt native.cgroupdriver=systemd \
          --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
          --insecure-registry=192.168.99.183 \
          $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $ADD_REGISTRY \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY\
          $REGISTRIES
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
MountFlags=slave
KillMode=process

[Install]
WantedBy=multi-user.target

重新加载systemctl服务：

[root@wecloud-test-harbor ~]# systemctl daemon-reload

重启docker服务：

[root@wecloud-test-harbor ~]# systemctl restart docker.service

再次进行登录harbor:

[root@wecloud-test-harbor ~]# docker login 192.168.99.180
Username: admin
Password: 
Login Succeeded

发现harbor登录成功，现在尝试推送镜像到仓库中：

[root@wecloud-test-harbor ~]# docker push 192.168.99.180/k8s/nginx:latest
The push refers to a repository [192.168.99.180/k8s/nginx]
8e5e010ce6ad: Pushed 
974dc1373097: Pushed 
3358360aedad: Pushed 
latest: digest: sha256:22650ea37ad4ccf8472330f141de0712c14fbad2b6792b6aba1687d3d2fa9aa5 size: 948

推送成功后可以在harbor的控制台进行查看：

发现镜像已经上传成功

---

拉取镜像

在另外一个机器上可以尝试拉取刚才上传的nginx镜像（同样需要配置/usr/lib/systemd/system/docker.service，启动项添加192.168.99.180的镜像地址），配置完成后进行拉取操作：

[root@wecloud-test-zhangchi-harbor ~]#  docker pull 192.168.99.180/k8s/nginx:latest
Trying to pull repository 192.168.99.180/k8s/nginx ... 
latest: Pulling from 192.168.99.180/k8s/nginx
2a72cbf407d6: Pull complete 
fefa2faca81f: Pull complete 
080aeede8114: Pull complete 
Digest: sha256:22650ea37ad4ccf8472330f141de0712c14fbad2b6792b6aba1687d3d2fa9aa5

拉取成功，并且镜像列表中可以看到新的镜像信息：

[root@wecloud-test-zhangchi-harbor ~]# docker images 
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
192.168.99.180/k8s/nginx                                latest              7f70b30f2cc6        5 days ago          108.7 MB

---

小结

本章介绍了单节点harbor的部署方案，非常的简单易行，如果部署多节点，更加实用的harbor方案。需要考虑更多的因素，后续将会给大家介绍多节点harbor的方案。