记录harbor 1.5升级到1.9

背景：

---

简明：

内网环境harbor1.5.0升级要1.9

要点：

（1）harbor数据库在v1.6版本做了很大的升级。数据库从原来的MySQL升级为postgresql数据库。根据官方文档的解释，旧版本的harbor升级到最新的v1.9,需先将harbor升级到v1.6,再升级到更高的版本。
（2）从v1.8.0开始，Harbor的配置已更改为.yml文件，迁移器将配置文件从harbor.cfg转换为harbor.yml。
（3）从v1.6.0开始，Harbor会在启动时自动尝试迁移数据库，因此如果从v1.6.0或更高版本升级，则无需调用迁移器工具来迁移数据库，只需要更新cfg文件即可。
（4）从v1.7.0开始，支持Helm Chart部署。

方案：

先从1.5升级到1.6，再从1.6升级到1.7，然后升级到1.9（按照官方文档的升级路线）

开始

---

一、harbor1.5.0升级到harbor1.6.3（既要升级db也要升级cfg）

（1）停下在运行的harbor

[root@bogon ~]# cd harbor
[root@bogon harbor]# ls
common                    docker-compose.notary.yml  ha          harbor.v1.5.0.tar.gz  LICENSE  prepare
docker-compose.clair.yml  docker-compose.yml         harbor.cfg  install.sh            NOTICE
[root@bogon harbor]# docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                    PORTS                                                              NAMES
717ef68de036        vmware/harbor-jobservice:v1.5.0        "/harbor/start.sh"       40 minutes ago      Up 40 minutes                                                                                harbor-jobservice
ee95661aa905        vmware/nginx-photon:v1.5.0             "nginx -g 'daemon ..."   40 minutes ago      Up 38 minutes (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp   nginx
85aa0167d66a        vmware/harbor-ui:v1.5.0                "/harbor/start.sh"       40 minutes ago      Up 40 minutes (healthy)                                                                      harbor-ui
cb256cbf09ed        vmware/clair-photon:v2.0.1-v1.5.0      "/docker-entrypoin..."   40 minutes ago      Up 40 minutes (healthy)   6060-6061/tcp                                                      clair
e3aa51500bdc        vmware/postgresql-photon:v1.5.0        "/entrypoint.sh po..."   40 minutes ago      Up 40 minutes (healthy)   5432/tcp                                                           clair-db
dce4338d4071        vmware/registry-photon:v2.6.2-v1.5.0   "/entrypoint.sh se..."   40 minutes ago      Up 40 minutes (healthy)   5000/tcp                                                           registry
43e6fbebc160        vmware/harbor-db:v1.5.0                "/usr/local/bin/do..."   40 minutes ago      Up 40 minutes (healthy)   3306/tcp                                                           harbor-db
eca25eea5b18        vmware/harbor-adminserver:v1.5.0       "/harbor/start.sh"       40 minutes ago      Up 40 minutes (healthy)                                                                      harbor-adminserver
d68e94c20695        vmware/redis-photon:v1.5.0             "docker-entrypoint..."   40 minutes ago      Up 40 minutes             6379/tcp                                                           redis
280da2ce97a9        vmware/harbor-log:v1.5.0               "/bin/sh -c /usr/l..."   40 minutes ago      Up 40 minutes (healthy)   127.0.0.1:1514->10514/tcp                                          harbor-log
[root@bogon harbor]# 
[root@bogon harbor]# docker-compose -f docker-compose.yml -f docker-compose.clair.yml down -v

（2）备份harbor目录

[root@bogon harbor]# cd ..
[root@bogon lanjian]# cp -r harbor harbor1.5.0-bak

（3）下载harbor 1.6.3镜像、数据迁移工具，以及1.5.0数据迁移工具

Get the lastest Harbor release package from Github

[root@localhost lanjian]# docker pull goharbor/harbor-migrator:v1.6.0
Trying to pull repository docker.io/goharbor/harbor-migrator ... 
v1.6.0: Pulling from docker.io/goharbor/harbor-migrator
51be32cd3c9d: Pull complete 
7e261d84cd9e: Pull complete 
f52c9d206429: Pull complete 
e399eeb55d79: Pull complete 
Digest: sha256:a8398de33c60bba7ad93ddcf977791e187e8b8553214f5589c749959bd265712
Status: Downloaded newer image for docker.io/goharbor/harbor-migrator:v1.6.0
[root@localhost lanjian]# docker images |grep harbor
docker.io/goharbor/harbor-migrator     v1.6.3              dac56e1d921f        11 months ago       799 MB
docker.io/goharbor/harbor-migrator     v1.6.0              22775c4e4066        14 months ago       803 MB
vmware/harbor-log                      v1.5.0              62bb6b8350d9        18 months ago       200 MB
vmware/harbor-jobservice               v1.5.0              aca9fd2e867f        18 months ago       194 MB
vmware/harbor-ui                       v1.5.0              1055166068d0        18 months ago       212 MB
vmware/harbor-adminserver              v1.5.0              019bc4544829        18 months ago       183 MB
vmware/harbor-db                       v1.5.0              82354dcf564f        18 months ago       526 MB
vmware/harbor-migrator                 v1.5.0              466c57ab0dc3        18 months ago       1.16 GB

（4）备份cfg和db

[root@localhost harbor]# docker run -it --rm -e DB_USR=root -e DB_PWD=root123 -v /data/database:/var/lib/mysql -v /home/lanjian/harbor/harbor.cfg:/harbor-migration/harbor-cfg/harbor.cfg -v /home/lanjian/backup:/harbor-migration/backup goharbor/harbor-migrator:v1.6.3 backup

（5）升级数据库框架（将MySQL框架升级为postgresql）并同步数据

[root@localhost harbor]# docker run -it --rm -e DB_USR=root -e DB_PWD=root123 -v /data/database:/var/lib/mysql -v /home/lanjian/harbor/harbor.cfg:/harbor-migration/harbor-cfg/harbor.cfg goharbor/harbor-migrator:v1.6.0 up

（6）升级harbor镜像及cfg并拷贝过去

[root@localhost lanjian]# mkdir harbor-1.6.3
[root@localhost lanjian]# cd 下载
[root@localhost 下载]# tar zxvf harbor-offline-installer-v1.6.3.tgz -C ../harbor1.6.3/
[root@localhost 下载]# cd ../harbor1.6.3/
[root@localhost harbor1.6.3]# ls
harbor
[root@localhost harbor1.6.3]# cd harbor/
[root@localhost harbor]# ls
common                          docker-compose.clair.yml   docker-compose.yml  harbor.cfg            install.sh  NOTICE               prepare
docker-compose.chartmuseum.yml  docker-compose.notary.yml  ha                  harbor.v1.6.3.tar.gz  LICENSE     open_source_license
[root@localhost harbor]#  docker run -it --rm -v /root/backup/harbor.cfg:/harbor-migration/harbor-cfg/harbor.cfg goharbor/harbor-migrator:v1.6.0 --cfg up
[root@localhost backup]# cp harbor.cfg /home/lanjian/harbor/
cp：是否覆盖"/home/lanjian/harbor/harbor.cfg"？ yes
[root@localhost backup]# 

（7）启动harbor1.6.3并验证

[root@localhost harbor]# ./install.sh

验证下版本是否完成升级，以及用户、镜像是否都正常

二、harbor1.6.3升级到1.7.0（相当于只需要升级cfg文件不用管db）

（1）同样的，停下正在运行的harbor

[root@localhost lanjian]# cd harbor
[root@localhost harbor]# docker-compose -f docker-compose.yml -f docker-compose.clair.yml down

（2）备份数据

[root@localhost lanjian]# mkdir harbor1.6.0-bak
[root@localhost lanjian]# mv harbor harbor1.6.0-bak/harbor
[root@localhost lanjian]# cp -r /data/database harbor1.6.0-bak/

（3）下载harbor v1.7.0离线安装包解压和migrate容器镜像

[root@localhost 下载]# tar zxvf harbor-offline-installer-v1.7.0.tgz
[root@localhost 下载]# mv harbor ../harbor
[root@localhost lanjian]# docker pull goharbor/harbor-migrator:v1.7.0

（4）升级harbor.cfg

官方文档：NOTE: The ${harbor_cfg} will be overwritten, you must move it to your installation directory after migration.
即是在备份目录里直接升级cfg文件的，然后再拷贝到新的harbor目录下，所以为防止意外，先对harbor.cfg再备份一次后再升级
PS：这里挂载volume时需要注意宿主机和容器都需要绝对路径，不然会报错，宿主机虽然可以相对路径，但其实所谓的相对路径指的是/var/lib/docker/volumes/，与宿主机的当前目录无关。

[root@localhost harbor]# cd ../harbor1.6.0-bak/
[root@localhost harbor]# cp harbor.cfg harbor.cfg.bak
[root@localhost harbor]# docker run -it --rm -v /home/lanjian/harbor1.6.0-bak/harbor/harbor.cfg:/harbor-migration/harbor-cfg/harbor.cfg docker.io/goharbor/harbor-migrator:v1.7.0 --cfg up
Please backup before upgrade, 
Enter y to continue updating or n to abort: y
The path of the migrated harbor.cfg is not set, the input file will be overwritten.
input version: 1.6.0, migrator chain: ['1.7.0']
migrating to version 1.7.0
Written new values to /harbor-migration/harbor-cfg/harbor.cfg

我们看到当前cfg文件已经被修改了，不是1.6的了，将其拷贝到1.7的安装路径下面即可

[root@localhost harbor]# cat harbor.cfg|grep version
#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version = 1.7.0
#Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
[root@localhost harbor]# mv harbor.cfg /home/lanjian/harbor/

（5）启动harbor v1.7.0并打开UI确认升级完成情况

[root@localhost harbor]# cd /home/lanjian/harbor
[root@localhost harbor]# ./install.sh

如果需要安装clair则带上参数 --with-clair哈

三、harbor 1.7升级到1.9（升级cfg为yml）

特别注意两点：
With the introduction of storage and artifact quotas in version 1.9.0, migration from 1.7.x and 1.8.x might take a few minutes. This is because the core walks through all blobs in the registry and populates the database with information about the layers and artifacts in projects.
因为引入了storage and artifact quotas，这次升级会比较费时间，核心会遍历仓库和数据库中的层信息

With the introduction of storage and artifact quotas in version 1.9.0, replication between version 1.9.0 and a previous version of Harbor does not work. You must upgrade all Harbor nodes to 1.9.0 if you have configured replication between them.
因为引入了storage and artifact quotas，所以同步复制只能再1.9的版本之间进行，如果其中一个仓库低于1.9则无法进行同步

（1）停止正在运行的harbor

[root@bogon harbor]# docker-compose -f docker-compose.yml -f docker-compose.clair.yml down -v

（2）备份数据

[root@bogon lanjian]# mkdir harbor1.7.0-bak
[root@bogon lanjian]# mv harbor harbor1.7.0-bak/harbor
[root@bogon lanjian]# cp -r /data/database harbor1.7.0-bak/

（3）下载harbor 1.9.0安装包解压及migrator镜像

[root@bogon lanjian]# docker pull goharbor/harbor-migrator:v1.9.0
[root@bogon 下载]# tar zxvf harbor-offline-installer-v1.9.0.tgz -C /home/lanjian/
[root@bogon lanjian]# cd harbor
[root@bogon harbor]# ls
harbor.v1.9.0.tar.gz  harbor.yml  install.sh  LICENSE  prepare

（4）升级cfg为yml文件

[root@bogon harbor]# docker run -it --rm -v /home/lanjian/harbor1.7.0-bak/harbor/harbor.cfg:/harbor-migration/harbor-cfg/harbor.yml -v /home/lanjian/harbor/harbor.yml:/harbor-migration/harbor-cfg-out/harbor.yml docker.io/goharbor/harbor-migrator:v1.9.0 --cfg up       

发现有报错：jinja2.exceptions.UndefinedError: ‘chart’ is undefined

Please backup before upgrade, 
Enter y to continue updating or n to abort: y
Command for config file migration: python ./cfg/run.py --input /harbor-migration/harbor-cfg/harbor.yml --output /harbor-migration/harbor-cfg-out/harbor.yml
input version: 1.7.0, migrator chain: ['1.8.0', '1.9.0']
migrating to version 1.8.0
migrating to version 1.9.0
Traceback (most recent call last):
  File "./cfg/run.py", line 70, in <module>
    main()
  File "./cfg/run.py", line 46, in main
    m.migrate(curr_input_path, curr_output_path)
  File "/harbor-migration/cfg/migrator_1_9_0/__init__.py", line 21, in migrate
    f.write(tpl.render(**config_dict))
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/harbor-migration/cfg/migrator_1_9_0/harbor.yml.jinja", line 122, in top-level template code
    absolute_url: {{ chart.absolute_url if chart.absolute_url == 'enabled' else 'disabled' }}
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 430, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'chart' is undefined

通过查看issue 9174以及issue 9146：
发现很多人都报了这个错误，原因时由于 1.7.x 版本的 harbar.cfg 中不存在下面的相关的配置（在1.8版本开始在模板中使用了chart，且为必须项Required Parameters for Harbor Deployment）：

chart:
  absolute_url: disabled

解决方案有两个：
（1）先升级到1.8.x版本，并在harbor.yml中加入上述chart配置。
（2）替换官方的migrator镜像为issue中大神thatsk提供的镜像。

为了避免麻烦，我直接选择方案二，替换掉官方的同步迁移镜像

[root@bogon harbor]# docker pull thatsk/harbor-migrator:v1.9.0
[root@bogon harbor]# docker run -it --rm -v /home/lanjian/harbor1.7.0-bak/harbor/harbor.cfg:/harbor-migration/harbor-cfg/harbor.yml -v /home/lanjian/harbor/harbor.yml:/harbor-migration/harbor-cfg-out/harbor.yml docker.io/thatsk/harbor-migrator:v1.9.0 --cfg up                        
Please backup before upgrade, 
Enter y to continue updating or n to abort: y
Command for config file migration: python ./cfg/run.py --input /harbor-migration/harbor-cfg/harbor.yml --output /harbor-migration/harbor-cfg-out/harbor.yml
input version: 1.7.0, migrator chain: ['1.8.0', '1.9.0']
migrating to version 1.8.0
migrating to version 1.9.0
Written new values to /harbor-migration/harbor-cfg-out/harbor.yml

升级成功

（5）启动harbor并检查UI

[root@bogon harbor]# ./install.sh --with-clair
[Step 0]: checking installation environment ...
✖ Need to upgrade docker package to 17.06.0+.
[root@bogon harbor]# docker --version
Docker version 1.13.1, build b2f74b2/1.13.1

提示docker版本太低，需要升级，那么就升吧，步骤：
先停止docker，删除旧版本，然后安装yum管理工具，然后添加国内镜像源，查看版本，安装指定的docker版本以及docker-compose版本：

[root@bogon bin]# systemctl stop docker
[root@bogon bin]# rpm -qa | grep docker
docker-1.13.1-94.gitb2f74b2.el7.centos.x86_64
docker-client-1.13.1-94.gitb2f74b2.el7.centos.x86_64
docker-common-1.13.1-94.gitb2f74b2.el7.centos.x86_64
[root@bogon bin]# yum remove docker-common-1.13.1-94.gitb2f74b2.el7.centos.x86_64

[root@bogon bin]# yum install -y yum-utils
[root@bogon bin]# yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
已加载插件：fastestmirror, langpacks
adding repo from: https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
grabbing file https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@bogon bin]# yum list docker-ce --showduplicates|sort -r
已加载插件：fastestmirror, langpacks
可安装的软件包
 * updates: mirrors.neusoft.edu.cn
Loading mirror speeds from cached hostfile
 * extras: mirrors.neusoft.edu.cn
 * epel: mirror.lzu.edu.cn
docker-ce.x86_64            3:19.03.5-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.4-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.3-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.2-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.1-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.0-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.9-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.8-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.7-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.6-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.5-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.4-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.3-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.2-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.1-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.0-3.el7                     docker-ce-stable
docker-ce.x86_64            18.06.3.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.2.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.1.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.06.0.ce-3.el7                    docker-ce-stable
docker-ce.x86_64            18.03.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            18.03.0.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.12.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.12.0.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.09.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.09.0.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.06.2.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.06.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.06.0.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.03.3.ce-1.el7                    docker-ce-stable
docker-ce.x86_64            17.03.2.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.03.1.ce-1.el7.centos             docker-ce-stable
docker-ce.x86_64            17.03.0.ce-1.el7.centos             docker-ce-stable
 * base: mirrors.nju.edu.cn
[root@bogon bin]# yum install docker-ce.x86_64.18.03.0.ce-1.el7.centos
[root@bogon bin]# systemctl start docker
[root@bogon bin]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@bogon bin]# docker version
Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:09:15 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:13:03 2018
  OS/Arch:      linux/amd64
  Experimental: false

PS：这里又趟坑了，在yum-config-manager时报错：

File "/bin/yum-config-manager", line 135
    except yum.Errors.RepoError, e:
                               ^
SyntaxError: invalid syntax

这时python升级为3.x后的问题，yum-config-manager还是用Python2写的，所以我们需要替换掉这个文件的Python版本：

[root@bogon bin]# head -n 1 /bin/yum-config-manager
#!/usr/bin/python -tt
[root@bogon bin]# ll /usr/bin/ |grep -i pytho
-rwxr-xr-x.   1 root root       11312 11月 14 2018 abrt-action-analyze-python
lrwxrwxrwx    1 root root          16 9月  29 23:10 python -> /usr/bin/python3
lrwxrwxrwx    1 root root           9 4月  27 2019 python2 -> python2.7
-rwxr-xr-x    1 root root        7216 4月   9 2019 python2.7
lrwxrwxrwx.   1 root root          24 3月   4 2019 python3 -> /usr/local/bin/python3.6
lrwxrwxrwx    1 root root          24 3月  14 2019 python_backup_20190314 -> /etc/alternatives/python
lrwxrwxrwx.   1 root root           7 3月   3 2019 python_old -> python2
[root@bogon bin]# vim /bin/yum-config-manager
[root@bogon bin]# head -n 1 /bin/yum-config-manager
#!/usr/bin/python2 -tt

继续升级docker-compose，这个简单些，可以通过pip安装：

[root@bogon harbor]# rm /usr/local/bin/docker-compose
[root@bogon harbor]# pip install docker-compose==1.18.0
[root@bogon harbor]# docker-compose version
docker-compose version 1.18.0, build 8dd22a9
docker-py version: 2.7.0
CPython version: 3.6.0
OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

好了，我们终于可以继续启动harbor了：

[root@bogon harbor]# ./install.sh --with-clair  
...
...
✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at http://192.168.137.7. 
For more details, please visit https://github.com/goharbor/harbor .

完成，检查下UI、镜像和log，升级成功！经历了那么多版本，终于完成了，奈斯！