Gitlab 升级、数据迁移

由于用的Gitlab版本比较旧了，计划升级并迁移到新的服务器。
从11.7.0 升级到 16.4.1 版本。

一、升级

1.1 确定升级路径

Gitlab升级是不能跨大版本升级的，根据官方升级路径来操作。
我是通过docker安装的Gitlab，所以只列出了docker的升级方式。

官方链接：https://docs.gitlab.com/ee/update/index.html#upgrade-paths

进去之后配置好目前版本、目标版本，就会给出具体的升级路径。以下是我的升级路径

1.2 升级前后检查

每次升级前后都要做的检查，以保证系统正常运行：

# 状态检查
gitlab-rake gitlab:check
# 数据加密检查
gitlab-rake gitlab:doctor:secrets

查看Gitlab UI：
能否登录、能否显示项目列表、能否拉取，提交代码等。

版本检查：

1.3 备份

GitLab 12.2 或更高版本：

docker exec -t  gitlab-backup create

需要注意的是，该命令只是备份了数据， 配置文件、ssl证书等内容是没有备份的，需要手动备份。
备份文件命名一般是时间戳+_gitlab_backup.tar
路径在backups里面

]# ls /home/git/data/backups/
1697632970_2023_10_18_16.1.5_gitlab_backup.tar

1.4 升级

# 按照顺序升级，升级之后记得检查。
docker run gitlab/gitlab-ce:11.11.8-ce.0
docker run gitlab/gitlab-ce:12.0.12-ce.0
docker run gitlab/gitlab-ce:12.1.17-ce.0
...
docker run gitlab/gitlab-ce:16.4.1-ce.0
docker run gitlab/gitlab-ce:15.4.6-ce.0

日志查看

docker logs -f --tail 100 git_web
docker compose logs --tail=100 -f # 需要在docker-compose.yml同一级目录里面执行

二、数据迁移

只能将备份还原到与创建备份时完全相同的 GitLab 版本和类型 （CE/EE）。
新服务器安装好Gitlab之后，按照下面的顺序执行命令

# Stop the processes that are connected to the database
# docker exec -it  gitlab-ctl stop puma
docker exec -it git_web gitlab-ctl stop puma
#docker exec -it  gitlab-ctl stop sidekiq
docker exec -it git_web gitlab-ctl stop sidekiq

# Verify that the processes are all down before continuing
# docker exec -it  gitlab-ctl status
 docker exec -it git_web  gitlab-ctl status

# Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name
# docker exec -it  gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
docker exec -it git_web   gitlab-backup restore BACKUP=1697632970_2023_10_18_16.1.5

# Restart the GitLab container
# docker restart 
docker restart git_web   

# Check GitLab
# docker exec -it  gitlab-rake gitlab:check SANITIZE=true
docker exec -it git_web  gitlab-rake gitlab:check SANITIZE=true

数据恢复之后，访问git，报错 网站启用了 HSTS，不能访问，重新更新了一下ssl证书可以正常访问了。

三、排错

1. 在 15.4.6 升级到 15.11.13 的时候报错：

[2023-10-17T04:35:35+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
gitlab_rails['smtp_tls'] and gitlab_rails['smtp_enable_starttls_auto'] are mutually exclusive. Set one of them to false. SMTP providers usually use port 465 for TLS and port 587 for STARTTLS.
Running handlers complete
[2023-10-17T04:35:35+00:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 06 seconds
[2023-10-17T04:35:35+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-10-17T04:35:35+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-10-17T04:35:35+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-10-17T04:35:35+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-10-17T04:35:35+00:00] FATAL: RuntimeError: gitlab_rails['smtp_tls'] and gitlab_rails['smtp_enable_starttls_auto'] are mutually exclusive. Set one of them to false. SMTP providers usually use port 465 for TLS and port 587 for STARTTLS. 

大概意思是在 gitlab.rb 的 SMTP 配置中,smtp_tls 和 smtp_enable_starttls_auto 这两个选项不能同时启用。
smtp_tls: 是否使用 TLS 加密SMTP连接。
smtp_enable_starttls_auto: 是否自动使用 STARTTLS命令启用TLS加密。

解决办法：
将其中一个的值改为 false 后重新部署即可。

2. Deploy in progress
   部署之后需要些时间，等几分钟再访问试试看。
   
3. can’t create Thread: Operation not permitted
   详细报错信息：

[2023-10-17T09:01:03+00:00] INFO: Loading cookbooks [[email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]]
Synchronizing cookbooks:

  ================================================================================
  Error Syncing Cookbooks:
  ================================================================================

  Unexpected Error:
  -----------------
  ThreadError: can't create Thread: Operation not permitted

  System Info:
  ------------
  chef_version=17.10.0
  platform=ubuntu
  platform_version=22.04
  ruby=ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
  program_name=/opt/gitlab/embedded/bin/cinc-client
  executable=/opt/gitlab/embedded/bin/cinc-client


  Running handlers:
[2023-10-17T09:01:03+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

can't create Thread: Operation not permitted

[2023-10-17T09:01:03+00:00] ERROR: Report handler GitLabHandler::Exception raised #
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/cookbooks/package/libraries/handlers/gitlab.rb:30:in `report'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:296:in `run_report_unsafe'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:284:in `run_report_safely'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:155:in `block in run_exception_handlers'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:153:in `each'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:153:in `run_exception_handlers'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/handler.rb:164:in `block in '
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:443:in `block in run_failed'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:442:in `each'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:442:in `run_failed'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:306:in `rescue in run'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:299:in `run'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `load'
[2023-10-17T09:01:03+00:00] ERROR: /opt/gitlab/embedded/bin/cinc-client:25:in `
'
  Running handlers complete
[2023-10-17T09:01:03+00:00] ERROR: Exception handlers complete
  Infra Phase failed. 0 resources updated in 01 seconds
[2023-10-17T09:01:03+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-10-17T09:01:03+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-10-17T09:01:03+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-10-17T09:01:03+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-10-17T09:01:03+00:00] FATAL: ThreadError: can't create Thread: Operation not permitted

官方回复：

# 链接：https://docs.gitlab.com/ee/install/docker.html#threaderror-cant-create-thread-operation-not-permitted

This error occurs when running a container built with newer versions on a host that doesn’t have support for the new clone3 function. In GitLab 16.0 and later, the container image includes the Ubuntu 22.04 Linux package which is built with this newer .glibcglibc

This problem is fixed with newer container runtime tools like Docker 20.10.10.

To resolve this issue, update Docker to version 20.10.10 or later.

解决方案：升级docker到 20.10.10 或者更高级版本即可。

4. 恢复数据的时候报错

2023-10-19 15:47:50 +0800 -- Unpacking backup ...
tar: 1697701149_2023_10_19_16.1.5_gitlab_backup.tar: Cannot open: Permission denied
tar: Error is not recoverable: exiting now
2023-10-19 15:47:50 +0800 -- Unpacking backup failed
2023-10-19 15:47:50 +0800 -- Deleting backup and restore PID file ... done

解决方案：权限不足导致的，进入容器修改权限就可以了

$ docker exec -it git_web bash

# cd /var/opt/gitlab/backups
# ll -h
-rw-------  1 root root 1.7G Oct 19 07:45 1697701149_2023_10_19_16.1.5_gitlab_backup.tar

# chown git:git 1697701149_2023_10_19_16.1.5_gitlab_backup.tar
# ll
-rw-------  1 git  git  1743431680 Oct 19 07:45 1697701149_2023_10_19_16.1.5_gitlab_backup.tar