Gitlab整理汇总（安装、非22端口克隆、升级及遇到问题、备份恢复）

欢迎关注个人公众号 DailyJobOps

原文地址：Gitlab整理汇总（安装、非22端口克隆、升级及遇到问题、备份恢复）

1、安装

如果不采用自建数据库，而是gitlab自集成的数据库，那么安装很简单

1.1、采用外置自建数据库

• 需要先成功安装数据库，确保数据库可以正常访问
• 在数据库中创建对应的数据库、账号、密码，然后授权
• 下载rpm包，执行yum install gitlab-ce.xxx.rpm
• 修改配置文件 /etc/gitlab/gitlab.rb
• 重载配置 gitlab-ctl reconfigure
• 重启服务 gitlab-ctl restart

1.2、采用Gitlab自集成数据库

• 下载rpm包，执行yum install gitlab-ce.xxx.rpm
• 重载配置 gitlab-ctl reconfigure
• 重启服务 gitlab-ctl restart

2、非22端口克隆

常规git仓库clone的方式为：

• http方式

  git clone http://gitlab.xxx.com/devops-group/devops-dbp-platform.git
  

• ssh方式（默认22端口）

  git clone [email protected]:devops-group/devops-dbp-platform.git
  

如果gitlab安装的ssh端口（其实就是安装主机的ssh服务的启动端口）是非22端口，比如是2022端口，则ssh方式克隆地址变更为

git clone ssh://[email protected]:2022/devops-group/devops-dbp-platform.git

注意和上面默认22端口的地址做对比，看看不一样的地方在哪里

---

3、升级

跨大版本升级的原则是：

先升级到下一个大版本中的最高版本，没有问题之后再如此继续直到升级到最后一个大版本中需要的版本

比如，实际环境中当前版本是 8.1.4 那么升级路线是 8.1.4 -> 8.17.8 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12

升级过程中需要注意一个数据库的问题，在 12.1 及之后版本，官方移除了对MySQL的支持，数据库使用PostgreSQL

image.png

升级步骤
1、先停止数据写入（当然如果在一个确定不会有写入的时候，比如下班之后，获取停止域名解析等可以不进行如下操作）

sudo gitlab-ctl stop unicorn 
sudo gitlab-ctl stop puma 
sudo gitlab-ctl stop sidekiq 

2、按照整理出来的升级路径下载对应的rpm包
这里推荐 https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7
3、更新升级

# 安装更新rpm包
rpm -Uvh gitlab-ce-10.8.7-ce.0.el7.x86_64.rpm
# 重新生效配置
gitlab-ctl reconfigure
# 重启服务
gitlab-ctl restart

4、验证当前版本是否正常
5、重复如上过程，升级到下一个版本

---

4、备份及恢复

备份配置

# 配置相关配置
# vim /etc/gitlab/gitlab.rb
# gitlab 访问域名
external_url 'http://192.168.8.127'  
# 备份配置
gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/data/gitlab/backups" 
gitlab_rails['backup_archive_permissions'] = 0644  
# 备份有效期，单位为秒，比如如下7天 ，也可以根据自己实际情况来做异地转存
# 这里的有效期是针对本地存储，而非remote storage，比如阿里云的OSS
gitlab_rails['backup_keep_time'] = 604800  

备份

gitlab-rake gitlab:backup:create

image.png

注意上述告警提示，需要自己备份如下两个文件，恢复的时候也确保这两个位置有对应的问题

• /etc/gitlab/gitlab.rb
• /etc/gitlab/gitlab-secrets.json

远端备份

另外从上述备份记录中注意到 Uploading backup archive to remote storage ... skipped， 从7.4版本支持直接转存本地备份到远程（Starting with GitLab 7.4 you can let the backup script upload the ‘.tar’ file it creates）

举例

gitlab_rails['backup_upload_connection'] = {
  'provider' => 'Aliyun',
  'aliyun_accesskey_id' => 'Access Key',
  'aliyun_secretekey_id' => 'Secret Key',
  'aliyun_oss_endpoint' => 'http://oss-cn-beijing-internal.aliyuncs.com',
  'aliyun_oss_bucket' => 'devops-backup',
  'aliyun_oss_location' => 'beijing'
}
# 对象存储bucket中的子目录
gitlab_rails['backup_upload_remote_directory'] = 'gitlab'

注意：一般备份不用长久保存，备份在OSS，为了节约成本，可以配置阿里云OSS bucket的生命周期，让其保留最近30天的备份即可

具体参考
https://docs.gitlab.com/12.10/ee/administration/object_storage.html

恢复

注意事项：
1、恢复的文件必须是在配置的备份路径下，比如这里的 /data/gitlab/backups
2、恢复的时候指定备份的序号，比如 637134260_2021_11_17_12.0.12_gitlab_backup.tar 这里取637134260_2021_11_17_12.0.12
3、备份文件的权限必须是 644
4、检查恢复状态 gitlab-rake gitlab:check SANITIZE=true

sudo gitlab-ctl stop unicorn 
sudo gitlab-ctl stop puma 
sudo gitlab-ctl stop sidekiq 
# Verify 
sudo gitlab-ctl status
# restore
gitlab-rake gitlab:backup:restore BACKUP=1629369869

5、内存消耗问题

#进程超时时间
unicorn['worker_timeout'] = 60
#进程数
unicorn['worker_processes'] = 10
#进程最小内存 200KB
unicorn['worker_memory_limit_min'] = "200 * 1 << 20"
#进程最大内存 300MB 
unicorn['worker_memory_limit_max'] = "300 * 1 << 20"
#并发数
sidekiq['concurrency'] = 16
#数据库缓存
postgresql['shared_buffers'] = "256MB"
#数据库并发数
postgresql['max_worker_processes'] = 8

6、升级过程问题整理：

1、mysql2 adapter 问题

报错信息：

NameError: uninitialized constant Mysql2::Client::SECURE_CONNECTION

解决方案：

升级到0.3.20 然后修改Gemfile.lock

-> https://github.com/brianmario/mysql2/issues/711
-> Use mysql2 0.3.17 or higher for MySQL 5.7 compatibility.

/opt/gitlab/embedded/bin/gem install -i/opt/gitlab/embedded/service/gem/ruby/2.1.0 mysql2 -v 0.3.20
vim /opt/gitlab/embedded/service/gitlab-rails/Gemfile.lock
mysql2 (0.3.20)

2、安装gitlab-ce之后，且在mysql2 adapter正常的情况，需要执行 gitlab-rake setup 进行数据库表安装

3、gitlab基于备份进行恢复报错

报错信息：

• ERROR 1227 (42000) at line 27: Access denied; you need (at least one of) the SUPER privilege(s) for this operation

解决方案：

管理员登录数据库然后给gitlab用户赋权 `**grant super on *.* to 'gitlab'@'localhost';**`

报错信息

• ERROR 1840 (HY000) at line 33: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty.

解决方案：

管理员登录数据库 执行 **reset master;**  因为gitlab使用的MySQL是本地自建，单节点不需要主从，故reset不影响；
还有一种方式是注销掉 备份文件中涉及到 `@@GLOBAL.GTID_PURGED`

4、权限报错

报错信息

Failed asserting that mode permissions on "/data/git-data/repositories" is 2770

解决方案

chmod 2770 /data/git-data/repositories

5、Bundler::GemNotFound

报错信息

Bundler::GemNotFound: Your bundle is locked to mysql2 (0.3.20), but that version could not be found in any of the sources listed in your Gemfile. If you haven't changed sources, that means the author of mysql2 (0.3.20) has removed it. You'll need to update your bundle to a different version of mysql2 (0.3.20) that hasn't been removed in order to install

昨天按照

/opt/gitlab/embedded/bin/gem install -i/opt/gitlab/embedded/service/gem/ruby/2.1.0 mysql2 -v 0.3.20

已经安装0.3.20版本，但是升级之后提示不存在，在 /opt/gitlab/embedded/service/gem/ruby/2.1.0/extensions/x86_64-linux/2.1.0/也看到对应的版本存在。
猜测是因为升级之后安装方式是否不一样

• /opt/gitlab/embedded/bin/gem list |grep mysql → 提示没有MySQL
• 重新安装，只不过不指定安装目录，/opt/gitlab/embedded/bin/gem install mysql2 -v 0.3.20
• 再次 /opt/gitlab/embedded/bin/gem list |grep mysql 提示安装成功
• 通过 find /opt/gitlab/ -name 'mysql2-0.3.20' 查找发现，新版本应该是从 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/ 下找扩展

image.png

6、升级到 9.5.10 报错

报错信息：

LoadError: cannot load such file -- peek-mysql2

解决方案：

[root@devops-gitlab-vpc ~]# /opt/gitlab/embedded/bin/gem install peek-mysql2
Fetching: peek-mysql2-1.2.0.gem (100%)
Successfully installed peek-mysql2-1.2.0
Parsing documentation for peek-mysql2-1.2.0
Installing ri documentation for peek-mysql2-1.2.0
Done installing documentation for peek-mysql2 after 0 seconds
1 gem installed

但是在安装了之后还是提示 cannot load such file -- peek-mysql2
最终经过排查和试验，在 Gemfile.lock 中 修改

gem 'peek-mysql2', '~> 1.1.0', group: :mysql 为 gem 'peek-mysql2' 和 之前 mysql2 的保持一致，重新 gitlab:check 该错误消失

7、升级到 9.5.10 报错

报错信息：

Mysql2::Error: SELECT command denied to user 'gitlab'@'localhost' for table 'user': SHOW FULL FIELDS FROM `mysql`.`user`

解决方案

mysql> grant select on mysql.* to gitlab@'localhost'; 
Query OK, 0 rows affected, 1 warning (0.00 sec)

报错信息：

Mysql2::Error: Thread stack overrun:  14000 bytes used of a 131072 byte stack, and 128000 bytes needed.  Use 'mysqld --thread_stack=#' to specify a bigger stack.: UPDATE `web_hooks` SET `job_events` = `web_hooks`.`build_events` WHERE `web_hooks`.`id` >= 1 AND `web_hooks`.`id` < 2

thread_stack=256K

Mysql2::Error: Duplicate column name 'job_events': ALTER TABLE web_hooks ADD job_events tinyint(1)

解决方案：

mysql> select job_events from web_hooks ;
+------------+
| job_events |
+------------+
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
|       NULL |
+------------+
18 rows in set (0.00 sec)

mysql> alter table web_hooks drop column job_events ;
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

报错信息：

Mysql2::Error: Trigger already exists: CREATE TRIGGER trigger_688beaaec90d_insert / Mysql2::Error: Trigger already exists: CREATE TRIGGER trigger_688beaaec90d_update

解决方案：

mysql> show triggers ;
+-----------------------------+--------+-----------+-------------------------------------------+--------+------------------------+-------------------+------------------+----------------------+----------------------+--------------------+
| Trigger                     | Event  | Table     | Statement                                 | Timing | Created                | sql_mode          | Definer          | character_set_client | collation_connection | Database Collation |
+-----------------------------+--------+-----------+-------------------------------------------+--------+------------------------+-------------------+------------------+----------------------+----------------------+--------------------+
| trigger_688beaaec90d_insert | INSERT | web_hooks | SET NEW.`job_events` = NEW.`build_events` | BEFORE | 2021-08-12 11:40:22.88 | STRICT_ALL_TABLES | gitlab@localhost | utf8                 | utf8_general_ci      | utf8_general_ci    |
| trigger_688beaaec90d_update | UPDATE | web_hooks | SET NEW.`job_events` = NEW.`build_events` | BEFORE | 2021-08-12 11:40:22.88 | STRICT_ALL_TABLES | gitlab@localhost | utf8                 | utf8_general_ci      | utf8_general_ci    |
+-----------------------------+--------+-----------+-------------------------------------------+--------+------------------------+-------------------+------------------+----------------------+----------------------+--------------------+
2 rows in set (0.00 sec)

mysql> drop trigger trigger_688beaaec90d_insert ;
Query OK, 0 rows affected (0.01 sec)

mysql> drop trigger trigger_688beaaec90d_update ;
Query OK, 0 rows affected (0.01 sec)

报错信息：

Mysql2::Error: Statement violates GTID consistency: CREATE TABLE ... SELECT.: CREATE TABLE issue_assignees AS
SELECT assignee_id AS user_id, id AS issue_id FROM issues WHERE assignee_id IS NOT NULL

问题分析：

MySQL5.6及以上的版本，开启了 enforce_gtid_consistency=true 功能导致的，MySQL官方解释说当启用 enforce_gtid_consistency 功能的时候，MySQL只允许能够保障事务安全，并且能够被日志记录的SQL语句被执行，像create table … select 和 create temporarytable语句，以及同时更新事务表和非事务表的SQL语句或事务都不允许执行。

解决方案：

# GTID_MODE = ON requires ENFORCE_GTID_CONSISTENCY = ON.
gtid_mode = 0
enforce_gtid_consistency = 0

8、升级 10.8.7 报错

报错信息

• mkmf.rb can't find header files for ruby at /opt/gitlab/embedded/lib/ruby/include/ruby.h

参考 https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3635

解决方案

[root@devops-gitlab-vpc embedded]# cd /opt/gitlab/embedded/lib/ruby/include
-bash: cd: /opt/gitlab/embedded/lib/ruby/include: No such file or directory
[root@devops-gitlab-vpc embedded]# cd /opt/gitlab/embedded/lib/ruby/
[root@devops-gitlab-vpc ruby]# ll
total 12
drwxr-xr-x 31 root root 4096 Aug 12 12:18 2.3.0
drwxr-xr-x  3 root root 4096 Aug 12 12:19 gems
drwxr-xr-x  3 root root 4096 Jul 26  2018 site_ruby
 
 
# complie ruby 2.3.0
wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.0.tar.gz
tar -zxvf ruby-2.3.0.tar.gz
cd ruby-2.3.0
./configure
make
 
cp -ra include /opt/gitlab/embedded/lib/ruby
 
cp -ra .ext/include/x86_64-linux /opt/gitlab/embedded/lib/ruby/include/ruby/
 
 
# Error: /opt/gitlab/embedded/lib/ruby/include/ruby/ruby.h:24:25: fatal error: ruby/config.h: No such file or directory
cd /opt/gitlab/embedded/lib/ruby/include/
cp ruby/x86_64-linux/ruby/config.h ruby/
 
 
# Error: make: *** No rule to make target `/include/x86_64-linux/ruby/config.h', needed by `client.o'.  Stop.
mkdir -p /include/x86_64-linux/ruby/
cp /opt/gitlab/embedded/lib/ruby/include/ruby/config.h /include/x86_64-linux/ruby/
 
 
最后在安装0.4.10
/opt/gitlab/embedded/bin/gem install mysql2 -v 0.4.10
 
 
# Error: LoadError: cannot load such file -- peek-mysql2
/opt/gitlab/embedded/bin/gem install peek-mysql2 -v 1.1.0
vim /opt/gitlab/embedded/service/gitlab-rails/Gemfile 
gem 'peek-mysql2', '~> 1.1.0', group: :mysql -> gem 'peek-mysql2', '~> 1.1.0'

再次 gitlab-ctl reconfigure

报错信息：

ERROR: Encountered unsupported config key 'gitlab_git_http_server' in /etc/gitlab/gitlab.rb.

gitlab_git_http_server['repo_root'] = "xxx" → gitlab_workhorse['repo_root'] = "xxx"

Error: Mysql2::Error: Illegal mix of collations (utf8_general_ci,IMPLICIT) and  (utf8_unicode_ci,IMPLICIT) for operation '=':         INSERT INTO user_synced_attributes_metadata   (user_id, provider, email_synced)

报错分析：

user_synced_attributes_metadata 采用默认字符集和字符集排序 utf8 和  utf8_general_ci （show variables like '%collation%';）

users 表和很多字段都是  utf8_unicode_ci 字符集排序，

解决方案：

根据上面的报错修复 email_provider 字符集排序。

mysql> alter table users modify email_provider varchar(255) COLLATE utf8_general_ci  DEFAULT NULL; 
Query OK, 115 rows affected (0.06 sec) Records: 115  Duplicates: 0  Warnings: 0

然后有问题，就把 users 表整个字符集都修改了

mysql> alter table users convert to character set utf8 collate utf8_general_ci ;

最后还是存在字符集排序的问题，修改整个数据库中不是 utf8_general_ci 的所有表 字符集排序

9、升级11.11.8 报错

报错信息

compiling statement.c
linking shared-object mysql2/mysql2.so
/bin/ld: unrecognized option '--compress-debug-sections=zlib'
/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status
make: *** [mysql2.so] Error 1

排查ld依赖

[root@devops-gitlab-vpc ruby-2.5.0]# ld --help |grep compress
[root@devops-gitlab-vpc ruby-2.5.0]# ld -v
GNU ld version 2.25.1-32.base.el7_4.1
[root@devops-gitlab-vpc ~]# ld -v
GNU ld version 2.27-44.base.el7

注意：
1、提示无 xxx 包的时候：

1、建议采用 /opt/gitlab/embedded/bin/gem install xxxx -v x.y.z 安装
2、至于 x.y.z 应该是多少，建议先从 Gemfile.lock 中获取 
grep xxx /opt/gitlab/embedded/service/gitlab-rails/Gemfile.lock

2、过程中需要不同版本的ruby相关头文件
wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.0.tar.gz
wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.0.tar.gz
这里在不同的版本安装的时候，清理之前版本的，重新copy最新版本

3、批量更新表字符集排序

4、Gemfile 文件中mysql2 和 peek-mysql2 的配置都不带后续的group
gem 'peek-mysql2', '~> 1.1.0', group: :mysql -> gem 'peek-mysql2', '~> 1.1.0'

10、Mysql2::Error: Incorrect string value 乱码

报错信息：

Mysql2::Error: Incorrect string value: '\xF0\x9F\x94\xA8\xE6\x89...' for column 'commit_title' at row 1: INSERT INTO `push_event_payloads` (`event_id`, `commit_count`, `ref_type`, `action`, `commit_from`, `commit_to`, `ref`, `commit_title`) VALUES (259421, 1, 0, 2, x'a2ad3a918aa4d9aac226c430b365bb5cb05f5619', x'1041d9cdfaed711af45455b65497284194aed02c', 'dev_test1', '1，修改手机空布局不居中问题。')

Fix:

解决方案

alter table push_event_payloads change commit_title commit_title varchar(70) character set utf8mb4 collate utf8mb4_general_ci ;

11、gitlab-ctl reconfigure 卡住不动

报错信息：

Error: gitlab-ctl reconfigure 过程中如果卡主不动

解决方案：

systemctl restart gitlab-runsvdir