gitlab remove project 失败

删除project失败，报500错误 “Whoops, something went wrong on our end.”

1. 查找日志

查找/var/log/gitlab/gitlab-rails/production.log ，对比执行过程，发现报错对应如下日志：
OpenSSL::Cipher::CipherError ():

官方推荐这个命令 gitlab-ctl tail gitlab-rails

Started GET "/-/metrics" for 127.0.0.1 at 2020-06-18 10:11:59 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 35ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
Started DELETE "/core/maxno" for 58.246.24.194 at 2020-06-18 10:12:03 +0800
Processing by ProjectsController#destroy as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "namespace_id"=>"core", "id"=>"maxno"}
Completed 500 Internal Server Error in 24ms (ActiveRecord: 2.9ms | Elasticsearch: 0.0ms)
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
app/models/concerns/token_authenticatable.rb:40:in `block in add_authentication_token_field'
app/services/projects/destroy_service.rb:13:in `async_execute'
lib/gitlab/metrics/instrumentation.rb:161:in `block in async_execute'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `async_execute'
app/controllers/projects_controller.rb:149:in `destroy'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:28:in `set_current_ip_address'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:445:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:439:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

2. google 解决办法。相信我，大部分的问题别人都碰到过。

https://gitlab.com/gitlab-org/gitlab-foss/-/issues/66002
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59413
https://docs.gitlab.com/ee/raketasks/backup_restore.html#when-the-secrets-file-is-lost

摘要：
just so I understand everything correctly, you had two machines, you export the projects from machine 1 and then imported it from machine 2 using the import/export feature right? I'm curious how you got the CI variables as CI variables are not exported.

OK, and was machine 2 set up with a fresh omnibus install? Or am backup? The reason I ask is we specify you need to reset everything that is encrypted A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted which in this case is the runner token.
Meaning you have to reset the tokens for runners. This can be done in two ways:

• Rails console: Project.find_by_full_path('root/my-project').update(runners_token: nil, runners_token_encrypted:nil)
• DB Console: UPDATE projects SET runners_token = null, runners_token_encrypted = null WHERE id = 28;

Note: The above is only done for 1 specific project you might have to do it for all projects. This will reset the registration token for the runners, all existing runners will keep working but new runners will have to use the new registration token.

进入DB控制台

[root@VM_1_17_centos ~]# gitlab-rails dbconsole
psql (10.7)
Type "help" for help.

gitlabhq_production=> 

Reset CI/CD variables

Check the ci_group_variables and ci_variables tables:
Those are the variables that you need to delete.

gitlabhq_production=> SELECT * FROM public."ci_group_variables";
 id | key | value | encrypted_value | encrypted_value_salt | encrypted_value_iv | group_id | protected | created_at | updated_at | masked | variable_type 
----+-----+-------+-----------------+----------------------+--------------------+----------+-----------+------------+------------+--------+---------------
(0 rows)

gitlabhq_production=> SELECT * FROM public."ci_variables";
 id | key | value | encrypted_value | encrypted_value_salt | encrypted_value_iv | project_id | protected | environment_scope | masked | variable_type 
----+-----+-------+-----------------+----------------------+--------------------+------------+-----------+-------------------+--------+---------------
(0 rows)

Reset Runner registration tokens

Clear all the tokens for projects, groups, and the whole instance:

gitlabhq_production=> UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE 16
gitlabhq_production=> UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE 33
gitlabhq_production=> UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE 1
gitlabhq_production=> UPDATE ci_runners SET token = null, token_encrypted = null;
UPDATE 0

3. 执行完后，都不用重启服务，立马见效