1 背景
在k8s中基于alpine做底层系统的容器进行git lfs push操作时，发现报错无法上传成功
Fatal error: Server error: http://git.ops.xxxxx.com/xxxx/yyyy.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
error: failed to push some refs to 'http://git.ops.xxxx.com/xxxx/yyyy.git'
而git clone、commit等操作均正常，并且git lfs push的操作在另一个机器A上能正常执行。
2 探索思路
2.1 从报错入手
先根据报错信息搜索，基本回答是此报错是git server(gitlab)侧发生错误，不过可以使用如下命令来打印trace跟踪信息(via: https://github.com/git-lfs/git-lfs/issues/3613#issuecomment-483852691)
GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push origin xxx/xxx
确实打印了更多信息，包括与Server的请求过程，如下：
07:18:13.442791 trace git-lfs: HTTP: POST http://xxxx:[email protected]/suncard/packages.git/info/lfs/objects/batch
> POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1
> Host: git.ops.yunnex.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 158
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)
>
{"operation":"upload","objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3}],"ref":{"name":"refs/heads/test-branch"}}07:18:13.618655 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Thu, 24 Sep 2020 07:18:13 GMT
< Etag: W/"0fb39c6aeeda2850fdcf244083961713"
< Server: nginx
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: 570154fd-b15f-4e8a-a735-b034e98dfbfd
< X-Runtime: 0.165423
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
07:18:13.619071 trace git-lfs: HTTP: {"objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}
{"objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}07:18:13.619307 trace git-lfs: tq: starting transfer adapter "basic"
07:18:13.619345 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
07:18:13.619383 trace git-lfs: xfer: adapter "basic" started
07:18:13.619413 trace git-lfs: xfer: adapter "basic" worker 1 starting
07:18:13.619537 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
07:18:13.619549 trace git-lfs: xfer: adapter "basic" worker 5 starting
07:18:13.619585 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
07:18:13.619642 trace git-lfs: xfer: adapter "basic" worker 0 starting
07:18:13.619673 trace git-lfs: xfer: adapter "basic" worker 6 starting
07:18:13.619681 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e"
07:18:13.619702 trace git-lfs: xfer: adapter "basic" worker 7 starting
07:18:13.619722 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
07:18:13.619732 trace git-lfs: xfer: adapter "basic" worker 2 starting
07:18:13.619775 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
07:18:13.619723 trace git-lfs: xfer: adapter "basic" worker 4 starting
07:18:13.619775 trace git-lfs: xfer: adapter "basic" worker 3 starting
07:18:13.619805 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
07:18:13.619788 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
07:18:13.619697 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
07:18:13.621028 trace git-lfs: HTTP: PUT http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
> PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3 HTTP/1.1
> Host: git.ops.yunnex.com
> Authorization: Basic * * * * *
> Content-Length: 3
> Content-Type: text/plain; charset=utf-8
> User-Agent: git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)
>
07:18:13.621492 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
dd
07:18:13.621527 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
07:18:13.621536 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
07:18:13.621563 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
07:18:13.621571 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
07:18:13.621579 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
07:18:13.621522 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
07:18:13.786463 trace git-lfs: HTTP: 500
< HTTP/1.1 500 Internal Server Error
< Content-Length: 2911
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Connection: keep-alive
< Content-Type: text/html; charset=utf-8
< Date: Thu, 24 Sep 2020 07:18:13 GMT
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Pragma: no-cache
< Server: nginx
< X-Request-Id: 0574e8c0-8ee8-4667-8141-1982d67c0bdc
< X-Runtime: 0.162882
<
07:18:13.786829 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e"
07:18:13.786927 trace git-lfs: tq: retrying object b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e: Fatal error: Server error: http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
07:18:13.787030 trace git-lfs: tq: enqueue retry #1 for "b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e" (size: 3)
07:18:13.787091 trace git-lfs: tq: sending batch of size 1
07:18:13.787212 trace git-lfs: api: batch 1 files
07:18:13.787330 trace git-lfs: HTTP: POST http://xxxxx:[email protected]/suncard/packages.git/info/lfs/objects/batch
...不断重试重复相同日志
其中关键请求如下
200 POST /suncard/packages.git/info/lfs/objects/batch
500 PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
同时到gitlab server执行tail -f /var/log/gitlab//.log，日志如下
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/suncard/packages.git/info/lfs/objects/batch" for 10.10.51.195 at 2020-09-24 15:23:16 +0800
Processing by Projects::LfsApiController#batch as JSON
Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>3}], "ref"=>{"name"=>"refs/heads/test-branch"}, "namespace_id"=>"suncard", "project_id"=>"packages.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>3}], "ref"=>{"name"=>"refs/heads/test-branch"}}}
Completed 200 OK in 138ms (Views: 0.3ms | ActiveRecord: 4.7ms)
Started PUT "/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3" for 10.10.51.195 at 2020-09-24 15:23:17 +0800
Processing by Projects::LfsStorageController#upload_finalize as HTML
Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>"3"}
Completed 500 Internal Server Error in 104ms (ActiveRecord: 2.6ms)
NoMethodError (undefined method `include?' for nil:NilClass):
app/controllers/projects/lfs_storage_controller.rb:55:in `tmp_filename'
app/controllers/projects/lfs_storage_controller.rb:23:in `upload_finalize'
lib/gitlab/i18n.rb:45:in `with_locale'
lib/gitlab/i18n.rb:51:in `with_user_locale'
app/controllers/application_controller.rb:294:in `set_locale'
lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/request_context.rb:18:in `call'
lib/gitlab/metrics/connection_rack_middleware.rb:31:in `call'
...不断重试重复相同日志
==> /var/log/gitlab/nginx/gitlab_access.log <==
10.10.51.195 - - [24/Sep/2020:15:23:16 +0800] "GET /suncard/packages.git/info/refs?service=git-upload-pack HTTP/1.1" 200 701 "-" "git/2.20.2"
10.10.51.195 - production [24/Sep/2020:15:23:16 +0800] "POST /suncard/packages.git/info/lfs/locks/verify HTTP/1.1" 404 1489 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
10.10.51.195 - production [24/Sep/2020:15:23:17 +0800] "POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1" 200 251 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
10.10.51.195 - production [24/Sep/2020:15:23:17 +0800] "PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3 HTTP/1.1" 500 2911 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
10.10.51.195 - production [24/Sep/2020:15:23:17 +0800] "POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1" 200 251 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
可见与client对应，500的报错信息如下，但是搜索后发现没有相关线索
NoMethodError (undefined method `include?' for nil:NilClass):
app/controllers/projects/lfs_storage_controller.rb:55:in `tmp_filename'
app/controllers/projects/lfs_storage_controller.rb:23:in `upload_finalize'
lib/gitlab/i18n.rb:45:in `with_locale'
2.2 对比测试
线索中断，想到机器A可以正常push，所以进行对比观察，执行相同操作，client日志如下
23:15:15.348869 trace git-lfs: HTTP: POST http://production:[email protected]/suncard/packages.git/info/lfs/objects/batch
> POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1
> Host: git.ops.yunnex.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 158
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)
>
{"operation":"upload","objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3}],"ref":{"name":"refs/heads/test-branch"}}23:15:15.525308 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Thu, 24 Sep 2020 15:15:15 GMT
< Etag: W/"486d174eca874b88d375fecca47236e2"
< Server: nginx
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: b87656c2-5593-4e19-bdc0-ea522657b9fc
< X-Runtime: 0.172955
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
23:15:15.525557 trace git-lfs: HTTP: {"objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}
{"objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}23:15:15.525647 trace git-lfs: tq: starting transfer adapter "basic"
23:15:15.525662 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
23:15:15.525669 trace git-lfs: xfer: adapter "basic" started
23:15:15.525700 trace git-lfs: xfer: adapter "basic" worker 0 starting
23:15:15.525705 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25"
23:15:15.525723 trace git-lfs: xfer: adapter "basic" worker 4 starting
23:15:15.525736 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
23:15:15.525759 trace git-lfs: xfer: adapter "basic" worker 6 starting
23:15:15.525770 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
23:15:15.525773 trace git-lfs: xfer: adapter "basic" worker 7 starting
23:15:15.525780 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
23:15:15.525777 trace git-lfs: xfer: adapter "basic" worker 5 starting
23:15:15.525789 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
23:15:15.525754 trace git-lfs: xfer: adapter "basic" worker 3 starting
23:15:15.525796 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
23:15:15.525731 trace git-lfs: xfer: adapter "basic" worker 2 starting
23:15:15.525803 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
23:15:15.525821 trace git-lfs: xfer: adapter "basic" worker 1 starting
23:15:15.525836 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
23:15:15.525855 trace git-lfs: HTTP: PUT http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3
> PUT /suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3 HTTP/1.1
> Host: git.ops.yunnex.com
> Authorization: Basic * * * * *
> Content-Length: 3
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)
>
23:15:15.525963 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
23:15:15.525968 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
23:15:15.525969 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
23:15:15.525970 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
23:15:15.525972 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
23:15:15.526006 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
23:15:15.526027 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
23:15:15.931456 trace git-lfs: HTTP: 200
< HTTP/1.1 200 OK
< Content-Length: 0
< Cache-Control: no-cache
< Connection: keep-alive
< Content-Type: text/html
< Date: Thu, 24 Sep 2020 15:15:15 GMT
< Server: nginx
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: e0ce367a-adce-44bd-b1ed-29a33f756bb5
< X-Runtime: 0.245919
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
<
23:15:15.931592 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25"
23:15:15.931664 trace git-lfs: xfer: adapter "basic" End()
23:15:15.931671 trace git-lfs: xfer: adapter "basic" worker 7 stopping
23:15:15.931675 trace git-lfs: xfer: adapter "basic" worker 2 stopping
23:15:15.931684 trace git-lfs: xfer: adapter "basic" worker 6 stopping
23:15:15.931685 trace git-lfs: xfer: adapter "basic" worker 3 stopping
23:15:15.931689 trace git-lfs: xfer: adapter "basic" worker 5 stopping
23:15:15.931692 trace git-lfs: xfer: adapter "basic" worker 1 stopping
23:15:15.931688 trace git-lfs: xfer: adapter "basic" worker 0 stopping
23:15:15.931682 trace git-lfs: xfer: adapter "basic" worker 4 stopping
23:15:15.931726 trace git-lfs: xfer: adapter "basic" stopped
23:15:15.931787 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
23:15:15.931801 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
23:15:15.931819 trace git-lfs: filepathfilter: rejecting "tmp" via []
23:15:15.931824 trace git-lfs: filepathfilter: accepting "tmp"
Uploading LFS objects: 100% (1/1), 3 B | 0 B/s, done
其中关键请求如下，其中500请求已变成200
200 POST /suncard/packages.git/info/lfs/objects/batch
200 PUT /suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3
而gitalb server对应日志如下
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/suncard/packages.git/info/lfs/objects/batch" for 192.168.1.25 at 2020-09-23 18:36:27 +0800
Processing by Projects::LfsApiController#batch as JSON
Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>4}], "ref"=>{"name"=>"refs/heads/test-branch"}, "namespace_id"=>"suncard", "project_id"=>"packages.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>4}], "ref"=>{"name"=>"refs/heads/test-branch"}}}
Completed 200 OK in 195ms (Views: 0.3ms | ActiveRecord: 6.1ms)
Started PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4/authorize" for 192.168.1.25 at 2020-09-23 18:36:28 +0800
Processing by Projects::LfsStorageController#upload_authorize as HTML
Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>"4"}
Completed 200 OK in 114ms (Views: 0.2ms | ActiveRecord: 3.0ms)
Started PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4" for 192.168.1.25 at 2020-09-23 18:36:28 +0800
Processing by Projects::LfsStorageController#upload_finalize as HTML
Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>"4"}
Completed 200 OK in 169ms (ActiveRecord: 51.1ms)
Updating statistics for project 32
==> /var/log/gitlab/nginx/gitlab_access.log <==
192.168.1.25 - production [23/Sep/2020:18:36:28 +0800] "POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1" 200 251 "-" "git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)"
192.168.1.25 - production [23/Sep/2020:18:36:28 +0800] "PUT /suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4 HTTP/1.1" 200 0 "-" "git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)"
变化如下
# 多了1个200 请求
PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4/authorize"
# 下列请求从500变为200
200 PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4"
gitlab server相同、操作的仓库分支相同的情况下，这样的差异推测是client客户端导致的了，先来看版本
git --version 1.8.3.1(正常) 2.20.4(异常)
git lfs version 2.4.2(正常) 2.5.1(异常)
其中正常的机器A是物理机centos用yum安装的git和git-lfs；异常的是跑在k8s集群中基于alpine的pod，是通过apk安装的，遂尝试apk降级git和git-lfs到机器A正常的版本。apk降级需要到其市场查找目标版本https://pkgs.alpinelinux.org/packages?name=git-lfs&branch=v3.9 。
再通过降通过降repository版本来安装低版本的软件包，原alpine是v3.9，现在尝试改为v3.8
apk add --no-cache --repository http://mirrors.aliyun.com/alpine/v3.8/main --repository http://mirrors.aliyun.com/alpine/v3.8/community 'git-lfs=2.4.2-r1'
再次执行原push等操作，终于正常了！以防万一，再到原本正常的机器A把git-lfs升级一下，看是否也异常。
安装过程：
mkdir /tmp/test
wget https://github.com/git-lfs/git-lfs/releases/download/v2.5.1/git-lfs-linux-amd64-v2.5.1.tar.gz
tar -xvf git-lfs-linux-amd64-v2.5.1.tar.gz
# 这样就可以在不影响原git-lfs安装的情况下进行测试，后续用/tmp/test/git-lfs进行push等操作
/tmp/test/git-lfs version
果不其然，也重现了相同异常。
3 结论
首先异常与git版本没有关系，只和git-lfs版本有关。之后又尝试了更低的2.3.x、更高的2.6.x~2.11.x。确定是从2.5.x开始的版本有异常。另外还需要关注gitlab的版本,目前为9.4.1
sudo gitlab-rake gitlab:env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.13.0
Sidekiq Version:5.0.0
Go Version: unknown
GitLab information
Version: 9.4.1
Revision: 6678b4e
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://git.ops.yunnex.com
HTTP Clone URL: http://git.ops.yunnex.com/some-group/some-project.git
SSH Clone URL: [email protected]:some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 5.3.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
目前只能推测该gitlab/lfs server版本与git lfs client push不兼容，所以采取的方案是lfs降为2.4.x，alpine使用v3.8 repository重新安装git-lfs。
“架构人生，迭代生命” ——深邃老夏，搜索summer_deep微信公众号可获取更多帮助