Docker registry垃圾回收
通过：
docker run -p 5000:5000 -v /netdata/xxxx/registry:/var/lib/registry registry.cn-hangzhou.aliyuncs.com/term/registry-gc:2.7 registry garbage-collect /var/lib/registry/config.yml搭建好registry服务后，docker push alpine:3.8镜像至registry服务。其中alpine的dockerfile内容如下：
FROM scratch
ADD rootfs.tar.xz /
CMD ["/bin/sh"]registry.cn-hangzhou.aliyuncs.com/term/registry-gc:2.7的dockerfile如下:
FROM alpine:3.4
RUN set -ex \
&& apk add --no-cache ca-certificates apache2-utils
COPY registry /bin/registry
COPY config.yml /etc/docker/registry/config.yml
VOLUME ["/var/lib/registry"]
EXPOSE 5000
COPY docker-entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/etc/docker/registry/config.yml"]config.yml内容如下:
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
maintenance:
readonly:
enabled: false
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3后端存储
镜像推送至registry后，可查看registry后端的本地存储目录结构如下：
docker
`-- registry
`-- v2
|-- blobs
| `-- sha256
| |-- 01
| | `-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- data
| |-- 1e
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| | `-- data
| `-- 5a
| `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| `-- data
`-- repositories
`-- alpine
|-- _layers
| `-- sha256
| |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- link
| `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| `-- link
|-- _manifests
| |-- revisions
| | `-- sha256
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| | `-- link
| `-- tags
| `-- 3.8
| |-- current
| | `-- link
| `-- index
| `-- sha256
| `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| `-- link
`-- _uploads查看Manifest
查看alpine:3.8的manifest文件内容：
curl -i -H "Accept:application/vnd.docker.distribution.manifest.v2+json" http://127.0.0.1:5000/v2/alpine/manifests/3.8Response Header:
HTTP/1.1 200 OK
Content-Length: 526
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667"
X-Content-Type-Options: nosniff
Date: Tue, 04 Sep 2018 11:59:14 GMTResponse body:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1512,
"digest": "sha256:0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 12456,
"digest": "sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f"
}
]
}执行垃圾回收
sudo docker run --rm --name gctest -v /home/term/registry:/var/lib/registry registry:2.5.1 garbage-collect -d /var/lib/registry/config.yml得到的结果如下：
alpine
alpine: marking manifest sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
alpine: marking blob sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
alpine: marking configuration sha256:0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
3 blobs marked, 0 blobs eligible for deletion发现所有的blobs都被mark, 无blobs可清理
删除Manifest
curl -X DELETE http://127.0.0.1:5000/v2/alpine/manifests/sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667从获取Manifest时的Header: Docker-Content-Digest里获取。
调用manifest API删除manifest文件后，再次查看registry后端存储目录结构:
docker
`-- registry
`-- v2
|-- blobs
| `-- sha256
| |-- 01
| | `-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- data
| |-- 1e
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| | `-- data
| |-- 5a
| | `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| | `-- data
| `-- a3
| `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
| `-- data
`-- repositories
`-- alpine
|-- _layers
| `-- sha256
| |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- link
| |-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| | `-- link
| `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
| `-- link
|-- _manifests
| |-- revisions
| | `-- sha256
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| `-- tags
`-- _uploads对比之前的目录，发现alpine下的tags目录内容已被删
再次垃圾回收
再次执行垃圾回收：
sudo docker run --rm --name gctest -v /home/term/registry:/var/lib/registry registry:2.5.1 garbage-collect -d /var/lib/registry/config.yml得到结果如下：
alpine
0 blobs marked, 4 blobs eligible for deletion
blob eligible for deletion: sha256:0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
blob eligible for deletion: sha256:1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
blob eligible for deletion: sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
blob eligible for deletion: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4以上为dry-run方式执行。
真正执行gc时，日志如下:
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/08/0873c923e00e0fd2ba78041bfb64a105e1ecb7678916d1f7776311e45bf5634b" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/11/11cd0b38bc3ceb958ffb2f9bd70be3fb317ce7d255c8a4c3f4af30e298aa1aab" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/1e/1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/5a/5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/8e/8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896
time="2018-09-05T01:50:32Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/01/0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc" go.version=go1.6.3 instance.id=ccafa0cc-4335-4535-b16a-9bff7886c896gc只清除blobs下的数据，但不会清除repositories里的内容及blobs下对应的目录。
docker
`-- registry
`-- v2
|-- blobs
| `-- sha256
| |-- 01
| |-- 08
| |-- 11
| |-- 1e
| |-- 5a
| |-- 8e
| `-- a3
`-- repositories
`-- alpine
|-- _layers
| `-- sha256
| |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- link
| |-- 11cd0b38bc3ceb958ffb2f9bd70be3fb317ce7d255c8a4c3f4af30e298aa1aab
| | `-- link
| |-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| |-- 8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6
| `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
| `-- link
|-- _manifests
| |-- revisions
| | `-- sha256
| | |-- 0873c923e00e0fd2ba78041bfb64a105e1ecb7678916d1f7776311e45bf5634b
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| `-- tags
`-- _uploads删除blobs
调用registry api删除blob：
curl -i -X DELETE http://127.0.0.1:5000/v2/alpine/blobs/sha256:5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3fResponse Header:
HTTP/1.1 202 Accepted
Content-Length: 0
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Wed, 05 Sep 2018 01:05:20 GMT
Content-Type: text/plain; charset=utf-8Reponse body:
docker
`-- registry
`-- v2
|-- blobs
| `-- sha256
| |-- 01
| | `-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- data
| |-- 1e
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| | `-- data
| |-- 5a
| | `-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| | `-- data
| `-- a3
| `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
| `-- data
`-- repositories
`-- alpine
|-- _layers
| `-- sha256
| |-- 0107b3d497f0786d2d586d6093015ac5c64140794798fe140728f78c21af7bdc
| | `-- link
| |-- 5a83255105eaf8d24ad660befe32d7b9d449889ffde99b3ec59caf8edee59a3f
| `-- a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
| `-- link
|-- _manifests
| |-- revisions
| | `-- sha256
| | `-- 1e7481c6b78ad4b425f1985e3324117075ab3ebe2ef99d54a578d0f8c1fe0667
| `-- tags
`-- _uploads对比之前目录结构，发现:docker/registry/v2/repositoires/文件被删。
Config内容
{
"architecture": "amd64",
"config": {
"ArgsEscaped": true,
"AttachStderr": false,
"AttachStdin": false,
"AttachStdout": false,
"Cmd": [
"/bin/sh"
],
"Domainname": "",
"Entrypoint": null,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Hostname": "",
"Image": "sha256:80c02d9956b63482b0e4344c56cc029d8c330e01a3f6093516b0a60b83118b9c",
"Labels": null,
"OnBuild": null,
"OpenStdin": false,
"StdinOnce": false,
"Tty": false,
"User": "",
"Volumes": null,
"WorkingDir": ""
},
"container": "bf174046ba695f7f6c1d841dd2894b2c34a96d73bb832c102b3ba2b6a0d20120",
"container_config": {
"ArgsEscaped": true,
"AttachStderr": false,
"AttachStdin": false,
"AttachStdout": false,
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/sh\"]"
],
"Domainname": "",
"Entrypoint": null,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Hostname": "bf174046ba69",
"Image": "sha256:80c02d9956b63482b0e4344c56cc029d8c330e01a3f6093516b0a60b83118b9c",
"Labels": {},
"OnBuild": null,
"OpenStdin": false,
"StdinOnce": false,
"Tty": false,
"User": "",
"Volumes": null,
"WorkingDir": ""
},
"created": "2018-09-04T09:32:26.252138866Z",
"docker_version": "17.06.2-ce",
"history": [
{
"created": "2018-09-04T09:32:25.454551817Z",
"created_by": "/bin/sh -c #(nop) ADD file:babb191309280a09293bff66ca4aead1ecad6119d9add3a803d489fa866d5282 in / "
},
{
"created": "2018-09-04T09:32:26.252138866Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"diff_ids": [
"sha256:2a7f9859cf97588d6533a9ac63c988b34beff0cc2758182b70e7a8cc8383d7f3"
],
"type": "layers"
}
}结论
- 要想调用registry API，请打开deletable=true的开关，可通过环境变量:
REGISTRY_STORAGE_DELETE_ENABLED=true指定, 也可在config.yml里配置 - 镜像清除，删除manifest即可，无需调用API删除layer，真正Blob清理需执行garbage-collect.
- 执行garbage-collect前，请将regsitry的
readonly=true开关打开，重启registry，保证gc时，registry只读。此时不允许docker push镜像，否则会清理掉gc过程中新push的blobs。清理完成后，将readonly开关关闭,再次重启registry。 - 目前registry暂无GC API，只能通过脚本执行GC工作。
坑点
- 目前registry不支持通过环境变量设置
readonly=true，在registry:2.5.1版本试过，会导致registry起不来，报错:
goroutine 1 [running]:
panic(0xc2a220, 0xc820438620)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/docker/distribution/registry/handlers.NewApp(0x7fd101c70ad8, 0xc8203ba780, 0xc82000d600, 0x7fd101c70ad8)
/go/src/github.com/docker/distribution/registry/handlers/app.go:135 +0xc44
github.com/docker/distribution/registry.NewRegistry(0x7fd101c70b18, 0xc8203ba780, 0xc82000d600, 0xc82000d600, 0x0, 0x0)
/go/src/github.com/docker/distribution/registry/registry.go:86 +0x2b0
github.com/docker/distribution/registry.glob.func1(0x14cc660, 0xc820347f50, 0x1, 0x1)
/go/src/github.com/docker/distribution/registry/registry.go:55 +0x2a9
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).execute(0x14cc660, 0xc820347f00, 0x1, 0x1, 0x0, 0x0)
/go/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:495 +0x6d4
github.com/docker/distribution/vendor/github.com/spf13/cobra.(*Command).Execute(0x14cc800, 0x0, 0x0)
/go/src/github.com/docker/distribution/vendor/github.com/spf13/cobra/command.go:560 +0x180
main.main()须通过额外挂载配置文件解决