gfengwong

容器启动加速-nydus

1. 概述
- 1.1. 使用nydus需要考虑的问题
- 1.2. 当前实践探索中遇到的问题
2. 容器集成nydus
- 2.1. contanierd 集成【核心】
  - 2.1.1. 版本说明
  - 2.1.2. 概念说明
  - 2.1.3. 构建
    - 2.1.3.1. 构建 buildkitd
    - 2.1.3.2. 安装nydus
    - 2.1.3.3. 启动buildkitd进程
    - 2.1.3.4. 构建镜像
  - 2.1.4. 部署
    - 2.1.4.1. 部署nydus-snapshotter
    - 2.1.4.2. 配置containerd支持nydus snapshotter
  - 2.1.5. 使用nydus镜像启动containerd容器
- 2.2. kubernertes集成
  - 2.2.1. 版本说明
  - 2.2.2. k8s集群准备
    - 2.2.2.1. 将k8s cri从docker替换为containerd
  - 2.2.3. 部署buildkit
    - 2.2.3.1. 构建buildkit镜像
    - 2.2.3.2. 按需修改yaml并部署
  - 2.2.4. 部署 nydus-snapshotter
    - 2.2.4.1. 更改nydus镜像
    - 2.2.4.2. 修改configmap
    - 2.2.4.3. 重启nydus-snapshotter
  - 2.2.5. 验证
    - 2.2.5.1. 在pod中构建nydus格式的镜像
    - 2.2.5.2. 使用pod启动nydus镜像容器
- 2.3. docker 集成
3. CI集成 – skaffold集成nydus
4. nydus镜像启动pod与ociv1镜像启动pod效果对比
- 4.1. wordpress镜像
  - 4.1.1. oci v1镜像容器pulling image时间
  - 4.1.2. nydus镜像容器pulling image时间
5. 问题记录
- 5.1. 镜像无法构建
- 5.2. containerd 命令速记
- 5.3. 镜像无法下载
- 5.4. 如何判断一个镜像是否是nydus镜像
  - 5.4.1. 通过本地manifest判断
  - 5.4.2. 通过nydusify check检查
- 5.5. nydus snapshotter无法启动
- 5.6. buildkit构建nydus镜像构建不成功
- 5.7. dragonflyoss仓库的镜像无法 inspect
- 5.8. nydus镜像与oci镜像manifest的区别
- 5.9. 构建的镜像list时告警
- 5.10. 问题 buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题
- 5.11. 运行oci-ref镜像导致nydusd进程z状态
  - 5.11.1. 现象
  - 5.11.2. 解决方式
- 5.12. nydus镜像和oci镜像拉取镜像时间相同
  - 5.12.1. nydus镜像启动 nydus-snapshotter日志
  - 5.12.2. oci镜像启动 nydus-snapshotter日志
6. 参考

1. 概述

Nydus是一个镜像加速器，主要通过容器启动时懒加载镜像提高启动速度, 大镜像可以秒级启动，后续的读写通过COW方式。以下是nydus的几个标签:

Nydus是蚂蚁、阿里云、字节等共建的开源容器镜像加速项目
CNCF孵化的项目Dragonfly中的一个子项目
新的镜像格式，不同于oci
良好的生态，兼容containerd、k8s、harbor、buildkit – 从构建、存储、部署、到CI集成，有丰富的解决方案
erofs+fscache+cachefiles 缓存方案: 使用内核缓存和只读文件系统进行io加速
- cachefiles: 获取文件缓存
- fscache: 管理cachefiles的缓存内容
- erofs: 将缓存以文件系统接口方式提供访问

用Nydus对容器进行加速，当前可用，可以使用(k8s+)buildkit构建镜像，使用(k8s+)containerd进行发布, 使用skaffold进行CI(但路途坎坷，该文档内容主要为这部分)；未来可期，使用fscache+erofs进行底层存储加速。

1.1. 使用nydus需要考虑的问题

如果要落地Nydus，需要考虑如下几个问题:

当前k8s container runtime 如果是docker，则需要切换成containerd，切换需要注意
- pod服务可能会中断
- 监控日志，原来通过docker接口，现在需要通过containerd接口 – 这部分k8s可以屏蔽
CI如何处理原有的docker build方式和现在buildkit构建方式
containerd功能相对简单，要用它来进行构建、运行等操作需要多个组件配合才行，要使用起来需要做不少操作:
- containerd
- nerdctl
- buildkitd
- buildctl …
- nydus-snapshotter 要使用nydus镜像的话需要
⚠️性能问题，写时复制从远端拉取，例如从镜像仓库拉取的话需要考虑
- 镜像仓库本身的IO性能
- 从pod到仓库的网络带宽
⚠️容器化使用nydus-snapshotter有待改进
现在nydus-snapshotter容器故障之后由于和容器启动需要nydus-snapshotter这样的循环依赖，所以无法自愈，需要
- 删除containerd中的nydus-snapshotter配置
- 重启containerd
- 重新配置containerd

1.2. 当前实践探索中遇到的问题

在实践探索中主要有如下问题:

涉及项目多，各项目版本不明确，遇到较多版本兼容问题
文档分散，不易查阅
实践类文档较少
文档落后代码较多
部分项目CI部分代码落后于主体代码

正在解决中的问题

构建出来的本地镜像缺少layer无法run

2. 容器集成nydus

2.1. contanierd 集成【核心】

2.1.1. 版本说明

组件	版本	备注
内核	5.18.2-1.el7.elrepo.x86_64
containerd	containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
buildkit	github.com/moby/buildkit 76aeaa55.m 76aeaa55ff4e1c575644bffbf0a43bea92215fbe.m	从https://github.com/nydusaccelerator/buildkit.git构建
nydus-image	v2.2.1
containerd-nydus-grpc	v0.8.0
nerdctl	1.3.1

2.1.2. 概念说明

nydus镜像: nydus格式的镜像，不同于现在的ociv1 ociv2，在使用时可进行懒加载，加快容器启动速度
nydus镜像构建
- buildkitd: 构建nydus镜像的工具，需要增加nydus兼容配置, 一般以守护进程运行，客户端通过 buildctl 命令通过sock文件或tcp进行与服务端连接，也支持daemonless
- nydus-image: nydus镜像工具，可以集成到buildkit中构建nydus格式的镜像，也可以单独使用进行nydus镜像构建、检查、导出等
nydus镜像运行
- containerd: 容器运行时
- nydus-snapshotter: 是一个containerd插件，用来支持contanierd使用nydus格式的镜像¹

buidlkit+nydus构建完镜像之后推送到仓库，可以是docker registry，也可以是harbor等
运行的时候每启动一个nydus镜像的容器，containerd会调用 nydus-snapshotter 去查看本地有没有缓存，如果没有的话启动 nydusd 进程去拉取,如果有的话启动nydusd fuse进程作为文件服务器提供给容器挂载，这里是懒加载的关键，容器需要什么nydusd就会从仓库拉什么，启动对应的snapshotter挂载进程。

/usr/bin/nydusd fuse --thread-num 10 --config /var/lib/containerd-nydus/config/chmadgaihqqap9ldb1eg/config.json --bootstrap /var/lib/containerd-nydus/snapshots/833/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/833/mnt --apisock /var/lib/containerd-nydus/socket/chmadgaihqqap9ldb1eg/api.sock --log-level debug --log-file /var/lib/containerd-nydus/logs/chmadgaihqqap9ldb1eg/nydusd.log

架构图如下，这里用的是fuse文件系统, erofs的社区文档比较少，暂时先用fuse

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jMiUFnzf-1686298186600)(null)]

如果用erofs

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vg8EBgO4-1686298186840)(null)]

2.1.3. 构建

2.1.3.1. 构建 buildkitd

官方社区仓库是 moby/buildkit, 这里用的nydusaccelerator的项目，是因为当前bug/新功能会先在nydusaccelerator发布，所以暂时先用这个项目

git clone https://github.com/nydusaccelerator/buildkit.git
cd buildkit
go build -tags=nydus -o ./bin/buildkitd ./cmd/buildkitd
mv ./bin/buildkitd /usr/bin/
chmod +x /usr/bin/buildkitd

2.1.3.2. 安装nydus

wget https://github.com/dragonflyoss/image-service/releases/download/v2.2.1/nydus-static-v2.2.1-linux-amd64.tgz
tar zxf nydus-static-v2.2.1-linux-amd64.tgz
cd nydus-static
sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr/bin

2.1.3.3. 启动buildkitd进程

buildkit 配置²

cat > /etc/buildkit/buildkitd.toml <<EOF
debug = false
[worker.oci]
  enabled = false

[worker.containerd]
  enabled = true
  # 构建成功的之后的镜像存储位置
  namespace = "default"
  address = "/run/containerd/containerd.sock"
  enabled = true
  platforms = [ "linux/amd64", "linux/arm64" ]
  gc = true
  # gckeepstorage sets storage limit for default gc profile, in MB.
  gckeepstorage = 9000
EOF

cat > /etc/systemd/system/buildkitd.service << EOF
[Unit]
Description=BuildKit
Documentation=https://github.com/moby/buildkit

[Service]
Environment="NYDUS_BUILDER=/usr/bin/nydus-image"
ExecStart=/usr/bin/buildkitd --config /etc/buildkit/buildkitd.toml

[Install]
WantedBy=multi-user.target
EOF

systemctl enable buildkitd
systemctl start buildkitd

2.1.3.4. 构建镜像

构建一个简单的hello world镜像

hello world程序

mkdir golang-helloworld

编辑 main.go

package main

import "fmt"

func main() {
  fmt.Println("Hello, world!")
}

构建可执行文件

go build -o hello main.go

Dockerfile

编辑dockerfile

FROM  ubuntu:20.04
COPY hello /
ENTRYPOINT [ "/hello" ]

构建

这里会构建一个nydus格式的镜像名称为reg.harbor.com/hello/hello-k8s:v0.0.4, push 为 true 则构建完成之后推送远端仓库

buildctl  build --progress=plain \
  --frontend=dockerfile.v0  \
  --local context=.  \
  --local dockerfile=.  \
  --output type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.4,compression=nydus,oci-mediatypes=true,push=true

校验

这里我们构建出来了nydus镜像，可以校验一下

nydusify check --target reg.harbor.com/hello/hello-k8s:v0.0.4
INFO[2023-05-24T10:59:56+08:00] Parsing image reg.harbor.com/hello/hello-k8s:v0.0.4
INFO[2023-05-24T10:59:56+08:00] Dumping OCI and Nydus manifests to ./output
...
INFO[2023-05-24T10:59:57+08:00] Verified Nydus image reg.harbor.com/hello/hello-k8s:v0.0.4

这里check的是远端镜像，一切符合预期,是nydus镜像³

cat output/nydus_manifest.json
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:3121880b13bcaf21d0748cc35139769cb353bdd8429e541da489d1833c0c9ea7",
    "size": 6940
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a",
      "size": 30779473,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    ...
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab",
      "size": 1067099,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4c73ce0e2ddc066d200bc119cc2b6f39938fd136b93fb8d47a590788cc91c91c",
      "size": 2258453,
      "annotations": {
        "containerd.io/snapshot/nydus-blob-ids": "[\"e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a\",\"15596146dcdb421faaf9881e63baa87731603f412d1f05aa78e2d8f5bb78f1d5\",\"68fb6d505c4733f3c39645876ab9ed30dbf46056b36630be2dafe48634af1e01\",\"32a3c81c8fb5b12c68d74e01b29cd02463d8e85c0296ac3e30874f0f38412ec9\",\"89fe8a5a29e872cb9afbdeffeb9307498e67f42a0d3fdeca754b7a55b8849083\",\"43f9d24cfd0900b5ccbc7030559df305d237932f5383f592128d165c723ca553\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab\"]",
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": ""
      }
    }
  ]
}

本地镜像暂时无法使用,使用会报错(这里是部署了nydus snapshotter之后)

nerdctl run --rm reg.harbor.com/hello/hello-k8s:v0.0.4 --snapshotter=nydus

INFO[0000] apply failure, attempting cleanup             error="failed to extract layer sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: failed to get reader from content store: content digest sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: not found" key="extract-504360049-gvw0 sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a"
FATA[0000] failed to extract layer sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: failed to get reader from content store: content digest sha256:e4aa28274bae00ca1bd04843009c93fba1f6cfae94fb9f60cc98e68e7dd0977a: not found

这里应该应该是nydus+fuse的bug，解决方法为在构建完成之后把本地镜像删除掉, 使用时通过远端拉取

nerdctl rmi reg.harbor.com/hello/hello-k8s:v0.0.4

本地镜像构建出来的manifest和远端一致，这里不能用是因为存储的时候少存储了snapshotter，具体问题待社区确认

2.1.4. 部署

2.1.4.1. 部署nydus-snapshotter

安装 containerd-nydus-grpc

wget https://github.com/containerd/nydus-snapshotter/releases/download/v0.8.0/nydus-snapshotter-v0.8.0-x86_64.tgz
tar zxf nydus-snapshotter-v0.8.0-x86_64.tgz
install -D -m 755 nydus-snapshotter/containerd-nydus-grpc /usr/bin

nydus-fusedev 配置

# nydus blob 缓存
mkdir /opt/nydus/cache

cat > /etc/nydus/nydusd-config.fusedev.json <<EOF
{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "https",
        "skip_verify": true,
        "timeout": 5,
        "connect_timeout": 5,
        "retry_limit": 4
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "/opt/nydus/cache"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4
  }
}
EOF

nydus snapshotter 主进程配置⁴

cat /etc/nydus/config.toml
version = 1
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
daemon_mode = "multiple"
cleanup_on_close = false

[system]
enable = true
address = "/run/containerd-nydus/system.sock"

[system.debug]
daemon_cpu_profile_duration_secs = 5
pprof_address = ""

[daemon]
nydusd_path = "/usr/bin/nydusd"
nydusimage_path = "/usr/bin/nydus-image"
# fusedev or fscache
fs_driver = "fusedev"
# How to process when daemon dies: "none", "restart" or "failover"
recover_policy = "restart"
# Specify a configuration file for nydusd
nydusd_config = "/etc/nydus/nydusd-config.fusedev.json"
# The fuse or fscache IO working threads started by nydusd
threads_number = 10

[log]
# Print logs to stdout rather than logging files
log_to_stdout = false
level = "debug"
log_rotation_compress = true
log_rotation_local_time = true
log_rotation_max_age = 7
log_rotation_max_backups = 5
log_rotation_max_size = 1

[metrics]
# Enable by assigning an address, empty indicates metrics server is disabled
address = ":9110"

[remote]
convert_vpc_registry = false

[remote.mirrors_config]
dir = ""

[remote.auth]
enable_kubeconfig_keychain = false
kubeconfig_path = ""
enable_cri_keychain = false
image_service_address = ""

[snapshot]
enable_nydus_overlayfs = false
sync_remove = false

[cache_manager]
disable = false
gc_period = "24h"
cache_dir = "/opt/nydus/cache/"

[image]
public_key_file = ""
validate_signature = false

[experimental]
enable_stargz = false
enable_referrer_detect = false

cat > /etc/systemd/system/nydus-snapshotter.service <<EOF
[Unit]
Description=nydus snapshotter
After=network.target
Before=containerd.service

[Service]
Type=simple
Environment=HOME=/root
ExecStart=/usr/local/bin/containerd-nydus-grpc --config /etc/nydus/config.toml
Restart=always
RestartSec=1
KillMode=process
OOMScoreAdjust=-999
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable nydus-snapshotter
systemctl start nydus-snapshotter

2.1.4.2. 配置containerd支持nydus snapshotter

编辑containerd配置 /etc/containerd/config.toml

version = 2
state = "/run/containerd"

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    # k8s原来用的container runtime是docker，则需要配置pause镜像，否则runtime改编之后k8s原有pod无法启动
    sandbox_image = "reg.harbor.com/kubernetes/pause:3.6"
    [plugins."io.containerd.grpc.v1.cri".cni]
      # cni插件路径
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      max_conf_num = 1
      conf_template = ""
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]

# Nydus插件主要配置
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
   snapshotter = "nydus"
   disable_snapshot_annotations = false
   discard_unpacked_layers = false

修改完毕重启containerd

systemctl restart containerd

2.1.5. 使用nydus镜像启动containerd容器

containerd也有类似docker的命令行 nerdctl

安装 nerdctl

wget https://github.com/containerd/nerdctl/releases/download/v1.3.1/nerdctl-1.3.1-linux-amd64.tar.gz
tar zxvf nerdctl-1.3.1-linux-amd64.tar.gz
install -D -m 755  nerdctl /usr/bin/

配置 nerdctl

默认使用 nydus-snapshotter

cat > /etc/nerdctl/nerdctl.toml <<EOF
debug          = false
debug_full     = false
address        = "unix:///var/run/containerd/containerd.sock"
namespace      = "default"
snapshotter    = "nydus"
experimental   = true
EOF

使用构建阶段产生的镜像运行容器

先删除本地镜像 nerdctl rmi reg.harbor.com/hello/hello-k8s:v0.0.4

nerdctl run --rm reg.harbor.com/hello/hello-k8s:v0.0.4
# 输出
reg.harbor.com/hello/hello-k8s:v0.0.4:                                      resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:c6ab4a56b80d503312322bda42fc31b038d6f6989d8d9e801c991787b8301132: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:3121880b13bcaf21d0748cc35139769cb353bdd8429e541da489d1833c0c9ea7:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4c73ce0e2ddc066d200bc119cc2b6f39938fd136b93fb8d47a590788cc91c91c:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.6 s                                                                    total:  2.2 Mi (3.6 MiB/s)
Hello, world!

使用 nerdctl 命令时可能会遇到 failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]" cni不支持的问题，需要升级cni

https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-arm64-v1.1.1.tgz
tar zxf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin

升级完毕即可用

到这里使用nydus加速containerd容器启动，基本完成。使用erofs待探索

2.2. kubernertes集成

2.2.1. 版本说明

组件	版本	备注
k8s	v1.22.10
containerd	containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
buildkitd	github.com/moby/buildkit 76aeaa55.m 76aeaa55ff4e1c575644bffbf0a43bea92215fbe.m	从https://github.com/nydusaccelerator/buildkit.git构建
nydus-snapshotter	v0.6.1	通过helm部署⁵,生产用的话需要更新下

2.2.2. k8s集群准备

首先需要一个 container runtime为containerd的集群

kubectl describe nodes |grep -i runtime
  Container Runtime Version:    docker://20.10.8
  Container Runtime Version:    docker://20.10.8
  Container Runtime Version:    docker://20.10.8

如果是docker，则需要更改为containerd

2.2.2.1. 将k8s cri从docker替换为containerd

此处操作比较粗糙，生产切换请慎重

更改 kubelet 配置

编辑/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

增加/在KUBELET_EXTRA_ARGS追加如内容

Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock"

创建 containerd 配置

新建 ``

version = 2
state = "/run/containerd"

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "reg.harbor.com/kubernetes/pause:3.6"
You have new mail in /var/spool/mail/root

重启containerd kubelet服务

systemctl daemon-reload
systemctl restart containerd kubelet

如果 kubelet 启动失败，请查看日志做对应处理 journalctl -xu kubelet -f

启动成功之后检查如下输出之后说明切换成功

kubectl describe nodes |grep -i runtime
  # 三个节点
  Container Runtime Version:    containerd://1.6.6
  Container Runtime Version:    containerd://1.6.6
  Container Runtime Version:    containerd://1.6.6

2.2.3. 部署buildkit

2.2.3.1. 构建buildkit镜像

社区的镜像不支持nydus，需要构建镜像

git clone https://github.com/nydusaccelerator/buildkit.git
cd buildkit

修改 Dockerfile

diff --git a/Dockerfile b/Dockerfile
index 8869138b..b006e584 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -97,7 +97,7 @@ ARG TARGETPLATFORM
 RUN --mount=target=. --mount=target=/root/.cache,type=cache \
   --mount=target=/go/pkg/mod,type=cache \
   --mount=source=/tmp/.ldflags,target=/tmp/.ldflags,from=buildkit-version \
-  CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd && \
+  CGO_ENABLED=0 xx-go build -ldflags "$(cat /tmp/.ldflags) -extldflags '-static'" -tags "osusergo netgo nydus static_build seccomp ${BUILDKITD_TAGS}" -o /usr/bin/buildkitd ./cmd/buildkitd && \
   xx-verify --static /usr/bin/buildkitd

 FROM scratch AS binaries-linux-helper
@@ -176,6 +176,7 @@ ARG ROOTLESSKIT_VERSION
 RUN git clone https://github.com/rootless-containers/rootlesskit.git /go/src/github.com/rootless-containers/rootlesskit
 WORKDIR /go/src/github.com/rootless-containers/rootlesskit
 ARG TARGETPLATFORM
+ENV GOPROXY "https://goproxy.cn,direct"
 RUN  --mount=target=/root/.cache,type=cache \
   git checkout -q "$ROOTLESSKIT_VERSION"  && \
   CGO_ENABLED=0 xx-go build -o /rootlesskit ./cmd/rootlesskit && \
@@ -203,6 +204,7 @@ RUN mkdir -p /out/nydus-static && tar xzvf nydus-static-$NYDUS_VERSION-$TARGETOS

 FROM buildkit-export AS buildkit-linux
 COPY --link --from=binaries / /usr/bin/
+COPY --from=nydus /out/nydus-static/* /usr/bin/
 ENTRYPOINT ["buildkitd"]

 FROM binaries AS buildkit-darwin

主要有三处修改:

编译增加nydus支持 – 必须
增加goproxy配置 – 国内拉依赖包会失败，需要配置goproxy
将nydus-image打入镜像，这步经测试可有可无

make build

会构建出两个镜像，我们只使用非rootless的 moby/buildkit:local

2.2.3.2. 按需修改yaml并部署

git clone https://github.com/nydusaccelerator/buildkit.git
cd buildkit

这里使用deployment+service.privileged ⁶

编辑 deployment+service.privileged.yaml，去掉了tls配置，增加了NYDUS_BUILDER环境变量，最终如下

镜像 shoery/buildkit:76aeaa55

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: buildkitd
  name: buildkitd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      labels:
        app: buildkitd
    spec:
      containers:
        - name: buildkitd
          env:
           - name: NYDUS_BUILDER
             value: /usr/bin/nydus-image
          image:  shoery/buildkit:76aeaa55
          args:
            - --addr
            - unix:///run/buildkit/buildkitd.sock
            - --addr
            - tcp://0.0.0.0:1234
          # the probe below will only work after Release v0.6.3
          readinessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          # the probe below will only work after Release v0.6.3
          livenessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          securityContext:
            privileged: true
          ports:
            - containerPort: 1234
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: buildkitd
  name: buildkitd
spec:
  ports:
    - port: 1234
      protocol: TCP
  selector:
    app: buildkitd

部署

kubectl apply -f examples/kubernetes/deployment+service.privileged.yaml

构建镜像时仅需客户端有 buildctl 即可, 通过 --addr tcp://0.0.0.0:1234 连接到 buildkitd

2.2.4. 部署 nydus-snapshotter

run_nydus_in_kubernetes

这里注意用 v0.8.2 以上的镜像版本

这里没太多修改的地方, 两个地方需要修改，其他按文档部署即可

git clone https://github.com/dragonflyoss/helm-charts.git
cd helm-charts

2.2.4.1. 更改nydus镜像

新建 config-nydus.yaml

name: nydus-snapshotter
image: ghcr.io/containerd/nydus-snapshotter
tag: v0.8.2
pullPolicy: IfNotPresent
hostNetwork: true
dragonfly:
  enable: false

containerRuntime:
  containerd:
    enable: true

2.2.4.2. 修改configmap

如果不增加nydus.toml配置，懒加载不会生效

编辑 charts/nydus-snapshotter/templates/nydus-snapshotter/snapshotter-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "nydus-snapshotter.fullname" . }}
  labels:
    app: {{ template "nydus-snapshotter.fullname" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    component: {{ .Values.name }}
data:
  config.json: |-
    {
      "device": {
        "backend": {
          "type": "registry",
          "config": {
          {{- if .Values.dragonfly.enable }}
            "mirrors": {{ mustToJson .Values.dragonfly.mirrorConfig }},
          {{- end }}
            "timeout": 5,
            "connect_timeout": 5,
            "retry_limit": 2
          }
        },
        "cache": {
          "type": "blobcache",
          "config": {
            "work_dir": "/var/lib/nydus/cache/"
          }
        }
      },
      "mode": "direct",
      "digest_validate": false,
      "iostats_files": false,
      "enable_xattr": true,
      "fs_prefetch": {
        "enable": true,
        "threads_count": 8,
        "merging_size": 1048576,
        "prefetch_all": true
      }
    }
  nydus.toml: |-
    version = 1
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
    daemon_mode = "multiple"
    cleanup_on_close = false
    [system]
    enable = true
    address = "/run/containerd-nydus/system.sock"
    [system.debug]
    daemon_cpu_profile_duration_secs = 5
    pprof_address = ""
    [daemon]
    nydusd_path = "/usr/local/bin/nydusd"
    nydusimage_path = "/usr/local/bin/nydus-image"
    fs_driver = "fusedev"
    recover_policy = "restart"
    nydusd_config = "/etc/nydus/config.json"
    threads_number = 10
    [log]
    log_to_stdout = false
    level = "debug"
    log_rotation_compress = true
    log_rotation_local_time = true
    log_rotation_max_age = 7
    log_rotation_max_backups = 5
    log_rotation_max_size = 1
    [metrics]
    address = ":9110"
    [remote]
    convert_vpc_registry = false
    [remote.mirrors_config]
    dir = ""
    [remote.auth]
    enable_kubeconfig_keychain = false
    kubeconfig_path = ""
    enable_cri_keychain = false
    image_service_address = ""
    [snapshot]
    enable_nydus_overlayfs = false
    sync_remove = false
    [cache_manager]
    disable = false
    gc_period = "24h"
    [image]
    public_key_file = ""
    validate_signature = false
    [experimental]
    enable_stargz = false
    enable_referrer_detect = false

安装

helm install --wait --timeout 10m --dependency-update \
    --create-namespace --namespace nydus-system \
    -f config-nydus.yaml \
    nydus-snapshotter charts/nydus-snapshotter

2.2.4.3. 重启nydus-snapshotter

后续维护中可能会有重启 nydus-snapshotter 的需求

由于修改containerd配置使用nyduys snapshotter之后, containerd和snapshotter会互相依赖，重启nydus-snapshotter容器会起不来，所以，每次重启都先把 /etc/containerd/config.toml 中如下部分去掉

[plugins."io.containerd.grpc.v1.cri".containerd]
discard_unpacked_layers = false
disable_snapshot_annotations = false
snapshotter = "nydus"
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

重启 containerd
重新创建 nydus-snapshotter pod

2.2.5. 验证

验证点: 在pod中构建nydus格式的镜像，使用pod启动nydus镜像容器

2.2.5.1. 在pod中构建nydus格式的镜像

要挂载 /root/.docker, 否则push镜像可能会没权限

挂载golang-helloworld的目的是将 hello 和 Dockerfile挂载进去

创建 build-nydus-image.yaml

apiVersion: v1
kind: Pod
metadata:
  name: build-pod
spec:
  restartPolicy: Never
  containers:
    - name: build-container
      image: buildkit:local
      command: ["buildctl", "--addr", "tcp://buildkitd.default:1234", "build", "--progress=plain", "--frontend=dockerfile.v0", "--local", "context=.", "--local", "dockerfile=.", "--output", "type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.6,compression=nydus,oci-mediatypes=true,push=true"]
      volumeMounts:
        - name: host-path-volume
          mountPath: /opt/golang-helloworld
        - name: docker-config-volume
          mountPath: /root/.docker
      workingDir: /opt/golang-helloworld
  volumes:
    - name: host-path-volume
      hostPath:
        path: /home/elrond/nydus/golang-helloworld
    - name: docker-config-volume
      hostPath:
        path: /root/.docker

命令说明

buildctl \
  # buildkitd的svc:port
  --addr tcp://default.buildkitd:1234 \
  build \
  # 输出的详细程度
  --progress=plain \
  # build frontend 类型
  --frontend=dockerfile.v0 \
  # 指定本地context路径
  --local context=.  \
  # 制定本地dockerfile路径
  --local dockerfile=.  \
  # 指定dockerfile文件，如果为空，默认值为Dockerfile
  --opt filename=hello.dockerfile
  # 镜像tag与类型、是否推送到远端
  --output type=image,name=reg.harbor.com/hello/hello-k8s:v0.0.6,compression=nydus,oci-mediatypes=true,push=true

开始构建

kubectl apply -f build-nydus-image.yaml

构建成功

k get po
NAME                         READY   STATUS      RESTARTS        AGE
build-pod                    0/1     Completed   0               23s

2.2.5.2. 使用pod启动nydus镜像容器

创建 nydus-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nydus-pod
spec:
  containers:
    - name: hello
      image: reg.harbor.com/hello/hello-k8s:v0.0.6
      imagePullPolicy: Always
      command: ["sh", "-c"]
      args:
        - tail -f /dev/null

运行

kubectl apply -f nydus-pod.yaml

这里可能会因为构建的镜像不正常，导致pod起不来，可以使用标准镜像 ghcr.io/dragonflyoss/image-service/nginx:nydus-latest 做测试

到这里在k8s上完成了nydus镜像的构建，和nydus镜像格式容器的启动

2.3. docker 集成

参照docker-env-setup

docker-nydus-graphdriver 尚在 Experimental 状态，且k8s在转向containerd，云原生社区对docker支持变弱，所以后续不考虑使用docker

3. CI集成 – skaffold集成nydus

编辑 skaffold.yaml

apiVersion: skaffold/v4beta4
kind: Config
build:
  tagPolicy:
    envTemplate:
      template: "{{.FOO}}"
  artifacts:
  - image: reg.harbor.com/hello/hello-k8s
    custom:
       buildCommand: buildctl --addr tcp://10.233.98.169:1234 build  --progress=plain --frontend=dockerfile.v0  --local context=. --local dockerfile=. --output type=image,name=reg.harbor.com/hello/hello-k8s:"{{.FOO}}",compression=nydus,oci-mediatypes=true,push=true --opt filename=hello.dockerfile
  local:
    push: true

入参:

–addr tcp://10.233.98.169 容器化的话，这个是svc的host和port
–local dockerfile=.: dockerfile路径
–output name=reg.harbor.com/hello/hello-k8s:“{{.FOO}}”: 镜像tag
–opt filename: dockerfile文件名

其他都是默认值

4. nydus镜像启动pod与ociv1镜像启动pod效果对比

镜像	ociv1拉取时间	nydus拉取时间
wordpress	12.26946259s	1.879414943s

4.1. wordpress镜像

先把镜像拉下来,推送到自己的仓库，然后转化一下推送到自己的仓库，用镜像启动k8s pod

docker pull wordpress:latest
docker tag wordpress:latest reg.harbor.com/hello/wordpress:latest
nydusify -D convert --nydus-image /usr/local/bin/nydus-image \
  --source reg.harbor.com/hello/wordpress:latest \
  --target reg.harbor.com/hello/wordpress:latest-nydus

4.1.1. oci v1镜像容器pulling image时间

新建 wordpress-ociv1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: wordpress-ociv1
spec:
  containers:
  - name: wordpress
    image: reg.harbor.com/hello/wordpress:latest
    imagePullPolicy: Always
    command:
      - sleep
      - "36000000000"
    resources:
      limits:
        cpu: "0.5"
        memory: "256Mi"

kubectl apply -f wordpress-ociv1.yaml
kubectl describe po wordpress-ociv1

Normal  Pulling    3m20s                 kubelet            Pulling image "reg.harbor.com/hello/wordpress:latest"
Normal  Pulled     3m7s                  kubelet            Successfully pulled image "reg.harbor.com/hello/wordpress:latest" in 12.26946259s

4.1.2. nydus镜像容器pulling image时间

新建 wordpress-nydus.yaml

apiVersion: v1
kind: Pod
metadata:
  name: wordpress-nydus
spec:
  containers:
  - name: wordpress
    image: reg.harbor.com/hello/wordpress:latest-nydus
    imagePullPolicy: Always
    command:
      - sleep
      - "36000000000"
    resources:
      limits:
        cpu: "0.5"
        memory: "256Mi"

Normal  Pulling    40s                kubelet            Pulling image "reg.harbor.com/hello/wordpress:latest-nydus"
Normal  Pulled     39s                kubelet            Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus" in 1.879414943s

5. 问题记录

5.1. 镜像无法构建

git clone https://github.com/nydusaccelerator/buildkit.git
cd buildkit
go build -tags=nydus -o ./bin/buildkitd ./cmd/buildkitd
mv ./bin/buildkitd /usr/bin/
chmod +x /usr/bin/buildkitd

原生的buildkit不支持nydus格式构建，需要重新编译支持
官方的buildkit对大镜像构建有bug，需要用nydus官方上游的仓库

5.2. containerd 命令速记

container与task⁷

主要用nerdctl工具

# 启动容器
ctr run -d docker.io/library/nginx:alpine my-container
# 容器列表
ctr -n default c ls
# task列表
ctr -n default task ls
# 删除容器
ctr -n default delete xxxxx
# 容器交互式命令
ctr task exec --tty --exec-id my-container my-container bash
# 镜像列表
ctr -n default images ls

# 获取contanierd镜像的manifest 先找到RepoDigests sha256
nerdctl inspect d88ba62a3cbb
cat /opt/docker/io.containerd.content.v1.content/blobs/sha256/d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3
# 转换镜像并推送到仓库
nydusify -D convert --nydus-image /usr/local/bin/nydus-image   --source  reg.harbor.com:443/library/golang@sha256:4b5cc6ca083856af8f58ab332f279b4401e1e46cc6d999957d5f88c563df24a7  --target reg.harbor.com:443/library/golang:1.18.7-nydus

nydusify -D convert --nydus-image /usr/local/bin/nydus-image   --source  ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5  --target reg.harbor.com/hello/ubuntu:nydus-nightly-v6

5.3. 镜像无法下载

nerdctl pull  ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5
ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5:                       resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:cb13d8b2c3b275fd392991b610e31462ab5dc588d4735b0c469ff867f86f617f:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:d9ea5e7108f5405da73ae48972441da2ab976da640264c4c06e07a473ebb86c3: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:d2197022013289d923e5b5aa9c9e22ec60bcdfa236948a02d24106b525f6171e:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f264de8a417d8927c431ef2a74286e24d716c3447a96ef7c7bd95d3af4ab3331:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c279d170bbf5c24d3b0b80bc1b49d147098d0f9f3f62cb010e98295b8d0e1f7c:    downloading    |+++++++++++++++++++++++++++++++++++++-| 31.0 MiB/31.1 MiB
docker.io/library/alpine:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:c0669ef34cdc14332c0f1ab0c2c01acb91d96014b172f1a76f3a39e63d1f0bda: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:5e2b554c1c45d22c9d1aa836828828e320a26011b76c08631ac896cbc3625e3e:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8a49fdb3b6a5ff2bd8ec6a86c05b2922a0f7454579ecc07637e94dfd1d0639b6:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 44.9s                                                                    total:  3.2 Mi (73.9 KiB/s)
FATA[0045] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2023-05-15T21:27:17+08:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/286be294817095497aef32271dac13b13e02756166a010917420039060f9aa9c/oci-hook.createRuntime.log: file already closed: unknown

解决升级cni到v1.1.1

5.4. 如何判断一个镜像是否是nydus镜像

如何判断一个镜像是否是nydus镜像⁸

5.4.1. 通过本地manifest判断

如果镜像在containerd上

nerdctl inspect d88ba62a3cbb
# 找到RepoDigests sha256 d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3

[
    {
        "Id": "sha256:13afb147bdb4382b43a833b646765dca8bc6fca92cc2a60b02e4fc718b356d97",
        "RepoTags": [
            "reg.harbor.com/hello/helloworld:v0.0.1"
        ],
        "RepoDigests": [
            "reg.harbor.com/hello/helloworld@sha256:d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3"
        ],
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-05-16T20:28:18.501558103+08:00",
        "Author": "",
        "Config": {
            "AttachStdin": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Entrypoint": [
                "/hello"
            ],
            "Labels": {
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.version": "22.04"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 79822848,
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",
                "sha256:95fc9a6f9c1a1ef74f6545d70c96011577f05741079cd8793aa134f4b1f51006",
                "sha256:719749732676f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24",
                "sha256:ed593dc031518310cd5ea3a5b20bfb2c73f6108badffb30b080de9edc251476d"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

查看manifest

# containerd的默认是/var/lib/containerd
cat /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/d88ba62a3cbb999055927cbd8a3dfcb2396abc2ea168c44a9e4e16df03f1b3a3

mediaType是nydus就是nydus镜像

{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:13afb147bdb4382b43a833b646765dca8bc6fca92cc2a60b02e4fc718b356d97",
    "size": 2137
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",
      "size": 32571260,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:95fc9a6f9c1a1ef74f6545d70c96011577f05741079cd8793aa134f4b1f51006",
      "size": 295017,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:719749732676f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24",
      "size": 1067099,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:bb55b253c36e5a008051aad37f2554a65ec6661476a08666d76e5ee1b84dbf4c",
      "size": 297871,
      "annotations": {
        "containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca\",\"95fc9a6f9c1a1ef74f6545d70c96011577f05741079cd8793aa134f4b1f51006\",\"719749732676f2e82e3baa4e1b0f5ea9cfa6aa83dde0c929a5ef8e78cacc7e24\"]",
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": ""
      }
    }
  ]
}

5.4.2. 通过nydusify check检查

nydusify check --target reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5

INFO[2023-05-18T10:46:09+08:00] Parsing image reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5
INFO[2023-05-18T10:46:11+08:00] Dumping OCI and Nydus manifests to ./output
INFO[2023-05-18T10:46:11+08:00] Pulling Nydus bootstrap to output/nydus_bootstrap
INFO[2023-05-18T10:46:11+08:00] Checking Nydus manifest
INFO[2023-05-18T10:46:11+08:00] Checking Nydus bootstrap
RAFS filesystem metadata is valid, referenced data blobs:
	 0: f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca, compressed data size 0x1e5beec, compressed file size 0x1e5c0ec, uncompressed file size 0x49b715a, chunks: 0x9d4, features: cap_toc tar-header
	 1: ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129, compressed data size 0x459da, compressed file size 0x45bda, uncompressed file size 0xb3c90, chunks: 0x1, features: cap_toc tar-header
	 2: 5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab, compressed data size 0x102213, compressed file size 0x102413, uncompressed file size 0x1ad499, chunks: 0x2, features: cap_toc tar-header
INFO[2023-05-18T10:46:11+08:00] Verified Nydus image reg.harbor.com/hello/helloworld:v0.0.11-nydus-nightly-v5

查看nydus manifest

cat output/nydus_manifest.json

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:f259d1eb0bf769d60aac8c78e4811a8eeefd811a27c24f6db6c36f72ac5c2324",
    "size": 2137
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",
      "size": 32571260,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129",
      "size": 295018,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab",
      "size": 1067099,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e26163305101a9bc6d05ad80945a8d360d99a48b0d3ef070cf0c2137655b6115",
      "size": 297868,
      "annotations": {
        "containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca\",\"ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab\"]",
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": ""
      }
    }
  ]
}

5.5. nydus snapshotter无法启动

可能是有另一个 nydus-snapshotter进程在启动 ps -ef|grep containerd-nydus-grpc

/usr/local/bin/containerd-nydus-grpc --root /opt/docker/io.containerd.snapshotter.v1.nydus --config-path /etc/nydus/nydusd-config.fusedev.json --config /etc/nydus/config.toml --log-to-stdout --log-level trace
INFO[2023-05-17T15:08:33.047605411+08:00] Start nydus-snapshotter. PID 25641 Version v0.8.0 FsDriver fusedev DaemonMode multiple
FATA[2023-05-17T15:08:37.018373244+08:00] failed to start nydus-snapshotter             error="failed to initialize snapshotter: create database: timeout"ß

5.6. buildkit构建nydus镜像构建不成功

buildctl  build  --progress=plain --frontend=dockerfile.v0  --local context=.  --local dockerfile=.  --output type=image,name=reg.harbor.com/hello/helloworld:v0.0.4-nydus-from-golang,compression=nydus,oci-mediatypes=true,push=true

#6 [2/2] COPY hello /
#6 ERROR: mount callback failed on /run/user/0/containerd-mount2534958279: apply nydus blob: unpack nydus blob: unpack nydus tar: unpack bootstrap from nydus: parse tar header: rpc error: code = OutOfRange desc = read past object length 54797046 bytes
------
 > [2/2] COPY hello /:
------
Dockerfile:2
--------------------
   1 |     FROM reg.harbor.com/library/golang:1.18.7-nydus
   2 | >>> COPY hello /
   3 |     ENTRYPOINT [ "/hello" ]
   4 |
--------------------
error: failed to solve: OutOfRange: failed to compute cache key: mount callback failed on /run/user/0/containerd-mount2534958279: apply nydus blob: unpack nydus blob: unpack nydus tar: unpack bootstrap from nydus: parse tar header: rpc error: code = OutOfRange desc = read past object length 54797046 bytes

nydus-image log

time="2023-05-17T15:34:24+08:00" level=debug msg="saved dapdgw7aeqou4bbcymhfxc94n as context:context:" span="[internal] load build context" spanID=7108c9f5c020f1dd traceID=3841af58efa9339c11d3c4107ad1e65a
time="2023-05-17T15:34:24+08:00" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown"
failed to compute cache key: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown
20015 0.0.0+unknown buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/solver.(*edge).createInputRequests.func1.1
	/root/buildkit/solver/edge.go:839
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
	/root/buildkit/solver/internal/pipe/pipe.go:82
runtime.goexit
	/root/sdk/go1.18/src/runtime/asm_amd64.s:1571

20015 0.0.0+unknown buildkitd --config /etc/buildkit/buildkitd.toml
main.unaryInterceptor.func1
	/root/buildkit/cmd/buildkitd/main.go:576
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
	/root/buildkit/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
	/root/buildkit/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
	/root/buildkit/api/services/control/control.pb.go:2440
google.golang.org/grpc.(*Server).processUnaryRPC
	/root/buildkit/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
	/root/buildkit/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/root/buildkit/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
	/root/sdk/go1.18/src/runtime/asm_amd64.s:1571

使用 nydusaccelerator/buildkit 构建buildkit之后未复现

5.7. dragonflyoss仓库的镜像无法 inspect

 nerdctl inspect ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xda8eb3]

goroutine 1 [running]:
github.com/containerd/nerdctl/pkg/inspecttypes/dockercompat.ImageFromNative(0xc0006cc000)
	/home/runner/work/nerdctl/nerdctl/pkg/inspecttypes/dockercompat/dockercompat.go:297 +0x253
github.com/containerd/nerdctl/pkg/cmd/image.Inspect.func1({0x13f0190?, _}, {{{0xc0006b0280, 0x3a}, 0x0, {{0xc0006ae0c0, 0x27}, {0xc0006c0050, 0x47}, 0x248, ...}, ...}, ...})
	/home/runner/work/nerdctl/nerdctl/pkg/cmd/image/inspect.go:52 +0x205
github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).Walk(0xc000547568, {0x13f0190, 0xc00014cd20}, {0x7ffc1ceb0732, 0x3a})
	/home/runner/work/nerdctl/nerdctl/pkg/idutil/imagewalker/imagewalker.go:81 +0x666
github.com/containerd/nerdctl/pkg/idutil/imagewalker.(*ImageWalker).WalkAll(0x0?, {0x13f0190, 0xc00014cd20}, {0xc0005477c0?, 0x1, 0x103b5e0?}, 0x1)
	/home/runner/work/nerdctl/nerdctl/pkg/idutil/imagewalker/imagewalker.go:98 +0xea
github.com/containerd/nerdctl/pkg/cmd/image.Inspect({_, _}, _, {_, _, _}, {{0x13e7d20, 0xc00012e008}, {0x0, 0x0, ...}, ...})
	/home/runner/work/nerdctl/nerdctl/pkg/cmd/image/inspect.go:64 +0x1a6
main.inspectAction(0xc000382600, {0xc00034e040, 0x1, 0x1?})
	/home/runner/work/nerdctl/nerdctl/cmd/nerdctl/inspect.go:147 +0xb7c
github.com/spf13/cobra.(*Command).execute(0xc000382600, {0xc000120170, 0x1, 0x1})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004300)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
main.xmain()
	/home/runner/work/nerdctl/nerdctl/cmd/nerdctl/main.go:136 +0x9f
main.main()
	/home/runner/work/nerdctl/nerdctl/cmd/nerdctl/main.go:119 +0x19

原因未知

5.8. nydus镜像与oci镜像manifest的区别

ociv1⁹

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 7023,
    "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 32654,
      "digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 16724,
      "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 73109,
      "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
    }
  ],
  "subject": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "size": 7682,
    "digest": "sha256:5b0bcabd1ed22e9fb1310cf6c2dec7cdef19f0ad69efa1f392e94a4333501270"
  },
  "annotations": {
    "com.example.key1": "value1",
    "com.example.key2": "value2"
  }
}

docker v2.2

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4710,
         "digest": "sha256:016db0f74f4d70593d50a571e76792f1223971e14fb791f9f6fbef3695846295",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4708,
         "digest": "sha256:1a3cd94d36b750c10782c41bf88458128686cf9b5b81da4bb136dae7c11e439f",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v5"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4708,
         "digest": "sha256:fac5d849d932a3a33d421f2918d6b36214fb8f048a04533e95bba5a1da263dc5",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4709,
         "digest": "sha256:6196a365b5ac3c585d65aeb73b68810fc9a24453d5e78ae6a8f229a0d8edf060",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4710,
         "digest": "sha256:1703758d9c663476da8b6e3ae929a359633d93f217dabb75fc09bda99b46aa84",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4708,
         "digest": "sha256:d4089c75201a6f692e55a9b5480ab7bb94a58937d4452d444f06863324d1ccd5",
         "platform": {
            "architecture": "mips64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4710,
         "digest": "sha256:bc495de2c53f8e0df08ee58ab06ef2e4fae9304826f649c8285c27a3ee9674eb",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4708,
         "digest": "sha256:bb98ee8ed8c2bf30b82a612775d9218cde446a7ea342d58b1826b2b29aeccaad",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

nydus

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:f259d1eb0bf769d60aac8c78e4811a8eeefd811a27c24f6db6c36f72ac5c2324",
    "size": 2137
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca",
      "size": 32571260,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129",
      "size": 295018,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab",
      "size": 1067099,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e26163305101a9bc6d05ad80945a8d360d99a48b0d3ef070cf0c2137655b6115",
      "size": 297868,
      "annotations": {
        "containerd.io/snapshot/nydus-blob-ids": "[\"f9ebd7c08b04c9dca8fd40cf8855097fb0f5285992224bfaa58afdf0cba82cca\",\"ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129\",\"5fdd275fe4099c6b7b4a3f8be50c2dc89bad3ab771d5fe454b9746471c6daaab\"]",
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": ""
      }
    }
  ]
}

5.9. 构建的镜像list时告警

nerdctl images
WARN[0000] failed to get unpacked size of image "reg.harbor.com/hello/helloworld:v0.0.7-nydus-nightly-v5" for platform "linux/amd64"  error="snapshot sha256:0ee114e6b61fec8478b90a426bdcbbc145d5857d84822922152dc8e09111f391 does not exist: not found"
REPOSITORY                            TAG                        IMAGE ID        CREATED           PLATFORM       SIZE     BLOB SIZE
reg.harbor.com/hello/helloworld    v0.0.7-nydus-nightly-v5    484d07c284ad    33 seconds ago    linux/amd64    0.0 B    345.5 MiB

本地镜像存储问题删除本地镜像即可，具体问题尚未可知

5.10. 问题 buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题

现象:

buildkit打的nydus镜像无法run ,run ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 没问题
使用nerdctl命令运行也一样

ctr run -d reg.harbor.com/hello/helloworld:v0.0.10-nydus-nightly-v5  hello

错误输出

INFO[0000] apply failure, attempting cleanup             error="failed to extract layer sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown" key="extract-962072102-FeAw sha256:f5294af206581b011d1423a0b7c580b44338d6fb790eaf0ebdcc838fcb107d29"
ctr: failed to extract layer sha256:ae29dc152fabfc7e6239b3a1f83501b5a48875711b7f53e6ebf7d96b22ad4129: failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type: unknown

构建完使用 nerdctl命令查看有告警信息

 nerdctl images
WARN[0000] failed to get unpacked size of image "reg.harbor.com/hello/helloworld:v0.0.10-nydus-nightly-v5" for platform "linux/amd64"  error="snapshot sha256:66e2cbc88c92b00741c687d56bedd11654d45c08dec2fcb38406683fb2695135 does not exist: not found"
REPOSITORY                                   TAG                         IMAGE ID        CREATED           PLATFORM       SIZE        BLOB SIZE
xxxx/hello/helloworld           v0.0.10-nydus-nightly-v5    b3faf19db1a6    11 minutes ago    linux/amd64    0.0 B       32.6 MiB

containerd配置

[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
   snapshotter = "nydus"
   disable_snapshot_annotations = false
   discard_unpacked_layers = false

环境说明

containerd containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
buildkitd -v buildkitd github.com/moby/buildkit v0.0.0+unknown 是通过ttps://github.com/nydusaccelerator/buildkit.git 构建的
nerdctl -v nerdctl version 1.3.1 在 /etc/nerdctl/nerdctl.toml 指定了 snapshotter

debug          = false
debug_full     = false
address        = "unix:///var/run/containerd/containerd.sock"
namespace      = "default"
snapshotter    = "nydus"
experimental   = true

buildkitd 启动命令

NYDUS_BUILDER=/usr/bin/nydus-image buildkitd --oci-worker-snapshotter=nydus --config /etc/buildkit/buildkitd.toml`

构建命令

buildctl  build  --progress=plain --frontend=dockerfile.v0  --local context=.  --local dockerfile=.  --output type=image,name=xxxx/helloworld:v0.0.10-nydus-nightly-v5,compression=nydus,oci-mediatypes=true

本地镜像存储问题删除本地镜像即可，问题同上，nydus-snapshotter+fusedev的问题，具体尚未可知，删除本地镜像使用远端镜像即可

5.11. 运行oci-ref镜像导致nydusd进程z状态

使用nydus zran artifact

使用zran artifact镜像启动容器导致nydusd进程d状态

5.11.1. 现象

pod 无法启动，describe po信息

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  33s                default-scheduler  Successfully assigned default/busybox-nydus-oci-ref to node1
  Normal   Pulled     31s                kubelet            Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 1.035131143s
  Normal   Pulled     25s                kubelet            Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 119.347118ms
  Warning  Failed     19s (x2 over 25s)  kubelet            Error: failed to create containerd container: wait until daemon is RUNNING: get daemon state: daemon socket /var/lib/containerd-nydus/socket/cho77etp707gng95ic00/api.sock: not found
  Normal   Pulling    5s (x3 over 32s)   kubelet            Pulling image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref"
  Normal   Pulled     5s                 kubelet            Successfully pulled image "reg.harbor.com/hello/wordpress:latest-nydus-oci-ref" in 101.618808ms

查看nydus-snapshotter log

2023-05-26 08:54:19.401685 +00:00] INFO [app/src/lib.rs:74] Program Version: v2.1.5, Git Commit: "ef2033c2e21d36938410069f07d32a5095a19398", Build Time: "2023-03-09T10:09:03.563891101Z", Profile: "release", Rustc Version: "rustc 1.61.0 (fe5b13d68 2022-05-18)"
[2023-05-26 08:54:19.401791 +00:00] INFO [src/bin/nydusd/main.rs:514] Set rlimit-nofile to 1000000, maximum 1048576
[2023-05-26 08:54:19.402224 +00:00] DEBUG [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/pseudo_fs.rs:161] pseudo fs iterate "/"
[2023-05-26 08:54:19.402452 +00:00] INFO [rafs/src/metadata/md_v6.rs:47] rafs superblock features: DIGESTER_SHA256 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESS_GZIP | PRESERVED_INLINED_CHUNK_DIGEST
[2023-05-26 08:54:19.402573 +00:00] ERROR [rafs/src/metadata/layout/v6.rs:1375] RafsV6Blob: idx 0 invalid ci_digest
[2023-05-26 08:54:19.402588 +00:00] ERROR [error/src/error.rs:21] Error:
	"invalid Rafs v6 blob entry"
	at rafs/src/metadata/layout/v6.rs:1495
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2023-05-26 08:54:19.402630 +00:00] ERROR [error/src/error.rs:21] Error:
	Rafs(FillSuperblock(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))
	at src/bin/nydusd/daemon.rs:140
	note: enable `RUST_BACKTRACE=1` env to display a backtrace
[2023-05-26 08:54:19.402669 +00:00] ERROR [src/bin/nydusd/main.rs:664] Failed in starting daemon: Invalid argument (os error 22)
Error: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
time="2023-05-26T08:54:19.404707985Z" level=error msg="Process 14029 has been a zombie"
time="2023-05-26T08:54:19.404759638Z" level=error msg="Nydusd cho77etp707gng95ic00 probably not started"
time="2023-05-26T08:54:19.412716804Z" level=error msg="Process 14029 has been a zombie"
time="2023-05-26T08:54:19.579192096Z" level=error msg="Process 14029 has been a zombie"
time="2023-05-26T08:54:19.719630411Z" level=error msg="Process 14029 has been a zombie"
time="2023-05-26T08:54:19.853129550Z" level=error msg="Process 14029 has been a zombie"
time="2023-05-26T08:54:20.023548928Z" level=error msg="Process 14029 has been a zombie"
time="2023-05-26T08:54:20.181968604Z" level=error msg="Process 14029 has been a zombie"

查看进程进入z状态

[root@node1 nydus]# ps -ef|grep 14029
root     14029 22492  0 08:54 ?        00:00:00 [nydusd] <defunct>
root     16385 15474  0 08:56 pts/2    00:00:00 grep --color=auto 14029

5.11.2. 解决方式

oci ref 要求 nydus-image/nydusd 2.2

5.12. nydus镜像和oci镜像拉取镜像时间相同

nydus镜像容器创建时最开始只会加载最后一层，所以会很快

{
  "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
  "digest": "sha256:ff67b2807cdb0005ac0137bb8d5a4cf453fead83247082f56351fa2ecc3acba0",
  "size": 1134668,
  "annotations": {
    "containerd.io/snapshot/nydus-bootstrap": "true",
    "containerd.io/snapshot/nydus-fs-version": "6"
  }
}

对比的时候记得把已经在本地的镜像删除掉

编辑 configmap 增加nydus.toml配置

kubectl edit cm -n nydus-system  nydus-snapshotter

apiVersion: v1
data:
  config.json: |-
    ...
  nydus.toml: |-
    version = 1
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
    daemon_mode = "multiple"
    cleanup_on_close = false
    [system]
    enable = true
    address = "/run/containerd-nydus/system.sock"
    [system.debug]
    daemon_cpu_profile_duration_secs = 5
    pprof_address = ""
    [daemon]
    nydusd_path = "/usr/local/bin/nydusd"
    nydusimage_path = "/usr/local/bin/nydus-image"
    fs_driver = "fusedev"
    recover_policy = "restart"
    nydusd_config = "/etc/nydus/config.json"
    threads_number = 10
    [log]
    log_to_stdout = false
    level = "debug"
    log_rotation_compress = true
    log_rotation_local_time = true
    log_rotation_max_age = 7
    log_rotation_max_backups = 5
    log_rotation_max_size = 1
    [metrics]
    address = ":9110"
    [remote]
    convert_vpc_registry = false
    [remote.mirrors_config]
    dir = ""
    [remote.auth]
    enable_kubeconfig_keychain = false
    kubeconfig_path = ""
    enable_cri_keychain = false
    image_service_address = ""
    [snapshot]
    enable_nydus_overlayfs = false
    sync_remove = false
    [cache_manager]
    disable = false
    gc_period = "24h"
    [image]
    public_key_file = ""
    validate_signature = false
    [experimental]
    enable_stargz = false
    enable_referrer_detect = false

重启nydus-snapshotter

去掉 /etc/containerd/config.toml 中的

[plugins."io.containerd.grpc.v1.cri".containerd]
discard_unpacked_layers = false
disable_snapshot_annotations = false
snapshotter = "nydus"
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"

重启 containerd

systemctl restart containerd

delete nydus-snapshot容器

再次创建即可

5.12.1. nydus镜像启动 nydus-snapshotter日志

nydus镜像会有.rs 输出，这部分是nydus代码用rust编写

time="2023-05-26T11:04:29.478658593Z" level=info msg="[Mounts] snapshot k8s.io/699/987c0c48cdfcbd8b451e7c892d2b922305c3c2fa07e2741ae5d003792dff96f9 ID 457 Kind Active"
time="2023-05-26T11:04:29.838641967Z" level=info msg="[Commit] snapshot with key \"k8s.io/700/extract-821649751-v57_ sha256:17f262ebfabe0a80a19356d4acdde01515c6488b4baf9f474ba805b205be6bcd\" snapshot id 458"
time="2023-05-26T11:04:29.862232365Z" level=info msg="[Commit] snapshot with key \"k8s.io/701/extract-848480697-roQj sha256:4969b12a639d311e01eb2997dca25b3ff4c15297cf1c4332246efc3da416ed90\" snapshot id 459"
time="2023-05-26T11:04:29.884272468Z" level=info msg="[Commit] snapshot with key \"k8s.io/702/extract-875046240-ES_o sha256:eea7439600aea67d552929f639492138db7736625ce46442b556a6773a7e65ad\" snapshot id 460"
time="2023-05-26T11:04:29.902972106Z" level=info msg="[Commit] snapshot with key \"k8s.io/703/extract-893877298-M0p7 sha256:995607f2a156cd5d860cc6cf21d5924430eea1497344c187cef6d8cb5958f08d\" snapshot id 461"
time="2023-05-26T11:04:29.930234997Z" level=info msg="[Commit] snapshot with key \"k8s.io/704/extract-921174867-zDfC sha256:977a5025ef78849159b33940f21f7bc078a4b449a197bf5a84809d18eafa2ecc\" snapshot id 462"
time="2023-05-26T11:04:29.953002744Z" level=info msg="[Commit] snapshot with key \"k8s.io/705/extract-940269515-m4B7 sha256:5008756b9b3af58134566161e811ab255d360cb9f6c03db6be9d1a82b10ea1ce\" snapshot id 463"
time="2023-05-26T11:04:29.985446333Z" level=info msg="[Commit] snapshot with key \"k8s.io/706/extract-963184826-zC2V sha256:dbb29c913fcddebd164aff00387b0cf2ae2605ca7c3ff8ae7fc81e3770c09d6d\" snapshot id 464"
time="2023-05-26T11:04:30.009608437Z" level=info msg="[Commit] snapshot with key \"k8s.io/707/extract-994808807-KS4S sha256:1dc2d75a8c5b4572fcaeb4ee74ee4a34f1f557cf380d85560bea88ddf8e656f1\" snapshot id 465"
time="2023-05-26T11:04:30.034452304Z" level=info msg="[Commit] snapshot with key \"k8s.io/708/extract-19386797-sAab sha256:2e352e3c2a94ff6a53bda70e8c78f1c0061bc5c09918d42f10ee42ca0ea6e05d\" snapshot id 466"
time="2023-05-26T11:04:30.051992401Z" level=info msg="[Commit] snapshot with key \"k8s.io/709/extract-43376118-0fYc sha256:c4655b7ee2454381d27aae6af1bbf99061da67f84943ebf11657db9841d2ce34\" snapshot id 467"
time="2023-05-26T11:04:30.069947216Z" level=info msg="[Commit] snapshot with key \"k8s.io/710/extract-61015950-IVNU sha256:c1775d8affc2abbc45ed4051e72fa8083f624f1a52f938e3dd4017ff2a551216\" snapshot id 468"
time="2023-05-26T11:04:30.088621057Z" level=info msg="[Commit] snapshot with key \"k8s.io/711/extract-79602317-8xBS sha256:7cd39c27735b21390c4f959ca1e12d38c2b73a5f173f5ffd1ee313fee85e317e\" snapshot id 469"
time="2023-05-26T11:04:30.107545892Z" level=info msg="[Commit] snapshot with key \"k8s.io/712/extract-98392114-LQ31 sha256:e835052588cf3a25d66b8a4f683b381ec2314cd293a61d7405861aedce7f1d8f\" snapshot id 470"
time="2023-05-26T11:04:30.138067726Z" level=info msg="[Commit] snapshot with key \"k8s.io/713/extract-129097712-x2We sha256:413163a369a3f699e6971333b345cb9655d22d8850e1713e20766dcc5fc6cf2d\" snapshot id 471"
time="2023-05-26T11:04:30.158431236Z" level=info msg="[Commit] snapshot with key \"k8s.io/714/extract-148574340-w72W sha256:856e93d9260062e3351e85ff977859c11f1c6fcaf4d7de56c50a4a40936ea6dc\" snapshot id 472"
time="2023-05-26T11:04:30.181674205Z" level=info msg="[Commit] snapshot with key \"k8s.io/715/extract-170870922-OaK7 sha256:bfdfb86a11099f8dfdcecb08a46dee87408cb00dedbb69d8c41bd4524e661fb6\" snapshot id 473"
time="2023-05-26T11:04:30.202089436Z" level=info msg="[Commit] snapshot with key \"k8s.io/716/extract-192117639-MjNO sha256:205b438eb857a4428cdfc1a492c3e9e445a7f6aad780f69b082e28471216c9a2\" snapshot id 474"
time="2023-05-26T11:04:30.224765412Z" level=info msg="[Commit] snapshot with key \"k8s.io/717/extract-214032149-U7eo sha256:a72248b4d79a6915f529bb00b97b576fec938895d81c0ebb125a02dd76f91169\" snapshot id 475"
time="2023-05-26T11:04:30.244866341Z" level=info msg="[Commit] snapshot with key \"k8s.io/718/extract-234486897-32MC sha256:f0fc4ec3e74f3762cc5c68435c8c93fb92fcac6ae34053cae00ae90dcafceab6\" snapshot id 476"
time="2023-05-26T11:04:30.428447831Z" level=info msg="[Commit] snapshot with key \"k8s.io/719/extract-255443005-VBbE sha256:3be1111cf1b2f35366acf9d164434690af4860fe8415c7be30faae3de1fde0f3\" snapshot id 477"
time="2023-05-26T11:04:30.495389473Z" level=info msg="Prepares active snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481, nydusd should start afterwards" key=k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 parent="k8s.io/720/sha256:3be1111cf1b2f35366acf9d164434690af4860fe8415c7be30faae3de1fde0f3"
time="2023-05-26T11:04:30.501028169Z" level=info msg="nydusd command: /usr/local/bin/nydusd fuse --config /var/lib/containerd-nydus/config/cho94flp707i10bb5ddg/config.json --bootstrap /var/lib/containerd-nydus/snapshots/477/fs/image/image.boot --mountpoint /var/lib/containerd-nydus/snapshots/477/mnt --apisock /var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock --log-level debug"
[2023-05-26 11:04:30.503482 +00:00] INFO [app/src/lib.rs:74] Program Version: v2.1.5, Git Commit: "ef2033c2e21d36938410069f07d32a5095a19398", Build Time: "2023-03-09T10:09:03.563891101Z", Profile: "release", Rustc Version: "rustc 1.61.0 (fe5b13d68 2022-05-18)"
[2023-05-26 11:04:30.503560 +00:00] INFO [src/bin/nydusd/main.rs:514] Set rlimit-nofile to 1000000, maximum 1048576
[2023-05-26 11:04:30.504138 +00:00] INFO [rafs/src/metadata/md_v6.rs:47] rafs superblock features: DIGESTER_BLAKE3 | EXPLICIT_UID_GID | HAS_XATTR | COMPRESS_ZSTD | PRESERVED_INLINED_CHUNK_DIGEST
[2023-05-26 11:04:30.504414 +00:00] INFO [storage/src/backend/connection.rs:263] backend config: ConnectionConfig { proxy: ProxyConfig { url: "", ping_url: "", fallback: false, check_interval: 5, use_http: false }, mirrors: [], skip_verify: false, timeout: 5, connect_timeout: 5, retry_limit: 2 }
[2023-05-26 11:04:30.631601 +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time: 1685100870
[2023-05-26 11:04:30.647334 +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time: 1685100870
[2023-05-26 11:04:30.668014 +00:00] INFO [storage/src/backend/registry.rs:271] cached bearer auth, next time: 1685100870
[2023-05-26 11:04:30.918676 +00:00] INFO [src/bin/nydusd/fs_service.rs:206] Rafs imported
[2023-05-26 11:04:30.920073 +00:00] INFO [src/bin/nydusd/fs_service.rs:102] Rafs mounted at /
[2023-05-26 11:04:30.920318 +00:00] INFO [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/transport/fusedev/linux_session.rs:337] mount source rafs dest /var/lib/containerd-nydus/snapshots/477/mnt with fstype fuse opts default_permissions,allow_other,fd=3,rootmode=40000,user_id=0,group_id=0 fd 3
[2023-05-26 11:04:30.920759 +00:00] INFO [src/bin/nydusd/daemon.rs:330] State machine(pid=32141): from Init to Ready, input [Mount], output [None]
[2023-05-26 11:04:30.921167 +00:00] INFO [src/bin/nydusd/daemon.rs:330] State machine(pid=32141): from Ready to Running, input [Start], output [Some(StartService)]
[2023-05-26 11:04:30.921189 +00:00] INFO [src/bin/nydusd/fusedev.rs:318] start 4 fuse servers
[2023-05-26 11:04:30.930559 +00:00] INFO [/cargo/registry/src/github.com-1ecc6299db9ec823/fuse-backend-rs-0.9.6/src/api/server/sync_io.rs:614] FUSE INIT major 7 minor 23
[2023-05-26 11:04:30.930808 +00:00] INFO [src/bin/nydusd/main.rs:660] Fuse daemon started!
[2023-05-26 11:04:30.931000 +00:00] INFO [src/bin/nydusd/api_server_glue.rs:406] HTTP API server running at /var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock
[2023-05-26 11:04:30.931090 +00:00] INFO [api/src/http.rs:911] http server started
[2023-05-26 11:04:30.961941 +00:00] INFO [api/src/http.rs:816] <--- Get Uri { string: "/api/v1/daemon" }
[2023-05-26 11:04:30.962087 +00:00] INFO [api/src/http.rs:821] ---> Get Status Code: OK, Elapse: Ok(146.605µs), Body Size: 866
time="2023-05-26T11:04:30.962507763Z" level=info msg="Nydus remote snapshot 477 is ready"
time="2023-05-26T11:04:30.962543380Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"
time="2023-05-26T11:04:30.964003080Z" level=info msg="Subscribe daemon cho94flp707i10bb5ddg liveness event, path=/var/lib/containerd-nydus/socket/cho94flp707i10bb5ddg/api.sock."
time="2023-05-26T11:04:30.969335490Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"
time="2023-05-26T11:04:30.969369349Z" level=info msg="Nydus remote snapshot 477 is ready"
time="2023-05-26T11:04:30.969416607Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"
time="2023-05-26T11:04:31.124174858Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"
time="2023-05-26T11:04:31.124228061Z" level=info msg="Nydus remote snapshot 477 is ready"
time="2023-05-26T11:04:31.124290212Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"
time="2023-05-26T11:04:31.355843306Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"
time="2023-05-26T11:04:31.355922313Z" level=info msg="Nydus remote snapshot 477 is ready"
time="2023-05-26T11:04:31.355974771Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"
time="2023-05-26T11:04:31.558021456Z" level=info msg="[Mounts] snapshot k8s.io/721/a779624ddf92028254a90857bb598a256a309433088bde9e57c23b0a214a5481 ID 478 Kind Active"
time="2023-05-26T11:04:31.558049891Z" level=info msg="Nydus remote snapshot 477 is ready"
time="2023-05-26T11:04:31.558096530Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/478/work upperdir=/var/lib/containerd-nydus/snapshots/478/fs lowerdir=/var/lib/containerd-nydus/snapshots/477/mnt]"

5.12.2. oci镜像启动 nydus-snapshotter日志

time="2023-05-26T11:06:30.246293363Z" level=info msg="[Mounts] snapshot k8s.io/722/3d4aa771e42ef99f0d65ce322670115c434297890eaf08bc260eecf2ec3461f7 ID 479 Kind Active"
time="2023-05-26T11:06:33.865401086Z" level=info msg="[Commit] snapshot with key \"k8s.io/723/extract-632530774-VjiP sha256:8cbe4b54fa88d8fc0198ea0cc3a5432aea41573e6a0ee26eca8c79f9fbfa40e3\" snapshot id 480"
time="2023-05-26T11:06:34.704875352Z" level=info msg="[Commit] snapshot with key \"k8s.io/725/extract-249815542-Zc-C sha256:61aba3d78f453ab50b4c0b687156ebb47b6d6dc290f57752d88b737f75a9835c\" snapshot id 481"
time="2023-05-26T11:06:37.941249262Z" level=info msg="[Commit] snapshot with key \"k8s.io/727/extract-722902017-GvJe sha256:d7027bd638e20f68a20d6ae247f2dda259aed1b46a6737bb70de28fa781a1769\" snapshot id 482"
time="2023-05-26T11:06:38.009096474Z" level=info msg="[Commit] snapshot with key \"k8s.io/729/extract-986715165-jE_Q sha256:f116aef2f51af904cebd6baaf86a911036f8540f0837f5081dff235b29135d32\" snapshot id 483"
time="2023-05-26T11:06:38.639504208Z" level=info msg="[Commit] snapshot with key \"k8s.io/731/extract-24233762-aGRw sha256:877496570f634566fc75289c3fd7cc9423f8b42f1b9cf3d5f1ed3703f0ec83e0\" snapshot id 484"
time="2023-05-26T11:06:38.676870045Z" level=info msg="[Commit] snapshot with key \"k8s.io/733/extract-661252421-FKvu sha256:41ca5e8fdea7603d4b22d1bd5dacf18cd8ae4a337c799224034fb38329be6428\" snapshot id 485"
time="2023-05-26T11:06:38.705044906Z" level=info msg="[Commit] snapshot with key \"k8s.io/735/extract-691144330-DHtG sha256:748bab432b39955f9824803e0fb9e2c838c1d82613f6331d4dd06806c6eb08a7\" snapshot id 486"
time="2023-05-26T11:06:38.790391324Z" level=info msg="[Commit] snapshot with key \"k8s.io/737/extract-721825026-kLLd sha256:a7317289186ffc46ccf46108f1a71012bda8de401a9772596b47ac2165279dde\" snapshot id 487"
time="2023-05-26T11:06:38.824921071Z" level=info msg="[Commit] snapshot with key \"k8s.io/739/extract-810354999-OX_U sha256:290e04ebe112a50211c9349710c52fcdba1e90c13c86b7d28fc9d5810fb4ff66\" snapshot id 488"
time="2023-05-26T11:06:39.284365821Z" level=info msg="[Commit] snapshot with key \"k8s.io/741/extract-838355879-06mH sha256:f56f832ba542550bd23ff7bd23bff350112f52c30a25fff73521d22231ae96d2\" snapshot id 489"
time="2023-05-26T11:06:39.318325143Z" level=info msg="[Commit] snapshot with key \"k8s.io/743/extract-302049626-ZzGp sha256:7edd07877746040a573221b32d42a30e71ac050a4b0b52e52637a109f125e07a\" snapshot id 490"
time="2023-05-26T11:06:39.346626770Z" level=info msg="[Commit] snapshot with key \"k8s.io/745/extract-332377616-20YV sha256:8e0edcf4b4ba88591fb18a3189e4ef180a3e907b3d3d374cec1688975c59210a\" snapshot id 491"
time="2023-05-26T11:06:39.373512591Z" level=info msg="[Commit] snapshot with key \"k8s.io/747/extract-359199786-wwaO sha256:0c386a4066a2ba95277419d44500293c0f4fdb017f83fc5f5c8e1047609bc057\" snapshot id 492"
time="2023-05-26T11:06:40.011031985Z" level=info msg="[Commit] snapshot with key \"k8s.io/749/extract-386775080-gUBX sha256:674062350681f3cc06dcc025867235baa315952732ac488d5692347118f5f027\" snapshot id 493"
time="2023-05-26T11:06:40.558428379Z" level=info msg="[Commit] snapshot with key \"k8s.io/751/extract-37315161-xwQT sha256:e723aac1be491e74d1de67af90ddff842b1428f56cc2b36e036c9bb1bf76fd99\" snapshot id 494"
time="2023-05-26T11:06:40.591120152Z" level=info msg="[Commit] snapshot with key \"k8s.io/753/extract-577254979-GuvE sha256:7ac16f932064276888bd1c8be5d3085c99fa945d1545f8b6e4969cc9ba7f932e\" snapshot id 495"
time="2023-05-26T11:06:40.617389492Z" level=info msg="[Commit] snapshot with key \"k8s.io/755/extract-604669307-p-5u sha256:4c8bfefa8feaa457402b6cfbb30bdb73dba02d105a432d4c3cdd3b4c02ba53d1\" snapshot id 496"
time="2023-05-26T11:06:40.650440923Z" level=info msg="[Commit] snapshot with key \"k8s.io/757/extract-630649889-v1Ai sha256:239e5e3542f395b783fe8f614b0299618c2e5fae5acff8d297f81b96bfb65384\" snapshot id 497"
time="2023-05-26T11:06:41.506705121Z" level=info msg="[Commit] snapshot with key \"k8s.io/759/extract-670259326-6llW sha256:bb99dc5df44865572a254bc0ee14ce436b1565097bb0b9010ad90f75dd4e9034\" snapshot id 498"
time="2023-05-26T11:06:41.551350458Z" level=info msg="[Commit] snapshot with key \"k8s.io/761/extract-536500627-OV2m sha256:057bf4d9b4e6fdb652c002e7fd7e34d86832392426ea9a6955fd02b803478958\" snapshot id 499"
time="2023-05-26T11:06:41.617638749Z" level=info msg="[Commit] snapshot with key \"k8s.io/763/extract-602137426-B0nU sha256:92125a743f1a248092f17a7e6e4dcaaa76483e6ff3f1b70ab362889b8c3833ed\" snapshot id 500"
time="2023-05-26T11:06:41.680736991Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"
time="2023-05-26T11:06:45.304017086Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"
time="2023-05-26T11:06:45.400978167Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"
time="2023-05-26T11:06:45.457585857Z" level=info msg="[Mounts] snapshot k8s.io/765/20e960589f1230a38b64c5677fd8eb75a06a0dc704944321b364f78fca4e8f10 ID 501 Kind Active"

6. 参考

参考链接
- docker使用nydus启动容器暂不支持
- buildkit-laily-fetch
- How to understand the ‘task’ concept in containerd
- Nerdctl 原生支持 Nydus 加速镜像
- Nydus | 容器镜像基础
- Install nydus for containerd
- how to check image manifest in containerd?
- run_nydus_in_kubernetes
- nydus社区钉钉支持
- v2list.manifest
主要涉及的github项目
- image-service
- moby/buildkit
- nydusaccelerator/buildkit
- nydus-snapshotter

nydus-snapshotter ↩︎
buildkitd.toml ↩︎
nydus-design ↩︎
nydus ↩︎
run_nydus_in_kubernetes ↩︎
buildkit-k8s-example ↩︎
How-to-understand-the-‘task’-concept-in-containerd ↩︎
how-to-check-image-manifest-in-containerd? ↩︎
opencontainers/image-spec ↩︎

你可能感兴趣的:(云原生,kubernetes,devops,docker,kubernetes,容器,云原生)

腾讯云技术深度探索：构建高效云原生微服务架构我的运维人生云原生架构腾讯云运维开发技术共享
腾讯云技术深度探索：构建高效云原生微服务架构在当今快速发展的技术环境中，云原生技术已成为企业数字化转型的关键驱动力。腾讯云作为行业领先的云服务提供商，不断推出创新的产品和技术，助力企业构建高效、可扩展的云原生微服务架构。本文将深入探讨腾讯云在微服务领域的最新进展，并通过一个实际案例展示如何在腾讯云平台上构建云原生应用。腾讯云微服务架构概览腾讯云微服务架构基于云原生理念，旨在帮助企业快速实现应用的容
PHP环境搭建详细教程好看资源平台前端 php
PHP是一个流行的服务器端脚本语言，广泛用于Web开发。为了使PHP能够在本地或服务器上运行，我们需要搭建一个合适的PHP环境。本教程将结合最新资料，介绍在不同操作系统上搭建PHP开发环境的多种方法，包括Windows、macOS和Linux系统的安装步骤，以及本地和Docker环境的配置。1.PHP环境搭建概述PHP环境的搭建主要分为以下几类：集成开发环境：例如XAMPP、WAMP、MAMP，这
Day17笔记-高阶函数 ~在杰难逃~ Python 笔记 python 开发语言 pycharm 数据分析
高阶函数【重点掌握】函数的本质：函数是一个变量，函数名是一个变量名，一个函数可以作为另一个函数的参数或返回值使用如果A函数作为B函数的参数，B函数调用完成之后，会得到一个结果，则B函数被称为高阶函数常用的高阶函数：map(),reduce(),filter(),sorted()1.map()map(func,iterable)，返回值是一个iterator【容器，迭代器】func:函数iterab
docker igotyback eureka 云原生
Docker容器的文件系统是隔离的，但是可以通过挂载卷（Volumes）或绑定挂载（BindMounts）将宿主机的文件系统目录映射到容器内部。要查看Docker容器的映射路径，可以使用以下方法：查看容器配置：使用dockerinspect命令可以查看容器的详细配置信息，包括挂载的卷。例如：bashdockerinspect在输出的JSON格式中，查找"Mounts"部分，这里会列出所有的挂载信息
ArrayList 源码解析程序猿进阶 Java基础 ArrayList List java 面试性能优化架构设计 idea
ArrayList是Java集合框架中的一个动态数组实现，提供了可变大小的数组功能。它继承自AbstractList并实现了List接口，是顺序容器，即元素存放的数据与放进去的顺序相同，允许放入null元素，底层通过数组实现。除该类未实现同步外，其余跟Vector大致相同。每个ArrayList都有一个容量capacity，表示底层数组的实际大小，容器内存储元素的个数不能多于当前容量。当向容器中添
react-intl——react国际化使用方案苹果酱0567 面试题汇总与解析 java 开发语言中间件 spring boot 后端
国际化介绍i18n：internationalization国家化简称，首字母+首尾字母间隔的字母个数+尾字母，类似的还有k8s(Kubernetes)React-intl是React中最受欢迎的库。使用步骤安装#usenpmnpminstallreact-intl-D#useyarn项目入口文件配置//index.tsximportReactfrom"react";importReactDOMf
06选课支付模块之基于消息队列发送支付通知消息 echo 云清学成在线 java rabbitmq 消息队列支付通知学成在线
消息队列发送支付通知消息需求分析订单服务作为通用服务，在订单支付成功后需要将支付结果异步通知给其他对接的微服务，微服务收到支付结果根据订单的类型去更新自己的业务数据技术方案使用消息队列进行异步通知需要保证消息的可靠性即生产端将消息成功通知到服务端：消息发送到交换机-->由交换机发送到队列-->消费者监听队列，收到消息进行处理，参考文章02-使用Docker安装RabbitMQ-CSDN博客生产者确
《 C++ 修炼全景指南：四》揭秘 C++ List 容器背后的实现原理，带你构建自己的双向链表 Lenyiin 技术指南 C++修炼全景指南 c++list 链表 stl
本篇博客，我们将详细讲解如何从头实现一个功能齐全且强大的C++List容器，并深入到各个细节。这篇博客将包括每一步的代码实现、解释以及扩展功能的探讨，目标是让初学者也能轻松理解。一、简介1.1、背景介绍在C++中，std::list是一个基于双向链表的容器，允许高效的插入和删除操作，适用于频繁插入和删除操作的场景。与动态数组不同，list允许常数时间内的插入和删除操作，支持双向遍历。这篇文章将详细
Ubuntu18.04 Docker部署Kinship(Django)项目过程 Dante617
1Docker的安装https://blog.csdn.net/weixin_41735055/article/details/1003551792下载镜像dockerpullprogramize/python3.6.8-dlib下载的镜像里包含python3.6.8和dlib19.17.03启动镜像dockerrun-it--namekinship-p7777:80-p3307:3306-p55
docker from指令的含义_多个FROM-含义 weixin_39722188 docker from指令的含义
小编典典什么是基本图片？一组文件，加上EXPOSE端口ENTRYPOINT和CMD。您可以添加文件并基于该基础图像构建新图像，Dockerfile并以FROM指令开头：后面提到的图像FROM是新图像的“基础图像”。这是否意味着如果我neo4j/neo4j在FROM指令中声明，则在运行映像时，neo数据库将自动运行并且可在端口7474的容器中使用？仅当您不覆盖CMD和时ENTRYPOINT。但是图像
Dockerfile FROM 两个 redDelta
Docker相关视频讲解：什么是容器Docker介绍实现"DockerfileFROM两个"的步骤步骤表格步骤操作1创建一个Dockerfile文件2写入FROM指令3构建第一个镜像4创建第二个Dockerfile文件5写入FROM指令6构建第二个镜像7合并两个镜像操作步骤说明步骤1：创建一个Dockerfile文件使用任意文本编辑器创建一个名为Dockerfile的文件。登录后复制#Docker
Dockerfile命令详解之 FROM 清风怎不知意容器化 java 前端 javascript
许多同学不知道Dockerfile应该如何写，不清楚Dockerfile中的指令分别有什么意义，能达到什么样的目的，接下来我将在容器化专栏中详细的为大家解释每一个指令的含义以及用法。专栏订阅传送门https://blog.csdn.net/qq_38220908/category_11989778.html指令不区分大小写。但是，按照惯例，它们应该是大写的，以便更容易地将它们与参数区分开来。(引用
Dockerfile（1） - FROM 指令详解小菠萝测试笔记 docker python java cmd 大数据
FROM指明当前的镜像基于哪个镜像构建dockerfile必须以FROM开头，除了ARG命令可以在FROM前面FROM[--platform=][AS]FROM[--platform=][:][AS]FROM[--platform=][@][AS]小栗子FROMalpine:latest一个dockerfile可以有多个FROM可以有多个FROM来创建多个镜像，或区分构建阶段，将一个构建阶段作为另
【从问题中去学习k8s】k8s中的常见面试题（夯实理论基础）（二十八）向往风的男子 k8s 学习 kubernetes 容器
本站以分享各种运维经验和运维所需要的技能为主《python零基础入门》：python零基础入门学习《python运维脚本》：python运维脚本实践《shell》：shell学习《terraform》持续更新中：terraform_Aws学习零基础入门到最佳实战《k8》从问题中去学习k8s《docker学习》暂未更新《ceph学习》ceph日常问题解决分享《日志收集》ELK+各种中间件《运维日常》
leetcode 11. 盛最多水的容器 Source_Chang
leetcode核心思想：双指针，数字小的那个指针移动classSolution{public:intmaxArea(vector&height){intleft=0;intright=height.size()-1;intmaxArea=0;while(left
Kubernetes数据持久化看清所苡看轻 kubernetes(k8s)emptyDir HostPath pv pvc kubernetes
在k8s中，Volume（数据卷）存在明确的生命周期（与包含该数据卷的容器组（pod）相同）。因此Volume的生命周期比同一容器组（pod）中任意容器的生命周期要更长，不管容器重启了多少次，数据都被保留下来。当然，如果pod不存在了，数据卷自然退出了。此时，根据pod所使用的数据卷类型不同，数据可能随着数据卷的退出而删除，也可能被真正持久化，并在下次容器组重启时仍然可以使用。从根本上来说，一个数
leetcode刷题day13|二叉树Part01（递归遍历、迭代遍历、统一迭代、层序遍历）小冉在学习 leetcode 算法职场和发展
递归遍历思路：使用递归的方式比较简单。1、递归函数的传参：因为最后输出一个数组，所以需要传入根节点和一个容器，本来想写数组，但发现长度不能确定，所以选择list。2、终止条件：当访问的节点为空时，return3、递归函数的逻辑：先访问一个节点，递归访问其他节点144.二叉树的前序遍历代码如下：classSolution{publicListpreorderTraversal(TreeNoderoo
Kubernetes部署MySQL数据持久化沫殇-MS Kubernetes MySQL数据库 kubernetes mysql 容器
一、安装配置NFS服务端1、安装nfs-kernel-server：sudoapt-yinstallnfs-kernel-server2、服务端创建共享目录#列出所有可用块设备的信息lsblk#格式化磁盘sudomkfs-text4/dev/sdb#创建一个目录：sudomkdir-p/data/nfs/mysql#更改目录权限：sudochown-Rnobody:nogroup/data/nfs
Kubernetes的3种数据持久化方式 Seal^_^ 【云原生】容器化与编排技术持续集成 #Kubernetes kubernetes 容器云原生 EmptyDir 面试 HostPath
Kubernetes的3种数据持久化方式1.EmptyDir2.HostPath3.PersistentVolume(PV)TheBegin点点关注，收藏不迷路Kubernetes提供了几种数据持久化方式，以满足不同场景的需求：1.EmptyDir用途：临时数据存储，Pod内容器间共享。特点：生命周期与Pod相同，Pod删除时数据也删除。2.HostPath用途：访问宿主机特定文件或目录。特点：增
【Kubernetes】常见面试题汇总（十一） summer.335 Kubernetes kubernetes 容器云原生
目录33.简述Kubernetes外部如何访问集群内的服务？34.简述Kubernetesingress？35.简述Kubernetes镜像的下载策略？33.简述Kubernetes外部如何访问集群内的服务？（1）对于Kubernetes，集群外的客户端默认情况，无法通过Pod的IP地址或者Service的虚拟IP地址：虚拟端口号进行访问。（2）通常可以通过以下方式进行访问Kubernetes集群
k8s中Service暴露的种类以及用法听说唐僧不吃肉 K8S kubernetes 容器云原生
一、说明在Kubernetes中，有几种不同的方式可以将服务（Service）暴露给外部流量。这些方式通过定义服务的spec.type字段来确定。二、详解1.ClusterIP定义：默认类型，服务只能在集群内部访问。作用：通过集群内部IP地址暴露服务。示例：spec:type:ClusterIPports:-port:80targetPo
Kubernetes 自定义控制器开发 IT回忆录 Kubenetes kubernetes
目录前言一、CRD二、创建数据库表（Mysql）二、控制器开发1.使用kubernetes的examplecontroller模板2.在controller.go中新增数据表监听方法3.修改tools工具生成资源对象结构体定义这里记录开发k8s控制器的一般方式，controller开发主要使用k8s提供的client-go库进行。前言Controller监听集群内部资源对象的变化，编辑资源对象(增
用kubedam搭建的k8s证书过期处理方法我滴鬼鬼呀wks k8s 1024程序员节
kubeadm部署的k8s证书过期1、查看证书过期时间kubeadmalphacertscheck-expiration若证书已经过期无法试用kubectl命令建议修改服务器时间到未过期的时间段2、配置kube-controller-manager.yaml文件cat/etc/kubernetes/manifests/kube-controller-manager.yamlapiVersion:v
k8s证书过期问题处理 olina_qin kubernetes 容器云原生
k8s证书过期问题处理opensslx509-in/etc/kubernetes/pki/apiserver.crt-noout-dateskubeadmcertsrenewallsystemctlrestartkubeleopensslx509-in/etc/kubernetes/pki/apiserver.crt-noout-text|grep"NotAfter"cp/etc/kubernet
Kubernetes Ingress 控制器（Nginx）安装与使用教程农优影
KubernetesIngress控制器（Nginx）安装与使用教程kubernetes-ingressNGINXandNGINXPlusIngressControllersforKubernetes项目地址:https://gitcode.com/gh_mirrors/ku/kubernetes-ingress1.项目目录结构及介绍在nginxinc/kubernetes-ingress仓库中，
【K8s】专题十一：Kubernetes 集群证书过期处理方法行者Sun1989 Kubernetes kubernetes 云原生容器
本文内容均来自个人笔记并重新梳理，如有错误欢迎指正！如果对您有帮助，烦请点赞、关注、转发、订阅专栏！专栏订阅入口Linux专栏|Docker专栏|Kubernetes专栏往期精彩文章【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法（续）【Docker】MySQL源码构建Docker镜
Java面试笔记记录6 今天背八股了吗 java 面试笔记
1.Spring是什么？特性？有哪些模块？Spring是一个轻量级、非入侵式的控制反转Ioc和面向切面AOP的框架。特性：1.Ioc和DISpring的核心就是一个大的工厂容器，可以维护所有对象的创建和依赖关系，Spring工厂用于生成Bean，并且管理Bean的生命周期，实现高内聚低耦合的设计理念。2.AOP编程Spring提供面向切面编程，可以方便实现对程序进行权限拦截、运行监控等切面功能。3
Docker学习十一：Kubernetes概述爱打羽球的程序猿 Docker学习系列 docker kubernetes 学习
一、Kubernetes简介2006年，Google提出了云计算的概念，当时的云计算领域还是以虚拟机为代表的云平台。2013年，Docker横空出世，Docker提出了镜像、仓库等核心概念，规范了服务的交付标准，使得复杂服务的落地变得更加简单，之后Docker又定义了OCI标准，Docker在容器领域称为事实的标准。但是，Docker诞生只是帮助定义了开发和交付标准，如果想要在生产环境中大批量的使
Cloud Native Weekly | 华为云抢先发布Redis5.0，红帽宣布收购混合云提供商 weixin_34302561 数据库 devops 大数据
1——华为云抢先发布Redis5.02——DigitalOceanK8s服务正式上线3——红帽宣布收购混合云提供商NooBaa4——微软发布多项AzureKubernetes服务更新1华为云抢先发布Redis5.012月17日，华为云在DCS2.0的基础上，快人一步，抢先推出了新的Redis5.0产品，这是一个崭新的突破。目前国内在缓存领域的发展普遍停留在Redis4.0阶段，华为云率先发布了Re
安装 `privoxy` 将 Socks5 转换为 HTTP 代理 MonkeyKing.sun 网络
(base)shgbitai@shgbitai-C9X299-PGF:~/tools$curl-xhttp://127.0.0.1:1080https://registry-1.docker.io/v2/curl:(56)ProxyCONNECTaborted(base)shgbitai@shgbitai-C9X299-PGF:~/tools$curl-xhttps://127.0.0.1:108
多线程编程之join()方法周凡杨 java JOIN 多线程编程线程
现实生活中，有些工作是需要团队中成员依次完成的，这就涉及到了一个顺序问题。现在有T1、T2、T3三个工人，如何保证T2在T1执行完后执行，T3在T2执行完后执行？问题分析：首先问题中有三个实体，T1、T2、T3，因为是多线程编程，所以都要设计成线程类。关键是怎么保证线程能依次执行完呢？ Java实现过程如下： public class T1 implements Runnabl
java中switch的使用 bingyingao java enum break continue
java中的switch仅支持case条件仅支持int、enum两种类型。用enum的时候，不能直接写下列形式。 switch (timeType) { case ProdtransTimeTypeEnum.DAILY: break; default: br
hive having count 不能去重 daizj hive 去重 having count 计数
hive在使用having count()是，不支持去重计数 hive (default)> select imei from t_test_phonenum where ds=20150701 group by imei having count(distinct phone_num)>1 limit 10; FAILED: SemanticExcep
WebSphere对JSP的缓存周凡杨 WAS JSP 缓存
对于线网上的工程，更新JSP到WebSphere后，有时会出现修改的jsp没有起作用，特别是改变了某jsp的样式后，在页面中没看到效果，这主要就是由于websphere中缓存的缘故，这就要清除WebSphere中jsp缓存。要清除WebSphere中JSP的缓存，就要找到WAS安装后的根目录。现服务
设计模式总结朱辉辉33 java 设计模式
1.工厂模式 1.1 工厂方法模式 (由一个工厂类管理构造方法) 1.1.1普通工厂模式(一个工厂类中只有一个方法) 1.1.2多工厂模式(一个工厂类中有多个方法) 1.1.3静态工厂模式(将工厂类中的方法变成静态方法) &n
实例：供应商管理报表需求调研报告老A不折腾 finereport 报表系统报表软件信息化选型
引言随着企业集团的生产规模扩张，为支撑全球供应链管理，对于供应商的管理和采购过程的监控已经不局限于简单的交付以及价格的管理，目前采购及供应商管理各个环节的操作分别在不同的系统下进行，而各个数据源都独立存在，无法提供统一的数据支持；因此，为了实现对于数据分析以提供采购决策，建立报表体系成为必须。业务目标 1、通过报表为采购决策提供数据分析与支撑 2、对供应商进行综合评估以及管理，合理管理和
mysql 林鹤霄
转载源：http://blog.sina.com.cn/s/blog_4f925fc30100rx5l.html mysql -uroot -p ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) [root@centos var]# service mysql
Linux下多线程堆栈查看工具(pstree、ps、pstack) aigo linux
原文：http://blog.csdn.net/yfkiss/article/details/6729364 1. pstree pstree以树结构显示进程$ pstree -p work | grep adsshd(22669)---bash(22670)---ad_preprocess(4551)-+-{ad_preprocess}(4552) &n
html input与textarea 值改变事件 alxw4616 JavaScript
// 文本输入框(input) 文本域(textarea)值改变事件 // onpropertychange(IE) oninput(w3c) $('input,textarea').on('propertychange input', function(event) { console.log($(this).val()) });
String类的基本用法百合不是茶 String
字符串的用法; // 根据字节数组创建字符串 byte[] by = { 'a', 'b', 'c', 'd' }; String newByteString = new String(by); 1,length() 获取字符串的长度 &nbs
JDK1.5 Semaphore实例 bijian1013 java thread java多线程 Semaphore
Semaphore类一个计数信号量。从概念上讲，信号量维护了一个许可集合。如有必要，在许可可用前会阻塞每一个 acquire()，然后再获取该许可。每个 release() 添加一个许可，从而可能释放一个正在阻塞的获取者。但是，不使用实际的许可对象，Semaphore 只对可用许可的号码进行计数，并采取相应的行动。 S
使用GZip来压缩传输量 bijian1013 java GZip
启动GZip压缩要用到一个开源的Filter：PJL Compressing Filter。这个Filter自1.5.0开始该工程开始构建于JDK5.0，因此在JDK1.4环境下只能使用1.4.6。 PJL Compressi
【Java范型三】Java范型详解之范型类型通配符 bit1129 java
定义如下一个简单的范型类， package com.tom.lang.generics; public class Generics<T> { private T value; public Generics(T value) { this.value = value; } }
【Hadoop十二】HDFS常用命令 bit1129 hadoop
1. 修改日志文件查看器 hdfs oev -i edits_0000000000000000081-0000000000000000089 -o edits.xml cat edits.xml 修改日志文件转储为xml格式的edits.xml文件，其中每条RECORD就是一个操作事务日志 2. fsimage查看HDFS中的块信息等 &nb
怎样区别nginx中rewrite时break和last ronin47
在使用nginx配置rewrite中经常会遇到有的地方用last并不能工作，换成break就可以，其中的原理是对于根目录的理解有所区别，按我的测试结果大致是这样的。 location / { proxy_pass http://test;
java-21.中兴面试题输入两个整数 n 和 m ，从数列 1 ， 2 ， 3.......n 中随意取几个数 , 使其和等于 m bylijinnan java
import java.util.ArrayList; import java.util.List; import java.util.Stack; public class CombinationToSum { /* 第21 题 2010 年中兴面试题编程求解：输入两个整数 n 和 m ，从数列 1 ， 2 ， 3.......n 中随意取几个数 , 使其和等
eclipse svn 帐号密码修改问题开窍的石头 eclipse SVN svn帐号密码修改
问题描述： Eclipse的SVN插件Subclipse做得很好，在svn操作方面提供了很强大丰富的功能。但到目前为止，该插件对svn用户的概念极为淡薄，不但不能方便地切换用户，而且一旦用户的帐号、密码保存之后，就无法再变更了。解决思路：删除subclipse记录的帐号、密码信息，重新输入
[电子商务]传统商务活动与互联网的结合 comsci 电子商务
某一个传统名牌产品，过去销售的地点就在某些特定的地区和阶层，现在进入互联网之后，用户的数量群突然扩大了无数倍，但是，这种产品潜在的劣势也被放大了无数倍，这种销售利润与经营风险同步放大的效应，在最近几年将会频繁出现。。。。如何避免销售量和利润率增加的
java 解析 properties-使用 Properties-可以指定配置文件路径 cuityang java properties
#mq xdr.mq.url=tcp://192.168.100.15:61618; import java.io.IOException; import java.util.Properties; public class Test { String conf = "log4j.properties"; private static final
Java核心问题集锦 darrenzhu java 基础核心难点
注意，这里的参考文章基本来自Effective Java和jdk源码 1)ConcurrentModificationException 当你用for each遍历一个list时，如果你在循环主体代码中修改list中的元素，将会得到这个Exception，解决的办法是： 1)用listIterator, 它支持在遍历的过程中修改元素， 2)不用listIterator, new一个
1分钟学会Markdown语法 dcj3sjt126com markdown
markdown 简明语法基本符号 *,-,+ 3个符号效果都一样，这3个符号被称为 Markdown符号空白行表示另起一个段落 `是表示inline代码，tab是用来标记代码段，分别对应html的code，pre标签换行单一段落( <p>) 用一个空白行连续两个空格会变成一个 <br> 连续3个符号，然后是空行
Gson使用二（GsonBuilder） eksliang json gson GsonBuilder
转载请出自出处：http://eksliang.iteye.com/blog/2175473 一.概述 GsonBuilder用来定制java跟json之间的转换格式二.基本使用实体测试类：温馨提示：默认情况下@Expose注解是不起作用的,除非你用GsonBuilder创建Gson的时候调用了GsonBuilder.excludeField
报ClassNotFoundException: Didn't find class "...Activity" on path: DexPathList gundumw100 android
有一个工程，本来运行是正常的，我想把它移植到另一台PC上，结果报： java.lang.RuntimeException: Unable to instantiate activity ComponentInfo{com.mobovip.bgr/com.mobovip.bgr.MainActivity}: java.lang.ClassNotFoundException: Didn't f
JavaWeb之JSP指令 ihuning javaweb
要点 JSP指令简介 page指令 include指令 JSP指令简介 JSP指令（directive）是为JSP引擎而设计的，它们并不直接产生任何可见输出，而只是告诉引擎如何处理JSP页面中的其余部分。 JSP指令的基本语法格式： <%@ 指令属性名="
mac上编译FFmpeg跑ios 啸笑天 ffmpeg
1、下载文件：https://github.com/libav/gas-preprocessor，复制gas-preprocessor.pl到/usr/local/bin/下，修改文件权限：chmod 777 /usr/local/bin/gas-preprocessor.pl 2、安装yasm-1.2.0 curl http://www.tortall.net/projects/yasm
sql mysql oracle中字符串连接 macroli oracle sql mysql SQL Server
有的时候，我们有需要将由不同栏位获得的资料串连在一起。每一种资料库都有提供方法来达到这个目的： MySQL: CONCAT() Oracle: CONCAT(), || SQL Server: + CONCAT() 的语法如下： Mysql 中 CONCAT(字串1, 字串2, 字串3, ...): 将字串1、字串2、字串3，等字串连在一起。请注意，Oracle的CON
Git fatal: unab SSL certificate problem: unable to get local issuer ce rtificate qiaolevip 学习永无止境每天进步一点点 git 纵观千象
// 报错如下： $ git pull origin master fatal: unable to access 'https://git.xxx.com/': SSL certificate problem: unable to get local issuer ce rtificate // 原因：由于git最新版默认使用ssl安全验证，但是我们是使用的git未设
windows命令行设置wifi surfingll windows wifi 笔记本wifi
还没有讨厌无线wifi的无尽广告么，还在耐心等待它慢慢启动么教你命令行设置笔记本电脑wifi： 1、开启wifi命令 netsh wlan set hostednetwork mode=allow ssid=surf8 key=bb123456 netsh wlan start hostednetwork pause 其中pause是等待输入，可以去掉 2、
Linux（Ubuntu）下安装sysv-rc-conf wmlJava linux ubuntu sysv-rc-conf
安装：sudo apt-get install sysv-rc-conf 使用：sudo sysv-rc-conf 操作界面十分简洁，你可以用鼠标点击，也可以用键盘方向键定位，用空格键选择，用Ctrl+N翻下一页，用Ctrl+P翻上一页，用Q退出。背景知识 sysv-rc-conf是一个强大的服务管理程序，群众的意见是sysv-rc-conf比chkconf
svn切换环境，重发布应用多了javaee标签前缀 zengshaotao javaee
更换了开发环境，从杭州，改变到了上海。svn的地址肯定要切换的，切换之前需要将原svn自带的.svn文件信息删除，可手动删除，也可通过废弃原来的svn位置提示删除.svn时删除。然后就是按照最新的svn地址和规范建立相关的目录信息，再将原来的纯代码信息上传到新的环境。然后再重新检出，这样每次修改后就可以看到哪些文件被修改过，这对于增量发布的规范特别有用。检出