简单介绍下什么是dind?使用场景是什么?
DinD即 Docker inside Docker, DinD在容器里有一个完整的docker构建系统,可直接在容器中完成镜像的构建,与之相对应的就是DooD ,通过挂载宿主机的docker.sock文件,调用宿主机的docker daemon去构建镜像。他们的主要使用场景有很多,比较常见的就是CICD场景中了,CICD需要构建镜像。我之前的文章使用的就是DooD的方式,通过挂载宿主机的docker.sock文件。而本讲介绍DinD,进行镜像构建并推送。简单说下DooD和Dind的优势与劣势:
DooD | DinD | |
---|---|---|
优势 | 同一台宿主机上的缓存可以通过同一个 Docker daemon 共享,这样构建镜像会比较快 | 1、明显支持并发,一个容器实例使用一个docker daemon进程; 2、容器实例之间相互隔离,不干扰,不使用缓存,比较干净 DinD劣势: |
劣势 | 1、由于不同容器实例挂在同一个宿主机的 Docker daemon 进程,所有实例里 docker 命令的权限也是共享的,也就是说不同容器实例可以查看甚至更新、删除到同一个 Docker daemon 下别的容器实例构建产生的镜像。这就是个安全问题; 2、DooD的并行效果不咋地,因为多个容器实例使用同一个Docker daemon进程。可能存在排队或者争抢。 3、DooD的方式会导致宿主机本地的存储被大量消耗,需要定期清理镜像;4、环境隔离性问题,不同容器构建镜像容易出现名字冲突问题 | 速度上还是没有DooD快,因为不能使用缓存。每个容器构建完后即销毁。 |
本讲想使用DinD的方式,因为我在实际开发中需要一个相互之间隔离,且每次使用镜像时,都需要重新拉取的环境,所以我选在DinD的方式来构建镜像。
先去dockerhub上找下docker镜像,里面有dind的版本。然后你可以参考tekton的dind-sidecar来构建镜像(参照参考文章[3]),但是由于构建镜像时需要拉取dockerhub镜像作为基础镜像,就需要加上镜像加速器,不然可能会拉取镜像失败导致构建镜像不成功。所以,我们需要构建一个带镜像加速器的docker:dind的镜像。参照 docker镜像的github地址.你可以找到dind的构建所需要的Dockerfile和entrypoint.sh文件。下面我贴一下我的Dockerfile和entrypoint.sh文件:
Dockerfile:(就改了基础镜像,改成了18.05,其他都没改)
FROM docker:18.05
# https://github.com/docker/docker/blob/master/project/PACKAGERS.md#runtime-dependencies
RUN set -eux; \
apk add --no-cache \
btrfs-progs \
e2fsprogs \
e2fsprogs-extra \
iptables \
openssl \
shadow-uidmap \
xfsprogs \
xz \
# pigz: https://github.com/moby/moby/pull/35697 (faster gzip implementation)
pigz \
; \
# only install zfs if it's available for the current architecture
# https://git.alpinelinux.org/cgit/aports/tree/main/zfs/APKBUILD?h=3.6-stable#n9 ("all !armhf !ppc64le" as of 2017-11-01)
# "apk info XYZ" exits with a zero exit code but no output when the package exists but not for this arch
if zfs="$(apk info --no-cache --quiet zfs)" && [ -n "$zfs" ]; then \
apk add --no-cache zfs; \
fi
# TODO aufs-tools
# set up subuid/subgid so that "--userns-remap=default" works out-of-the-box
RUN set -x \
&& addgroup -S dockremap \
&& adduser -S -G dockremap dockremap \
&& echo 'dockremap:165536:65536' >> /etc/subuid \
&& echo 'dockremap:165536:65536' >> /etc/subgid
# https://github.com/docker/docker/tree/master/hack/dind
ENV DIND_COMMIT 37498f009d8bf25fbb6199e8ccd34bed84f2874b
RUN set -eux; \
wget -O /usr/local/bin/dind "https://raw.githubusercontent.com/docker/docker/${DIND_COMMIT}/hack/dind"; \
chmod +x /usr/local/bin/dind
COPY dockerd-entrypoint.sh /usr/local/bin/
VOLUME /var/lib/docker
EXPOSE 2375 2376
ENTRYPOINT ["dockerd-entrypoint.sh"]
CMD []
entrypony.sh:(增加了阿里云镜像加速地址)
#!/bin/sh
set -eu
_tls_ensure_private() {
local f="$1"; shift
[ -s "$f" ] || openssl genrsa -out "$f" 4096
}
_tls_san() {
{
ip -oneline address | awk '{ gsub(/\/.+$/, "", $4); print "IP:" $4 }'
{
cat /etc/hostname
echo 'docker'
echo 'localhost'
hostname -f
hostname -s
} | sed 's/^/DNS:/'
[ -z "${DOCKER_TLS_SAN:-}" ] || echo "$DOCKER_TLS_SAN"
} | sort -u | xargs printf '%s,' | sed "s/,\$//"
}
_tls_generate_certs() {
local dir="$1"; shift
# if ca/key.pem || !ca/cert.pem, generate CA public if necessary
# if ca/key.pem, generate server public
# if ca/key.pem, generate client public
# (regenerating public certs every startup to account for SAN/IP changes and/or expiration)
# https://github.com/FiloSottile/mkcert/issues/174
local certValidDays='825'
if [ -s "$dir/ca/key.pem" ] || [ ! -s "$dir/ca/cert.pem" ]; then
# if we either have a CA private key or do *not* have a CA public key, then we should create/manage the CA
mkdir -p "$dir/ca"
_tls_ensure_private "$dir/ca/key.pem"
openssl req -new -key "$dir/ca/key.pem" \
-out "$dir/ca/cert.pem" \
-subj '/CN=docker:dind CA' -x509 -days "$certValidDays"
fi
if [ -s "$dir/ca/key.pem" ]; then
# if we have a CA private key, we should create/manage a server key
mkdir -p "$dir/server"
_tls_ensure_private "$dir/server/key.pem"
openssl req -new -key "$dir/server/key.pem" \
-out "$dir/server/csr.pem" \
-subj '/CN=docker:dind server'
cat > "$dir/server/openssl.cnf" <<-EOF
[ x509_exts ]
subjectAltName = $(_tls_san)
EOF
openssl x509 -req \
-in "$dir/server/csr.pem" \
-CA "$dir/ca/cert.pem" \
-CAkey "$dir/ca/key.pem" \
-CAcreateserial \
-out "$dir/server/cert.pem" \
-days "$certValidDays" \
-extfile "$dir/server/openssl.cnf" \
-extensions x509_exts
cp "$dir/ca/cert.pem" "$dir/server/ca.pem"
openssl verify -CAfile "$dir/server/ca.pem" "$dir/server/cert.pem"
fi
if [ -s "$dir/ca/key.pem" ]; then
# if we have a CA private key, we should create/manage a client key
mkdir -p "$dir/client"
_tls_ensure_private "$dir/client/key.pem"
chmod 0644 "$dir/client/key.pem" # openssl defaults to 0600 for the private key, but this one needs to be sared with arbitrary client contexts
openssl req -new \
-key "$dir/client/key.pem" \
-out "$dir/client/csr.pem" \
-subj '/CN=docker:dind client'
cat > "$dir/client/openssl.cnf" <<-'EOF'
[ x509_exts ]
extendedKeyUsage = clientAuth
EOF
openssl x509 -req \
-in "$dir/client/csr.pem" \
-CA "$dir/ca/cert.pem" \
-CAkey "$dir/ca/key.pem" \
-CAcreateserial \
-out "$dir/client/cert.pem" \
-days "$certValidDays" \
-extfile "$dir/client/openssl.cnf" \
-extensions x509_exts
cp "$dir/ca/cert.pem" "$dir/client/ca.pem"
openssl verify -CAfile "$dir/client/ca.pem" "$dir/client/cert.pem"
fi
}
# no arguments passed
# or first arg is `-f` or `--some-option`
if [ "$#" -eq 0 ] || [ "${1#-}" != "$1" ]; then
# set "dockerSocket" to the default "--host" *unix socket* value (for both standard or rootless)
uid="$(id -u)"
if [ "$uid" = '0' ]; then
dockerSocket='unix:///var/run/docker.sock'
else
# if we're not root, we must be trying to run rootless
: "${XDG_RUNTIME_DIR:=/run/user/$uid}"
dockerSocket="unix://$XDG_RUNTIME_DIR/docker.sock"
fi
case "${DOCKER_HOST:-}" in
unix://*)
dockerSocket="$DOCKER_HOST"
;;
esac
# add our default arguments
if [ -n "${DOCKER_TLS_CERTDIR:-}" ] \
&& _tls_generate_certs "$DOCKER_TLS_CERTDIR" \
&& [ -s "$DOCKER_TLS_CERTDIR/server/ca.pem" ] \
&& [ -s "$DOCKER_TLS_CERTDIR/server/cert.pem" ] \
&& [ -s "$DOCKER_TLS_CERTDIR/server/key.pem" ] \
; then
# generate certs and use TLS if requested/possible (default in 19.03+)
set -- dockerd \
--host="$dockerSocket" \
--host=tcp://0.0.0.0:2376 \
--tlsverify \
--tlscacert "$DOCKER_TLS_CERTDIR/server/ca.pem" \
--tlscert "$DOCKER_TLS_CERTDIR/server/cert.pem" \
--tlskey "$DOCKER_TLS_CERTDIR/server/key.pem" \
--registry-mirror=https://cduvuqsh.mirror.aliyuncs.com \
"$@"
DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="${DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS:-} -p 0.0.0.0:2376:2376/tcp"
else
# TLS disabled (-e DOCKER_TLS_CERTDIR='') or missing certs
set -- dockerd \
--host="$dockerSocket" \
--host=tcp://0.0.0.0:2375 \
"$@"
DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS="${DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS:-} -p 0.0.0.0:2375:2375/tcp"
fi
fi
if [ "$1" = 'dockerd' ]; then
# explicitly remove Docker's default PID file to ensure that it can start properly if it was stopped uncleanly (andthus didn't clean up the PID file)
find /run /var/run -iname 'docker*.pid' -delete || :
uid="$(id -u)"
if [ "$uid" != '0' ]; then
# if we're not root, we must be trying to run rootless
if ! command -v rootlesskit > /dev/null; then
echo >&2 "error: attempting to run rootless dockerd but missing 'rootlesskit' (perhaps the 'docker:ind-rootless' image variant is intended?)"
exit 1
fi
user="$(id -un 2>/dev/null || :)"
if ! grep -qE "^($uid${user:+|$user}):" /etc/subuid || ! grep -qE "^($uid${user:+|$user}):" /etc/subgid; thn
echo >&2 "error: attempting to run rootless dockerd but missing necessary entries in /etc/subuid an/or /etc/subgid for $uid"
exit 1
fi
: "${XDG_RUNTIME_DIR:=/run/user/$uid}"
export XDG_RUNTIME_DIR
if ! mkdir -p "$XDG_RUNTIME_DIR" || [ ! -w "$XDG_RUNTIME_DIR" ] || ! mkdir -p "$HOME/.local/share/docker" | [ ! -w "$HOME/.local/share/docker" ]; then
echo >&2 "error: attempting to run rootless dockerd but need writable HOME ($HOME) and XDG_RUNTIME_IR ($XDG_RUNTIME_DIR) for user $uid"
exit 1
fi
if [ -f /proc/sys/kernel/unprivileged_userns_clone ] && unprivClone="$(cat /proc/sys/kernel/unprivileged_usrns_clone)" && [ "$unprivClone" != '1' ]; then
echo >&2 "error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/poc/sys/kernel/unprivileged_userns_clone) set to 1"
exit 1
fi
if [ -f /proc/sys/user/max_user_namespaces ] && maxUserns="$(cat /proc/sys/user/max_user_namespaces)" && [ $maxUserns" = '0' ]; then
echo >&2 "error: attempting to run rootless dockerd but need 'user.max_user_namespaces' (/proc/sys/ser/max_user_namespaces) set to a sufficiently large value"
exit 1
fi
# TODO overlay support detection?
exec rootlesskit \
--net="${DOCKERD_ROOTLESS_ROOTLESSKIT_NET:-kit}" \
--mtu="${DOCKERD_ROOTLESS_ROOTLESSKIT_MTU:-1500}" \
--disable-host-loopback \
--port-driver=builtin \
--copy-up=/etc \
--copy-up=/run \
${DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS:-} \
"$@"
elif [ -x '/usr/local/bin/dind' ]; then
# if we have the (mostly defunct now) Docker-in-Docker wrapper script, use it
set -- '/usr/local/bin/dind' "$@"
fi
else
# if it isn't `dockerd` we're trying to run, pass it through `docker-entrypoint.sh` so it gets `DOCKER_HOST` set apropriately too
set -- docker-entrypoint.sh "$@"
fi
exec "$@"
我在entrypoint.sh中增加了registry-mirror的指定,写了我的阿里云镜像加速地址。
运行
docker build -t registru.nanjun/tekton/docker:v18.05-dind
制作完dind镜像,下面开始跑tekton的taskrun啦
参考tekton的dind-sidecar的yaml,task.yaml如下:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: docker-in-docker-demo
namespace: nanjun
spec:
steps:
- image: docker
name: client
script: |
#!/usr/bin/env sh
cat > Dockerfile << EOF
FROM ubuntu
RUN apt-get update
ENTRYPOINT ["echo", "hello"]
EOF
docker build -t hello . && docker run hello
docker images
volumeMounts:
- mountPath: /var/run/
name: dind-socket
sidecars: #sidecar模式
- image: registry.nanjun/tekton/docker:v18.05-dind
name: server
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/docker
name: dind-storage
- mountPath: /var/run/
name: dind-socket
volumes:
- name: dind-storage
emptyDir: {}
- name: dind-socket
emptyDir: {}
你需要了解k8s的sidecar模式(边车模式),我这边的dind使用是每次运行taskrun,都会在pod中起一个sidecar的容器,作为docker daemon,另一个client容器将构建请求发送到这个server容器,构建完,client本地运行一个容器并查看镜像列表,然后退出,退出后pod的中所有容器都会销毁,sidecar容器也会被销毁。这样就实现了在隔离的环境中构建镜像,因为下一次构建镜像还是同样的过程,但是和上一次的完全无关。
下面是taskrun.yaml:
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
name: docker-in-docker-demo
namespace: nanjun
spec:
taskRef:
name: docker-in-docker-demo
DinD可容入tekton CI中,作为构建镜像的一种选择,用户可自由选择使用dood还是dind的方式,使用缓存就用dood,使用隔离,就用dind。本地使用文件将在github地址查看:https://github.com/fishingfly/sidecar-dind, 如使用有问题加我微信:nanjun_1224, 欢迎关注微信公众号“云原生书记”,我们一起成长!
参考文章
[1]:https://hub.docker.com/_/docker
[2]:https://github.com/docker-library/docker
[3]:https://github.com/tektoncd/pipeline/blob/master/examples/v1alpha1/taskruns/dind-sidecar.yaml