OpenShift 4.6 新特性 - 用 Windows MachineConfig Operator 管理 Windows Container
《OpenShift 4.x HOL教程汇总》
文章目录
- 查看Windows节点的Kubernete版本
- 安装访问Windows节点的SSH Jumphost
- 登录到Windows节点
- 在Windows节点操作容器环境
- 向Windows节点部署应用
- 用MachineSet扩展Windows节点
- 其他参考
说明：
- 本文主要演示如何在OpenShift 4.6上操作部署和管理Windows Container。
- 本文所用OpenShift 4.6集群中需要有运行Windows Server 2019 Datacenter节点，使用它来运行Windows Container，
- 需要提前安装配置好 Windows Machine Config Operator。
查看Windows节点的Kubernete版本
- 查看集群节点基本信息。可以看到ip-10-0-156-110.ap-southeast-1.compute.internal节点的Kubernetes版本是v1.19.0-rc.2.1023+f5121a6a6a02dd，和其他节点不同。
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-146-72.ap-southeast-1.compute.internal Ready master 63m v1.19.0+db1fc96
ip-10-0-154-146.ap-southeast-1.compute.internal Ready worker 49m v1.19.0+db1fc96
ip-10-0-156-110.ap-southeast-1.compute.internal Ready worker 15m v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-178-234.ap-southeast-1.compute.internal Ready master 63m v1.19.0+db1fc96
ip-10-0-189-77.ap-southeast-1.compute.internal Ready worker 50m v1.19.0+db1fc96
ip-10-0-195-53.ap-southeast-1.compute.internal Ready master 63m v1.19.0+db1fc96
- 查询集群中运行Windows的节点，该节点就是ip-10-0-156-110.ap-southeast-1.compute.internal。我们后面用的
就是你环境中的这个Windows节点名。
$ oc get nodes -l kubernetes.io/os=windows -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-156-110.ap-southeast-1.compute.internal Ready worker 16m v1.19.0-rc.2.1023+f5121a6a6a02dd 10.0.156.110 <none> Windows Server 2019 Datacenter 10.0.17763.1457 docker://19.3.11
安装访问Windows节点的SSH Jumphost
- 创建内容如下的deploy-sshproxy.sh文件。
#!/bin/bash
set -e
# Configuration via env. variables:
# Namespace where the bastion should run. The namespace will be created.
SSH_BASTION_NAMESPACE="${SSH_BASTION_NAMESPACE:-openshift-ssh-bastion}"
# Directory with bastion yaml files. Can be either local directory or http(s) URL.
BASEDIR="${BASEDIR:-https://raw.githubusercontent.com/eparis/ssh-bastion/master/deploy}"
clean_up () {
ARG=$?
rm -f "${RSATMP}" "${RSATMP}.pub"
rm -f "${ECDSATMP}" "${ECDSATMP}.pub"
rm -f "${ED25519TMP}" "${ED25519TMP}.pub"
rm -f "${CONFIGFILE}"
exit $ARG
}
trap clean_up EXIT
create_host_keys() {
RSATMP=$(mktemp -u)
/usr/bin/ssh-keygen -q -t rsa -f "${RSATMP}" -C '' -N ''
ECDSATMP=$(mktemp -u)
/usr/bin/ssh-keygen -q -t ecdsa -f "${ECDSATMP}" -C '' -N ''
ED25519TMP=$(mktemp -u)
/usr/bin/ssh-keygen -q -t ed25519 -f "${ED25519TMP}" -C '' -N ''
CONFIGFILE=$(mktemp)
echo 'HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
PermitRootLogin no
AuthorizedKeysFile /home/core/.ssh/authorized_keys
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
' > "${CONFIGFILE}"
oc -n "${SSH_BASTION_NAMESPACE}" create secret generic ssh-host-keys --from-file="ssh_host_rsa_key=${RSATMP},ssh_host_ecdsa_key=${ECDSATMP},ssh_host_ed25519_key=${ED25519
TMP},sshd_config=${CONFIGFILE}"
}
# Non-namespaced objects
oc apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: ${SSH_BASTION_NAMESPACE}
labels:
openshift.io/run-level: "0"
EOF
oc apply -f "${BASEDIR}/clusterrole.yaml"
# using oc apply to modifty any already existing clusterrolebinding
oc create clusterrolebinding ssh-bastion --clusterrole=ssh-bastion --user="system:serviceaccount:${SSH_BASTION_NAMESPACE}:ssh-bastion" -o yaml --dry-run=client | oc apply -f
-
# Namespaced objects
oc -n "${SSH_BASTION_NAMESPACE}" apply -f "${BASEDIR}/service.yaml"
oc -n "${SSH_BASTION_NAMESPACE}" get secret ssh-host-keys &>/dev/null || create_host_keys
oc -n "${SSH_BASTION_NAMESPACE}" apply -f "${BASEDIR}/serviceaccount.yaml"
oc -n "${SSH_BASTION_NAMESPACE}" apply -f "${BASEDIR}/role.yaml"
oc -n "${SSH_BASTION_NAMESPACE}" create rolebinding ssh-bastion --clusterrole=ssh-bastion --user="system:serviceaccount:${SSH_BASTION_NAMESPACE}:ssh-bastion" -o yaml --dry-ru
n=client | oc apply -f -
oc -n "${SSH_BASTION_NAMESPACE}" apply -f "${BASEDIR}/deployment.yaml"
retry=120
while [ $retry -ge 0 ]
do
retry=$((retry-1))
bastion_host=$(oc get service -n "${SSH_BASTION_NAMESPACE}" ssh-bastion -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' || true)
if [ -n "${bastion_host}" ]; then
break
fi
bastion_ip=$(oc get service -n "${SSH_BASTION_NAMESPACE}" ssh-bastion -o jsonpath='{.status.loadBalancer.ingress[0].ip}' || true)
if [ -n "${bastion_ip}" ]; then
break
fi
sleep 1
done
if [ -n "${bastion_host}" ]; then
echo "Waiting for ${bastion_host} to show up in DNS"
retry=120
while [ $retry -ge 0 ]
do
retry=$((retry-1))
if ! ((retry % 10)); then
echo "...Still waiting for DNS..."
fi
if nslookup "${bastion_host}" > /dev/null ; then
break
else
sleep 2
fi
done
else
bastion_host="${bastion_ip}"
fi
echo "The bastion address is ${bastion_host}"
echo "You may want to use https://raw.githubusercontent.com/eparis/ssh-bastion/master/ssh.sh to easily ssh through the bastion to specific nodes."
- 执行命令，部署可以访问Windows节点的SSH Jumphost相关资源，它们被部署到openshift-ssh-bastion项目中。
$ bash deploy-sshproxy.sh
namespace/openshift-ssh-bastion created
clusterrole.rbac.authorization.k8s.io/ssh-bastion created
clusterrolebinding.rbac.authorization.k8s.io/ssh-bastion created
service/ssh-bastion created
secret/ssh-host-keys created
serviceaccount/ssh-bastion created
role.rbac.authorization.k8s.io/ssh-bastion created
rolebinding.rbac.authorization.k8s.io/ssh-bastion created
deployment.apps/ssh-bastion created
Waiting for a396bfd3fdd644f29bed71e530a1341d-1824562040.ap-southeast-1.elb.amazonaws.com to show up in DNS
...Still waiting for DNS...
...Still waiting for DNS...
...Still waiting for DNS...
...Still waiting for DNS...
...Still waiting for DNS...
...Still waiting for DNS...
The bastion address is a396bfd3fdd644f29bed71e530a1341d-1824562040.ap-southeast-1.elb.amazonaws.com
You may want to use https://raw.githubusercontent.com/eparis/ssh-bastion/master/ssh.sh to easily ssh through the bastion to specific nodes.
- 执行命令，查看部署资源的状态。
$ oc get all -n openshift-ssh-bastion
NAME READY STATUS RESTARTS AGE
pod/ssh-bastion-5fcf8d7d9b-d5788 1/1 Running 0 2m28s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ssh-bastion LoadBalancer 172.30.4.134 a396bfd3fdd644f29bed71e530a1341d-1824562040.ap-southeast-1.elb.amazonaws.com 22:30421/TCP 2m32s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ssh-bastion 1/1 1 1 2m28s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ssh-bastion-5fcf8d7d9b 1 1 1 2m28s
登录到Windows节点
说明：以下操作都需要在Terminal1的Linux Bash中执行。
- 创建内容如下的sshcmd.sh文件，我们稍后会用它来登录Windows节点并进入PowerShell。
#!/bin/bash
ssh_key_param=''
if [ ! -z "$SSH_KEY_PATH" ]; then
ssh_key_param="-i $SSH_KEY_PATH"
fi
ingress_host="$(oc get service --all-namespaces -l run=ssh-bastion -o go-template='{{ with (index (index .items 0).status.loadBalancer.ingress 0) }}{{ or .hostname .ip }}{{end}}')"
ssh $ssh_key_param -t -o StrictHostKeyChecking=no -o ProxyCommand="ssh $ssh_key_param -A -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -W %h:%p core@${ingress_host}"
administrator@$1 "powershell"
- 执行命令，开启一个ssh shell session。
$ eval `ssh-agent`
- 加入当前用户的ssh key
$ ssh-add ${HOME}/.ssh/id_rsa
- 执行脚本，进入
节点，此时就会自动进入Powershell中。
$ bash sshcmd.sh <WINDOWS_NODE>
Warning: Permanently added 'a396bfd3fdd644f29bed71e530a1341d-1824562040.ap-southeast-1.elb.amazonaws.com,13.229.174.64' (ECDSA) to the list of known hosts.
Warning: Permanently added 'ip-10-0-156-110.ap-southeast-1.compute.internal' (ECDSA) to the list of known hosts.
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
PS C:\Users\Administrator>
在Windows节点操作容器环境
说明：以下操作全部在Terminal2的Powershell中执行。
- 查看Windows节点中运行容器。
PS C:\Users\Administrator> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
- 查看Windows节点中的容器服务进程。
PS C:\Users\Administrator> Get-Process | ?{ $_.ProcessName -match "kube|overlay|docker" }
Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName
------- ------ ----- ----- ------ -- -- -----------
414 22 68952 46184 198.47 2544 0 dockerd
205 16 25884 32348 3.20 3848 0 hybrid-overlay-node
467 24 45244 70212 34.83 3032 0 kubelet
235 21 29772 40076 11.30 4676 0 kube-proxy
- 查看Windows节点中的镜像。
PS C:\Users\Administrator> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
- 执行命令，把mcr.microsoft.com/windows/servercore镜像pull到Windows节点本地。
PS C:\Users\Administrator> docker pull mcr.microsoft.com/windows/servercore:ltsc2019
- 再次查看Windows节点本地的镜像。
PS C:\Users\Administrator> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mcr.microsoft.com/windows/servercore ltsc2019 71d3f266f3af 13 days ago 5.1GB
向Windows节点部署应用
说明：以下操作如未说明，都需要在Terminal1的Linux Bash中执行。
- 查看Windows节点中的“Taints”属性。下面将用该属性作为选择条件将Pod只部署在Windows节点中。
$ oc describe node <WINDOWS_NODE> |grep -i Taint
Taints: os=Windows:NoSchedule
- 创建内容如下的WinWebServer.yaml文件。注意：在部署中定义了“tolerations”，使用它和Windows节点上的“Taint”属性，让该部署只运行到Windows节点。
apiVersion: v1
kind: Service
metadata:
name: win-webserver
labels:
app: win-webserver
spec:
ports:
# the port that this service should serve on
- port: 80
targetPort: 80
selector:
app: win-webserver
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
selector:
matchLabels:
app: win-webserver
replicas: 1
template:
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
tolerations:
- key: "os"
value: "Windows"
Effect: "NoSchedule"
containers:
- name: windowswebserver
image: mcr.microsoft.com/windows/servercore:ltsc2019
imagePullPolicy: IfNotPresent
command:
- powershell.exe
- -command
- $listener = New-Object System.Net.HttpListener; $listener.Prefixes.Add('http://*:80/'); $listener.Start();Write-Host('Listening at http://*:80/'); while ($listener.IsListening) { $context = $listener.GetContext(); $response = $context.Response; $content='>>>Windows Container Web Server > > >'; $buffer = [System.Text.Encoding]::UTF8.GetBytes($content); $response.ContentLength64 = $buffer.Length; $response.OutputStream.Write($buffer, 0, $buffer.Length); $response.Close(); };
nodeSelector:
beta.kubernetes.io/os: windows
- 在default项目中创建WinWebServer.yaml的对象。
$ oc create -n default -f WinWebServer.yaml
service/win-webserver created
deployment.apps/win-webserver created
- 查看default项目中的pod状态，直到状态变为“Running”。确认该pod是运行在
上了。
$ oc get pods -n default -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
win-webserver-549cd7495d-gsgsl 1/1 Running 0 75s 10.132.0.2 ip-10-0-156-110.ap-southeast-1.compute.internal <none> <none>
- 在Terminal2中的Powershell中执行以下命令，确认此时在Windows节点中已经有运行的Pod了。
PS C:\Users\Administrator> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
929bc257e78a 71d3f266f3af "powershell.exe -com…" 2 minutes ago Up 2 minutes k8s_windowswebserver_win-webserver-549cd7495d-gsgsl_default_c0967447-1e00-4791-8a95-8e154c6a5810_0
c48c9f36a728 mcr.microsoft.com/oss/kubernetes/pause:1.3.0 "cmd /S /C pauseloop…" 2 minutes ago Up 2 minutes k8s_POD_win-webserver-549cd7495d-gsgsl_default_c0967447-1e00-4791-8a95-8e154c6a5810_0
- 重新在Terminal1中执行命令，根据service创建route。
$ oc expose svc/win-webserver
route.route.openshift.io/win-webserver exposed
- 通过route访问win-webserver应用，确认可以正常访问运行在Window节点中的Pod。
$ curl -s http://$(oc get route win-webserver -n default -o jsonpath='{.spec.host}')
<html><body><H1>Windows Container Web Server</H1></body></html>[ec2-user@bastion ~]$
- 执行命令，进入运行在Window节点中的容器。
$ oc exec -it $(oc get pods -l app=win-webserver -o name) powershell
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
PS C:\>
- 从Window节点中的容器中退出。
PS C:\> exit
用MachineSet扩展Windows节点
- 查看OpenShift中包含Windows的machinesets，以下将用
代表它。
$ oc get machinesets -n openshift-machine-api | grep windows
cluster-pek-2b3a-qxnvj-windows-ap-southeast-1a 1 1 1 1 54m
- 查看OpenShift中运行Windows的machine。
$ oc get machines -n openshift-machine-api -l machine.openshift.io/os-id=Windows
NAME PHASE TYPE REGION ZONE AGE
cluster-pek-2b3a-qxnvj-windows-ap-southeast-1a-lv7rq Running m5a.large ap-southeast-1 ap-southeast-1a 56m
- 执行命令扩展
。
$ oc scale machineset <WINDOWS_MACHINE_SET> --replicas=2 -n openshift-machine-api
machineset.machine.openshift.io/cluster-pek-2b3a-qxnvj-windows-ap-southeast-1a scaled
- 查看node的变化情况。需要一段时间部署新的node，最后确认OpenShift集群会有2个Windows节点。
$ oc get nodes -l kubernetes.io/os=windows -w
NAME STATUS ROLES AGE VERSION
ip-10-0-156-110.ap-southeast-1.compute.internal Ready worker 64m v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal NotReady <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal NotReady <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal NotReady <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal NotReady <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal NotReady <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal Ready <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal Ready <none> 0s v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal Ready worker 1s v1.19.0-rc.2.1023+f5121a6a6a02dd
$ oc get nodes -l kubernetes.io/os=windows
NAME STATUS ROLES AGE VERSION
ip-10-0-156-110.ap-southeast-1.compute.internal Ready worker 66m v1.19.0-rc.2.1023+f5121a6a6a02dd
ip-10-0-158-52.ap-southeast-1.compute.internal Ready worker 17s v1.19.0-rc.2.1023+f5121a6a6a02dd
其他参考
https://www.openshift.com/blog/announcing-the-community-windows-machine-config-operator-on-openshift-4.6