概述
在前面一份文档中介绍了如何下载离线镜像并打包镜像仓库文件，接下来将介绍如何使用镜像仓库文件进行离线安装。
环境
一台基础架构节点：172.31.20.100
一台bootstrap节点：172.31.20.101
三台master节点：172.31.20.102-104
两台worker节点：172.31.20.105-106
网关：172.31.20.254
DNS Server：172.31.0.121
搭建本地镜像仓库(在基础架构节点上)
- 安装工具
yum -y install podman httpd-tools
- 创建准备给本地私有镜像仓库的相关目录
mkdir -p /opt/registry/{auth,certs,data}
- 创建证书，然后会提示你输入相关信息，
注意：Common Name (eg, your name or your server's hostname) 字段要填写镜像仓库的域名，不能使用IP。其他的可以空着，这一步我错了几次，如下：
cd /opt/registry/certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt
Generating a 4096 bit RSA private key
....................................++
................................................................................................................................................................................................................................................................................................................++
writing new private key to 'domain.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:cn
State or Province Name (full name) []:shanghai
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:registry.ocp4.poc.com
Email Address []:
- 创建密码文件
htpasswd -bBc /opt/registry/auth/htpasswd admin admin
- 准备好镜像压缩文件解压
cd /opt/registry/data/
tar -zxvf ocp4.3.8-images.tar.gz
# ll
drwxr-xr-x. 3 root root 22 Apr 20 07:49 docker
-rw-r--r--. 1 root root 6620484921 Apr 20 22:34 ocp4.3.8-images.tar.gz
- 使用一个镜像来部署私有镜像仓库：
podman run -d --name mirror-registry -p 5000:5000 \
-v /opt/registry/data:/var/lib/registry:z \
-v /opt/registry/auth:/auth:z \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v /opt/registry/certs:/certs:z \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
docker.io/library/registry:2
注意需要放行特定的防火墙端口，我这里直接关闭防火墙，所以不开放了，也可以按照官方文档开通防火墙：
firewall-cmd --add-port=5000/tcp --zone=internal --permanent
firewall-cmd --add-port=5000/tcp --zone=public --permanent
firewall-cmd --reload
- 更新Linux系统证书
cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust
- 测试镜像仓库是否能正常访问：
需要在/etc/hosts文件中添加域名解析，否则无法解析 registry.ocp4.poc.com地址。
curl -u admin:admin -k https://registry.ocp4.poc.com:5000/v2/_catalog
{"repositories":[]}
- 将镜像仓库的用户名密码镜像base64加密
echo -n ’admin:admin' | base64 -w0
YWRtaW46YWRtaW4=
- 安装jq，为了能够将密钥文件进行json格式化，好看些。
yum install jq
- 去下载pull-secret.text文件，然后json格式化生成新的文件
mkdir -p ~/.openshift
cat ./pull-secret.text | jq . > pull-secret
- 文件内容如下：
{
"auths": {
"cloud.openshift.com": {
"auth": "b3BlbnNo...",
"email": "[email protected]"
},
"quay.io": {
"auth": "b3BlbnNo...",
"email": "[email protected]"
},
"registry.connect.redhat.com": {
"auth": "NTE3Njg5Nj...",
"email": "[email protected]"
},
"registry.redhat.io": {
"auth": "NTE3Njg5Nj...",
"email": "[email protected]"
}
}
}
- 然后添加一下私有镜像仓库的内容，用于将quay的镜像同步到私有镜像仓库过程中需要做认证，这就是个认证密码文件，类似如下。
{ "auths": {
"registry.ocp4.poc.com:5000": {
"auth": "YWRtaW46YWRtaW4=",
"email": "[email protected]"
}
}
}
在基础架构节点上安装相关环境
- 下载github代码
注意要使用新版本的ansible，我使用的是2.9.6，否则ansible会报错ssh-key生成的问题。
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm -E %rhel).noarch.rpm
yum -y install ansible git
git clone https://github.com/RedHatOfficial/ocp4-helpernode
cd ocp4-helpernode
- 准备ansible相关文件
cp docs/examples/vars.yaml .
- 修改mac地址等相关参数如下：
---
disk: sda
helper:
name: "helper"
ipaddr: "172.31.20.100"
dns:
domain: "poc.com"
clusterid: "ocp4"
forwarder1: "172.31.0.121"
forwarder2: "8.8.4.4"
dhcp:
router: "172.31.20.254"
bcast: "172.31.20.255"
netmask: "255.255.255.0"
poolstart: "172.31.20.101"
poolend: "172.31.20.109"
ipid: "172.31.20.0"
netmaskid: "255.255.255.0"
bootstrap:
name: "bootstrap"
ipaddr: "172.31.20.101"
macaddr: "00:50:56:b7:fa:e1"
masters:
- name: "master00"
ipaddr: "172.31.20.102"
macaddr: "00:50:56:b7:87:b4"
- name: "master01"
ipaddr: "172.31.20.103"
macaddr: "00:50:56:b7:87:95"
- name: "master02"
ipaddr: "172.31.20.104"
macaddr: "00:50:56:b7:a8:32"
workers:
- name: "worker00"
ipaddr: "172.31.20.105"
macaddr: "00:50:56:b7:db:1e"
- name: "worker01"
ipaddr: "172.31.20.106"
macaddr: "00:50:56:b7:29:74"
- 先下载相关文件基础文件，再导入基础架构节点，否则在执行ansible下载非常慢
ocp_bios: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.8/rhcos-4.3.8-x86_64-metal.x86_64.raw.gz"
ocp_initramfs: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.8/rhcos-4.3.8-x86_64-installer-initramfs.x86_64.img"
ocp_install_kernel: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.8/rhcos-4.3.8-x86_64-installer-kernel-x86_64"
ocp_client: "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.3.8/openshift-client-linux-4.3.8.tar.gz"
ocp_installer: "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.3.8/openshift-install-linux-4.3.8.tar.gz"
- 将下载好的文件导入的路径，如果没有对应的路径，手动创建。
mkdir -p /var/www/html/install/
mkdir -p /var/lib/tftpboot/rhcos/
cp openshift-client-linux-4.3.8.tar.gz /usr/local/src/openshift-client-linux.tar.gz
cp openshift-install-linux-4.3.8.tar.gz /usr/local/src/openshift-install-linux.tar.gz
cp rhcos-4.3.8-x86_64-metal.x86_64.raw.gz /var/www/html/install/bios.raw.gz
cp rhcos-4.3.8-x86_64-installer-initramfs.x86_64.img /var/lib/tftpboot/rhcos/initramfs.img
cp rhcos-4.3.8-x86_64-installer-kernel-x86_64 /var/lib/tftpboot/rhcos/kernel
- 执行ansible
ansible-playbook -e @vars.yaml tasks/main.yml
- 在named里面添加一条解析registry的dns的A记录
cat /var/named/zonefile.db
$TTL 1W
@ IN SOA ns1.ocp4.poc.com. root (
2020042803 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS ns1.ocp4.poc.com.
IN MX 10 smtp.ocp4.poc.com.
;
;
ns1 IN A 172.31.20.100
smtp IN A 172.31.20.100
;
helper IN A 172.31.20.100
registry IN A 172.31.20.100
;
- 重启named
systemctl restart named
- 查看环境的工具使用
/usr/local/bin/helpernodecheck
Usage:
helpernodecheck {dns-masters|dns-workers|dns-etcd|dns-other|install-info|haproxy|services|nfs-info}
- 创建Ignition文件
mkdir ~/ocp4
cd ~/ocp4
mkdir -p ~/.openshift
- 查看helper_rsa文件是否存在
现在的ansible会帮助我们创建密钥文件，也可以关闭：
ls -1 ~/.ssh/helper_rsa
- 准备创建install-config文件
cd ~/ocp4
cat < install-config.yaml
apiVersion: v1
baseDomain: poc.com
compute:
- hyperthreading: Enabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: ocp4
networking:
clusterNetworks:
- cidr: 10.254.0.0/16
hostPrefix: 24
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '$(< ~/.openshift/pull-secret)'
sshKey: '$(< ~/.ssh/helper_rsa.pub)'
EOF
- 补充install-config.yaml的内容
主要增加修改镜像仓地址为内部镜像仓库地址，已经相应的https证书。注意additionalTrustBundle证书那里需要空两格。
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIFzTCCA7WgAwIBAgIJALtWkVirVDe2MA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNV
BAYTAmNuMREwDwYDVQQIDAhzaGFuZ2hhaTEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5
MRwwGgYDVQQKDBNEZWZhdWx0IENvbXBhbnkgTHRkMSYwJAYDVQQDDB1yZWdpc3Ry
eS5vY3A0Lm9mZmxpbmUuenNwLmNvbTAeFw0yMDA0MjAxMzMwMjFaFw0yMTA0MjAx
MzMwMjFaMH0xCzAJBgNVBAYTAmNuMREwDwYDVQQIDAhzaGFuZ2hhaTEVMBMGA1UE
BwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZhdWx0IENvbXBhbnkgTHRkMSYw
JAYDVQQDDB1yZWdpc3RyeS5vY3A0Lm9mZmxpbmUuenNwLmNvbTCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAMLfCbYdD9vciz1IEBauLNo38zwWqUX5HKJm
OARTkAA4CUsmCsa37oFpgKxy4Q8v2kwL3xvyMz5PtO2hBCiHUL07JOpaUn9o4lHC
Nn0+gABFEJIcTe0INPY0DwYXR9w3Nwo3AghMtNCPlaNWvedf4yGYOAVnHHZtj42h
zQnZtnD6ML8YR59xXWlb7zdHIICcwKwxAJLmHoy0zHqQiXskvEh2GKIOeWT1cM1h
vYwchlZbQM6zVbj4udONWkLnn8VWzzuCu1Aeqha4xEd5zkER4wcfR2Vhqn1gCKyH
VylFOBIUNVBQ8dVAQFlHGns+vJWGUMofiB074RvW4dumMui8CEJdwWvo7NeJZk+S
inyCNx5tGg2EgQ43IBV3aP8Q7xFz3/J0FSfcLBDj7Qa0mdtXqPyZRpqRILccvfpg
wjewXbPlDwfR5Iu2R3BjKxzpyRtc9YvG4M6y5J2kfFCQNPx9l4MbTGQxCycpH0CN
ocMSVIdPum76eNFDdXZDloM5LBndsuNRkHhB3RsNutCG47nG6C9JGgUxnfPJGnAI
l5ZBAppSBWMw2q9zi4JZDiU6TgUZfYja5fxa4G4bjNDTLjADyJfIDyPpDrDeA2uV
JRUZeif2+vxn1DtlDliTklL+ywPT4RsojqrCgNd0Q4qFvChb2BJgPwsMYqiLlON5
S4AF2tJNAgMBAAGjUDBOMB0GA1UdDgQWBBR1yN1IwiEnVIX8OwWPYhZicKzDMzAf
BgNVHSMEGDAWgBR1yN1IwiEnVIX8OwWPYhZicKzDMzAMBgNVHRMEBTADAQH/MA0G
CSqGSIb3DQEBCwUAA4ICAQCVl05IwjiY0D/IYpzI+lsmogejIekH9umIV7YEZ7Yk
npLpJxsrBdDYD0itShKkJhDPHbw8H3wU9H3m//mzWf1zdZB/Qa0ZqpcaIz+EeTnZ
TMSSoGEXXfr8jdlYgYaz+4pDUeqpUAbceboWFtrLBuG4dWoAGbL0/Err1odL+wjH
dYiiKQR4cdk1kH3ziwGgJRdAKk+WlRE5wx6+H5KfXVVoiSbFxFxZ8yJfIVZQR7yw
SqXeF8w97WGaMXpX/Ly9VcjjpKy5rnKk7KRpyZPSOSJC198LCS91tOdOriMRdMVS
4jlnRcXsIRUiwVENxlOEOKJLfH1cNFJZpelH1NrZP0pR/aSQs7+JYHTVb9f8xKxx
+yeZ3WlLsVDFjf0tDFHjKuxDoKWSbMClKS3IKJygKe+Q6faOogoMKLssoyZ7M3U6
JHysCVci5MnH0OA+aZTEgyYu8tW5wExcMRNjWnh8oB7bghTDTm6T3DZdDZGyOfcF
RdPRPNFWr4j9Qy96MCVUzyoEC5vKu+UrPSDj2HPjtr0CW31/DKN0rXUbH1oPhPtQ
Yv00nleDBH3x3MDG1lL0mcee4d580pXDSMG1kFl+RHwyyue5oPcwxOWTP2gR5Su/
noMCFwVx/9HWo1BVHJXHrBIXKa7NUbgQmrsorVz+3EXGKKVPdXigRozq/4wjeu8J
3Q==
-----END CERTIFICATE-----
imageContentSources:
- mirrors:
- registry.ocp4.poc.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- registry.ocp4.poc.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- 创建manifests文件：
openshift-install create manifests
- 将master设置为不可调度
sed -i 's/mastersSchedulable: true/mastersSchedulable: false/g' manifests/cluster-scheduler-02-config.yml
- 创建ignition文件：
openshift-install create ignition-configs
- 放到对应路径：
cp ~/ocp4/*.ign /var/www/html/ignition/
restorecon -vR /var/www/html/
chmod o+r /var/www/html/ignition/*.ign
- 开始安装
openshift-install wait-for bootstrap-complete --log-level debug
- 输出日志：
DEBUG OpenShift Installer 4.3.8
DEBUG Built from commit f7a2f7cf9ec3201bb8c9ebb677c05d21c72e3cc5
INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4.poc.com:6443...
DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4.poc.com:6443/version?timeout=32s: EOF
INFO API v1.16.2 up
INFO Waiting up to 30m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
安装结束后
- 准备认证文件
export KUBECONFIG=/root/ocp4/auth/kubeconfig
# 或
cp ~/ocp4/auth/kubeconfig ~/.kube/config
- 查看csr请求
oc get csr
oc get csr --no-headers | awk '{print $1}' | xargs oc adm certificate approve
oc get csr | grep 'system:node'
- 部署nfs storageclass
helpernodecheck nfs-setup
- 将镜像仓库设置为可管理
oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
- 将镜像仓库以route暴露出去
oc patch configs.imageregistry.operator.openshift.io/cluster --type merge -p '{"spec":{"defaultRoute":true}}'
- 设置镜像仓库的存储为pvc
$ oc edit configs.imageregistry.operator.openshift.io
storage:
pvc:
claim:
- 查看image-registry是否ready：
oc get clusteroperator image-registry
- 查看pvc是否创建：
oc get pvc --all-namespaces
- 拿到集群输出信息去登录：
openshift-install wait-for install-complete
参考链接
https://github.com/RedHatOfficial/ocp4-helpernode/blob/master/docs/quickstart.md
https://docs.openshift.com/container-platform/4.3/installing/installing_bare_metal/installing-restricted-networks-bare-metal.html