openshift 3.10部署文档
机器
| ip | hostname |
|---|---|
| 10.39.47.63 | openshift-master |
| 10.39.47.64 | openshift-node-64 |
| 10.39.47.65 | openshift-node-65 |
| 10.39.47.66 | openshift-node-66 |
系统
[root@openshift-master ansible]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
每台都执行yum升级并安装必要的软件包
yum clean all && yum update -y && yum install epel-release -y && yum install lrzsz tree -y
每台都配置系统时区为GMT+8
timedatectl set-timezone Asia/Shanghai
每台都安装docker，并配置docker mirror源
yum install docker -y
cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com"]
}
每个节点都要执行
添加host映射
vi /etc/hosts
10.39.47.63 openshift-master openshift-master.example.com
10.39.47.64 openshift-node-64 openshift-node-64.example.com
10.39.47.65 openshift-node-65 openshift-node-65.example.com
10.39.47.66 openshift-node-66 openshift-node-66.example.com
格式化磁盘(每个节点)
[root@openshift-node-66 ~]# fdisk /dev/vdc
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x658c9b83.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-545259519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-545259519, default 545259519):
Using default value 545259519
Partition 1 of type Linux and of size 260 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@openshift-node-66 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 100G 0 disk
└─vda1 253:1 0 100G 0 part /
vdb 253:16 0 16G 0 disk [SWAP]
vdc 253:32 0 260G 0 disk
└─vdc1 253:33 0 260G 0 part
[root@openshift-node-66 ~]# bi
bind biosdecode biosdevname
[root@openshift-node-66 ~]# b
badblocks bg blkid btrfs btrfs-map-logical bunzip2 bzip2
base64 bind blockdev btrfsck btrfs-select-super busctl bzip2recover
basename biosdecode bond2team btrfs-convert btrfstune bzcat bzless
bash biosdevname bootctl btrfs-debug-tree btrfs-zero-log bzcmp bzmore
bashbug blkdeactivate break btrfs-find-root build-locale-archive bzdiff
bashbug-64 blkdiscard bridge btrfs-image builtin bzgrep
[root@openshift-node-66 ~]# bl
blkdeactivate blkdiscard blkid blockdev
[root@openshift-node-66 ~]# bl
blkdeactivate blkdiscard blkid blockdev
[root@openshift-node-66 ~]# blkid
/dev/vda1: UUID="277fd82b-0856-4e23-8371-1f11823281b7" TYPE="ext4"
/dev/vdb: LABEL="YUNIFYSWAP" UUID="48eb1df6-1663-4a52-ab30-040d552c2d76" TYPE="swap"
[root@openshift-node-66 ~]# mkfs.xfs /dev/vdc1
meta-data=/dev/vdc1 isize=512 agcount=4, agsize=17039296 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=68157184, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=33279, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@openshift-node-66 ~]# blkid
/dev/vda1: UUID="277fd82b-0856-4e23-8371-1f11823281b7" TYPE="ext4"
/dev/vdb: LABEL="YUNIFYSWAP" UUID="48eb1df6-1663-4a52-ab30-040d552c2d76" TYPE="swap"
/dev/vdc1: UUID="ade4f022-1f2d-4854-a34e-876d874b71f8" TYPE="xfs"
[root@openshift-node-66 ~]# vi /etc/fstab
[root@openshift-node-66 ~]#
[root@openshift-node-66 ~]#
[root@openshift-node-66 ~]#
[root@openshift-node-66 ~]#
[root@openshift-node-66 ~]#
[root@openshift-node-66 ~]#
[root@openshift-node-66 ~]# systemctl stop docker
[root@openshift-node-66 ~]# mount -a
[root@openshift-node-66 ~]# systemctl start docker
[root@openshift-node-66 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 100G 0 disk
└─vda1 253:1 0 100G 0 part /
vdb 253:16 0 16G 0 disk [SWAP]
vdc 253:32 0 260G 0 disk
└─vdc1 253:33 0 260G 0 part /var/lib/docker
需要打开selinux
[root@openshift-master ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
Master节点
在master节点上生成免密码登录的ssh key
[root@openshift-master ansible] yum install -y ansible
[root@openshift-master ansible]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
6e:63:84:52:ae:ca:0e:9f:7c:e8:3e:fa:cd:e9:9e:21 root@openshift-master
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . |
| o . |
| . o S |
| o o |
|. E.o = |
| =o*.+ o . |
|.*@=B |
+-----------------+
[root@openshift-master ansible]# ssh-copy-id 10.39.47.63
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Warning: Permanently added '10.39.47.63' (ECDSA) to the list of known hosts.
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '10.39.47.63'"
and check to make sure that only the key(s) you wanted were added.
[root@openshift-master ansible]# ssh-copy-id 10.39.47.64
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Warning: Permanently added '10.39.47.64' (ECDSA) to the list of known hosts.
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '10.39.47.64'"
and check to make sure that only the key(s) you wanted were added.
[root@openshift-master ansible]# ssh-copy-id 10.39.47.65
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Warning: Permanently added '10.39.47.65' (ECDSA) to the list of known hosts.
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '10.39.47.65'"
and check to make sure that only the key(s) you wanted were added.
安装ansible-openshift脚本
yum install centos-release-openshift-origin310 -y
yum install openshift-ansible -y
每台都安装docker，并配置docker mirror源
yum install docker ansible -y
升级ansible(openshift对ansible要求严格)
yum install python-pip python-devel -y
yum install gcc glbc-devel zlib-devel rpm-build openssl-devel -y
pip install -U pip
pip install cryptography -U
pip install ansible --upgrade
需要升级到ansible 2.7.1
升级ansible的文档参考Ansible系列（一） Centos7 安装ansible ，解决cryptography版本低引起的故障问题
TASK [openshift_control_plane : Wait for all control plane pods to become ready] *****************************************************************************
FAILED - RETRYING: Wait for all control plane pods to become ready (60 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (59 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (58 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (57 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (56 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (55 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (54 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (53 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (52 retries left).
FAILED - RETRYING: Wait for all control plane pods to become ready (51 retries left).
这个原因是没有生etcd-ca的证书/etc/origin/master/master.etcd-ca.crt
root@openshift-master ~]# docker ps -a | grep api
5c20e93ac530 ebcfed580e6b "/bin/bash -c '#!/..." 2 minutes ago Exited (255) 2 minutes ago k8s_api_master-api-openshift-master_kube-system_9ca23c5815da8ed1d3dca61d87e1f6ab_77
7f55a8778021 docker.io/openshift/origin-pod:v3.10.0 "/usr/bin/pod" 5 hours ago Up 5 hours k8s_POD_master-api-openshift-master_kube-system_9ca23c5815da8ed1d3dca61d87e1f6ab_0
[root@openshift-master ~]# docker logs 5c20e93ac530
...
Invalid MasterConfig /etc/origin/master/master-config.yaml
etcdClientInfo.ca: Invalid value: "/etc/origin/master/master.etcd-ca.crt": could not read file: stat /etc/origin/master/master.etcd-ca.crt: no such file or directory
相关的issue
https://github.com/openshift/openshift-ansible/issues/8376
https://bugzilla.redhat.com/show_bug.cgi?id=1638699
https://github.com/openshift/openshift-ansible/issues/10368
配置 ansible 库存文件
[root@openshift-master ~]# cat /etc/ansible/hosts
# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
#ansible_become=true
openshift_deployment_type=origin
# uncomment the following to enable htpasswd authentication; defaults to AllowAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_ca_cert_expire_days=3650
openshift_node_cert_expire_days=3650
openshift_master_cert_expire_days=3650
etcd_ca_default_days=3650
#This variable overrides the default subdomain to use for exposed routes
openshift_hosted_manage_registry=false
openshift_disable_check=memory_availability,disk_availability,docker_image_availability
openshift_enable_service_catalog=false
template_service_broker_install=false
ansible_service_broker_install=false
# host group for masters
[masters]
openshift-master.example.com
# host group for etcd
[etcd]
openshift-master.example.com
# host group for nodes, includes region info
[nodes]
openshift-master.example.com openshift_node_group_name='node-config-master-infra'
openshift-node-64.example.com openshift_node_group_name='node-config-compute'
openshift-node-65.example.com openshift_node_group_name='node-config-compute'
openshift-node-66.example.com openshift_node_group_name='node-config-compute'
执行准备工作命令
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml
安装openshift
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml
卸载openshift
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml
添加本地主hosts
➜ # cat /etc/hosts
10.39.47.63 openshift-master
创建用户，参考设置openshift用户登录
[root@openshift-master ~]# htpasswd -b /etc/origin/master/htpasswd dev dev
Adding password for user dev
[root@openshift-master ~]# cat /etc/origin/master/htpasswd
dev:$apr1$kKt0XW0C$SjIX59pIkhk7jYB59uXA..
[root@openshift-master ~]# oc login -u system:admin
Logged into "https://openshift-master:8443" as "system:admin" using existing credentials.
You have access to the following projects and can switch between them with 'oc project ':
* default
kube-public
kube-system
management-infra
openshift
openshift-infra
openshift-logging
openshift-node
openshift-sdn
openshift-web-console
Using project "default".
[root@openshift-master ~]# oc login
Authentication required for https://openshift-master:8443 (openshift)
Username: dev
Password:
Login successful.
You don't have any projects. You can try to create a new project, by running
oc new-project
[root@openshift-master ~]# oc get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "dev" cannot list pods in the namespace "kube-system": User "dev" cannot list pods in project "kube-system"
[root@openshift-master ~]# oc logout
Logged "dev" out on "https://openshift-master:8443"
[root@openshift-master ~]# oc get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:anonymous" cannot list pods in the namespace "kube-system": User "system:anonymous" cannot list pods in project "kube-system"
[root@openshift-master ~]# oc get nodes
Error from server (Forbidden): nodes is forbidden: User "system:anonymous" cannot list nodes at the cluster scope: User "system:anonymous" cannot list all nodes in the cluster
[root@openshift-master ~]# oc login -u system:admin
Logged into "https://openshift-master:8443" as "system:admin" using existing credentials.
You have access to the following projects and can switch between them with 'oc project ':
* default
kube-public
kube-system
management-infra
openshift
openshift-infra
openshift-logging
openshift-node
openshift-sdn
openshift-web-console
Using project "default".
[root@openshift-master ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
openshift-master Ready infra,master 2h v1.10.0+b81c8f8
openshift-node-64 Ready compute 29m v1.10.0+b81c8f8
openshift-node-65 Ready compute 29m v1.10.0+b81c8f8
openshift-node-66 Ready compute 29m v1.10.0+b81c8f8
访问以下地址确保 OpenShift Origin 可以正确访问：
访问 https://openshift-master:8443 用户名和密码就是上一步创建的:dev/dev
登录
参考
example_inventories
index
Openshift集群部署.md
设置openshift用户登录