Kubernetes学习笔记-Part.05 基础环境准备
目录
Part.01 Kubernets与docker
Part.02 Docker版本
Part.03 Kubernetes原理
Part.04 资源规划
Part.05 基础环境准备
Part.06 Docker安装
Part.07 Harbor搭建
Part.08 K8s环境安装
Part.09 K8s集群构建
Part.10 容器回退
第五章 基础环境准备
5.1.SSH免密登录
在master01、master02、master03上生成公钥，配置免密登录到其他节点
ssh-keygen -t rsa -f ~/.ssh/id_rsa -C username_root
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 [email protected]
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 [email protected]
5.2.ansbile配置
在外网服务器上，下载ansible及相关依赖包
yum install -y epel-release
yumdownloader --resolve --destdir /opt/ansible/ ansible
上传至master01上，并进行安装
rpm -ivh /opt/ansible/*
安装完成后查询版本
[root@localhost ~]# ansible --version
ansible 2.9.27
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
配置ansible和rhel-system-roles，创建配置文件
mkdir /root/ansible
cd /root/ansible
cp /etc/ansible/ansible.cfg /root/ansible/
修改配置文件，/root/ansible/ansible.cfg
[defaults]
inventory = /root/ansible/inventory
ask_pass = false
remote_user = root
配置inventory文件，/root/ansible/inventory
[k8s:children]
master
worker
harbor
[master]
192.168.111.1 hostname=master01
192.168.111.2 hostname=master02
192.168.111.3 hostname=master03
[worker]
192.168.111.11 hostname=worker01
192.168.111.12 hostname=worker02
[harbor]
192.168.111.20 hostname=harbor01
测试
[root@master01 ansible]# ansible all -m ping
192.168.111.3 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
192.168.111.12 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
192.168.111.11 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
192.168.111.1 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
192.168.111.2 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
192.168.111.20 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
5.3.修改主机名
创建playbook，/root/ansible/hostname.yml
---
- name: modify hostname
hosts: all
tasks:
- name: modify hostname permanently
raw: "echo {{ hostname | quote }} > /etc/hostname"
- name: modify hostname temporarily
shell: hostname {{ hostname | quote }}
执行并确认
[root@master01 ansible]# ansible-playbook hostname.yml
PLAY [modify hostname] ****************************************************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************************************
ok: [192.168.111.11]
ok: [192.168.111.12]
ok: [192.168.111.1]
ok: [192.168.111.2]
ok: [192.168.111.3]
ok: [192.168.111.20]
TASK [modify hostname permanently] ****************************************************************************************************************************
changed: [192.168.111.2]
changed: [192.168.111.1]
changed: [192.168.111.11]
changed: [192.168.111.3]
changed: [192.168.111.12]
changed: [192.168.111.20]
TASK [modify hostname temporarily] ****************************************************************************************************************************
changed: [192.168.111.3]
changed: [192.168.111.11]
changed: [192.168.111.1]
changed: [192.168.111.2]
changed: [192.168.111.12]
changed: [192.168.111.20]
PLAY RECAP ****************************************************************************************************************************************************
192.168.111.1 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.11 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.12 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.2 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.20 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.3 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
[root@master01 ansible]# ansible all -m shell -a 'hostname'
192.168.111.3 | CHANGED | rc=0 >>
master03
192.168.111.11 | CHANGED | rc=0 >>
worker01
192.168.111.1 | CHANGED | rc=0 >>
master01
192.168.111.2 | CHANGED | rc=0 >>
master02
192.168.111.12 | CHANGED | rc=0 >>
worker02
192.168.111.20 | CHANGED | rc=0 >>
harbor01
5.4.修改hosts列表
在master01上修改主机列表，/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.111.1 master01.k8s.local master01
192.168.111.2 master02.k8s.local master02
192.168.111.3 master03.k8s.local master03
192.168.111.11 worker01.k8s.local worker01
192.168.111.12 worker02.k8s.local worker02
192.168.111.20 harbor01.k8s.local harbor01
分发至其他节点
ansible all -m template -a 'src=/etc/hosts dest=/etc/hosts'
5.5.关闭firewall和SELinux
关闭firewall
ansible all -m service -a 'name=firewalld state=stopped enabled=no'
确认状态
[root@master01 ansible]# ansible all -m shell -a 'systemctl status firewalld | grep Active'
192.168.111.11 | CHANGED | rc=0 >>
Active: inactive (dead)
192.168.111.12 | CHANGED | rc=0 >>
Active: inactive (dead)
192.168.111.1 | CHANGED | rc=0 >>
Active: inactive (dead)
192.168.111.3 | CHANGED | rc=0 >>
Active: inactive (dead)
192.168.111.2 | CHANGED | rc=0 >>
Active: inactive (dead)
192.168.111.20 | CHANGED | rc=0 >>
Active: inactive (dead)
关闭SELinux
ansible all -m selinux -a 'policy=targeted state=disabled'
确认状态
[root@localhost ansible]# ansible all -m shell -a 'getenforce'
192.168.111.1 | CHANGED | rc=0 >>
Permissive
192.168.111.11 | CHANGED | rc=0 >>
Permissive
192.168.111.3 | CHANGED | rc=0 >>
Permissive
192.168.111.2 | CHANGED | rc=0 >>
Permissive
192.168.111.12 | CHANGED | rc=0 >>
Permissive
192.168.111.20 | CHANGED | rc=0 >>
Permissive
5.6.配置系统Yum源
【master01】配置CentOS镜像Yum源
mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom/
rm -f /etc/yum.repos.d/*
创建repo文件，/etc/yum.repos.d/local.repo
[centos]
name=centos
baseurl=file:///mnt/cdrom
gpgcheck=0
enabled=1
更新yum源
yum clean all
yum makecache fast
安装httpd服务
yum install -y httpd
systemctl enable --now httpd
配置http服务指向CentOS源
mkdir /var/www/html/centos
umount /mnt/cdrom/
mount /dev/cdrom /var/www/html/centos/
删除原有repo文件
ansible all -m shell -a 'rm -f /etc/yum.repos.d/*.repo'
配置所有节点的系统Yum源
ansible all -m yum_repository -a 'name="centos" description="centos" baseurl="http://master01.k8s.local/centos" enabled=yes gpgcheck=no'
ansible all -m shell -a 'yum clean all'
ansible all -m shell -a 'yum makecache fast'
ansible all -m shell -a 'yum update -y'
5.7.安装基础软件
安装vim等基础软件，/root/ansible/packages.yml
---
- hosts: all
tasks:
- name: install packages
yum:
name:
- pciutils
- bash-completion
- vim
- chrony
- net-tools
state: present
执行并确认
[root@master01 ansible]# ansible-playbook packages.yml
PLAY [all] ****************************************************************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************************************
ok: [192.168.111.3]
ok: [192.168.111.1]
ok: [192.168.111.12]
ok: [192.168.111.11]
ok: [192.168.111.2]
ok: [192.168.111.20]
TASK [install packages] ***************************************************************************************************************************************
ok: [192.168.111.2]
ok: [192.168.111.11]
ok: [192.168.111.1]
ok: [192.168.111.12]
ok: [192.168.111.20]
changed: [192.168.111.3]
PLAY RECAP ****************************************************************************************************************************************************
192.168.111.1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.11 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.12 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.2 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.20 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
192.168.111.3 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
5.8.NTP时钟
以master01为时钟源，其余节点从master01进行时钟同步
服务端（master01）
修改配置文件，/etc/chrony.conf
# 不指定外部NTP源
# 允许本网段其节点作为客户端访问
allow 192.168.111.0/24
# 如果时间服务可不用，则使用本地时间作为标准时间授权，层数为10
local stratum 10
重启服务
systemctl restart chronyd
客户端（mster02/worker01/worker02/harbor01）
在外网服务器上下载ansible system role的安装包
yumdownloader --resolve rhel-system-roles
将安装包上传至master01的/opt/ansible/下，并进行安装
[root@localhost ~]# rpm -ivh /opt/ansible/python-netaddr-0.7.5-9.el7.noarch.rpm
warning: /opt/ansible/python-netaddr-0.7.5-9.el7.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:python-netaddr-0.7.5-9.el7 ################################# [100%]
[root@localhost ~]# rpm -ivh /opt/ansible/rhel-system-roles-1.7.3-4.el7_9.noarch.rpm
warning: /opt/ansible/rhel-system-roles-1.7.3-4.el7_9.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:rhel-system-roles-1.7.3-4.el7_9 ################################# [100%]
安装ntp时钟，/root/ansible/timesync.yml
---
- hosts: 192.168.111.2,192.168.111.3,worker,harbor
vars:
timesync_ntp_servers:
- hostname: 192.168.111.1
iburst: yes
roles:
- rhel-system-roles.timesync
执行
ansible-playbook /root/ansible/timesync.yml
确认时钟同步情况
[root@master01 ansible]# ansible 192.168.111.2,192.168.111.3,worker,harbor -m shell -a 'chronyc sources -v'
192.168.111.12 | CHANGED | rc=0 >>
210 Number of sources = 1
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local 10 6 377 46 +5212ns[ +19us] +/- 73us
192.168.111.3 | CHANGED | rc=0 >>
210 Number of sources = 1
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local 10 6 17 30 -261ns[ -62us] +/- 966us
192.168.111.11 | CHANGED | rc=0 >>
210 Number of sources = 1
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local 10 6 377 35 -17us[ -20us] +/- 130us
192.168.111.20 | CHANGED | rc=0 >>
210 Number of sources = 1
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local 10 6 377 25 -4152ns[-7463ns] +/- 96us
192.168.111.2 | CHANGED | rc=0 >>
210 Number of sources = 1
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* master01.k8s.local 10 6 377 27 -52us[ -50us] +/- 191us
5.9.关闭swap
临时关闭：
ansible all -m shell -a 'swapoff -a'
永久关闭：
ansible all -m shell -a 'sed -ri "s/.*swap.*/#&/" /etc/fstab'
5.10.启用ipvs转发
在kubernetes中service有两种代理模型，一种是基于iptables的，一种是基于ipvs的；ipvs转发性能更好。
在master01-03上开启ipvs转发
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
赋予执行权限并执行
chmod +x /etc/sysconfig/modules/ipvs.modules
/bin/bash /etc/sysconfig/modules/ipvs.modules
5.11.启用网桥过滤及内核转发
bridge-nf-call-iptables这个内核参数，表示bridge设备在二层转发时也去调用iptables配置的三层规则（包含conntrack），所以开启这个参数就能够解决Service同节点通信问题。
在master01上创建/etc/sysctl.d/k8s.conf，添加网桥过滤及内核转发配置
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
分发至其他节点
ansible all -m template -a 'src=/etc/sysctl.d/k8s.conf dest=/etc/sysctl.d/'
ansible all -m shell -a 'modprobe br_netfilter'
验证是否生效
[root@master01 ansible]# ansible all -m shell -a 'sysctl --system | grep -A3 k8s'
192.168.111.3 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.1 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.12 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.11 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.2 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
192.168.111.20 | CHANGED | rc=0 >>
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1