k8s安装

目标：
    192.168.248.151 k8s-master
    192.168.248.161 k8s-node01
    192.168.248.171 k8s-node02
步骤记录：
cd /etc/yum.repos.d/
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

vi /etc/yum.repos.d/kubernetes.repo

    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
    gpgcheck=0
    gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    enabled=1

systemctl stop firewalld.service
systemctl disable firewalld.service

mv CentOS-Base.repo CentOS-Base.repo.bak
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache

yum -y remove kubelet
yum -y remove kubeadm
yum -y remove kubectl

yum install -y kubectl-1.13.4-0.x86_64
yum install -y kubelet-1.13.4-0.x86_64
yum install -y kubeadm-1.13.4-0.x86_64

vi /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

    192.168.248.151 www.k8s-master.com master
    192.168.248.161 www.k8s-node01.com node01
    192.168.248.151 www.k8s-node02.com node02

systemctl start docker.service

注：下面重启网络服务或主机后效果不再，若要其自动执行，可将命令echo "1" > /proc/sys/net/ipv4/ip_forward 写入脚本/etc/rc.d/rc.local 或者 在/etc/sysconfig/network脚本中添加 FORWARD_IPV4="YES"
echo 1 > bridge-nf-call-iptables
echo 1 > bridge-nf-call-ip6tables
echo 1 > /proc/sys/net/ipv4/ip_forward

vim /etc/sysconfig/kubelet
     KUBELET_EXTRA_ARGS="--fail-swap-on=false"

mkdir -p $HOME/.kube
cd $HOME/.kube
vi kube.sh
    docker pull mirrorgooglecontainers/kube-apiserver-amd64:v1.13.4
    docker pull mirrorgooglecontainers/kube-controller-manager-amd64:v1.13.4
    docker pull mirrorgooglecontainers/kube-scheduler-amd64:v1.13.4
    docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.13.4
    docker pull mirrorgooglecontainers/pause:3.1
    docker pull mirrorgooglecontainers/etcd-amd64:3.2.24
    docker pull coredns/coredns:1.2.6

chmod 775 kube.sh     
./kube.sh

docker tag mirrorgooglecontainers/kube-apiserver-amd64:v1.13.4 k8s.gcr.io/kube-apiserver:v1.13.4
docker tag mirrorgooglecontainers/kube-controller-manager-amd64:v1.13.4 k8s.gcr.io/kube-controller-manager:v1.13.4
docker tag mirrorgooglecontainers/kube-scheduler-amd64:v1.13.4 k8s.gcr.io/kube-scheduler:v1.13.4
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.13.4 k8s.gcr.io/kube-proxy:v1.13.4
docker tag docker.io/coredns/coredns:1.2.6 k8s.gcr.io/coredns:1.2.6
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1

-----------------k8s-master/node都需要执行以上步骤--------------------

# kubeadm init --kubernetes-version=v1.13.4 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
    [init] Using Kubernetes version: v1.13.4
    [preflight] Running pre-flight checks
            [WARNING Swap]: running with swap on is not supported. Please disable swap
            [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.3. Latest validated version: 18.06
    [preflight] Pulling images required for setting up a Kubernetes cluster
    [preflight] This might take a minute or two, depending on the speed of your internet connection
    [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Activating the kubelet service
    [certs] Using certificateDir folder "/etc/kubernetes/pki"
    [certs] Generating "ca" certificate and key
    [certs] Generating "apiserver" certificate and key
    [certs] apiserver serving cert is signed for DNS names [www.k8s-master.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.248.151]
    [certs] Generating "apiserver-kubelet-client" certificate and key
    [certs] Generating "etcd/ca" certificate and key
    [certs] Generating "etcd/server" certificate and key
    [certs] etcd/server serving cert is signed for DNS names [www.k8s-master.com localhost] and IPs [192.168.248.151 127.0.0.1 ::1]
    [certs] Generating "etcd/peer" certificate and key
    [certs] etcd/peer serving cert is signed for DNS names [www.k8s-master.com localhost] and IPs [192.168.248.151 127.0.0.1 ::1]
    [certs] Generating "apiserver-etcd-client" certificate and key
    [certs] Generating "etcd/healthcheck-client" certificate and key
    [certs] Generating "front-proxy-ca" certificate and key
    [certs] Generating "front-proxy-client" certificate and key
    [certs] Generating "sa" key and public key
    [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
    [kubeconfig] Writing "admin.conf" kubeconfig file
    [kubeconfig] Writing "kubelet.conf" kubeconfig file
    [kubeconfig] Writing "controller-manager.conf" kubeconfig file
    [kubeconfig] Writing "scheduler.conf" kubeconfig file
    [control-plane] Using manifest folder "/etc/kubernetes/manifests"
    [control-plane] Creating static Pod manifest for "kube-apiserver"
    [control-plane] Creating static Pod manifest for "kube-controller-manager"
    [control-plane] Creating static Pod manifest for "kube-scheduler"
    [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
    [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
    [apiclient] All control plane components are healthy after 38.009827 seconds
    [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
    [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "www.k8s-master.com" as an annotation
    [kubelet-check] Initial timeout of 40s passed.
    [mark-control-plane] Marking the node www.k8s-master.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
    [mark-control-plane] Marking the node www.k8s-master.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
    [bootstrap-token] Using token: 0nk9c6.q1ae3b18jyjdvwv6
    [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
    [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
    [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
    [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy

    Your Kubernetes master has initialized successfully!

    To start using your cluster, you need to run the following as a regular user:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/

    You can now join any number of machines by running the following on each node
    as root:

      kubeadm join 192.168.248.151:6443 --token 0nk9c6.q1ae3b18jyjdvwv6 --discovery-token-ca-cert-hash sha256:78962ed6e497acbcba86daa2709e19c212324db134449a64a06e17c41e86eeca

[root@www .kube]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@www .kube]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@www .kube]# export KUBECONFIG=/etc/kubernetes/admin.conf
[root@www .kube]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

k8s-node安装
-----------------k8s-master/node共同的执行步骤结束，执行以下命令--------------------
kubeadm join 192.168.248.151:6443 --token 0nk9c6.q1ae3b18jyjdvwv6 --discovery-token-ca-cert-hash sha256:78962ed6e497acbcba86daa2709e19c212324db134449a64a06e17c41e86eeca
-----------------即k8s-master初始化之后生成的加入master的语句-----------------------
遇到的问题：
1.禁用swap
    swapoff -a
    同时
    vi /etc/fstab
    注释掉swap那一行
    free m 查看swap使用
2.Unable to update cni config: No networks found in /etc/cni/net.d
    Apr 08 19:32:35 k8s-master kubelet[62363]: E0408 19:32:35.989153   62363 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
    解决方式：vi /etc/cni/net.d/10-flannel.conf 加入
    {
        "name": "cbr0",
        "type": "flannel",
        "delegate": {
            "isDefaultGateway": true
        }
    }

参考文献：
1.Kubernetes特性
2.Kubernetes 集群安装
3.kubeadm部署kubernetes集群1
4.从集群外访问k8s的pod 的几种方式--hostNetwork
5.永久关闭linux swap
6.Docker集群管理工具-Kubernetes1.10.6（k8s）四节点集群使用kubeadm部署流程(一)
7.k8s集群加入节点时出现configmaps "kubelet-config-1.11" is forbidden错误的解决方式
8.Kubernetes之kubectl常用命令
9.Kubeadm踩坑
10.centos7环境下kubeadm方式安装kubernates1.13
11.centos7安装指定版本docker和kubeadm安装kubernetes
12.Kubernetes14--Kubeadm详解
13.【kubernetes/k8s概念】CNI详解
14.kubernetes nodes notready 解决思路
15.k8s 集群部署问题整理  
16.Unable to update cni config: No networks found in /etc/cni/net.d