k8s-1.7.10内网部署安装笔记

昨天做了一天的flannel网络问题终于得到了解决，原来是flannel的版本问题。

安装1.9.1时使用的kubernetes-cni是高版本的，因此可以与自动获取的高版本flannel兼容，没出什么问题。

安装1.7.10时使用的cni则是0.5低版本的，因此不能兼容需要手动获取0.8.0版本的flannel

 

下面进行完整的kubernetes-v1.7.10版本内网安装过程：

---

一、安装docker、go

 

1、安装docker，可以直接用apt安装，版本目前为1.13. 

   apt-get update && apt-get install docker.io

2、安装go，这里使用的是最新版本，从官网下载安装

    http://www.golangtc.com/download

   目前是go1.9.2.linux-amd64.tar.gz，下载后解压

tar -xzf go1.9.2.linux-amd64.tar.gz  -C /usr/local

 设置环境变量 ：

vim ~/.bashrc

 末尾加入

export GOPATH=/opt/go
export GOROOT=/usr/local/go
export GOARCH=386
export GOOS=linux
export GOBIN=$GOROOT/bin/
export GOTOOLS=$GOROOT/pkg/tool/
export PATH=$PATH:$GOBIN:$GOTOOLS

保存后重新加载环境变量

source ~/.bashrc

任意目录下进行安装完成验证

go version

出现版本信息即成功。

 

二、安装kubeadm等工具

1、从可以fq的机子上通过

apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat </etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

获取相应版本的kubeadm、kubectl、kubelet和kubernetes-cni（apt-cache madison ***查看版本）
然后在/var/cache中找到四个文件，

2、在内网计算机安装工具

先安装ebtables和socat工具

apt-get install ebtables
apt-get install socat

再依次安装

dpkg -i kubernetes-cni_0.5.1-00_amd64.deb
dpkg -i kubectl_1.7.10-00_amd64.deb
dpkg -i kubelet_1.7.10-00_amd64.deb
dpkg -i kubeadm_1.7.10-00_amd64.deb

三、获取镜像

1、从fq机上传镜像，上传到自己的docker hub中

先打TAG:

docker tag quay.io/coreos/flannel:v0.8.0-amd64 eavan/flannel:v0.8.0-amd64
docker tag gcr.io/google_containers/kube-apiserver-amd64:v1.7.10 eavan/kube-apiserver-amd64:v1.7.10
docker tag gcr.io/google_containers/kube-proxy-amd64:v1.7.10 eavan/kube-proxy-amd64:v1.7.10
docker tag gcr.io/google_containers/kube-controller-manager-amd64:v1.7.10 eavan/kube-controller-manager-amd64:v1.7.10
docker tag gcr.io/google_containers/kube-scheduler-amd64:v1.7.10 eavan/kube-scheduler-amd64:v1.7.10
docker tag gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 eavan/k8s-dns-sidecar-amd64:1.14.5
docker tag gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 eavan/k8s-dns-kube-dns-amd64:1.14.5
docker tag gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 eavan/k8s-dns-dnsmasq-nanny-amd64:1.14.5
docker tag gcr.io/google_containers/etcd-amd64:3.0.17 eavan/etcd-amd64:3.0.17
docker tag gcr.io/google_containers/pause-amd64:3.0 eavan/pause-amd64:3.0

再通过docker push 命令一一上传，现在已在eavan/中，不必再次上传

 

2、内网机获取镜像

docker pull 一一获取镜像

docker tag 一一反向打上tag

全部打完tag 后可以通过docker rmi 来删除其他的镜像

ps:可以通过编写脚本文件自动执行减少工作量

在master节点上，del.sh

#!/bin/bash
images=(
kube-proxy-amd64:v1.7.10
kube-controller-manager-amd64:v1.7.10
kube-apiserver-amd64:v1.7.10
kube-scheduler-amd64:v1.7.10
k8s-dns-sidecar-amd64:1.14.5
k8s-dns-kube-dns-amd64:1.14.5
k8s-dns-dnsmasq-nanny-amd64:1.14.5
etcd-amd64:3.0.17
pause-amd64:3.0
)
for imageName in ${images[@]} ; do
docker pull eavan/$imageName
docker tag eavan/$imageName gcr.io/google_containers/$imageName
docker rmi eavan/$imageName
done

在node节点上，del.sh

#!/bin/bash 
images=( 
kube-proxy-amd64:v1.7.10  
etcd-amd64:3.0.17 
pause-amd64:3.0 
) 
for imageName in ${images[@]} ; do 
docker pull eavan/$imageName
docker tag eavan/$imageName gcr.io/google_containers/$imageName 
docker rmi eavan/$imageName 
done

四、master初始化

hostname xxx
vi /etc/hostname 
vi /etc/hosts
reboot
kubeadm init --kubernetes-version=v1.7.10 --pod-network-cidr=10.244.0.0/16

初始化完成后按照打印的要求做。

（若想要将master节点布置为可以运行pod的节点）

kubectl taint nodes --all node-role.kubernetes.io/master-

五、安装flannel

wget https://raw.githubusercontent.com/coreos/flannel/v0.8.0/Documentation/kube-flannel-rbac.yml
wget https://raw.githubusercontent.com/coreos/flannel/v0.8.0/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel-rbac.yml
kubectl apply -f kube-flannel.yml

成功后created 5个文件

通过

kubectl get pod --all-namespaces -o wide

查看全部pod，都running则完成。

六、8080端口及DNS问题

1、解决curl localhost:8080无法访问

vi /etc/kubernetes/manifests/kube-apiserver.yaml

找到insecure-port，原始值为0，按照正常来说0端口代表所有端口，但在本此问题中，0代表默认为6443，因此apiserver不通过8080端口开放，需要将0改为8080即可

（若改完后显示6443拒绝访问，等待或者systemctl restart kubelet）

改完后通过curl localhost:8080即可看到api，也可以通过lsof -i:8080观察8080端口的监听

 

2、解决DNS总是crashloopback的问题

修改Kubelet启动参数

vim /etc/systemd/system/kubelet.service.d

找到并添加

KUBELET_DNS_ARGS=--address=192.168.xxx.xxx(本机Ip)

 

3、解决DNS无法解析域名的问题

测试DNS是否正常工作：

创建busybox.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

使用该文件创建pod

kubectl create -f busybox.yaml

等待pod进入running状态

kubectl get pods busybox

一旦pod处于running状态时，可以使用exec nslookup来查询状态：

kubectl exec -ti busybox -- nslookup kubernetes.default

发现出现域名无法解析的状况（can't find ）

① 修改/etc/resolv.conf，添加nameserver 192.168.224.2(虚拟机dns域名)

systemctl restart docker 
systemctl restart kubelet

② 修改防火墙

iptables -P FORWARD ACCEPT

再次查询状态即发现已成功。

ps：修改/etc/resolv.conf仅为一次性，当重启网络或reboot后会失效

永久性修改：

vim /etc/network/interfaces 

添加

dns-nameservers 192.168.224.2 即可

 

七、dashboard安装

注意：目前只安装上1.6.2成功

https://www.cnblogs.com/aguncn/p/7158881.html

创建kubernetes-dashboard-rbac.yaml

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin 
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system

创建kubernetes-dashboard.yaml

# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.6 (RBAC enabled).
#
# Example usage: kubectl create -f 

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
      - name: kubernetes-dashboard
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/kubernetes-dashboard-amd64:v1.6.1
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          # - --apiserver-host=http://my-address:port
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 9090
  selector:
    k8s-app: kubernetes-dashboard

再create便可成功。

通过

kubectl get svc -n kube-system

查看dashboard的端口，然后打开浏览器，通过localhost:xxx访问

 

八、heapster安装

mkdir /etc/kubernetes/heapster
cd heapster
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml

修改docker images 来源:

registry.cn-shenzhen.aliyuncs.com/rancher_cn/heapster-grafana-amd64:v4.4.3

registry.cn-shenzhen.aliyuncs.com/rancher_cn/heapster-amd64:v1.4.2

registry.cn-shenzhen.aliyuncs.com/rancher_cn/heapster-influxdb-amd64:v1.3.3

 

kubectl create -f ./

即可