记minikube启动失败，爬坑排错

最近在学习使用Minikube运行一个本地单节点Kubernetes集群，没想到在启动minikube的时候就遇到了很多坑，只能一个一个排。

理论基础我就不再阐述了，我这里只介绍安装启动步骤，和我遇到的坑，及解决办法。

安装kubectl

	curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

	chmod +x ./kubectl

	mv ./kubectl /usr/local/bin/

	kubectl version

安装VirtualBox

	wget http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo

	[root@localhost yum.repos.d]# cat virtualbox.repo 
	[virtualbox]
	name=Oracle Linux / RHEL / CentOS-$releasever / $basearch - VirtualBox
	baseurl=http://download.virtualbox.org/virtualbox/rpm/el/$releasever/$basearch
	enabled=1
	gpgcheck=1
	repo_gpgcheck=1
	gpgkey=https://www.virtualbox.org/download/oracle_vbox.asc
	
	//更新源
	yum  upgrade

	//查看VirtualBox版本
	yum search VirtualBox

	已加载插件：product-id, search-disabled-repos, subscription-manager
	This system is not registered with an entitlement server. You can use subscription-manager to register.
	
	VirtualBox-4.3.x86_64 : Oracle VM VirtualBox
	VirtualBox-5.0.x86_64 : Oracle VM VirtualBox
	VirtualBox-5.1.x86_64 : Oracle VM VirtualBox
	VirtualBox-5.2.x86_64 : Oracle VM VirtualBox
	VirtualBox-6.0.x86_64 : Oracle VM VirtualBox

	//安装VirtualBox-6.0
	yum -y install VirtualBox-6.0

安装Minikube，并启动

	Minikube需要开启内核虚拟化。所以在安装之前，通过egrep -o "vmx|svm" /proc/cpuinfo查看本机是否开启。

	//安装Minikube
	curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
    && chmod +x minikube

	mv ./minikube /usr/local/bin

	//启动
	minikube start 

	特别注意：minikube start --vm-driver=none 
	这是说表示minikube运行在宿主机不需要KVM或者VirtualBox，但根据官方文档的介绍，我们还是不采用这种方法。

此时错误就来了，而且一来一大堆，只能一个一个慢慢解决

We support Virtualbox starting with version 5. Your VirtualBox install is “WARNING: The vboxdrv kernel module is not loaded. Either there is no module available for the current kernel (3.10.0-862.11.6.el7.x86_64) or it failed to load. Please recompile the kernel module and install it by sudo /sbin/vboxconfig You will not be able to start VMs until this problem is fixed. 5.2.18r124319”. Please upgrade at https://www.virtualbox.org.

一大堆看不懂的，反正启动报错了，我们执行/sbin/vboxconfig，查看我们缺少哪些步骤。

vboxdrv.sh: Stopping VirtualBox services.
vboxdrv.sh: Starting VirtualBox services.
vboxdrv.sh: Building VirtualBox kernel modules.
This system is currently not set up to build kernel modules.
Please install the Linux kernel “header” files matching the current kernel
for adding new hardware support to the system.
The distribution packages containing the headers are probably:
kernel-devel kernel-devel-3.10.0-862.11.6.el7.x86_64

所以解决这个错误的关键就是：
Please install the Linux kernel “header” files matching the current kernel
执行命令行：yum install kernel-devel

安装过程就省略了，下载完成后再次执行/sbin/vboxconfig发现还是提示相同的问题

	//如果输出的是下面内容，则表示正常。
	vboxdrv.sh: Stopping VirtualBox services.
	vboxdrv.sh: Starting VirtualBox services.
	vboxdrv.sh: Building VirtualBox kernel modules

还是提示安装kernel-devel，我们考虑uname -r的内核版本和安装kernel-devel版本不兼容，我们需要将内核版本升级到我们安装的版本。

	//查看内核版本
	uname -r
	3.10.0-693.21.1.el7.x86_64

	ll /usr/src/kernels/
	total 4
	drwxr-xr-x 22 root root 4096 Jun 21 17:14 3.10.0-957.21.3.el7.x86_64

升级内核

	//更新yum源
	yum update


	//安装grub2（centos7)
	yum install kernel grub2


	//配置grub
	//这是未修改前的配置：
	[root@localhost ~]# cat /etc/default/grub 
	GRUB_TIMEOUT=5
	GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
	GRUB_DEFAULT=saved
	GRUB_DISABLE_SUBMENU=true
	GRUB_TERMINAL_OUTPUT="console"
	GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet"
	GRUB_DISABLE_RECOVERY="true"


	//修改/etc/default/grub，将GRUB_CMDLINE_LINUX后面的单词quiet删除，其他不变，保存退出：
	[root@localhost ~]# cat /etc/default/grub 
	GRUB_TIMEOUT=5
	GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
	GRUB_DEFAULT=saved
	GRUB_DISABLE_SUBMENU=true
	GRUB_TERMINAL_OUTPUT="console"
	GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb"
	GRUB_DISABLE_RECOVERY="true"


	//配置内核启动文件
	mkdir /boot/grub
	ln -s /boot/grub2/grub.cfg /boot/grub/grub.cfg
	grub2-mkconfig -o /boot/grub/grub.cfg
	touch /.autorelabel


	//重启等待完成
	[root@localhost ~]# reboot
	[root@localhost ~]# uname -r
	3.10.0-957.27.2.el7.x86_64

再次执行/sbin/vboxconfig

	[root@localhost ~]# /sbin/vboxconfig
	vboxdrv.sh: Stopping VirtualBox services.
	vboxdrv.sh: Starting VirtualBox services.
	vboxdrv.sh: Building VirtualBox kernel modules.

启动Minikube，启动过程时间较长，不要中断

minikube start

• minikube v1.3.1 on Redhat 7.4
  ! Please don’t run minikube as root or with ‘sudo’ privileges. It isn’t necessary with virtualbox driver.
• Tip: Use 'minikube start -p ’ to create a new cluster, or ‘minikube delete’ to delete this one.
• Using the running virtualbox “minikube” VM …
• Waiting for the host to be provisioned …
• Preparing Kubernetes v1.15.2 on Docker 18.09.8 …
  E0910 15:19:22.217027 23391 start.go:558] Error caching images: Caching images for kubeadm: caching images: caching image /root/.minikube/cache/images/k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64_1.14.13: fetching remote image: Get https://k8s.gcr.io/v2/: dial tcp 108.177.97.82:443: connect: connection refused
• Unable to load cached images: loading cached images: loading image /root/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: stat /root/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1: no such file or directory
• Relaunching Kubernetes using kubeadm …

X Error restarting cluster: waiting for apiserver: timed out waiting for the condition

• Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  • https://github.com/kubernetes/minikube/issues/new/choose

报错：Error restarting cluster: waiting for apiserver: timed out waiting for the condition

	 执行minikube delete，minikube start 解决

执行minikube start 会拉取镜像，但是由于需要fq，所以会报错拉取不下来，我们可以选择利用docker拉取镜像，最后再修改tag

//拉取镜像

	docker pull mirrorgooglecontainers/kube-apiserver:v1.15.2
	docker pull mirrorgooglecontainers/kube-controller-manager:v1.15.2
	docker pull mirrorgooglecontainers/kube-scheduler:v1.15.2
	docker pull mirrorgooglecontainers/kube-proxy:v1.15.2
	docker pull mirrorgooglecontainers/pause:3.1
	docker pull mirrorgooglecontainers/etcd:3.3.10
	docker pull coredns/coredns:1.3.1

//修改标签

	docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.15.3 k8s.gcr.io/kube-proxy:v1.15.3
	docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.15.3 k8s.gcr.io/kube-scheduler:v1.15.3
	docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.15.3 k8s.gcr.io/kube-controller-manager:v1.15.3
	docker tag docker.io/mirrorgooglecontainers/kube-apiserver-amd64:v1.15.2 k8s.gcr.io/kube-apiserver-amd64:v1.15.2
	docker tag docker.io/mirrorgooglecontainers/pause:3.1  k8s.gcr.io/pause:3.1
	docker tag docker.io/mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
	docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

//标签修改后，查看镜像

	[root@localhost docker]# docker images 
	REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE
	mirrorgooglecontainers/kube-proxy                v1.15.3             232b5c793146        3 weeks ago         82.4MB
	k8s.gcr.io/kube-proxy                            v1.15.3             232b5c793146        3 weeks ago         82.4MB
	mirrorgooglecontainers/kube-controller-manager   v1.15.3             e77c31de5547        3 weeks ago         159MB
	k8s.gcr.io/kube-controller-manager               v1.15.3             e77c31de5547        3 weeks ago         159MB
	mirrorgooglecontainers/kube-scheduler            v1.15.3             703f9c69a5d5        3 weeks ago         81.1MB
	k8s.gcr.io/kube-scheduler                        v1.15.3             703f9c69a5d5        3 weeks ago         81.1MB
	mirrorgooglecontainers/kube-apiserver-amd64      v1.15.2             34a53be6c9a7        5 weeks ago         207MB
	k8s.gcr.io/kube-apiserver-amd64                  v1.15.2             34a53be6c9a7        5 weeks ago         207MB
	coredns/coredns                                  1.3.1               eb516548c180        8 months ago        40.3MB
	k8s.gcr.io/coredns                               1.3.1               eb516548c180        8 months ago        40.3MB
	mirrorgooglecontainers/etcd                      3.3.10              2c4adeb21b4f        9 months ago        258MB
	k8s.gcr.io/etcd                                  3.3.10              2c4adeb21b4f        9 months ago        258MB
	mirrorgooglecontainers/pause                     3.1                 da86e6ba6ca1        20 months ago       742kB
	k8s.gcr.io/pause     

//rmi删除不用的镜像，最后再查看保留下来的镜像

修改标签后，它们具有相同的image ID，此时如果通过删除imageID的方式是不可行的。我们可以通过删除REPOSITORY:TAG的方式删除镜像。

例如：

	docker rmi mirrorgooglecontainers/pause:3.1

	[root@localhost docker]# docker images 
	REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
	k8s.gcr.io/kube-proxy                v1.15.3             232b5c793146        3 weeks ago         82.4MB
	k8s.gcr.io/kube-controller-manager   v1.15.3             e77c31de5547        3 weeks ago         159MB
	k8s.gcr.io/kube-scheduler            v1.15.3             703f9c69a5d5        3 weeks ago         81.1MB
	k8s.gcr.io/kube-apiserver-amd64      v1.15.2             34a53be6c9a7        5 weeks ago         207MB
	k8s.gcr.io/coredns                   1.3.1               eb516548c180        8 months ago        40.3MB
	k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        9 months ago        258MB
	k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        20 months ago       742kB

启动加上参数，使用本地的镜像

export http_proxy=http://192.168.157.19:5000;export https_proxy=http://192.168.157.19:5000;

minikube start --docker-env HTTP_PROXY=${http_proxy} --docker-env HTTPS_PROXY=${https_proxy} --docker-env NO_PROXY=locahost,127.0.0.1,192.168.157.19

• export http_proxy命令是添加命令行代理，主要是为了让minikube可以在命令行通过proxy去下载相关文件
• –docker-env http_proxy…，设置虚拟机中docker daemon的环境变量，这里的环境变量http_proxy表示虚拟机中docker daemon使用的代理
• –docker-env no_proxy，设置虚拟机中docker daemon不使用代理的地址段