k8s集群终于跑正常了
redis和docker这两个词语会自动被修改为首字母大写并链接到知识库，所以在这里先写一遍，后面就不会被改写了。
0、具体操作见 （vmware中搭建k8s），virtulbox也是相同的流程。
1、学习k8s，读的是这本书《KUBERNETES权威指南 从DOCKET到KURBERNETES实践全接触.pdf》
2、这边书刚开始讲的是在单机上搭建一个k8s的hello world，用的是centos7.
于是我也在virtulbox中安装了centos7，并成功运行hello world。
3、然后，开始尝试集群了。在网上找了个教程，比较简单，很快就成功了，kubectl get nodes能看到各node了。
但是，应用跑的不正常，有的节点可以访问，有的节点不可以。而且从不同节点访问，查到的数据不相同，似乎是多个独立的系统。怀疑是iptables中cluster ip的规则有问题。
接着，集群坏了，k8s的基础服务都启动不了。怀疑是不是因为我创建这些虚机时，用的是链接式拷贝，是不是原始的虚机安装了其它软件，导致k8s集群启动不了。
4、删掉重来，用完全拷贝的方式建立虚机。
一切正常，但是在启动redis-master-controller.yaml时，docker中无法建立容器，
用kubectl describe pod redis-master命令排查，发现下拉不了镜像。其实这些镜像已经存在于docker中了（搭建单机k8s时，自动pull的）。只是镜像名字前面多加了docker.io/ 我修改了redis-master-controller.yaml中镜像的名字，仍然pull失败。奇怪。
5、今天定位了网络问题（见virtualbox虚拟机无法上网），网络搞通后，hello world终于正常运行了。
之前的一些疑问，有答案了
1、随便访问哪个node的ip（比如http://192.168.56.251:30001/,http://192.168.56.252:30001/）(http://192.168.56.250:30001/是不行的)，都可以访问到服务的（proxy自动转的）
2、在内部时，访问真实端口也是可以的。
[root@centm ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend 10.254.218.57
kubernetes 10.254.0.1
redis-master 10.254.142.174
redis-slave 10.254.201.123
curl 10.254.218.57:80 可以通。
ping 10.254.218.57 不通
3、node中多了一个127.0.0.1 ，不知道为什么
[root@centm ~]# kubectl get nodes
NAME STATUS AGE
127.0.0.1 NotReady 15d
cents1 Ready 1d
cents2 Ready 1d
[root@centm ~]# ps -ef|grep kube
kube 578 1 0 Jan20 ? 00:15:55 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080
kube 588 1 0 Jan20 ? 00:01:09 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080
kube 2079 1 0 Jan20 ? 00:08:11 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota
4、clusterip是一个虚ip，实际是iptables中的几个转发规则。
[root@cents2 ~]# ip a
1: lo:
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3:
link/ether 08:00:27:58:5d:6e brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 82058sec preferred_lft 82058sec
inet6 fe80::b171:84d0:5173:de63/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8:
link/ether 08:00:27:7a:24:14 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.252/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe7a:2414/64 scope link
valid_lft forever preferred_lft forever
4: flannel.1:
link/ether fa:5a:c7:c5:aa:e5 brd ff:ff:ff:ff:ff:ff
inet 172.16.80.0/16 scope global flannel.1
valid_lft forever preferred_lft forever
5: docker0:
link/ether 02:42:72:2f:1f:ae brd ff:ff:ff:ff:ff:ff
inet 172.16.80.1/24 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:72ff:fe2f:1fae/64 scope link
valid_lft forever preferred_lft forever
7: vethc56c1d4@if6:
link/ether 92:c8:3d:3f:b9:49 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::90c8:3dff:fe3f:b949/64 scope link
valid_lft forever preferred_lft forever
9: vethf961994@if8:
link/ether d6:be:4b:6e:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::d4be:4bff:fe6e:2681/64 scope link
valid_lft forever preferred_lft forever
11: vethe4cd28e@if10:
link/ether ee:55:55:df:4e:50 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::ec55:55ff:fedf:4e50/64 scope link
valid_lft forever preferred_lft forever
[root@cents2 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*filter
:INPUT ACCEPT [27:4324]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [25:2821]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-SERVICES - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Sun Jan 22 00:41:01 2017
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [2:119]
:POSTROUTING ACCEPT [2:119]
:DOCKER - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-63GTHXGNEQIFF6GY - [0:0]
:KUBE-SEP-77PLGVXVTAKNHL2K - [0:0]
:KUBE-SEP-7R2ESD4YYXMXFEFZ - [0:0]
:KUBE-SEP-GIMIRAR4ZAKGMA2Q - [0:0]
:KUBE-SEP-LYGBYJFMWSAWPLXU - [0:0]
:KUBE-SEP-Y7WMR7EBCL4N3QJX - [0:0]
:KUBE-SEP-ZDWRYP3AMCRYOGNR - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]
:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0]
:KUBE-SVC-GYQQTB6TY565JPRW - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.16.80.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-63GTHXGNEQIFF6GY -s 172.16.62.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-63GTHXGNEQIFF6GY -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.62.4:80
-A KUBE-SEP-77PLGVXVTAKNHL2K -s 172.16.80.3/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-77PLGVXVTAKNHL2K -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.3:80
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -s 172.16.80.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.80.2:6379
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -s 192.168.56.250/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.56.250:6443
-A KUBE-SEP-LYGBYJFMWSAWPLXU -s 172.16.62.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LYGBYJFMWSAWPLXU -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.62.3:6379
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -s 172.16.62.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 172.16.62.2:6379
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -s 172.16.80.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.4:80
-A KUBE-SERVICES -d 10.254.218.57/32 -p tcp -m comment --comment "default/frontend: cluster IP" -m tcp --dport 80 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.254.142.174/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
-A KUBE-SERVICES -d 10.254.201.123/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O33
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Y7WMR7EBCL4N3QJX
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LYGBYJFMWSAWPLXU
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-7R2ESD4YYXMXFEFZ
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-63GTHXGNEQIFF6GY
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-77PLGVXVTAKNHL2K
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -j KUBE-SEP-ZDWRYP3AMCRYOGNR
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
COMMIT
# Completed on Sun Jan 22 00:41:01 2017
----------------------------------------------------
尝试了本地卷
[root@centm ~]# cat redis-master-controller_with_volume.yaml
apiVersion: v1
kind: ReplicationController
metadata:
labels:edis-master
name: redis-master
spec:
replicas: 1
selector:
name: redis-master
template:
metadata:
labels:
name: redis-master
spec:
volumes:
- name: "gf-dir1"
hostPath:
path: "/tmp"
containers:
- name: master
ports:: docker.io/kubeguide/redis-master:latest
- containerPort : 6379
volumeMounts:
- name: "gf-dir1"
mountPath: "/gf1"