Cloud in Action: Play with Minikube/Kubernetes for container orchestration
Containers are revolutionizing the way that people build, package and deploy software, but when you deploy an application with hundreds of containers, you would have to think about challenges like service discovery, load balancing, health checks and auto-scaling etc, and this is exactly where container orchestration platforms become extremely useful and important.
There are many orchestration tools available for Docker containers, Kubernetes created by Google and based on its 15 years of experience in running containerized applications, has recently become the most popular and widely used orchestration platform, and actually formed the foundation for next generation PaaS. Today we are going to deploy a single-node Kubernetes cluster – Minikube, and get some hands-on experience with container orchestration and see what features Kubernetes can provide.
Two hosts are prepared for this test: Host1 (10.0.0.137) for Docker; Host2 (10.0.0.141) for Kubectl, Minikube and its VM(192.168.99.100).
Install and Run Minikube
Create a docker image and move to Minikube VM
Interact with Minikube
– Create a Pod
– Create a Deployment
– Deploy and expose a Service
– Exprience Autoscaling and Self-Healing
Install and Run Minikube
https://kubernetes.io/docs/getting-started-guides/minikube/
////////////////////////////// Host2
// Install virtualbox
gset@ubuntu:~$ sudo apt-get install virtualbox
// Install kubectl which provides a CLI for running commands against Kubernetes clusters.
gset@ubuntu:~$ curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl –s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
gset@ubuntu:~$ chmod +x ./kubectl
gset@ubuntu:~$ sudo mv ./kubectl /usr/local/bin/kubectl
// Install Minikube
gset@ubuntu:~$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.23.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
// Run Minikube
gset@ubuntu:~$ su root
root@ubuntu:/home/gset# minikube version
minikube version: v0.23.0
root@ubuntu:/home/gset# minikube start
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
// Check the Minikube VM is running
root@ubuntu:/home/gset# vboxmanage list runningvms
"minikube" {7cd6d16b-5deb-4023-8034-e624a567abfa}
root@ubuntu:/home/gset# ip route
default via 10.0.0.2 dev eth0 proto static metric 100
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.141 metric 100
169.254.0.0/16 dev eth0 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.99.0/24 dev vboxnet0 proto kernel scope link src 192.168.99.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
root@ubuntu:/home/gset# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0c:29:4f:57:58
inet addr:10.0.0.141 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::b478:ddf1:2b69:84f7/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:907 errors:0 dropped:0 overruns:0 frame:0
TX packets:741 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:367877 (367.8 KB) TX bytes:85635 (85.6 KB)
vboxnet0 Link encap:Ethernet HWaddr 0a:00:27:00:00:00
inet addr:192.168.99.1 Bcast:192.168.99.255 Mask:255.255.255.0
inet6 addr: fe80::800:27ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:87 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:12222 (12.2 KB)
// Get the IP address of Minikube VM
root@ubuntu:/home/gset# minikube ip
192.168.99.100
// ssh Minikube VM
root@ubuntu:/home/gset# minikube ssh
root@ubuntu:/home/gset# kubectl get all --all-namespaces
root@ubuntu:/home/gset# minikube dashboard
Create a docker image and move to Minikube VM
////////////////////////////// Host1
gset@ubuntu:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
// Write a web server application with Python and Flask to return the IP addresses of client, local and server
gset@ubuntu:~$ cd xgf
gset@ubuntu:~/xgf$ gedit start.py
from flask import Flask,request
import socket,fcntl,struct
def get_ip_address(ifname):
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
return socket.inet_ntoa(fcntl.ioctl(s.fileno(),0x8915,struct.pack('256s', ifname[:15]))[20:24])
app = Flask(__name__)
@app.route('/')
def hello_world():
client_ip = request.remote_addr+':'+ str(request.environ.get('REMOTE_PORT'))
server_ip = request.host
local_ip = get_ip_address('eth0')+':80'
result = 'Client IP:' + client_ip + ', Server IP:' + server_ip + ', Local IP:' + local_ip + '\r\nHello World, this is Richard Xue !\r\n'
print result
return result
if __name__ == '__main__':
app.run(host='0.0.0.0',port=80)
// Create a container with the image ‘flaskweb’ and run the web application
// The image ‘flaskweb’ was made last time, please refer to the web page: https://blog.51cto.com/8493144/2044680
gset@ubuntu:~$ docker run -it -v /home/gset/xgf:/xgf -p 8080:80 --name xgfweb flaskweb
root@74a2a034ec94:/#
root@74a2a034ec94:/# ls xgf
start.py
root@74a2a034ec94:/# cp xgf/start.py ./
root@74a2a034ec94:/# python start.py
* Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
Client IP:10.0.0.137:44008, Server IP:10.0.0.137:8080, Local IP:172.17.0.2:80, Hello World, this is Richard Xue !
10.0.0.137 - - [03/Dec/2017 01:39:44] "GET / HTTP/1.1" 200 –
// Test the container xgfweb in Host1
gset@ubuntu:~$ curl 10.0.0.137:8080 && echo
Client IP:10.0.0.137:44010, Server IP:10.0.0.137:8080, Local IP:172.17.0.2:80, Hello World, this is Richard Xue !
gset@ubuntu:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
74a2a034ec94 flaskweb "/bin/bash" 6 minutes ago Up 6 minutes 0.0.0.0:8080->80/tcp xgfweb
// Create the image flaskweb:1.0.0 with the container xgfweb
gset@ubuntu:~$ docker commit -m "flask web server" -a "Richard Xue" xgfweb flaskweb:1.0.0
sha256:043406ad245b615741b4523b5b3c58ada9791534725f75602fbdc7bd287a9566
gset@ubuntu:~$
gset@ubuntu:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
flaskweb 1.0.0 043406ad245b 5 seconds ago 394MB
flaskweb latest 52637ad9b6a1 3 days ago 394MB
ubuntu 14.04 d6ed29ffda6b 2 weeks ago 221MB
// Create the image flaskweb:2.2.2 which can automatically run start.py
gset@ubuntu:~$ mkdir temp
gset@ubuntu:~$ cd temp
gset@ubuntu:~/temp$
gset@ubuntu:~/temp$ gedit dockerfile
FROM flaskweb:1.0.0
CMD python start.py
gset@ubuntu:~/temp$ docker build -t flaskweb:2.2.2 .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM flaskweb:1.0.0
---> 043406ad245b
Step 2/2 : CMD python start.py
---> Running in 84ad5ac1b903
---> 28570edf26f8
Removing intermediate container 84ad5ac1b903
Successfully built 28570edf26f8
Successfully tagged flaskweb:2.2.2
gset@ubuntu:~/temp$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
flaskweb 2.2.2 28570edf26f8 36 seconds ago 394MB
flaskweb 1.0.0 043406ad245b 3 minutes ago 394MB
flaskweb latest 52637ad9b6a1 3 days ago 394MB
ubuntu 14.04 d6ed29ffda6b 2 weeks ago 221MB
// Save the image flaskweb:2.2.2 into myweb.tar
gset@ubuntu:~/temp$ docker save -o myweb.tar flaskweb:2.2.2
gset@ubuntu:~/temp$ ls
dockerfile myweb.tar
// Copy myweb.tar to Host2
gset@ubuntu:~/temp$ scp ./myweb.tar [email protected]:/home/gset
[email protected]'s password:
myweb.tar 100% 391MB 130.2MB/s 00:03
////////////////////////////// Host2
// Copy myweb.tar from Host2 to its Minikube VM
root@ubuntu:/home/gset# ls -l | grep myweb.tar
-rw------- 1 gset gset 409503232 Dec 2 17:49 myweb.tar
root@ubuntu:/home/gset#
root@ubuntu:/home/gset# scp ./myweb.tar [email protected]:/home/docker
The authenticity of host '192.168.99.100 (192.168.99.100)' can't be established.
ECDSA key fingerprint is SHA256:Y9aeWTI8hm7KyVwRihXvwZVksLhi57Fj4ini6sWuS7w.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.99.100' (ECDSA) to the list of known hosts.
[email protected]'s password: tcuser
myweb.tar 100% 391MB 39.1MB/s 00:10
////////////////////////////// Minikube VM of Host2
// Save the image flaskweb:2.2.2 in Minikube VM
$ pwd
/home/docker
$ ls -l
total 399908
-rw------- 1 docker docker 409503232 Dec 3 03:36 myweb.tar
$ docker load < myweb.tar
c08b59ef4a3d: Loading layer 229.9MB/229.9MB
c088f4b849d4: Loading layer 209.9kB/209.9kB
02323b2bcb37: Loading layer 15.36kB/15.36kB
cd514e6bdf2f: Loading layer 5.632kB/5.632kB
59482791e4b2: Loading layer 3.072kB/3.072kB
b8fa682b301b: Loading layer 179.2MB/179.2MB
a8a77ada417c: Loading layer 4.608kB/4.608kB
2fe9f759980b: Loading layer 38.4kB/38.4kB
569de8133137: Loading layer 4.608kB/4.608kB
Loaded image: flaskweb:2.2.2
$ docker images
Interact with Minikube – Create a Pod
A Pod models an application-specific "logical host" and can contain different application containers which are relatively tightly coupled. Pods are the atomic unit on the Kubernetes platform.
////////////////////////////// Host2
root@ubuntu:/home/gset# gedit myapp.yaml
apiVersion: v1
kind: Pod
metadata:
name: yxhweb
labels:
app: yxhweb
spec:
containers:
- name: yxhweb
image: flaskweb:2.2.2
ports:
- containerPort: 80
volumeMounts:
- name: mystorage
mountPath: /xgf
volumes:
- name: mystorage
hostPath:
path: /home/gset/xgf
root@ubuntu:/home/gset# kubectl create -f myapp.yaml // Create a Pod
pod "yxhweb" created
root@ubuntu:/home/gset# kubectl exec yxhweb hostname
root@ubuntu:/home/gset# kubectl exec yxhweb -- cat /etc/hosts
root@ubuntu:/home/gset# kubectl exec yxhweb -- cat /etc/resolv.conf
// Getting a shell to a Container
root@ubuntu:/home/gset# kubectl exec -it yxhweb -- /bin/bash
////////////////////////////// Pod
root@yxhweb:/# ls
bin dev hello.py lib media opt root sbin start.py tmp var
boot etc home lib64 mnt proc run srv sys usr xgf
root@yxhweb:/#
root@yxhweb:/# ping www.google.com
PING www.google.com (74.125.28.105) 56(84) bytes of data.
64 bytes from pc-in-f105.1e100.net (74.125.28.105): icmp_seq=1 ttl=61 time=36.1 ms
64 bytes from pc-in-f105.1e100.net (74.125.28.105): icmp_seq=2 ttl=61 time=35.9 ms
root@yxhweb:/# exit
exit
////////////////////////////// Host2
root@ubuntu:/home/gset# kubectl get pods
NAME READY STATUS RESTARTS AGE
yxhweb 1/1 Running 0 8m
root@ubuntu:/home/gset# kubectl describe pods yxhweb
root@ubuntu:/home/gset# kubectl delete pods yxhweb // Delete a Pod
pod "yxhweb" deleted
Interact with Minikube – Create a Deployment
The Deployment instructs Kubernetes how to create and update instances of your application. Once you've created a Deployment, the Kubernetes master schedules mentioned application instances onto individual Nodes in the cluster. Once the application instances are created, a Kubernetes Deployment Controller continuously monitors those instances. If the Node hosting an instance goes down or is deleted, the Deployment controller replaces it. This provides a self-healing mechanism to address machine failure or maintenance.
////////////////////////////// Host2
// Create a deployment with flaskweb:2.2.2
root@ubuntu:/home/gset# kubectl run yxhweb --image=flaskweb:2.2.2 --port=80 // Create a deployment
deployment "yxhweb" created
root@ubuntu:/home/gset# kubectl get pods
NAME READY STATUS RESTARTS AGE
yxhweb-5cc454d658-5vrjd 1/1 Running 0 59s
root@ubuntu:/home/gset# kubectl exec yxhweb-5cc454d658-x8kjt – hostname
root@ubuntu:/home/gset# kubectl exec yxhweb-5cc454d658-x8kjt -- ls –l
root@ubuntu:/home/gset# kubectl describe pods yxhweb
root@ubuntu:/home/gset# kubectl describe deployment yxhweb
////////////////////////////// Minikube VM of Host2
// The IP address of pod is not reachable outside the cluster
// Check the deployment yxhweb in Minikube VM of Host2
$ docker ps | grep yxhweb
862a00aae886 28570edf26f8 "/bin/sh -c 'pytho..." 11 minutes ago Up 11 minutes k8s_yxhweb_yxhweb-5cc454d658-5vrjd_default_aa106b95-d7dc-11e7-bf5e-08002750abcf_0
912cb5f202f6 gcr.io/google_containers/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_yxhweb-5cc454d658-5vrjd_default_aa106b95-d7dc-11e7-bf5e-08002750abcf_0
$ curl 172.17.0.2:80 && echo
Client IP:172.17.0.1:58568, Server IP:172.17.0.2, Local IP:172.17.0.2:80, Hello World, this is Richard Xue !
////////////////////////////// Host2
root@ubuntu:/home/gset# kubectl delete deployment yxhweb // Delete a deployment
deployment "yxhweb" deleted
Interact with Minikube – Deploy and expose a Service
A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them - sometimes called a micro-service. The set of Pods targeted by a Service is (usually) determined by a Label Selector.
////////////////////////////// Host2
root@ubuntu:/home/gset# kubectl run xgfweb --image=flaskweb:2.2.2 --port=80 // Create a deployment
root@ubuntu:/home/gset# kubectl expose deployment xgfweb --type=NodePort // Create and expose a service
root@ubuntu:/home/gset# kubectl scale --replicas=3 deployment/xgfweb
root@ubuntu:/home/gset# kubectl get pods
NAME READY STATUS RESTARTS AGE
xgfweb-666d44f966-bcmkk 1/1 Running 0 1m
xgfweb-666d44f966-j4c4m 1/1 Running 0 1m
xgfweb-666d44f966-jjzmt 1/1 Running 0 1m
root@ubuntu:/home/gset# kubectl describe service xgfweb
root@ubuntu:/home/gset# kubectl describe deployment xgfweb
// The service xgfweb (10.0.0.226:80) which contains 3 pods (172.17.0.4:80, 172.17.0.6:80, 172.17.0.7:80 ) is exposed to the outside by 192.168.99.100:31006
// Check the service xgfweb in Host2
root@ubuntu:/home/gset# minikube service xgfweb
root@ubuntu:/home/gset# minikube service xgfweb --url
http://192.168.99.100:31006
root@ubuntu:/home/gset# curl $(minikube service xgfweb --url)
root@ubuntu:/home/gset# curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:51270, Server IP:192.168.99.100:31006, Local IP:172.17.0.6:80, Hello World, this is Richard Xue !
root@ubuntu:/home/gset# curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:51272, Server IP:192.168.99.100:31006, Local IP:172.17.0.4:80, Hello World, this is Richard Xue !
root@ubuntu:/home/gset# curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:51280, Server IP:192.168.99.100:31006, Local IP:172.17.0.7:80, Hello World, this is Richard Xue !
////////////////////////////// Minikube VM of Host2
// The IP addresses of pod and service are not reachable outside the cluster
// Check the service xgfweb in Minikube VM of Host2
$ docker ps | grep xgfweb
29c31d315cc8 28570edf26f8 "/bin/sh -c 'pytho..." 12 minutes ago Up 12 minutes k8s_xgfweb_xgfweb-666d44f966-n6789_default_13209a15-d7df-11e7-bf5e-08002750abcf_0
6d96143a4817 28570edf26f8 "/bin/sh -c 'pytho..." 12 minutes ago Up 12 minutes k8s_xgfweb_xgfweb-666d44f966-8dhf6_default_1321429e-d7df-11e7-bf5e-08002750abcf_0
529010d88f57 gcr.io/google_containers/pause-amd64:3.0 "/pause" 12 minutes ago Up 12 minutes k8s_POD_xgfweb-666d44f966-n6789_default_13209a15-d7df-11e7-bf5e-08002750abcf_0
79fc5453c4ff gcr.io/google_containers/pause-amd64:3.0 "/pause" 12 minutes ago Up 12 minutes k8s_POD_xgfweb-666d44f966-8dhf6_default_1321429e-d7df-11e7-bf5e-08002750abcf_0
d161f4f916e2 28570edf26f8 "/bin/sh -c 'pytho..." 13 minutes ago Up 13 minutes k8s_xgfweb_xgfweb-666d44f966-5kf2z_default_00877ead-d7df-11e7-bf5e-08002750abcf_0
475590734b03 gcr.io/google_containers/pause-amd64:3.0 "/pause" 13 minutes ago Up 13 minutes k8s_POD_xgfweb-666d44f966-5kf2z_default_00877ead-d7df-11e7-bf5e-08002750abcf_0
// Check by the exposed IP addr and port
$ curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:58630, Server IP:192.168.99.100:31006, Local IP:172.17.0.6:80, Hello World, this is Richard Xue !
$ curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:58636, Server IP:192.168.99.100:31006, Local IP:172.17.0.4:80, Hello World, this is Richard Xue !
$ curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:58640, Server IP:192.168.99.100:31006, Local IP:172.17.0.7:80, Hello World, this is Richard Xue !
// Check by the cluster-IP addr and port
$ curl 10.0.0.226:80 && echo
Client IP:10.0.2.15:34912, Server IP:10.0.0.226, Local IP:172.17.0.4:80, Hello World, this is Richard Xue !
$ curl 10.0.0.226:80 && echo
Client IP:10.0.2.15:34914, Server IP:10.0.0.226, Local IP:172.17.0.6:80, Hello World, this is Richard Xue !
$ curl 10.0.0.226:80 && echo
Client IP:10.0.2.15:34918, Server IP:10.0.0.226, Local IP:172.17.0.7:80, Hello World, this is Richard Xue !
// by the IP addr and port of pod
$ curl 172.17.0.4:80 && echo
Client IP:172.17.0.1:60324, Server IP:172.17.0.4, Local IP:172.17.0.4:80, Hello World, this is Richard Xue !
$ curl 172.17.0.6:80 && echo
Client IP:172.17.0.1:47780, Server IP:172.17.0.6, Local IP:172.17.0.6:80, Hello World, this is Richard Xue !
$ curl 172.17.0.7:80 && echo
Client IP:172.17.0.1:35806, Server IP:172.17.0.7, Local IP:172.17.0.7:80, Hello World, this is Richard Xue !
Interact with Minikube – Exprience Autoscaling and Self-Healing
////////////////////////////// Host2
// Delete some pods and then check the service and deployment
root@ubuntu:/home/gset# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1
xgfweb NodePort 10.0.0.226
root@ubuntu:/home/gset#
root@ubuntu:/home/gset# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
xgfweb 3 3 3 3 45m
yxhweb 1 1 1 1 1h
root@ubuntu:/home/gset#
root@ubuntu:/home/gset# kubectl get pods
NAME READY STATUS RESTARTS AGE
xgfweb-666d44f966-5kf2z 1/1 Running 0 45m
xgfweb-666d44f966-8dhf6 1/1 Running 0 45m
xgfweb-666d44f966-n6789 1/1 Running 0 45m
yxhweb-5cc454d658-5vrjd 1/1 Running 0 1h
root@ubuntu:/home/gset#
root@ubuntu:/home/gset# kubectl delete pods xgfweb-666d44f966-5kf2z
pod "xgfweb-666d44f966-5kf2z" deleted
root@ubuntu:/home/gset# kubectl delete pods xgfweb-666d44f966-8dhf6
pod "xgfweb-666d44f966-8dhf6" deleted
root@ubuntu:/home/gset#
root@ubuntu:/home/gset# kubectl get pods
NAME READY STATUS RESTARTS AGE
xgfweb-666d44f966-44gqz 1/1 Running 0 1m
xgfweb-666d44f966-5kf2z 0/1 Terminating 0 46m
xgfweb-666d44f966-7kgt4 1/1 Running 0 1m
xgfweb-666d44f966-8dhf6 1/1 Terminating 0 46m
xgfweb-666d44f966-n6789 1/1 Running 0 46m
yxhweb-5cc454d658-5vrjd 1/1 Running 0 1h
root@ubuntu:/home/gset#
// Kubernetes can restart containers that fail, replaces and reschedules containers when nodes die, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve
root@ubuntu:/home/gset# kubectl get pods
NAME READY STATUS RESTARTS AGE
xgfweb-666d44f966-44gqz 1/1 Running 0 1m
xgfweb-666d44f966-7kgt4 1/1 Running 0 1m
xgfweb-666d44f966-n6789 1/1 Running 0 46m
yxhweb-5cc454d658-5vrjd 1/1 Running 0 1h
root@ubuntu:/home/gset#
root@ubuntu:/home/gset# kubectl describe service xgfweb
root@ubuntu:/home/gset# curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:51382, Server IP:192.168.99.100:31006, Local IP:172.17.0.8:80, Hello World, this is Richard Xue !
root@ubuntu:/home/gset# curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:51388, Server IP:192.168.99.100:31006, Local IP:172.17.0.8:80, Hello World, this is Richard Xue !
root@ubuntu:/home/gset# curl 192.168.99.100:31006 && echo
Client IP:172.17.0.1:51392, Server IP:192.168.99.100:31006, Local IP:172.17.0.9:80, Hello World, this is Richard Xue !
root@ubuntu:/home/gset# kubectl get all --all-namespaces
////////////////////////////// Host2
root@ubuntu:/home/gset# kubectl delete deployment xgfweb
deployment "xgfweb" deleted
root@ubuntu:/home/gset# kubectl delete service xgfweb
service "xgfweb" deleted