Centos7下Kubernetes配置使用GlusterFS
操作环境
网络拓扑
环境说明：
OS：
CentOS Linux release 7.4.1708 Client:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
Go version: go1.8.3
Git commit: ec8512b/1.12.6
Built: Mon Dec 11 16:08:42 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
Go version: go1.8.3
Git commit: ec8512b/1.12.6
Built: Mon Dec 11 16:08:42 2017
OS/Arch: linux/amd64
Kubernetes v1.5.2glusterfs 3.12.3操作步骤
Kubernetes环境已经完成部署完成，部署步骤参照 Centos7 下Kubernetes集群安装部署。
部署Glusterfs
这里有5台服务器，10.10.200.226、10.10.200.227作为GlusterFS存储服务器，10.10.200.224、
10.10.200.229、10.10.200.230调用GlusterFS，同样也需要安装Gluster。
1.分别在5台服务器上安装Gluster
[root@k8s-master ~]#yum -y install centos-release-gluster
[root@k8s-master ~]#yum -y install glusterfs glusterfs-fuse glusterfs-server
[root@k8s-master ~]#systemctl enable glusterd
[root@k8s-master ~]#systemctl start glusterd
编辑/etc/hosts文件
[root@k8s-master ~]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.200.226 glusterfs-node1
10.10.200.227 glusterfs-node2
10.10.200.224 k8s-master
10.10.200.229 k8s-node1
10.10.200.230 k8s-node2
[root@gluster-node1 ~]#gluster peer probe k8s-master
[root@gluster-node1 ~]#gluster peer probe k8s-node1
[root@gluster-node1 ~]#gluster peer probe k8s-node2
[root@gluster-node1 ~]#gluster peer probe gluster-node2[root@k8s-master yaml]# gluster peer status
Number of Peers: 4
Hostname: glusterfs-node2
Uuid: 3c5c5994-0c05-4be0-9ec3-a431f193a1f0
State: Peer in Cluster (Connected)
Hostname: glusterfs-node1
Uuid: 31ac3dff-94a3-4166-991b-bf4727113539
State: Peer in Cluster (Connected)
Hostname: 10.10.200.229
Uuid: 7c2f03fe-e2cb-412e-a91f-2baf873610ef
State: Peer in Cluster (Connected)
Other names:
10.10.200.229
Hostname: 10.10.200.230
Uuid: 314a7676-d1eb-4c58-9617-8c510c93961d
State: Peer in Cluster (Connected)
3.创建volume
这里只选择10.10.200.226/10.10.200.227来进行创建volume
[root@gluster-node1 brick1]# gluster volume create mysql-volume replica 2 glusterfs-node2:/data/brick1/mysql-volume/ glusterfs-node1:/data/brick1/mysql-volume/
[root@gluster-node1 brick1]# gluster volume start mysql-volume
volume start: mysql-volume: success
查看volume状态
[root@gluster-node1 brick1]# gluster volume status
Status of volume: mysql-volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick glusterfs-node2:/data/brick1/mysql-vo
lume 49152 0 Y 2976
Brick glusterfs-node1:/data/brick1/mysql-vo
lume 49152 0 Y 7835
Self-heal Daemon on localhost N/A N/A Y 7856
Self-heal Daemon on k8s-master N/A N/A Y 2025
Self-heal Daemon on glusterfs-node2 N/A N/A Y 2997
Task Status of Volume mysql-volume
------------------------------------------------------------------------------
There are no active volume tasks
配置Kubernetes
可以在https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/glusterfs上面找
到kuernetes配置gluster的文件glusterfs-endpoint.json/glusterfs-pod.json/glusterfs-services.json。
这里只需要glusterfs-endpoint.json，glusterfs-service.json。
修改glusterfs-endpoint.json文件如下，主要是ip地址的修改
[root@k8s-master yaml]# vi glusterfs-endpoints.json
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "glusterfs-cluster"
},
"subsets": [
{
"addresses": [
{
"ip": "10.10.200.226"
}
],
"ports": [
{
"port": 1
}
]
},
{
"addresses": [
{
"ip": "10.10.200.227"
}
],
"ports": [
{
"port": 1
}
]
}
]
}
[root@k8s-master yaml]# vi glusterfs-service.json
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "glusterfs-cluster"
},
"spec": {
"ports": [
{"port": 1}
]
}
}
[root@k8s-master yaml]# kubectl apply -f glusterfs-endpoints.json
endpoints "glusterfs-cluster" created
[root@k8s-master yaml]# kubectl apply -f glusterfs-service.json
service "glusterfs-cluster" created
[root@k8s-master yaml]# kubectl get ep
NAME ENDPOINTS AGE
glusterfs-cluster 10.10.200.226:1,10.10.200.227:1 31s
kubernetes 10.10.200.224:6443 6d
[root@k8s-master yaml]# kubectl get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
glusterfs-cluster 172.17.62.135 1/TCP 33s
kubernetes 10.254.0.1 443/TCP 6d
[root@k8s-master yaml]# vi glusterfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: gluster-dev-volume1
labels:
name: mysql1
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
glusterfs:
endpoints: "glusterfs-cluster"
path: "mysql-volume"
readOnly: false
[root@k8s-master yaml]# vi glusterfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: glusterfs-mysql1
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
selector:
matchLabels:
name: "mysql1"
[root@k8s-master yaml]# kubectl apply -f glusterfs-pv.yaml
persistentvolume "gluster-dev-volume1" created
[root@k8s-master yaml]# kubectl apply -f glusterfs-pvc.yaml
persistentvolumeclaim "glusterfs-mysql1" created
[root@k8s-master yaml]# kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
gluster-dev-volume1 10Gi RWX Retain Bound default/glusterfs-mysql1 8s
[root@k8s-master yaml]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
glusterfs-mysql1 Bound gluster-dev-volume1 10Gi RWX 6s
以上就完成了kubernetes配置glusterfs的全过程，下面我们通过运行mysql测试一下，编写mysql deployment yaml如下
[root@k8s-master yaml]# vi mysql-deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
name: mysql
spec:
type: NodePort
ports:
- name: mysqlport
port: 3306
nodePort: 32006
selector:
name: mysql
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mysql
spec:
replicas: 1
template:
metadata:
labels:
name: mysql
spec:
containers:
- name: mysqlcontainer
image: docker.io/mysql
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_ROOT_PASSWORD
value: root123456
ports:
- containerPort : 3306
volumeMounts:
- name: gluster-mysql-data
mountPath: "/var/lib/mysql"
volumes:
- name: gluster-mysql-data
persistentVolumeClaim:
claimName: glusterfs-mysql1
启动该文件
[root@k8s-master yaml]# kubectl apply -f mysql-deployment.yaml
service "mysql" created
deployment "mysql" configured查看状态
[root@k8s-master yaml]# kubectl get service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
glusterfs-cluster 172.17.62.135 1/TCP 13m
kubernetes 10.254.0.1 443/TCP 6d
mysql 172.17.231.221 3306:32006/TCP 1m
[root@k8s-master yaml]# kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-3736381796-5j11t 1/1 Running 0 1m
nginx-controller-1hxkz 1/1 Running 3 3d
nginx-controller-3xcl8 1/1 Running 3 3d
[root@k8s-master yaml]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
mysql-3736381796-5j11t 1/1 Running 0 1m 172.17.99.3 10.10.200.230
nginx-controller-1hxkz 1/1 Running 3 3d 172.17.99.2 10.10.200.230
nginx-controller-3xcl8 1/1 Running 3 3d 172.17.17.2 10.10.200.229 [root@k8s-node2 ~]# mysql -h 10.10.200.230 -P 32006 -uroot -proot123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.20 MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]>