2020-10-13
在k8s中搭建es集群的思路如下图：
k8s搭建es服务集群结构示意图
service1、service2、service3用于集群内服务互通，service用于集群服务对外提供访问。
方式一：使用自定义存储卷
statefulset的部署文件 es-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-sts-01
spec:
selector:
matchLabels:
app: es
es-cluster: es01
serviceName: "elasticsearch-svc"
replicas: 1
template:
metadata:
labels:
app: es
es-cluster: es01
spec:
terminationGracePeriodSeconds: 10
volumes:
- name: data-storage
persistentVolumeClaim:
claimName: es-nfs-pvc-1 #使用第一块存储卷
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
ports:
- containerPort: 9200
name: es-cli
- containerPort: 9300
name: es-inner
env:
- name: cluster.name
value: es-luster
- name: node.name
value: es01
- name: cluster.initial_master_nodes #用于第一次启动时引导集群的创建，集群组建成功后就可以不需要了
value: es01,es02,es03 #对应各个es的node.name
- name: discovery.seed_hosts
value: es-svc-inner-02,es-svc-inner-02
- name: http.cors.enabled
value: "true"
- name: http.cors.allow-origin
value: "*"
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: data-storage
---
kind: StatefulSet
metadata:
name: es-sts-02
spec:
selector:
matchLabels:
app: es
es-cluster: es02
serviceName: "elasticsearch-svc"
replicas: 1
template:
metadata:
labels:
app: es
es-cluster: es02
spec:
terminationGracePeriodSeconds: 10
volumes:
- name: data-storage
persistentVolumeClaim:
claimName: es-nfs-pvc-2 #使用第二块存储卷
containers:
- name: es-02
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
ports:
- containerPort: 9200
name: es-cli
- containerPort: 9300
name: es-inner
env:
- name: cluster.name
value: es-cluster
- name: node.name
value: es02
- name: cluster.initial_master_nodes # 最好每个都要配置这个列表，而且要与es01配置的一致
value: es01,es02,es03
- name: discovery.seed_hosts
value: es-svc-inner-01,es-svc-inner-03
- name: http.cors.enabled
value: "true"
- name: http.cors.allow-origin
value: "*"
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: data-storage
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-sts-03
spec:
selector:
matchLabels:
app: es
es-cluster: es03
serviceName: "elasticsearch-svc"
replicas: 1
template:
metadata:
labels:
app: es
es-cluster: es03
spec:
terminationGracePeriodSeconds: 10
volumes:
- name: data-storage
persistentVolumeClaim:
claimName: es-nfs-pvc-3 #使用第三块存储卷
containers:
- name: es-03
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
ports:
- containerPort: 9200
name: es-cli
- containerPort: 9300
name: es-iner
env:
- name: cluster.name
value: es-cluster
- name: node.name
value: es03
- name: cluster.initial_master_nodes # 最好每个都要配置这个列表，而且要与es01配置的一致
- name: discovery.seed_hosts
value: es-svc-inner-01,es-svc-inner-02
- name: http.cors.enabled
value: "true"
- name: http.cors.allow-origin
value: "*"
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: data-storage
存储卷的部署：persistant-voulmes.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: es-nfs-pv-1
labels:
pv: es-nfs-pv-1
spec:
capacity:
storage: 1Gi
#volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: nfs
nfs:
path: /data/nfs/es
server: k8snode-01
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: es-nfs-pv-2
labels:
pv: es-nfs-pv-2
spec:
capacity:
storage: 1Gi
#volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: nfs
nfs:
path: /data/nfs/es
server: k8snode-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: es-nfs-pv-3
labels:
pv: es-nfs-pv-3
spec:
capacity:
storage: 1Gi
#volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Recycle
storageClassName: nfs
nfs:
path: /data/nfs/es
server: k8snode-03
##claim部署
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: es-nfs-pvc-1
spec:
accessModes:
- ReadWriteOnce
#volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: nfs
selector:
matchLabels:
pv: "es-nfs-pv-1"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: es-nfs-pvc-2
spec:
accessModes:
- ReadWriteOnce
#volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: nfs
selector:
matchLabels:
pv: "es-nfs-pv-2"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: es-nfs-pvc-3
spec:
accessModes:
- ReadWriteOnce
#volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: nfs
selector:
matchLabels:
pv: "es-nfs-pv-3"
注：存储卷的部署不成功的问题，提供下思路：需要对应的服务器上启动nfs服务和相关路径的读写权限。
Service的部署：es-service.yaml
##用于对外提供服务的Service
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-svc
labels:
app: elasticsearch-svc
spec:
type: NodePort
ports:
- port: 9200
name: es-cli
nodePort: 30001
selector:
app: es
---
##用于集群内部服务间互通的Service
apiVersion: v1
kind: Service
metadata:
name: es-svc-inner-01
labels:
app: es-svc-inner-01
spec:
type: ClusterIP
ports:
- port: 9300
name: es-port
selector:
es-cluster: es01
---
apiVersion: v1
kind: Service
metadata:
name: es-svc-inner-02
labels:
app: es-svc-inner-02
spec:
type: ClusterIP
ports:
- port: 9300
name: es-port
selector:
es-cluster: es02
---
apiVersion: v1
kind: Service
metadata:
name: es-svc-inner-03
labels:
app: es-svc-inner-03
spec:
type: ClusterIP
ports:
- port: 9300
name: es-port
selector:
es-cluster: es03
方式二：使用阿里云存储
使用阿里云存储就不需要上面方试一那样麻创建PV和PVC了。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-sts-01
namespace: elasticsearch
spec:
selector:
matchLabels:
app: es
es-cluster: es01
serviceName: "elasticsearch-svc"
replicas: 1 # by default is 1
template:
metadata:
labels:
app: es
es-cluster: es01
spec:
securityContext:
fsGroup: 1000 #使用这个获得读写权限
terminationGracePeriodSeconds: 10
volumes:
- name: aliyun-nas-01
flexVolume:
driver: "alicloud/nas"
options:
server: "xxxxxx.xxxxxx.nas.aliyuncs.com"
path: "/elasticsearch/test/node01"
vers: "3"
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
ports:
- containerPort: 9200
name: es-cli
- containerPort: 9300
name: es-iner
env:
- name: cluster.name
value: es-cluster
- name: node.name
value: es01
- name: cluster.initial_master_nodes
value: es01,es02,es03
- name: discovery.seed_hosts
value: es-svc-inner-02,es-svc-inner-02
- name: http.cors.enabled
value: "true"
- name: http.cors.allow-origin
value: "*"
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: aliyun-nas-01
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-sts-02
namespace: elasticsearch
spec:
selector:
matchLabels:
app: es
es-cluster: es02
serviceName: "elasticsearch-svc"
replicas: 1
template:
metadata:
labels:
app: es
es-cluster: es02
spec:
securityContext:
fsGroup: 1000
terminationGracePeriodSeconds: 10
volumes:
- name: aliyun-nas-02
flexVolume:
driver: "alicloud/nas"
options:
server: "xxxxxx.xxxxxx.nas.aliyuncs.com"
path: "/elasticsearch/test/node02"
vers: "3"
containers:
- name: es-02
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
ports:
- containerPort: 9200
name: es-cli
- containerPort: 9300
name: es-iner
env:
- name: cluster.name
value: es-cluster
- name: node.name
value: es02
- name: cluster.initial_master_nodes
value: es01,es02,es03
- name: discovery.seed_hosts
value: es-svc-inner-01,es-svc-inner-03
- name: http.cors.enabled
value: "true"
- name: http.cors.allow-origin
value: "*"
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: aliyun-nas-02
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-sts-03
namespace: elasticsearch
spec:
selector:
matchLabels:
app: es
es-cluster: es03
serviceName: "elasticsearch-svc"
replicas: 1
template:
metadata:
labels:
app: es
es-cluster: es03
spec:
securityContext:
fsGroup: 1000
terminationGracePeriodSeconds: 10
volumes:
- name: aliyun-nas-03
flexVolume:
driver: "alicloud/nas"
options:
server: "xxxxxx.xxxxxx.nas.aliyuncs.com"
path: "/elasticsearch/test/node03"
vers: "3"
containers:
- name: es-03
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.2
ports:
- containerPort: 9200
name: es-cli
- containerPort: 9300
name: es-iner
env:
- name: cluster.name
value: es-cluster
- name: node.name
value: es03
- name: cluster.initial_master_nodes
value: es01,es02,es03
- name: discovery.seed_hosts
value: es-svc-inner-01,es-svc-inner-02
- name: http.cors.enabled
value: "true"
- name: http.cors.allow-origin
value: "*"
volumeMounts:
- mountPath: "/usr/share/elasticsearch/data"
name: aliyun-nas-03
经验证，需要搭建三个es实例才能保证其中一个挂了之后集群还能正常运行，官方也推荐最好配置三个以上的mater node，以防数据的丢失。
ps: Elasticsearch默认单机情况下会自动引导组建集群，但是只要配置了
discovery.seed_providers
discovery.seed_hosts
cluster.initial_master_nodes
三个中的一个Elasticsearch就不会自动引导集群了。