K8s (6) kafka部署

整个安装步骤如下。

$ git clone https://github.com/Yolean/kubernetes-kafka
$ cd kubernetes-kafka
$ kubectl apply -f 00-namespace.yml    # 创建namespace ‘kafka'
$ kubectl apply -f 01-test-namespace.yml    # 创建namespace ‘test-kafka'

# 创建storage class
$ kubectl apply -f ./configure/docker-storageclass-broker.yml 
$ kubectl apply -f ./configure/docker-storageclass-zookeeper.yml

# 创建local persistent volume， 5个先
$ sudo mkdir /var/local/kakfa
$ sudo chmod a+rwx /var/local/kafka   #不一定是最安全的方式，但是DEV/test没问题

apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka0
  labels:
    type: local
spec:
  storageClassName: kafka-zookeeper
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  hostPath:
    # this is the local path
    path: "/val/local/kafka"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kafka1
  labels:
    type: local
spec:
  storageClassName: kafka-zookeeper
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  hostPath:
    # this is the local path
    path: "/val/local/kafka"
。。。

$kubectl apply -f create-local.yaml

$ kubectl apply -f ./zookeeper  # apply all the yaml files in the directory

$ kubectl describe pod zoo-0 --namespace=kafka

# pods zoo-0 and zoo-1 拿不到storage，所以Hang住了。一番检查之后，发现它们request了一个不存在的storage class：kafka-zookeeper-regional
# 在github上报告了。

# 决定在51zoo.yml里面把storage class改了。因为这个regional是为云准备的，本地没啥区别。
$ kubectl apply -f 51zoo.yml 
The StatefulSet "zoo" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

# 只好删除
$ kubectl delete statefulsets zoo --namespace=kafka

# 重新创建statefulsets还是不行，不能bind storage。why？？？

# 只好创建新的storage class：kafka-zookeeper-regional

# 自动就bind上了。

# 照样建立storage volume for kafka，10Gi
$ kubectl apply -f ./kafka

# POD还是失败
$ kubectl -n kafka logs kafka-0 -c init-config
+ cp /etc/kafka-configmap/log4j.properties /etc/kafka/
+ KAFKA_BROKER_ID=0
+ SEDS=("s/#init#broker.id=#init#/broker.id=$KAFKA_BROKER_ID/")
+ LABELS=kafka-broker-id=0
+ ANNOTATIONS=
+ hash kubectl
++ kubectl get node rep1 '-o=go-template={{index .metadata.labels "failure-domain.beta.kubernetes.io/zone"}}'
Error from server (Forbidden): nodes "node1" is forbidden: User "system:serviceaccount:kafka:default" cannot get resource "nodes" in API group "" at the cluster scope
+ ZONE=

#原来需要把RBAC来设置好
$ kubectl apply -f ./rbac-namespace-default/

#这样kafka就启动完成了 

安装成功，但是Storage不对，暂时for test。

测试需要Launch另外一个pod

$ kubectl run temp-kafka --image solsson/kafka --rm -ti --command -- bash

$ kubectl exec  -ti /bin/bash

# ./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --create --partitions 1 --replication-factor 1 --topic test 
# ./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --list
# ./bin/kafka-console-producer.sh --broker-list broker.kafka:9092 --topic test
# ./bin/kafka-console-consumer.sh --bootstrap-server broker.kafka:9092 --topic test --from-beginning