Cassandra集群化Docker完整部署

               Thingsboard下Cassandra集群化Docker部署

                                                                                 徐景周

一、环境

• CentOS
• 安装Nano

apt-get update

apt-get install nano

 

• Oracle JDK：1.8及以上

查找Java安装路径

which java

ls /usr/bin/java (上一步结果路径)

ls -lrt /etc/alternatives/java (上一步结果路径)

ls /usr/lib/jvm/java-8-openjdk-amd64 (最终路径)

 

• 关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

 

二、单机部署(Docker版)

  1. 停止宿主机Cassandra(如有必要)

sudo service cassandra stop

sudo chkconfig cassandra off

sudo service cassandra status

 

  2. 安装Docker

• 安装

         请浏览【参考文献 1】。

• 修改Docker默认目录

sudo service docker stop

mv /var/lib/docker /home/docker

ln -s /home/docker /var/lib/docker


sudo yum install -y nano

sudo nano /usr/lib/systemd/system/docker.service

文件修改项如下：

ExecStart=/usr/bin/dockerd --graph /home/docker -H fd:// --containerd=/run/containerd/containerd.soc


sudo systemctl daemon-reload

sudo service docker restart

sudo service docker status

docker info | grep Dir

 

• Docker默认磁盘空间

df -h

docker info

docker exec -it tb-cassandra df -h

 

• 自启动

sudo systemctl enable docker

 

3. Docker中安装Cassandra

1. 操作指令

• 安装

docker stop tb-cassandra
docker rm tb-cassandra

docker run --name tb-cassandra -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 -d cassandra:3.0

• 启动

docker start tb-cassandra

• 检查

docker ps -a

• 自启动

docker update --restart=always 容器ID

 

 2. 实现细节

    请浏览【参考文献 2】和【参考文献 3】。

 

4. 修改Cassandra配置

 1. 操作指令

• 进入容器bash

docker exec -it  tb-cassandra bash

• 启动cql环境

cqlsh

• cql环境下，修改族名

update system.local set cluster_name = 'Thingsboard Cluster' where key='local';

• cql环境下，查看族名

select cluster_name from system.local;

• cql环境下，退出

exit

• 刷新配置

nodetool flush system

• 修改cassandra.yaml配置文件

nano /etc/cassandra/cassandra.yaml

或者采用文件替换方式（先退出docker bash）

docker exec -it tb-cassandra bash

ls etc/cassandra/cassandra.yaml

exit;


docker ps

docker cp cassandra.yaml 容器id:/etc/cassandra/cassandra.yaml


docker exec -it tb-cassandra bash

cat /etc/cassandra/cassandra.yaml

Cassandra.yaml配置文件修改

 创建目录及修改属性

ls /var/lib/cassandra/

mkdir /var/lib/cassandra/data

mkdir /var/lib/cassandra/commitlog

mkdir /var/lib/cassandra/saved_caches

mkdir /var/lib/cassandra/hints

chown 777 /var/lib/cassandra/data

chown 777 /var/lib/cassandra/commitlog

chown 777 /var/lib/cassandra/saved_caches

chown 777 /var/lib/cassandra/hints

 修改文件下面的选项

cluster_name: 'Thingsboard Cluster’

hints_directory: /var/lib/cassandra/hints

data_file_directories:

     - /var/lib/cassandra/data

commitlog_directory: /var/lib/cassandra/commitlog

saved_caches_directory: /var/lib/cassandra/saved_caches

seeds: "127.0.0.1”

listen_address: localhost

# broadcast_address: 1.2.3.4

start_rpc: true

rpc_address: 0.0.0.0

broadcast_rpc_address: 1.2.3.4

• 退出docker bash

exit;

• 安装Thingsboard对应数据库及数据表（顺序：schema-ts.cql -> schema-entities.cql）

docker ps

docker cp schema-ts.cql 容器id:/root/schema-ts.cql

docker cp schema-entities.cql 容器id:/root/schema-entities.cql


docker exec -it tb-cassandra bash

ls /root

cqlsh -f /root/schema-entities.cql

cqlsh -f /root/schema-ts.cql

exit;

• 重启Cassandra

docker restart tb-cassandra

• 查看Cassandra是否启动正常

docker logs tb-cassandra

实现细节

    请浏览【参考文献4】和【参考文献 5】。

 

5. 修改Thingsboard配置

• 修改文件thingsboard.conf（/usr/share/thingsboard/conf）

# DB Configuration
export DATABASE_ENTITIES_TYPE=sql

export DATABASE_TS_TYPE=cassandra

export SPRING_JPA_DATABASE_PLATFORM=org.hibernate.dialect.PostgreSQLDialect

export SPRING_DRIVER_CLASS_NAME=org.postgresql.Driver

export SPRING_DATASOURCE_URL=jdbc:postgresql://192.168.1.131:5432/thingsboard

export SPRING_DATASOURCE_USERNAME=postgres

export SPRING_DATASOURCE_PASSWORD=123456


export CASSANDRA_CLUSTER_NAME=Thingsboard Cluster

export CASSANDRA_KEYSPACE_NAME=thingsboard

export CASSANDRA_URL=192.168.1.131:9042

export CASSANDRA_USE_CREDENTIALS=false

export CASSANDRA_USERNAME=

export CASSANDRA_PASSWORD=

• 重启Thingsboard

sudo service thingsboard restart

nano /var/log/thingsboard/thingsboard.log

• 重启Mosquitto(可选)

ps -aux | grep mosquitto

kill -9 PID

mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d

• 重启TB-Gateway

sudo service tb-gateway restart

nano /var/log/tb-gateway/tb-gateway.log

 

6. Cassandra调优

  实现细节

  请浏览【参考文献7】和【参考文献8】。

 

7. 测试

• 打开Thingsboard后，运行测试脚本。请浏览【参考文献20】。

node /opt/mqtt/mqtt_gateway_mqtt_sz_new.js

• 查看tb-cassandra容器中Cassandra数据

docker exec -it tb-cassandra bash

cqlsh

select count(1) from thingsboard.ts_kv_latest_cf;

exit

exit;

 

8. Cassandra查询工具

DataStax Studio

Cassandra Manager

Razorsql

 

9. 配置示例

192.168.1.131（Cassandra + Postgres）
192.168.1.59（Thingsboard + TB-Gateway）

 

三、单机部署(非Docker版)

如果不希望基于Docker，直接在机器上安装Cassandra的情况。

1. 停止Docker版Cassandra(如有必要)

sudo service docker stop

sudo systemctl disable docker

docker info


# 杀掉Cassandra端口占用程序
netstat -tunlp
kill PID

2. 实现细节

    请浏览【参考文献6】和“单机部署（Docker版）-修改Cassandra配置”章节。

 

四、集群部署(Docker版)

1. 停止宿主机Cassandra(如有必要)

sudo service cassandra stop

sudo chkconfig cassandra off

sudo service cassandra status

2. 安装Docker

    请参考单机部署中对应章节。

3. 安装Ping和IFConfig

apt-get update

apt-get install inetutils-ping

apt-get install net-tools

4. 修改主机名

# 查看主机名
Hostnamectl


# 133机器
hostnamectl set-hostname 133


# 134机器
hostnamectl set-hostname 134


# 135机器
hostnamectl set-hostname 135


# 136机器
hostnamectl set-hostname 136

5. 安装IPTables

sudo yum install -y iptables-services

systemctl enable iptables.service

/usr/libexec/iptables/iptables.init save


systemctl daemon-reload

sudo service docker restart

6. Docker之间跨主机通讯(SWARM方式)

    实现细节请浏览【参考文献9】。

例如：将133做为Manager机，其它做为Work机。
①　Manager上创建指令(133机器)
docker swarm init --advertise-addr=192.168.1.133

②　Work上加入指令(136机器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.136 192.168.1.133:2377

③　Work上加入指令(135机器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.135 192.168.1.133:2377

④　Work上加入指令(135机器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.134 192.168.1.133:2377

⑤　Manager上查看集群状态(133机器)
docker node ls

⑥　Manager上创建overlay网络(133机器)
docker network create -d overlay --attachable cassandra1-net
docker service update --network-add cassandra1-net --network-rm cassandranet my-cassandra

docker network create -d overlay --attachable cassandranet

docker network create -d overlay --subnet=172.172.0.0/24 --attachable  cassandranet

# Manager机器和其它Work机器上都可查看
docker network ls

⑦　Manager上创建swarm服务(133机器)
docker service rm my-cassandra
docker network rm cassandranet
docker service create --name my-cassandra --network cassandra1-net --replicas 1 --publish published=8080,target=80 cassandra

# 黄色标记为机器数
docker service create --name my-cassandra --network cassandranet --replicas 5 --publish published=8080,target=80 cassandra

docker service ls

⑧　Manager上查看新建服务是否连接到新建overlay网络(133机器)
docker network inspect cassandranet 

docker network inspect cassandra1-net

⑨　查看服务
docker ps -a

⑩　查看节点情况
docker node ls
docker service ps my-cassandra

7. Docker中安装Cassandra

• 操作指令

# 安装
docker stop tb-cassandra

docker rm tb-cassandra


# 133机器
sudo docker run -d --name tb-cassandra --network=cassandranet --ip 172.172.0.133 --restart always -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 cassandra:3.0


# 134机器
sudo docker run -d --network=cassandranet --ip 172.172.0.134 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0


# 135机器
sudo docker run -d --network=cassandranet --ip 172.172.0.135 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0


# 136机器
sudo docker run -d --network=cassandranet --ip 172.172.0.136 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0


# 启动
docker start tb-cassandra


# 检查
docker ps -a


# 自启动
docker update --restart=always 容器ID

8. 修改Cassandra配置

①　按单机部署相关章节，执行前6步。

②　在修改后cassandra.yaml 基础上，再修改下面几项内容
1)	# 选取一半数量的节点物理IP，做为Seeds
例如：
- seeds: "192.168.1.133,192.168.1.134"

2)	# 替换成各节点自己的物理IP地址
例如；
# 容器IP(docker inspect tb-cassandra)
listen_address: 172.172.0.133
# hostIP
broadcast_address: 192.168.1.133

3)	# 同步策略
endpoint_snitch: GossipingPropertyFileSnitch 

③　修改文件cassandra-rackdc.properties
1)	dc=datacenter1
2)	rack=rack1

④　修改文件cassandra-env.sh
1)	#JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=192.168.1.134"
2)	LOCAL_JMX=no
3)	JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
4)	JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"

⑤　分别替换各个容器节点cassandra中对应的文件：cassandra.yaml、cassandra-rackdc.properties、cassandra-env.sh文件

例如：
docker exec -it tb-cassandra bash
ls etc/cassandra/cassandra.yaml
exit;

docker ps

# 133机器
# cassandra.yaml 
docker cp /root/cassandra_133.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp  /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties

# cassandra-env.sh
docker cp /root/cassandra-env-133.sh  容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access  容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password  容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
ls /etc/cassandra/
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access

more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties

# 134机器
# cassandra.yaml 
docker cp  /root/cassandra_134.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp  /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties

# cassandra-env.sh
docker cp /root/cassandra-env-134.sh  容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access  容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password  容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access

more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties

# 135机器
# cassandra.yaml 
docker cp  /root/cassandra_135.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp  /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties

# cassandra-env.sh
docker cp /root/cassandra-env-135.sh  容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access  容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password  容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access

more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties

# 136机器
# cassandra.yaml 
docker cp  /root/cassandra_136.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp  /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties

# cassandra-env.sh
docker cp /root/cassandra-env-136.sh  容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access  容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password  容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access

more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties

# 退出
exit;

⑥　安装Thingsboard数据库及数据表（顺序：schema-entities.cql -> schema-ts.cql）
docker ps
docker cp /root/schema-entities.cql 容器id:/root/schema-entities.cql
docker cp /root/schema-ts.cql 容器id:/root/schema-ts.cql

docker exec -it tb-cassandra bash
ls /root
cqlsh -f /root/schema-entities.cql
cqlsh -f /root/schema-ts.cql
exit;

⑦　重启Cassandra
docker stop tb-cassandra
docker start tb-cassandra

⑧　查看Cassandra是否启动正常
docker logs tb-cassandra

 

9. 查看集群情况

各节点的cassandra.yaml文件依次替换后，分别启动各个Cassandra节点，查看集群情况。

例如：

# 任一机器
docker exec -it tb-cassandra bash

nodetool -h 192.168.1.133 -u cassandra -pw cassandra status

nodetool -h 192.168.1.134 -u cassandra -pw cassandra status

nodetool -h 192.168.1.135 -u cassandra -pw cassandra status

nodetool -h 192.168.1.136 -u cassandra -pw cassandra status


nodetool -h 192.168.1.133 -u cassandra -pw cassandra ring

nodetool -h 192.168.1.134 -u cassandra -pw cassandra ring

nodetool -h 192.168.1.135 -u cassandra -pw cassandra ring

nodetool -h 192.168.1.136 -u cassandra -pw cassandra ring


cqlsh 192.168.1.133 -u cassandra -p cassandra

ping 172.172.0.134
ping 172.172.0.135
ping 172.172.0.136

exit;


# Manager(133机器
docker node ls

docker network inspect cassandranet

10. 修改Thingsboard配置

	在单机部署修改后thingsboard.conf的基础上，修改下面内容（/usr/share/thingsboard/conf）：
1)	Export CASSANDRA_URL=192.168.1.133:9042,192.168.1.134:9042,192.168.1.135:9042,192.168.1.136:9042
2)	export CASSANDRA_USE_CREDENTIALS=true
3)	export CASSANDRA_USERNAME=cassandra
4)	export CASSANDRA_PASSWORD=cassandra

	重启Thingsboard
sudo service thingsboard restart
nano /var/log/thingsboard/thingsboard.log

	重启Mosquitto(可选，MQTT)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d

	重启TB-Gateway
sudo service tb-gateway restart
nano /var/log/tb-gateway/tb-gateway.log

 

11. 集群工具

Nodetool

JConsole


例如：
nodetool status

nodetool info

nodetool proxyhistograms

12. 修复无法启动的容器

Docker容器中，由于个别文档(例如：cassandra-env.sh)导致cassandra无法启动。


例如：
cd /home/docker

find ./ -name 'cassandra-env.sh'

对搜索到的文档进行比对修改，再重启容器即可！

13. 配置示例

192.168.1.133（Cassandra）

192.168.1.134（Cassandra）

192.168.1.135（Cassandra）

192.168.1.136（Cassandra）

192.168.1.131（Postgres）

192.168.1.59（Thingsboard + TB-Gateway）

 

五、集群部署(非Docker版)

如果不希望基于Docker，直接在机器上安装Cassandra集群的情况。

1. 停止Docker版Cassandra(如有必要)

sudo service docker stop

sudo systemctl disable docker

docker info

2. 关闭运行中YUM

rm -f  /var/run/yum.pid 

3. 安装Cassandra

    请浏览【参考文献6】

4. 修改配置文件

①　修改Cassandra默认族名

例如：
cqlsh
update system.local set cluster_name = 'Thingsboard Cluster' where key='local';
select cluster_name from system.local;
exit
nodetool flush system

②　修改配置文件
  请浏览【参考文献18】、【参考文献19】，以及“单机部署（Docker版）-修改Cassandra配置”章节内容，进行修改。

例如：
sudo nano /etc/thingsboard/conf/thingsboard.conf
sudo nano /etc/cassandra/default.conf/cassandra.yaml

备注：其中修改的文件(cassandra.yaml和thingsboard.conf)

③　安装Thingsboard数据库及数据表
备注：数据库及表只需任选一台安装，其它节点会定时自动同步。执行顺序：schema-entities.cql -> schema-ts.cql

例如：
# 安装在133机器
ls /root
cqlsh 192.168.1.133 -u cassandra -p cassandra -f /root/schema-entities.cql
cqlsh 192.168.1.133 -u cassandra -p cassandra -f /root/schema-ts.cql

# 查看是否创建
cqlsh 192.168.1.133 -u cassandra -p cassandra
select * from thingsboard.user;
select count(1) from thingsboard.ts_kv_latest_cf;
exit

 

5. 查看集群情况

    重启各cassandra节点，查看集群情况。

例如：

# 如果出现前后datacenter不一致问题，执行下面语句
rm -rf /var/lib/cassandra/data/*

ls /var/lib/cassandra/data


# 杀掉Cassandra端口占用程序
netstat -tunlp

kill PID


sudo systemctl stop cassandra

sudo systemctl start cassandra

nano /var/log/cassandra/cassandra.log

sudo systemctl status cassandra


# 查看集群
nodetool status

cqlsh 192.168.1.133 -u cassandra -p cassandra

nodetool -h 192.168.1.133 -u cassandra -pw cassandra status

nodetool -h 192.168.1.133 -u cassandra -pw cassandra ring


# 查看库、表同步状况
# 136机器
cqlsh 192.168.1.136 -u cassandra -p cassandra

select * from thingsboard.user;

select count(1) from thingsboard.ts_kv_latest_cf;

# 135机器
cqlsh 192.168.1.135 -u cassandra -p cassandra

select * from thingsboard.user;


# 查看集群节点
select peer from system.peers;


# Cassandra各节点成功启动后，查看日志(cassandra.log)

6. 修改system_auth表的配置

此步骤可选，可视情况而定。

例如：
# 133机器上，”dc1”为数据中心名(可通过nodetool status获取)
cqlsh 192.168.1.133 -u cassandra -p cassandra
ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy','dc1': '5'};
exit
nodetool repair

# 其它机器上进行同步
nodetool repair

7. 测试

替换配置文件(thingsboard.conf)后，重启Thingsboard和TB-Gateway，运行测试脚本进行测试！

例如：

# 重启Mosquitto(可选)
ps -aux | grep mosquitto

kill -9 PID

mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d


# 重启Thingsboard
sudo systemctl restart thingsboard

nano /var/log/thingsboard/thingsboard.log

sudo systemctl status thingsboard


# 重启TB-Gateway
sudo systemctl restart tb-gateway

nano /var/log/tb-gateway/tb-gateway.log

sudo systemctl status tb-gateway


# 运行测试脚本
node /opt/mqtt/mqtt_gateway_mqtt_sz_new.js


# Thingsboard成功运行后，查看日志(thingsboard.log)

3. 查看数据库负载均衡及同步情况

测试成功运行后，可在任一台Cassandra节点查看库状况。

例如：

# 133机器上
cqlsh 192.168.1.133 -u cassandra -p cassandra

select count(1) from thingsboard.ts_kv_latest_cf;

exit

 

9. 配置示例

192.168.1.133（Cassandra）

192.168.1.134（Cassandra）

192.168.1.135（Cassandra）

192.168.1.136（Cassandra）

192.168.1.131（Postgres）

192.168.1.59（Thingsboard + TB-Gateway）

 

参考文献

1. https://docs.docker.com/install/linux/docker-ce/centos/
2. https://hub.docker.com/_/cassandra/
3. https://thingsboard.io/docs/user-guide/install/docker/
4. https://stackoverflow.com/questions/22006887/cassandra-saved-cluster-name-test-cluster-configured-name
5. https://stackoverflow.com/questions/36133127/how-to-configure-cassandra-for-remote-connection
6. https://thingsboard.io/docs/user-guide/install/rhel/?rhelThingsboardDatabase=hybrid
7. https://docs.datastax.com/en/ddac/doc/datastax_enterprise/config/configRecommendedSettings.html#configRecommendedSettings
8. https://www.jianshu.com/p/5bacb06e334b
9. https://docs.docker.com/v17.12/network/network-tutorial-overlay/#walkthrough
10. https://cloud.tencent.com/developer/article/1110563
11. https://www.ibm.com/developerworks/cn/opensource/os-cn-cassandraxu1/index.html
12. https://www.jianshu.com/p/b970876ee3cf
13. https://blog.csdn.net/chenxingzhen001/article/details/8729623
14. https://stackoverflow.com/questions/38405983/apache-cassandra-3-7-snitch-issue-cannot-start-data-center/39150744#39150744
15. https://blog.csdn.net/walkerjong/article/details/52221126
16. https://www.cnblogs.com/51kata/p/5276407.html
17. https://www.cnblogs.com/capcom923/p/deploy_dockerize_cassandra_HA_cluster.html
18. https://thingsboard.io/docs/user-guide/install/cluster-setup/
19. https://thingsboard.io/docs/reference/performance-tests/