搭建基于clickhouse集群的zookeeper+kafka的数据分析架构

1 安装java环境

yum search java

yum install java-1.8.0

2 安装zookeeper

mkdir /app/

cd /app/

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

上面的下载不了用下面的

tar -zxf apache-zookeeper-3.7.0-bin.tar.gz 解压

3 更改配置文件

cd /conf

cp zoo.sample.cfg zoo.cfg 复制配置文件

vim zoo.cfg 修改

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/app/apache-zookeeper-3.7.0-bin/data

dataLog=/app/apache-zookeeper-3.7.0-bin/log

# the port at which the clients will connect

clientPort=2181

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60

#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

## Metrics Providers

#

# https://prometheus.io Metrics Exporter

#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider

#metricsProvider.httpPort=7000

#metricsProvider.exportJvmInfo=true

server.1=ch201:2888:3888

server.2=ch202:2888:3888

server.3=ch203:2888:3888

4 修改hosts

192.168.211.100 ch201

192.168.211.188 ch202

192.168.211.186 ch203

5 修改myid，每台机器不同

mkdir -p /app/apache-zookeeper-3.7.0-bin/data

mkdir -p /app/apache-zookeeper-3.7.0-bin/log

cd /app/apache-zookeeper-3.7.0-bin/data

vim myid 只写一个数字字符

以上三台机器修改为1，2，3

6 全部启动zookeeper

到bin下面 ./zkServer.sh start

然后./zkServer.sh status 查看状态

安装clickhouse，参考另一篇教程

vim /etc/clickhouse-server/config.xml

 ::


/etc/clickhouse-server/config.d/metrika.xml

vim /etc/clickhouse-server/config.d/metrika.xml

        
                 
                        
                                 true
                                
                                        ch201
                                        9000
                                
                        
                        
                                
                                        true
                                        ch202
                                        9000
                                
                        
                        
                                true
                                
                                        ch203
                                        9000
                                
                        
                
        
        
        
          
                ch201
                2181
          

          
                ch202
                2181
          
          
                ch203
                2181
          
        
         
                01
                01
                ch201
        
         
           ::/0
        

        
        
          10000000000

          0.01
          lz4
        
        

修改配置文件

以上三台机器同样配置，macro改掉

同样执行sudo clickhouse start.启动集群

看到服务启动后登录clickhouse查看

clickhouse-client直接登录

select * from system.clusters查看集群状态

安装kafka

cd /app

wget Apache Downloads

wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz 版本不同，皆可用

tar -zxf kafka_2.13-2.8.1.tgz解压

cd /config/

vim server.properties

broker.id=1 这个值为myid的值，每天机器唯一

listeners=PLAINTEXT://192.168.211.100:9092 本机ip，kafka的地址

zookeeper.connect=192.168.211.100:2181,192.168.211.186:2181,192.168.211.188:2181 集群三台机器的ip

cd /bin

./kafka-server-start.sh /app/kafka_2.13-2.8.1/config/server.properties

如果报错日志问题，将日志目录删除重新启动即可

nohup ./kafka-server-start.sh /apps/kafka_2.13-2.8.1/config/server.properties >>/tmp/kafka-logs/kafka.log 2>&1 & 后台启动，可查看/tmp下面的日志