和standalone模式不同，distributed Connect天然地结合了Kafka提供的负载均衡和故障转移功能，能够自动地在多节点机器上平衡负载。用户可以增减机器来实现整体系统的高伸缩性。
启动
bin/connect-distributed.sh config/connect-distributed.properties
distributed 模式中我们不需要指定 source 和 sink 的配置文件。distributed 模式中的 connector 只能通过 REST API 来创建和管理。
列出 connector
curl http://localhost:8083/connectors
# 输出信息
[] dnslookup: 0.004501
connect: 0.004817
appconnect: 0.000000
pretransfer: 0.004931
starttransfer: 0.127923
total: 0.127956
size: 2
创建 file source connector
curl -i -X POST -H "Content-type:application/json" -H "Accept:application/json" -d '{"name":"test-file-source","config":{"connector.class":"FileStreamSource", "tasks.max":"1","topic":"connect-file-test","file":"foo.txt"}}' http://localhost:8083/connectors
# 输出信息
HTTP/1.1 201 Created
Date: Thu, 08 Nov 2018 02:41:02 GMT
Location: http://localhost:8083/connectors/test-file-source
Content-Type: application/json
Content-Length: 185
Server: Jetty(9.4.11.v20180605)
{"name":"test-file-source","config":{"connector.class":"FileStreamSource","tasks.max":"1","topic":"connect-file-test","file":"foo.txt","name":"test-file-source"},"tasks":[],"type":null} dnslookup: 0.004267
connect: 0.004562
appconnect: 0.000000
pretransfer: 0.004601
starttransfer: 0.624504
total: 0.624588
size: 185
创建 file sink connector
curl -i -X POST -H "Content-type:application/json" -H "Accept:application/json" -d '{"name":"test-file-sink","config":{"connector.class":"FileStreamSink", "tasks.max":"1","topics":"connect-file-test","file":"bar.txt"}}' http://localhost:8083/connectors
# 输出信息
HTTP/1.1 201 Created
Date: Thu, 08 Nov 2018 02:42:23 GMT
Location: http://localhost:8083/connectors/test-file-sink
Content-Type: application/json
Content-Length: 180
Server: Jetty(9.4.11.v20180605)
{"name":"test-file-sink","config":{"connector.class":"FileStreamSink","tasks.max":"1","topics":"connect-file-test","file":"bar.txt","name":"test-file-sink"},"tasks":[],"type":null} dnslookup: 0.005152
connect: 0.005507
appconnect: 0.000000
pretransfer: 0.005540
starttransfer: 0.518425
total: 0.518574
size: 180
获取当前所有connector以检查之前的两个connector是否已被创建出来：
curl http://localhost:8083/connectors
# 输出信息
["test-file-source","test-file-sink"] dnslookup: 0.005082
connect: 0.005479
appconnect: 0.000000
pretransfer: 0.005525
starttransfer: 0.010020
total: 0.010046
size: 37
可以看到两个connector都已经被创建出来了。
查询某个 connector
curl http://localhost:8083/connectors/test-file-sink/config
# 输出信息
{"connector.class":"FileStreamSink","name":"test-file-sink","file":"bar.txt","tasks.max":"1","topics":"connect-file-test"} dnslookup: 0.005147
connect: 0.005572
appconnect: 0.000000
pretransfer: 0.005606
starttransfer: 0.011680
total: 0.011751
size: 122
查看运行状态
curl http://localhost:8083/connectors/test-file-source/status
# 输出信息
{"name":"test-file-source","connector":{"state":"RUNNING","worker_id":"192.168.3.132:8083"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"192.168.3.132:8083"}],"type":"source"} dnslookup: 0.005213
connect: 0.005614
appconnect: 0.000000
pretransfer: 0.005667
starttransfer: 0.017550
total: 0.017602
size: 178
测试
下面开始写入输入文件foo.txt：
echo "hello" >> ./foo.txt
echo "kafka connect example" >> ./foo.txt
echo "distributed mode test" >> ./foo.txt
bar.txt 已经被创建，查看内容：
hello
kafka connect example
distributed mode test
删除 connector
curl -i -X DELETE http://localhost:8083/connectors/test-file-source
curl -i -X DELETE http://localhost:8083/connectors/test-file-sink