部署storm统计服务开启zookeeper、kafka（ACL认证）
| 服务IP |
模块 |
版本信息 |
| 10.130.41.42 |
zookeeper |
zookeeper-3.6.3 |
| 10.130.41.43 |
kafka |
kafka_2.11-2.3.1 |
| 10.130.41.44 |
storm |
apache-storm-0.9.6 |
zookeeper部署
- 部署mongodb_1服务器的zookeeper
- 传安装部署包和配置文件
[root@mongodb_1 ~]# su - storm
[storm@mongodb_1 ~]$ rz -be ====> mongodb_1 ~]$ rz -be storm_node1.tar.gz
[storm@mongodb_1 ~]$ tar xvf storm_node1.tar.gz
[storm@mongodb_1 ~]$ cd storm_node1
[storm@mongodb_1 storm_node1]$ mv * .bash_profile ../
[storm@mongodb_1 ~]$ source .bash_profile ;java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
[storm@mongodb_1 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_1 conf]$ vim zoo.cfg
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_1 conf]$ vim zk_server_jaas.conf
Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="zk_cluster"
password="zk_cluster_passwd"
user_admin="Qnsoft@123";
};
#拷贝kafka acl lib
[storm@mongodb_1 zookeeper-3.6.3]$ cd /home/storm/zookeeper-3.6.3
[storm@mongodb_1 zookeeper-3.6.3]$ mkdir zk_sasl_dependencies
[storm@mongodb_1 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_1 libs]$ cp kafka-clients-2.3.1.jar lz4-java-1.6.0.jar slf4j-api-1.7.26.jar snappy-java-1.1.7.3.jar ~/zookeeper-3.6.3/lib/
[storm@mongodb_1 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_1 ~]$ echo "1" > /home/storm/zookeeper-3.6.3/data/myid
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ vim zkEnv.sh #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_1 bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_1 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
- 部署mongodb_2服务器的zookeeper
- 上传安装部署包和配置文件
[root@mongodb_2 ~]# su - storm
[storm@mongodb_2 ~]$ rz -be ====> mongodb_1 ~]$ rz -be storm_node2.tar.gz
[storm@mongodb_2 ~]$ tar xvf storm_node2.tar.gz
[storm@mongodb_2 ~]$ cd storm_node2
[storm@mongodb_2 storm_node1]$ mv * .bash_profile ../
[storm@mongodb_2 ~]$ source .bash_profile ;java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
[storm@mongodb_2 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_2 conf]$ vim zoo.cfg
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_2:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_2 conf]$ vim zk_server_jaas.conf
Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="zk_cluster"
password="zk_cluster_passwd"
user_admin="Qnsoft@123";
};
#拷贝kafka acl lib
[storm@mongodb_2 zookeeper-3.6.3]$ cd /home/storm/zookeeper-3.6.3
[storm@mongodb_2 zookeeper-3.6.3]$ mkdir zk_sasl_dependencies
[storm@mongodb_2 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_2 libs]$ cp kafka-clients-2.3.1.jar lz4-java-1.6.0.jar slf4j-api-1.7.26.jar snappy-java-1.1.7.3.jar ~/zookeeper-3.6.3/lib/
[storm@mongodb_2 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_2 ~]$ echo "1" > /home/storm/zookeeper-3.6.3/data/myid
[storm@mongodb_2 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_2 bin]$ vim zkEnv.sh #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_2 bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_2 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
- 部署mongodb_3服务器的zookeeper
- 上传安装部署包和配置文件
[root@mongodb_3 ~]# su - storm
[storm@mongodb_3 ~]$ rz -be ====> mongodb_1 ~]$ rz -be storm_node3.tar.gz
[storm@mongodb_3 ~]$ tar xvf storm_node3.tar.gz
[storm@mongodb_3 ~]$ cd storm_node3
[storm@mongodb_3 storm_node1]$ mv * .bash_profile ../
[storm@mongodb_3 ~]$ source .bash_profile ;java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
[storm@mongodb_3 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_3 conf]$ vim zoo.cfg
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_3:2182:3181
server.2:mongodb_3:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_3 conf]$ vim zk_server_jaas.conf
Server {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="zk_cluster"
password="zk_cluster_passwd"
user_admin="Qnsoft@123";
};
#拷贝kafka acl lib
[storm@mongodb_3 zookeeper-3.6.3]$ cd /home/storm/zookeeper-3.6.3
[storm@mongodb_3 zookeeper-3.6.3]$ mkdir zk_sasl_dependencies
[storm@mongodb_3 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_3 libs]$ cp kafka-clients-2.3.1.jar lz4-java-1.6.0.jar slf4j-api-1.7.26.jar snappy-java-1.1.7.3.jar ~/zookeeper-3.6.3/lib/
[storm@mongodb_3 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_3 ~]$ echo "1" > /home/storm/zookeeper-3.6.3/data/myid
[storm@mongodb_3 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_3 bin]$ vim zkEnv.sh #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_3 bin]$ ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_3 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
- zookeeper设置用户密码，并设置连接zookeeper白名单。
[storm@mongodb_2 bin]$ ./zkCli.sh -server mongodb_2:2182,mongodb_1:2182,mongodb_3:2182
[zk: localhost:2181(CONNECTED) 0] addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 1] setAcl / auth:admin:123456:cdrwa
[zk: localhost:2181(CONNECTED) 2] getAcl /
'digest,'admin:0uek/hZ/V9fgiM35b0Z2226acMQ=
: cdrwa
[storm@mongodb_2 bin]$ ./zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /
Insufficient permission : /
[zk: localhost:2181(CONNECTED) 1] addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 2] ls /
[test, zookeeper]
[zk: localhost:2181(CONNECTED) 8] setAcl / ip:10.130.41.42:cdrwa,ip:10.130.41.43:cdrwa,ip:10.130.41.44:cdrwa,ip:10.130.41.86:cdrwa,ip:10.130.41.119:cdrwa,ip:127.0.0.1:cdrwa,auth:kafka:cdrwa
初始化storm配置
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ ./zkCli.sh
[zk: localhost:2181(CONNECTED) 2] addauth digest admin:123456
create /storm config
create /storm/config 1
kafka部署
[storm@mongodb_1 ~]$ cd ~/kafka_2.11-0.8.2.2/config/
[storm@mongodb_1 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0
[storm@mongodb_1 config]$ more sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Qnsoft@123";
[storm@mongodb_1 config]$ cat kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkacluster"
password="Qnsoft@123"
user_kafkacluster="Qnsoft@123"
user_admin="Qnsoft@123"
};
Client{
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="Qnsoft@123";
};
[storm@mongodb_1 ~]$mkdir /home/storm/kafka_2.11-0.8.2.2/log -p
[storm@mongodb_1 ~]$cd /home/storm/kafka_2.11-0.8.2.2/bin
[storm@mongodb_1 bin]$ vim kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"
[storm@mongodb_1 ~]$ ./kafka-server-start.sh -daemon config/server.properties
[storm@mongodb_2 ~]$ cd ~/kafka_2.11-0.8.2.2/config/
[storm@mongodb_2 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_2:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0
[storm@mongodb_2 config]$ more sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Qnsoft@123";
[storm@mongodb_2 config]$ cat kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkacluster"
password="Qnsoft@123"
user_kafkacluster="Qnsoft@123"
user_admin="Qnsoft@123"
};
Client{
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="Qnsoft@123";
};
[storm@mongodb_2 ~]$mkdir /home/storm/kafka_2.11-0.8.2.2/log -p
[storm@mongodb_2 ~]$cd /home/storm/kafka_2.11-0.8.2.2/bin
[storm@mongodb_2 bin]$ vim kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"
[storm@mongodb_2 ~]$ ./kafka-server-start.sh -daemon config/server.properties
[storm@mongodb_3 ~]$ cd ~/kafka_2.11-0.8.2.2/config/
[storm@mongodb_3 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_3:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0
[storm@mongodb_3 config]$ more sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Qnsoft@123";
[storm@mongodb_3 config]$ cat kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkacluster"
password="Qnsoft@123"
user_kafkacluster="Qnsoft@123"
user_admin="Qnsoft@123";
};
Client{
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="Qnsoft@123";
};
[storm@mongodb_3 ~]$mkdir /home/storm/kafka_2.11-0.8.2.2/log -p
[storm@mongodb_3 ~]$cd /home/storm/kafka_2.11-0.8.2.2/bin
[storm@mongodb_3 bin]$ vim kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"
[storm@mongodb_3 ~]$ ./kafka-server-start.sh -daemon config/server.properties
- 初始化创建topics,在其中一台kafka上面操作即可
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ cd kafka_2.11-0.8.2.2/
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic agentStateDetail
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic call_detail
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic chatLog
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ivr_message
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic new_r_ags_e
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic t_srvappraise
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ent_record_bx_table
[storm@mongodb_1 kafka_2.11-0.8.2.2]$ bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ent_record_fastdfs_url
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic ent_record_fastdfs_url
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic ent_record_bx_table
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic new_r_ags_e
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic chatLog
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic call_detail
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic agentStateDetail
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --add --allow-principal User:admin --producer --topic t_srvappraise
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181/kafka --list
Current ACLs for resource `Topic:LITERAL:t_srvappraise`:
User:admin has Allow permission for operations: Describe from hosts: *
User:admin has Allow permission for operations: Create from hosts: *
User:admin has Allow permission for operations: Write from hosts: *
Current ACLs for resource `Topic:LITERAL:new_r_ags_e`:
User:admin has Allow permission for operations: Describe from hosts: *
User:admin has Allow permission for operations: Create from hosts: *
User:admin has Allow permission for operations: Write from hosts: *
Current ACLs for resource `Topic:LITERAL:ent_record_fastdfs_url`:
User:admin has Allow permission for operations: Describe from hosts: *
User:admin has Allow permission for operations: Create from hosts: *
User:admin has Allow permission for operations: Write from hosts: *
Current ACLs for resource `Topic:LITERAL:ent_record_bx_table`:
User:admin has Allow permission for operations: Describe from hosts: *
User:admin has Allow permission for operations: Create from hosts: *
User:admin has Allow permission for operations: Write from hosts: *
Current ACLs for resource `Topic:LITERAL:chatLog`:
User:admin has Allow permission for operations: Describe from hosts: *
User:admin has Allow permission for operations: Create from hosts: *
User:admin has Allow permission for operations: Write from hosts: *
Current ACLs for resource `Topic:LITERAL:agentStateDetail`:
User:admin has Allow permission for operations: Describe from hosts: *
User:admin has Allow permission for operations: Create from hosts: *
User:admin has Allow permission for operations: Write from hosts: *
storm部署
[storm@mongodb_1 ~]$ cd ~/apache-storm-0.9.6/conf/
[storm@mongodb_1 conf]$ vim storm.yaml
storm.zookeeper.servers:
- "mongodb_1"
- "mongodb_2"
- "mongodb_3"
nimbus.host: "mongodb_1"
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-0.9.6/data"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
- 6705
- 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.digest.1: admin=123456
[storm@mongodb_1 ~]$ mkdir -p /home/storm/apache-storm-0.9.6/data
[storm@mongodb_1 ~]$ cd /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_1 bin]$ nohup ./storm nimbus &
[storm@mongodb_1 bin]$ nohup ./storm ui &
[storm@mongodb_1 bin]$ jps #查看是否正常运行
12770 Jps
12519 core
12473 nimbus
8602 QuorumPeerMain
11007 Kafka
[storm@mongodb_2 ~]$ cd ~/apache-storm-0.9.6/conf/
[storm@mongodb_2 conf]$ vim storm.yaml
storm.zookeeper.servers:
- "mongodb_1"
- "mongodb_2"
- "mongodb_3"
nimbus.host: "mongodb_1"
storm.local.dir: "/home/storm/apache-storm-0.9.6/data"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
- 6705
- 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.digest.1: admin=123456
[storm@mongodb_2 ~]$ mkdir -p /home/storm/apache-storm-0.9.6/data
[storm@mongodb_2 ~]$ cd /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_2 bin]$ nohup ./storm supervisor &
[storm@mongodb_2 bin]$ jps #查看是否正常运行
21075 supervisor
20968 Kafka
20794 QuorumPeerMain
21150 Jps
[storm@mongodb_3 ~]$ cd ~/apache-storm-0.9.6/conf/
[storm@mongodb_3 conf]$ vim storm.yaml
storm.zookeeper.servers:
- "mongodb_1"
- "mongodb_2"
- "mongodb_3"
nimbus.host: "mongodb_1"
storm.local.dir: "/home/storm/apache-storm-0.9.6/data"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
- 6705
- 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.digest.1: admin=123456
[storm@mongodb_3 ~]$ mkdir -p /home/storm/apache-storm-0.9.6/data
[storm@mongodb_3 ~]$ cd /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_3 bin]$ nohup ./storm supervisor &
[storm@mongodb_3 bin]$ jps #查看是否正常运行
20962 Kafka
21074 supervisor
20791 QuorumPeerMain
21147 Jps
- storm的主节点nimbus在第一台上启动拓扑图
- 启动拓扑图时需要指定stormStatics.jar 包，如果使用新storm安装包没有，请自行找测试部要stormStatics.jar包
[storm@mongodb_1 bin]$ cd /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.BxRecordTopology BxRecordTopology mongodb_1,mongodb_2,mongodb_3 2181 ent_record_bx_table /brokers
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.FastdfsUrlTopology FastdfsUrlTopology mongodb_1,mongodb_2,mongodb_3 2181 ent_record_fastdfs_url /brokers
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.TSrvappraiseTopology TSrvappraiseTopology mongodb_1,mongodb_2,mongodb_3 2181 t_srvappraise /brokers
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.RAGSETopology RAGSETopology mongodb_1,mongodb_2,mongodb_3 2181 new_r_ags_e /brokers
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.IVRMESSAGETopology IVRMESSAGETopology mongodb_1,mongodb_2,mongodb_3 2181 ivr_message /brokers
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.CallDetailTopology CallDetailTopology mongodb_1,mongodb_2,mongodb_3 2181 call_detail /brokers
[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.AgentStateDetailTopology AgentStateDetailTopology mongodb_1,mongodb_2,mongodb_3 2181 agentStateDetail /brokers
- 使用storm-ui查看刚启动拓扑是否正常
- 访问第一台的服务器ip端口是8888
