JBoss 系列五十三:JBoss 7/WildFly 集群之 HornetQ Messaging中讨论了JBoss 7/WildFly 集群中HornetQ Messaging的高可用性和负载均衡的原理和配置注意事项,本文通过实验,模拟一个HornetQ Messaging的集群环境,演示说明JBoss 7/WildFly 集群中HornetQ Messaging的高可用性和负载均衡。本实验运行示意图:
如图所示本实验中:
我们需要配置live消息服务器和backup消息服务器,我们分别给出node1和node2的配置。
在<subsystem xmlns="urn:jboss:domain:messaging:中配置hornetq-server:
<subsystem xmlns="urn:jboss:domain:messaging:1.3"> <hornetq-server> <persistence-enabled>true</persistence-enabled> <cluster-password>clusterPassword1!</cluster-password> <backup>false</backup> <allow-failback>true</allow-failback> <failover-on-shutdown>false</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>NIO</journal-type> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>group-1</backup-group-name> <connectors> <netty-connector name="netty" socket-binding="messaging" /> <netty-connector name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> </netty-connector> <in-vm-connector name="in-vm" server-id="1" /> </connectors> <acceptors> <netty-acceptor name="netty" socket-binding="messaging" /> <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> <param key="direct-deliver" value="false" /> </netty-acceptor> <in-vm-acceptor name="in-vm" server-id="1" /> </acceptors> <broadcast-groups> <broadcast-group name="bg-group1"> <socket-binding>messaging-group</socket-binding> <broadcast-period>5000</broadcast-period> <connector-ref>netty</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group1"> <socket-binding>messaging-group</socket-binding> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="my-cluster"> <address>jms</address> <connector-ref>netty</connector-ref> <discovery-group-ref discovery-group-name="dg-group1" /> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="send" roles="guest" /> <permission type="consume" roles="guest" /> <permission type="createNonDurableQueue" roles="guest" /> <permission type="deleteNonDurableQueue" roles="guest" /> </security-setting> </security-settings> <address-settings> <!--default for catch all --> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>1000</redistribution-delay> <max-size-bytes>10485760</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> <message-counter-history-day-limit>10 </message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty" /> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> </pooled-connection-factory> </jms-connection-factories> </hornetq-server> <hornetq-server name="backup-server"> <persistence-enabled>true</persistence-enabled> <cluster-password>clusterPassword1!</cluster-password> <backup>true</backup> <allow-failback>true</allow-failback> <failover-on-shutdown>false</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>NIO</journal-type> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>group-2</backup-group-name> <paging-directory path="messagingpagingbackupa" /> <bindings-directory path="messagingbindingsbackupa" /> <journal-directory path="messagingjournalbackupa" /> <large-messages-directory path="messaginglargemessagesbackupa" /> <connectors> <netty-connector name="netty" socket-binding="messaging-backup" /> <in-vm-connector name="in-vm" server-id="2" /> </connectors> <acceptors> <netty-acceptor name="netty" socket-binding="messaging-backup" /> <in-vm-acceptor name="in-vm" server-id="2" /> </acceptors> <broadcast-groups> <broadcast-group name="bg-group1"> <socket-binding>messaging-group</socket-binding> <broadcast-period>5000</broadcast-period> <connector-ref>netty</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group1"> <socket-binding>messaging-group</socket-binding> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="my-cluster"> <address>jms</address> <connector-ref>netty</connector-ref> <discovery-group-ref discovery-group-name="dg-group1" /> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="send" roles="guest" /> <permission type="consume" roles="guest" /> <permission type="createNonDurableQueue" roles="guest" /> <permission type="deleteNonDurableQueue" roles="guest" /> </security-setting> </security-settings> <address-settings> <!--default for catch all --> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>1000</redistribution-delay> <max-size-bytes>10485760</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> <message-counter-history-day-limit>10 </message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty" /> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> </pooled-connection-factory> </jms-connection-factories> </hornetq-server> </subsystem>
<socket-binding name="messaging-backup" port="5446" />
在<subsystem xmlns="urn:jboss:domain:messaging:中配置hornetq-server
<subsystem xmlns="urn:jboss:domain:messaging:1.3"> <hornetq-server> <persistence-enabled>true</persistence-enabled> <cluster-password>clusterPassword1!</cluster-password> <backup>false</backup> <allow-failback>true</allow-failback> <failover-on-shutdown>false</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>NIO</journal-type> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>group-2</backup-group-name> <connectors> <netty-connector name="netty" socket-binding="messaging" /> <netty-connector name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> </netty-connector> <in-vm-connector name="in-vm" server-id="1" /> </connectors> <acceptors> <netty-acceptor name="netty" socket-binding="messaging" /> <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50" /> <param key="direct-deliver" value="false" /> </netty-acceptor> <in-vm-acceptor name="in-vm" server-id="1" /> </acceptors> <broadcast-groups> <broadcast-group name="bg-group1"> <socket-binding>messaging-group</socket-binding> <broadcast-period>5000</broadcast-period> <connector-ref>netty</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group1"> <socket-binding>messaging-group</socket-binding> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="my-cluster"> <address>jms</address> <connector-ref>netty</connector-ref> <discovery-group-ref discovery-group-name="dg-group1" /> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="send" roles="guest" /> <permission type="consume" roles="guest" /> <permission type="createNonDurableQueue" roles="guest" /> <permission type="deleteNonDurableQueue" roles="guest" /> </security-setting> </security-settings> <address-settings> <!--default for catch all --> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>1000</redistribution-delay> <max-size-bytes>10485760</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> <message-counter-history-day-limit>10 </message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty" /> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> </pooled-connection-factory> </jms-connection-factories> </hornetq-server> <hornetq-server name="backup-server"> <persistence-enabled>true</persistence-enabled> <cluster-password>clusterPassword1!</cluster-password> <backup>true</backup> <allow-failback>true</allow-failback> <failover-on-shutdown>false</failover-on-shutdown> <shared-store>false</shared-store> <journal-type>NIO</journal-type> <journal-min-files>2</journal-min-files> <check-for-live-server>true</check-for-live-server> <backup-group-name>group-1</backup-group-name> <paging-directory path="messagingpagingbackupa" /> <bindings-directory path="messagingbindingsbackupa" /> <journal-directory path="messagingjournalbackupa" /> <large-messages-directory path="messaginglargemessagesbackupa" /> <connectors> <netty-connector name="netty" socket-binding="messaging-backup" /> <in-vm-connector name="in-vm" server-id="2" /> </connectors> <acceptors> <netty-acceptor name="netty" socket-binding="messaging-backup" /> <in-vm-acceptor name="in-vm" server-id="2" /> </acceptors> <broadcast-groups> <broadcast-group name="bg-group1"> <socket-binding>messaging-group</socket-binding> <broadcast-period>5000</broadcast-period> <connector-ref>netty</connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group1"> <socket-binding>messaging-group</socket-binding> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="my-cluster"> <address>jms</address> <connector-ref>netty</connector-ref> <discovery-group-ref discovery-group-name="dg-group1" /> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="send" roles="guest" /> <permission type="consume" roles="guest" /> <permission type="createNonDurableQueue" roles="guest" /> <permission type="deleteNonDurableQueue" roles="guest" /> </security-setting> </security-settings> <address-settings> <!--default for catch all --> <address-setting match="#"> <dead-letter-address>jms.queue.DLQ</dead-letter-address> <expiry-address>jms.queue.ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <redistribution-delay>1000</redistribution-delay> <max-size-bytes>10485760</max-size-bytes> <address-full-policy>BLOCK</address-full-policy> <message-counter-history-day-limit>10 </message-counter-history-day-limit> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/ConnectionFactory" /> </entries> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty" /> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory" /> </entries> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1.0</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <ha>true</ha> <block-on-acknowledge>true</block-on-acknowledge> <retry-interval>1000</retry-interval> <retry-interval-multiplier>1</retry-interval-multiplier> <reconnect-attempts>-1</reconnect-attempts> <transaction mode="xa" /> <connectors> <connector-ref connector-name="in-vm" /> </connectors> <entries> <entry name="java:/JmsXA" /> </entries> </pooled-connection-factory> </jms-connection-factories> </hornetq-server> </subsystem>
<socket-binding name="messaging-backup" port="5446" />
注意,如下我实验过程中配置文件standalone-full-ha-node1.xml和standalone-full-ha-node2.xml。
https://github.com/kylinsoong/cluster/blob/master/demo/jms/standalone-full-ha-node1.xml
https://github.com/kylinsoong/cluster/blob/master/demo/jms/standalone-full-ha-node2.xml
我们需要创建Application User来与JBoss进行交换,具体使用JBOSS_HOME/bin/add_user.sh或JBOSS_HOME/bin/add_user.bat创建Application User democlient/passowrd1!。
本实验需要部署消息队列到JBoss,根据之前系列(JBoss 系列六:JBoss 7/WildFly中配置使用JMS消息队列)部署DistributedQueue到JBoss。我们可以使用拷贝queue-jms.xml文件到deployments目录下面的方式部署DistributedQueue。queue-jms.xml内容如下所示:
<?xml version="1.0" encoding="UTF-8"?> <messaging-deployment xmlns="urn:jboss:messaging-deployment:1.0"> <hornetq-server> <jms-destinations> <jms-queue name="DistributedQueue"> <entry name="queue/DistributedQueue" /> <entry name="java:jboss/exported/jms/queue/DistributedQueue"/> <durable>true</durable> </jms-queue> </jms-destinations> </hornetq-server> </messaging-deployment>
部署完成我们通过如下命令分别启动两个节点:
./standalone.sh -c standalone-full-ha-node1.xml -b 10.66.218.47 -bmanagement=10.66.218.47 -u 239.255.100.100 -Djboss.node.name=node1 ./standalone.sh -c standalone-full-ha-node2.xml -b 10.66.218.48 -bmanagement=10.66.218.48 -u 239.255.100.100 -Djboss.node.name=node2
我们通过如下步骤测试JBoss 7/WildFly消息服务高可用:
1. 发送连续10条消息
运行JMSClient通过node1发送连续的10条消息到集群,我们发现node1处理第一条和第三条消息,输出如下日志
05:20:09,741 INFO [stdout] (Thread-8 (HornetQ-client-global-threads-62067681)) 1: JBoss 7/WildFly HornetQ Messaging High Available 05:20:19,750 INFO [stdout] (Thread-8 (HornetQ-client-global-threads-62067681)) 3: JBoss 7/WildFly HornetQ Messaging High Available
05:20:11,626 INFO [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 2: JBoss 7/WildFly HornetQ Messaging High Available 05:20:21,648 INFO [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 4: JBoss 7/WildFly HornetQ Messaging High Available
2. 关闭node1,故障转移高可用测试
当node1处理第1条和第3条消息后,还有3条消息(5,7,9)未被处理,这个时候我们强制关闭node1,模拟发生故障,这时5,7,9消息被分发到node2,node2处理5,7,9消息,node2处理日志如下
05:20:31,658 INFO [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 6: JBoss 7/WildFly HornetQ Messaging High Available 05:20:41,679 INFO [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 8: JBoss 7/WildFly HornetQ Messaging High Available 05:20:51,688 INFO [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 10: JBoss 7/WildFly HornetQ Messaging High Available 05:21:28,059 INFO [stdout] (Thread-8 (HornetQ-client-global-threads-1878339755)) 5: JBoss 7/WildFly HornetQ Messaging High Available 05:21:38,068 INFO [stdout] (Thread-8 (HornetQ-client-global-threads-1878339755)) 7: JBoss 7/WildFly HornetQ Messaging High Available 05:21:48,076 INFO [stdout] (Thread-8 (HornetQ-client-global-threads-1878339755)) 9: JBoss 7/WildFly HornetQ Messaging High Available