JBoss 系列五十四：JBoss 7/WildFly 集群之 HornetQ Messaging - II（示例）

概述

JBoss 系列五十三：JBoss 7/WildFly 集群之 HornetQ Messaging中讨论了JBoss 7/WildFly 集群中HornetQ Messaging的高可用性和负载均衡的原理和配置注意事项，本文通过实验，模拟一个HornetQ Messaging的集群环境，演示说明JBoss 7/WildFly 集群中HornetQ Messaging的高可用性和负载均衡。本实验运行示意图：

如图所示本实验中：

• 两个JBoss节点（node1和node2）构成一个集群，两个节点上分别有live消息服务器和backup消息服务器，且node1上的live消息服务器和node2上的backup消息服务器位于group-1，而node2上的live消息服务器和node1上的backup消息服务器位于group-2
• 消息队列DistributedQueue部署在两个节点上的live服务器，消息驱动Bean（MessageDrivenBean）作为消息队列的消费者监听于DistributedQueue之上
  
• JMS客户端（JMSClient）通过node1发送连续的10条消息到集群
  
• JBoss node1部署于物理机器10.66.218.47上，JBoss node2部署于物理机器10.66.218.48上
  

JBoss 端配置

我们需要配置live消息服务器和backup消息服务器，我们分别给出node1和node2的配置。

node1配置

在<subsystem xmlns="urn:jboss:domain:messaging:中配置hornetq-server：

		<subsystem xmlns="urn:jboss:domain:messaging:1.3">
			<hornetq-server>
				<persistence-enabled>true</persistence-enabled>
				<cluster-password>clusterPassword1!</cluster-password>
				<backup>false</backup>
				<allow-failback>true</allow-failback>
				<failover-on-shutdown>false</failover-on-shutdown>
				<shared-store>false</shared-store>
				<journal-type>NIO</journal-type>
				<journal-min-files>2</journal-min-files>
				<check-for-live-server>true</check-for-live-server>
				<backup-group-name>group-1</backup-group-name>
				<connectors>
					<netty-connector name="netty" socket-binding="messaging" />
					<netty-connector name="netty-throughput"
						socket-binding="messaging-throughput">
						<param key="batch-delay" value="50" />
					</netty-connector>
					<in-vm-connector name="in-vm" server-id="1" />
				</connectors>
				<acceptors>
					<netty-acceptor name="netty" socket-binding="messaging" />
					<netty-acceptor name="netty-throughput"
						socket-binding="messaging-throughput">
						<param key="batch-delay" value="50" />
						<param key="direct-deliver" value="false" />
					</netty-acceptor>
					<in-vm-acceptor name="in-vm" server-id="1" />
				</acceptors>
				<broadcast-groups>
					<broadcast-group name="bg-group1">
						<socket-binding>messaging-group</socket-binding>
						<broadcast-period>5000</broadcast-period>
						<connector-ref>netty</connector-ref>
					</broadcast-group>
				</broadcast-groups>

				<discovery-groups>
					<discovery-group name="dg-group1">
						<socket-binding>messaging-group</socket-binding>
						<refresh-timeout>10000</refresh-timeout>
					</discovery-group>
				</discovery-groups>
				<cluster-connections>
					<cluster-connection name="my-cluster">
						<address>jms</address>
						<connector-ref>netty</connector-ref>
						<discovery-group-ref discovery-group-name="dg-group1" />
					</cluster-connection>
				</cluster-connections>
				<security-settings>
					<security-setting match="#">
						<permission type="send" roles="guest" />
						<permission type="consume" roles="guest" />
						<permission type="createNonDurableQueue" roles="guest" />
						<permission type="deleteNonDurableQueue" roles="guest" />
					</security-setting>
				</security-settings>
				<address-settings>
					<!--default for catch all -->
					<address-setting match="#">
						<dead-letter-address>jms.queue.DLQ</dead-letter-address>
						<expiry-address>jms.queue.ExpiryQueue</expiry-address>
						<redelivery-delay>0</redelivery-delay>
						<redistribution-delay>1000</redistribution-delay>
						<max-size-bytes>10485760</max-size-bytes>
						<address-full-policy>BLOCK</address-full-policy>
						<message-counter-history-day-limit>10
						</message-counter-history-day-limit>
					</address-setting>
				</address-settings>
				<jms-connection-factories>
					<connection-factory name="InVmConnectionFactory">
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/ConnectionFactory" />
						</entries>
					</connection-factory>
					<connection-factory name="RemoteConnectionFactory">
						<connectors>
							<connector-ref connector-name="netty" />
						</connectors>
						<entries>
							<entry name="java:jboss/exported/jms/RemoteConnectionFactory" />
						</entries>
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1.0</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
					</connection-factory>
					<pooled-connection-factory name="hornetq-ra">
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
						<transaction mode="xa" />
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/JmsXA" />
						</entries>
					</pooled-connection-factory>
				</jms-connection-factories>
			</hornetq-server>
			<hornetq-server name="backup-server">
				<persistence-enabled>true</persistence-enabled>
				<cluster-password>clusterPassword1!</cluster-password>
				<backup>true</backup>
				<allow-failback>true</allow-failback>
				<failover-on-shutdown>false</failover-on-shutdown>
				<shared-store>false</shared-store>
				<journal-type>NIO</journal-type>
				<journal-min-files>2</journal-min-files>
				<check-for-live-server>true</check-for-live-server>
				<backup-group-name>group-2</backup-group-name>
				<paging-directory path="messagingpagingbackupa" />
				<bindings-directory path="messagingbindingsbackupa" />
				<journal-directory path="messagingjournalbackupa" />
				<large-messages-directory path="messaginglargemessagesbackupa" />
				<connectors>
					<netty-connector name="netty" socket-binding="messaging-backup" />
					<in-vm-connector name="in-vm" server-id="2" />
				</connectors>
				<acceptors>
					<netty-acceptor name="netty" socket-binding="messaging-backup" />
					<in-vm-acceptor name="in-vm" server-id="2" />
				</acceptors>
				<broadcast-groups>
					<broadcast-group name="bg-group1">
						<socket-binding>messaging-group</socket-binding>
						<broadcast-period>5000</broadcast-period>
						<connector-ref>netty</connector-ref>
					</broadcast-group>
				</broadcast-groups>

				<discovery-groups>
					<discovery-group name="dg-group1">
						<socket-binding>messaging-group</socket-binding>
						<refresh-timeout>10000</refresh-timeout>
					</discovery-group>
				</discovery-groups>
				<cluster-connections>
					<cluster-connection name="my-cluster">
						<address>jms</address>
						<connector-ref>netty</connector-ref>
						<discovery-group-ref discovery-group-name="dg-group1" />
					</cluster-connection>
				</cluster-connections>
				<security-settings>
					<security-setting match="#">
						<permission type="send" roles="guest" />
						<permission type="consume" roles="guest" />
						<permission type="createNonDurableQueue" roles="guest" />
						<permission type="deleteNonDurableQueue" roles="guest" />
					</security-setting>
				</security-settings>
				<address-settings>
					<!--default for catch all -->
					<address-setting match="#">
						<dead-letter-address>jms.queue.DLQ</dead-letter-address>
						<expiry-address>jms.queue.ExpiryQueue</expiry-address>
						<redelivery-delay>0</redelivery-delay>
						<redistribution-delay>1000</redistribution-delay>
						<max-size-bytes>10485760</max-size-bytes>
						<address-full-policy>BLOCK</address-full-policy>
						<message-counter-history-day-limit>10
						</message-counter-history-day-limit>
					</address-setting>
				</address-settings>
				<jms-connection-factories>
					<connection-factory name="InVmConnectionFactory">
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/ConnectionFactory" />
						</entries>
					</connection-factory>
					<connection-factory name="RemoteConnectionFactory">
						<connectors>
							<connector-ref connector-name="netty" />
						</connectors>
						<entries>
							<entry name="java:jboss/exported/jms/RemoteConnectionFactory" />
						</entries>
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1.0</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
					</connection-factory>
					<pooled-connection-factory name="hornetq-ra">
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
						<transaction mode="xa" />
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/JmsXA" />
						</entries>
					</pooled-connection-factory>
				</jms-connection-factories>
			</hornetq-server>
		</subsystem>

在<socket-binding-group name="standard-sockets"添加messaging-backup socket-binding如下：

<socket-binding name="messaging-backup" port="5446" />

node2配置

在<subsystem xmlns="urn:jboss:domain:messaging:中配置hornetq-server

		<subsystem xmlns="urn:jboss:domain:messaging:1.3">
			<hornetq-server>
				<persistence-enabled>true</persistence-enabled>
				<cluster-password>clusterPassword1!</cluster-password>
				<backup>false</backup>
				<allow-failback>true</allow-failback>
				<failover-on-shutdown>false</failover-on-shutdown>
				<shared-store>false</shared-store>
				<journal-type>NIO</journal-type>
				<journal-min-files>2</journal-min-files>
				<check-for-live-server>true</check-for-live-server>
				<backup-group-name>group-2</backup-group-name>
				<connectors>
					<netty-connector name="netty" socket-binding="messaging" />
					<netty-connector name="netty-throughput"
						socket-binding="messaging-throughput">
						<param key="batch-delay" value="50" />
					</netty-connector>
					<in-vm-connector name="in-vm" server-id="1" />
				</connectors>
				<acceptors>
					<netty-acceptor name="netty" socket-binding="messaging" />
					<netty-acceptor name="netty-throughput"
						socket-binding="messaging-throughput">
						<param key="batch-delay" value="50" />
						<param key="direct-deliver" value="false" />
					</netty-acceptor>
					<in-vm-acceptor name="in-vm" server-id="1" />
				</acceptors>
				<broadcast-groups>
					<broadcast-group name="bg-group1">
						<socket-binding>messaging-group</socket-binding>
						<broadcast-period>5000</broadcast-period>
						<connector-ref>netty</connector-ref>
					</broadcast-group>
				</broadcast-groups>

				<discovery-groups>
					<discovery-group name="dg-group1">
						<socket-binding>messaging-group</socket-binding>
						<refresh-timeout>10000</refresh-timeout>
					</discovery-group>
				</discovery-groups>
				<cluster-connections>
					<cluster-connection name="my-cluster">
						<address>jms</address>
						<connector-ref>netty</connector-ref>
						<discovery-group-ref discovery-group-name="dg-group1" />
					</cluster-connection>
				</cluster-connections>
				<security-settings>
					<security-setting match="#">
						<permission type="send" roles="guest" />
						<permission type="consume" roles="guest" />
						<permission type="createNonDurableQueue" roles="guest" />
						<permission type="deleteNonDurableQueue" roles="guest" />
					</security-setting>
				</security-settings>
				<address-settings>
					<!--default for catch all -->
					<address-setting match="#">
						<dead-letter-address>jms.queue.DLQ</dead-letter-address>
						<expiry-address>jms.queue.ExpiryQueue</expiry-address>
						<redelivery-delay>0</redelivery-delay>
						<redistribution-delay>1000</redistribution-delay>
						<max-size-bytes>10485760</max-size-bytes>
						<address-full-policy>BLOCK</address-full-policy>
						<message-counter-history-day-limit>10
						</message-counter-history-day-limit>
					</address-setting>
				</address-settings>

				<jms-connection-factories>
					<connection-factory name="InVmConnectionFactory">
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/ConnectionFactory" />
						</entries>
					</connection-factory>
					<connection-factory name="RemoteConnectionFactory">
						<connectors>
							<connector-ref connector-name="netty" />
						</connectors>
						<entries>
							<entry name="java:jboss/exported/jms/RemoteConnectionFactory" />
						</entries>
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1.0</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
					</connection-factory>
					<pooled-connection-factory name="hornetq-ra">
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
						<transaction mode="xa" />
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/JmsXA" />
						</entries>
					</pooled-connection-factory>
				</jms-connection-factories>
			</hornetq-server>
			<hornetq-server name="backup-server">
				<persistence-enabled>true</persistence-enabled>
				<cluster-password>clusterPassword1!</cluster-password>
				<backup>true</backup>
				<allow-failback>true</allow-failback>
				<failover-on-shutdown>false</failover-on-shutdown>
				<shared-store>false</shared-store>
				<journal-type>NIO</journal-type>
				<journal-min-files>2</journal-min-files>
				<check-for-live-server>true</check-for-live-server>
				<backup-group-name>group-1</backup-group-name>
				<paging-directory path="messagingpagingbackupa" />
				<bindings-directory path="messagingbindingsbackupa" />
				<journal-directory path="messagingjournalbackupa" />
				<large-messages-directory path="messaginglargemessagesbackupa" />
				<connectors>
					<netty-connector name="netty" socket-binding="messaging-backup" />
					<in-vm-connector name="in-vm" server-id="2" />
				</connectors>
				<acceptors>
					<netty-acceptor name="netty" socket-binding="messaging-backup" />
					<in-vm-acceptor name="in-vm" server-id="2" />
				</acceptors>
				<broadcast-groups>
					<broadcast-group name="bg-group1">
						<socket-binding>messaging-group</socket-binding>
						<broadcast-period>5000</broadcast-period>
						<connector-ref>netty</connector-ref>
					</broadcast-group>
				</broadcast-groups>

				<discovery-groups>
					<discovery-group name="dg-group1">
						<socket-binding>messaging-group</socket-binding>
						<refresh-timeout>10000</refresh-timeout>
					</discovery-group>
				</discovery-groups>
				<cluster-connections>
					<cluster-connection name="my-cluster">
						<address>jms</address>
						<connector-ref>netty</connector-ref>
						<discovery-group-ref discovery-group-name="dg-group1" />
					</cluster-connection>
				</cluster-connections>
				<security-settings>
					<security-setting match="#">
						<permission type="send" roles="guest" />
						<permission type="consume" roles="guest" />
						<permission type="createNonDurableQueue" roles="guest" />
						<permission type="deleteNonDurableQueue" roles="guest" />
					</security-setting>
				</security-settings>
				<address-settings>
					<!--default for catch all -->
					<address-setting match="#">
						<dead-letter-address>jms.queue.DLQ</dead-letter-address>
						<expiry-address>jms.queue.ExpiryQueue</expiry-address>
						<redelivery-delay>0</redelivery-delay>
						<redistribution-delay>1000</redistribution-delay>
						<max-size-bytes>10485760</max-size-bytes>
						<address-full-policy>BLOCK</address-full-policy>
						<message-counter-history-day-limit>10
						</message-counter-history-day-limit>
					</address-setting>
				</address-settings>

				<jms-connection-factories>
					<connection-factory name="InVmConnectionFactory">
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/ConnectionFactory" />
						</entries>
					</connection-factory>
					<connection-factory name="RemoteConnectionFactory">
						<connectors>
							<connector-ref connector-name="netty" />
						</connectors>
						<entries>
							<entry name="java:jboss/exported/jms/RemoteConnectionFactory" />
						</entries>
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1.0</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
					</connection-factory>
					<pooled-connection-factory name="hornetq-ra">
						<ha>true</ha>
						<block-on-acknowledge>true</block-on-acknowledge>
						<retry-interval>1000</retry-interval>
						<retry-interval-multiplier>1</retry-interval-multiplier>
						<reconnect-attempts>-1</reconnect-attempts>
						<transaction mode="xa" />
						<connectors>
							<connector-ref connector-name="in-vm" />
						</connectors>
						<entries>
							<entry name="java:/JmsXA" />
						</entries>
					</pooled-connection-factory>
				</jms-connection-factories>
			</hornetq-server>
		</subsystem>

在<socket-binding-group name="standard-sockets"添加messaging-backup socket-binding如下：

<socket-binding name="messaging-backup" port="5446" />

注意，如下我实验过程中配置文件standalone-full-ha-node1.xml和standalone-full-ha-node2.xml。

https://github.com/kylinsoong/cluster/blob/master/demo/jms/standalone-full-ha-node1.xml

https://github.com/kylinsoong/cluster/blob/master/demo/jms/standalone-full-ha-node2.xml

创建Application User

我们需要创建Application User来与JBoss进行交换，具体使用JBOSS_HOME/bin/add_user.sh或JBOSS_HOME/bin/add_user.bat创建Application User democlient/passowrd1!。

部署高可用实验应用

本实验需要部署消息队列到JBoss，根据之前系列（JBoss 系列六：JBoss 7/WildFly中配置使用JMS消息队列）部署DistributedQueue到JBoss。我们可以使用拷贝queue-jms.xml文件到deployments目录下面的方式部署DistributedQueue。queue-jms.xml内容如下所示：

<?xml version="1.0" encoding="UTF-8"?>
<messaging-deployment xmlns="urn:jboss:messaging-deployment:1.0">
	<hornetq-server>
		<jms-destinations>
			<jms-queue name="DistributedQueue">
				<entry name="queue/DistributedQueue" />
				<entry name="java:jboss/exported/jms/queue/DistributedQueue"/>
				<durable>true</durable>
			</jms-queue>
		</jms-destinations>
	</hornetq-server>
</messaging-deployment>

本实验应用代码位于github（ https://github.com/kylinsoong/cluster/tree/master/demo/jms），使用 系列一方法下载编译代码，会生成部署应用cluster-demo-jms.jar，我们可以使用拷贝cluster-demo-jms.jar文件到deployments目录下面的方式部署应用。

部署完成我们通过如下命令分别启动两个节点：

./standalone.sh -c standalone-full-ha-node1.xml -b 10.66.218.47 -bmanagement=10.66.218.47 -u 239.255.100.100 -Djboss.node.name=node1
./standalone.sh -c standalone-full-ha-node2.xml -b 10.66.218.48 -bmanagement=10.66.218.48 -u 239.255.100.100 -Djboss.node.name=node2

高可用和负载均衡过程测试

我们通过如下步骤测试JBoss 7/WildFly消息服务高可用：

1. 发送连续10条消息
运行JMSClient通过node1发送连续的10条消息到集群，我们发现node1处理第一条和第三条消息，输出如下日志

05:20:09,741 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-62067681)) 1: JBoss 7/WildFly HornetQ Messaging High Available
05:20:19,750 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-62067681)) 3: JBoss 7/WildFly HornetQ Messaging High Available

与此同时node2处理第二条和第四条消息，输出如下日志

05:20:11,626 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 2: JBoss 7/WildFly HornetQ Messaging High Available
05:20:21,648 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 4: JBoss 7/WildFly HornetQ Messaging High Available

此处体现了JBoss 7/WildFly消息服务器的负载均衡功能，10条消息分别给node1分发五条，给node2分发五条，node1分发五条为（第1，3，5，7，9），node2分发五条为（第2，4，6，8，10）。

2. 关闭node1，故障转移高可用测试
当node1处理第1条和第3条消息后，还有3条消息（5，7，9）未被处理，这个时候我们强制关闭node1，模拟发生故障，这时5，7，9消息被分发到node2，node2处理5，7，9消息，node2处理日志如下

05:20:31,658 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 6: JBoss 7/WildFly HornetQ Messaging High Available
05:20:41,679 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 8: JBoss 7/WildFly HornetQ Messaging High Available
05:20:51,688 INFO  [stdout] (Thread-7 (HornetQ-client-global-threads-1878339755)) 10: JBoss 7/WildFly HornetQ Messaging High Available
05:21:28,059 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1878339755)) 5: JBoss 7/WildFly HornetQ Messaging High Available
05:21:38,068 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1878339755)) 7: JBoss 7/WildFly HornetQ Messaging High Available
05:21:48,076 INFO  [stdout] (Thread-8 (HornetQ-client-global-threads-1878339755)) 9: JBoss 7/WildFly HornetQ Messaging High Available

注意，处理第5条和第10条消息之间的间隔为30多秒，这说明故障转移的时间为30多秒。