Ashwini Kuntamukkala - Technology Enthusiast

Monday, February 24, 2014

ActiveMQ - Network of Brokers Explained

Objective

This 7 part blog series is to share about how to create network of ActiveMQ brokers in order to achieve high availability and scalability.

Why network of brokers?

ActiveMQ message broker is a core component of messaging infrastructure in an enterprise. It needs to be highly available and dynamically scalable to facilitate communication between dynamic heterogeneous distributed applications which have varying capacity needs.

Scaling enterprise applications on commodity hardware is a rage nowadays. ActiveMQ caters to that very well by being able to create a network of brokers to share the load.

Many times applications running across geographically distributed data centers need to coordinate messages. Running message producers and consumers across geographic regions/data centers can be architected better using network of brokers.

ActiveMQ uses transport connectors over which it communicates with message producers and consumers. However, in order to facilitate broker to broker communication, ActiveMQ uses network connectors.

A network connector is a bridge between two brokers which allows on-demand message forwarding.

In other words, if Broker B1 initiates a network connector to Broker B2 then the messages on a channel (queue/topic) on B1 get forwarded to B2 if there is at least one consumer on B2 for the same channel. If the network connector was configured to be duplex, the messages get forwarded from B2 to B1 on demand.

This is very interesting because it is now possible for brokers to communicate with each other dynamically.

In this 7 part blog series, we will look into the following topics to gain understanding of this very powerful ActiveMQ feature:

Network Connector Basics - Part 1
Duplex network connectors - Part 2
Load balancing consumers on local/remote brokers - Part 3
Load-balance consumers/subscribers on remote brokers

Queue: Load balance remote concurrent consumers - Part 4
Topic: Load Balance Durable Subscriptions on Remote Brokers - Part 5

Store/Forward messages and consumer failover - Part 6

How to prevent stuck messages

Virtual Destinations - Part 7

To give credit where it is due, the following URLs have helped me in creating this blog post series.

Advanced Messaging with ActiveMQ by Dejan Bosanac [Slides 32-36]
Understanding ActiveMQ Broker Networks by Jakub Korab

Prerequisites

ActiveMQ 5.8.0 – To create broker instances
Apache Ant – To run ActiveMQ sample producer and consumers for demo.

We will use multiple ActiveMQ broker instances on the same machine for the ease of demonstration.

Network Connector Basics - Part 1

The following diagram shows how a network connector functions. It bridges two brokers and is used to forward messages from Broker-1 to Broker-2 on demand if established by Broker-1 to Broker-2.

A network connector can be duplex so messages could be forwarded in the opposite direction; from Broker-2 to Broker-1, once there is a consumer on Broker-1 for a channel which exists in Broker-2. More on this in Part 2

Setup network connector between broker-1 and broker-2

Create two broker instances, say broker-1 and broker-2

Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/bin
Ashwinis-MacBook-Pro:bin akuntamukkala$ ./activemq-admin create ../bridge-demo/broker-1

Ashwinis-MacBook-Pro:bin akuntamukkala$ ./activemq-admin create ../bridge-demo/broker-2

Since we will be running both brokers on the same machine, let's configure broker-2 such that there are no port conflicts.

Edit /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/activemq.xml

Change transport connector to 61626 from 61616
Change AMQP port from 5672 to 6672 (won't be using it for this blog)

Edit /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/jetty.xml

Change web console port to 9161 from 8161

Configure Network Connector from broker-1 to broker-2
Add the following XML snippet to /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml

<networkConnectors>
<networkConnector
name="T:broker1->broker2"
uri="static:(tcp://localhost:61626)"
duplex="false"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector
name="Q:broker1->broker2"
uri="static:(tcp://localhost:61626)"
duplex="false"
decreaseNetworkConsumerPriority="true"
networkTTL="2"
dynamicOnly="true">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
</networkConnectors>

The above XML snippet configures two network connectors "T:broker1->broker2" (only topics as queues are excluded) and "Q:broker1->broker2" (only queues as topics are excluded). This allows for nice separation between network connectors used for topics and queues.

The name can be arbitrary although I prefer to specify the [type]:[source broker]->[destination broker].

The URI attribute specifies how to connect to broker-2

Start broker-2

Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/bin

Ashwinis-MacBook-Pro:bin akuntamukkala$ ./broker-2 console

Start broker-1

Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/bin

Ashwinis-MacBook-Pro:bin akuntamukkala$ ./broker-1 console

Logs on broker-1 show 2 network connectors being established with broker-2

INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626
INFO | Connector vm://broker-1 Started
INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626
INFO | Network connection between vm://broker-1#24 and tcp://localhost/127.0.0.1:61626@52132(broker-2) has been established.
INFO | Network connection between vm://broker-1#26 and tcp://localhost/127.0.0.1:61626@52133(broker-2) has been established.

Web Console on broker-1 @ http://localhost:8161/admin/connections.jsp shows the two network connectors established to broker-2

The same on broker-2 does not show any network connectors since no network connectors were initiated by broker-2

Let's see this in action

Let's produce 100 persistent messages on a queue called "foo.bar" on broker-1.

Ashwinis-MacBook-Pro:example akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/example
Ashwinis-MacBook-Pro:example akuntamukkala$ ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100

broker-1 web console shows that 100 messages have been enqueued in queue "foo.bar"

http://localhost:8161/admin/queues.jsp

Let's start a consumer on a queue called "foo.bar" on broker-2. The important thing to note here is that the destination name "foo.bar" should match exactly.

Ashwinis-MacBook-Pro:example akuntamukkala$ ant consumer -Durl=tcp://localhost:61626 -Dtopic=false -Dsubject=foo.bar

We find that all the 100 messages from broker-1's foo.bar queue get forwarded to broker-2's foo.bar queue consumer.

broker-1 admin console at http://localhost:8161/admin/queues.jsp

broker-2 admin console @ http://localhost:9161/admin/queues.jsp shows that the consumer we had started has consumed all 100 messages which were forwarded on-demand from broker-1

broker-2 consumer details on foo.bar queue

broker-1 admin console shows that all 100 messages have been dequeued [forwarded to broker-2 via the network connector].

broker-1 consumer details on "foo.bar" queue shows that the consumer is created on demand: [name of connector]_[destination broker]_inbound_[source broker]

Thus we have seen the basics of network connector in ActiveMQ.

As always, please feel to comment about anything that can be improved. Your inputs are welcome!

Stay tuned for Part 2...

Wednesday, January 1, 2014

Temp, Store and Memory Percent Usage in ActiveMQ

In order to effectively use ActiveMQ, it is very important to understand how ActiveMQ manages memory and disk resources to handle non-persistent and persistent messages.
ActiveMQ has three key parameters which need to be kept under check.

Temp Percent Usage

This is the % of assigned disk storage that has been used up to spool non-persistent messages
Non persistent messages are those that don't survive broker restart

Store Percent Usage

This is the % of assigned disk space that has been used up to store persistent messages

Memory Percent Usage

This is the % of assigned memory of the broker that has been used up to keep track of destinations, cache messages etc. This value needs to be lesser than -Xmx (Max JVM heap size)

This blog attempts to clarify how store,temp and memory % usage of a single node ActiveMQ broker instance are calculated. We are using ActiveMQ version 5.8.0 for this explanation.

Once we gain clarity on how ActiveMQ operates these values, we can fine tune ActiveMQ using key configuration settings in order to handle the following use cases.

Large number of destinations (queues/topics)

The destinations could be created/deleted as needed

Slow consumers

This is huge issue when consumers are unable to keep up with the rate at which messages are being produced.

Message Burst

Rapid influx of large number of messages with huge payload size for a brief period of time

Inappropriate Resource utilization

Few destinations chewing up resources causing others to starve

Scaling Strategies

If you are interested to know how ActiveMQ can be scaled horizontally, please refer to a slide deck created by Bosanac Dejan. You can find it here

It contains different ActiveMQ topologies which can be used effectively to meet volume throughput in addition to various parameters to tune ActiveMQ. I found it extremely useful.

Let's dig right in...

The following XML snippet is taken from configuration activemq.xml. The values specified for memoryUsage, storeUsage and tempUsage are for discussion purposes only.

Memory Usage

256MB of JVM memory is available for the broker. Not to be confused with -Xmx parameter.

Store Usage:

This is the disk space used by persistent messages (using KahaDB)

Temp Usage:

This is the disk space used by non-persistent message, assuming we are using default KahaDB. ActiveMQ spools non-persistent messages to disk in order to prevent broker running out of memory

Understanding Temp Usage

Broker availability is critical for message infrastructure. Hence producer flow control is a protection mechanism that prevents a runaway producer from pumping non-persistent messages into a destination when there are no consumers or when consumer(s) is unable to keep up with the rate at which messages are being produced into the destination.

Let’s take an example of producing non-persistent messages having 1MB payload size into a destination “foo.bar” in a local broker instance

C:\apache-activemq-5.8.0\example>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Ddurable=false -DmessageSize=1048576

The producer eventually hangs as temp % usage hits 100%

Since the messages are non-persistent, they are going to be stored in tmp_storage on the disk

ActiveMQ provides mechanism to tune memory usage per destination. Here we have a generic policy for all queues where producer flow control is enabled and destination memory limit is 100MB (again this is only for illustration purposes).

<policyEntry queue=">" optimizedDispatch="true" producerFlowControl="true" cursorMemoryHighWaterMark="30" memoryLimit="100 mb" >

The temp % usage is calculated as follows:

(Size of the tmp_storage folder / temp usage memory limit ) * 100
So in our case:
265,025,856/(256*1024*1024) * 100 = 99.8 ~ 100% as shown in the broker console.

The following log message shows up in activemq.log

INFO | Usage(default:temp:queue://foo.bar:temp) percentUsage=99%, usage=268535808, limit=268435456, percentUsageMinDelta=1%;Parent:Usage(default:temp
) percentUsage=100%, usage=268535808, limit=268435456, percentUsageMinDelta=1%: Temp Store is Full (99% of 268435456). Stopping producer (ID:AKUNTAMU-
1-61270-1388528939599-1:1:1:1) to prevent flooding queue://foo.bar. See http://activemq.apache.org/producer-flow-control.html for more info (blocking

for: 1421s)

Let's take another example...

Consider the following system usage configuration. We have reduced tempUsage to 50MB while keeping the same destination level policy.

<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="256 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="512 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 mb"/>
</tempUsage>
</systemUsage>

</systemUsage>

In this case we find that temp usage balloons to 191%

temp_storage stops growing at close to 96MB and producer hangs..

Temp percent usage is 191% because (95.5MB / 50 MB)*100 where 95.5 MB is size of the folder and 50MB is temp usage limit.

The destination has a limit of 100MB so the temp_storage didn't grow past it. It is sort of confusing which is caused by the fact that temp usage limit is less that per destination memory limit.

Store Usage

Let’s repeat the same test with persistent messages.

The system usage is configured as follows:

<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="256 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="512 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="256 mb"/>
</tempUsage>
</systemUsage>

</systemUsage>

Per destination policy is as follows:

<policyEntry queue=">" optimizedDispatch="true" producerFlowControl="true" cursorMemoryHighWaterMark="30" memoryLimit="100 mb" >

Let's produce 1MB persistent messages into a queue named "foo.bar"

C:\apache-activemq-5.8.0\example>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Ddurable=true -DmessageSize=1048576

Producer hangs after 512 messages

The following log statement appears in broker log file

INFO | Usage(default:store:queue://foo.bar:store) percentUsage=99%, usage=537210471, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:st
ore) percentUsage=100%, usage=537210471, limit=536870912,percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (ID: AKUNTAMU-1-31754-1388571228628-1:1:1:1) to prevent flooding queue://foo.bar. See http://activemq.apache.org/producer-flow-control.html for more info (
blocking for: 155s)

Broker store usage is 100% as shown below.

Since the messages are persistent, they need to be saved onto the file system. Store usage limit is 512MB.

The above screenshot shows the kahadb folder where persistent messages is 543 MB (512MB for the messages and other database related files)

Memory Usage

In the above example, the memory usage percentage is 11. How did that come about?

As per the destination policy, the memory allocated per destination is 100MB and the cursorMemoryHighWaterMark is specified to be 30. So 30% of 100MB is 30MB. Hence 30MB is used to store messages in memory for faster processing in addition to be being stored in the KahaDB. .

The memory usage limit is configured to be 256MB. So 30MB is ~ 11% of 256

(30/256) * 100 ~ 11%

So if we were to have 9 such queues where similar situation was to occur then we would have exhausted broker memory usage as 11 % * 9 = 99% ~ 100%

Memory usage is the amount of memory used by the broker for storing messages. Many a times, this can become a bottleneck as once this space is full, the broker will stall the producers. There are trade-offs between fast processing and effective memory management.

If we keep more messages in memory, the processing is faster. However the memory consumption will be very high. On the contrary, if messages are kept on the disk then processing will become slow.

Conclusion

We have seen in this blog how store, temp and memory usage work in ActiveMQ. % of store and temp usage cannot be configured per destination while % of memory usage can be because of cursorMemoryHighWaterMark.

Hope you found this information useful. The examples given here are for explanation purposes only and not meant to be production ready. You will need to do proper capacity planning and determine your broker topology for optimal configuration. Feel free to reach out if any comments!

References

http://blog.raulkr.net/2012/08/demystifying-producer-flow-control-and.html

http://tmielke.blogspot.com/2011/02/observations-on-activemqs-temp-storage.html

http://activemq.apache.org/javalangoutofmemory.html

http://www.slideshare.net/dejanb/advanced-messaging-with-apache-activemq -Bosanac Dejan

http://www.pepperdust.org/?p=150

http://stackoverflow.com/questions/2361541/how-do-you-scale-your-activemq-vertically

Monday, August 19, 2013

Active 5.8.0 Master Slave Web Console Fix

Recently I ran into an issue when I attempted to use ActiveMQ 5.8.0 Web Console to monitor a two node Master/Slave ActiveMQ cluster.

The ActiveMQ 5.8.0 Web Console (hosted in a Tomcat container) connects to remote ActiveMQ instance over JMX. In ActiveMQ 5.8.0, the JMX connector is enabled for both Master and Slave nodes. Due to which the Web Console is unable to determine which node is the master node. Hence the issue.

In order to get around this problem, I created a class which extends the RemoteJMXBrokerFacade class by adding intelligence to an overridden findBrokers(...) method such that the Web Console considers an ActiveMQ node as a master node if and only if it is able to fetch a reference to the BrokerViewMBean.

The above operation throws an IllegalStateException if the broker was to be slave node.

The Web Console gets the list of JMX Connectors for all the ActiveMQ Brokers in a cluster via the following property

-Dwebconsole.jmx.url=service:jmx:rmi:///jndi/rmi://amq-node-1-machine:12099/jmxrmi,service:jmx:rmi:///jndi/rmi://amq-node-2-machine:12099/jmxrmi

The following is the source code:

package org.apache.activemq.web;

import java.io.IOException;
import java.util.Set;

import javax.management.MBeanServerConnection;
import javax.management.MBeanServerInvocationHandler;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;

import org.apache.activemq.broker.jmx.BrokerViewMBean;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * This class helps ActiveMQ Web Console determine if the remote broker instance is master or slave
 * 
 * In the webconsole-properties.xml, replace the <code>org.apache.activemq.web.RemoteJMXBrokerFacade</code> class 
 * with <code>org.apache.activemq.web.MasterSlaveAwareJMXBrokerFacade</code>
 * <pre>
 * {@code
 *  <bean id="brokerQuery" class="org.apache.activemq.web.MasterSlaveAwareJMXBrokerFacade" autowire="constructor" destroy-method="shutdown">
    <property name="configuration" ref="configuration"/>
    <property name="brokerName"><null/></property>
   </bean>
 * }
 * </pre>
 * @author AKUNTAMUKKALA
 *
 */
public class MasterSlaveAwareJMXBrokerFacade extends RemoteJMXBrokerFacade {

 private static final transient Logger LOG = LoggerFactory
   .getLogger(MasterSlaveAwareJMXBrokerFacade.class);

 @Override
 protected Set<ObjectName> findBrokers(MBeanServerConnection connection)
   throws IOException, MalformedObjectNameException {

  LOG.debug("Finding brokers");
  ObjectName name;

  name = new ObjectName("org.apache.activemq:type=Broker,brokerName=*");

  Set<ObjectName> brokers = connection.queryNames(name, null);
  

  try {
   ObjectName brokerObjectName = (ObjectName) brokers.iterator().next();
   BrokerViewMBean mbean = (BrokerViewMBean) MBeanServerInvocationHandler
     .newProxyInstance(connection, brokerObjectName,
       BrokerViewMBean.class, true);

   LOG.debug("Broker Id : " + mbean.getBrokerId());

  } catch (Throwable t) {
   // TODO Auto-generated catch block
   LOG.error("Exception occurred while trying to retrieve BrokerViewMBean", t);
   throw new RuntimeException(t);
  }
  return brokers;
 }

}

If the Web Console attempts to connect to slave broker, the following exception gets thrown.

SEVERE: Servlet.service() for servlet [jsp] in context with path [/activemq-web-console-5.8.0] threw exception [javax.el.ELException: Error reading 'brokerName' on type org.apache.activemq.web.RemoteJMXBrokerFacade] with root cause
java.lang.IllegalStateException: Broker is not yet started.
 at org.apache.activemq.broker.jmx.BrokerView.safeGetBroker(BrokerView.java:459)
 at org.apache.activemq.broker.jmx.BrokerView.getBrokerName(BrokerView.java:75)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
 at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
 at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
 at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
 at javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
 at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
 at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
 at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
 at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
 at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
 at sun.rmi.transport.Transport$1.run(Transport.java:159)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)

This solution has worked for me. I would like to know if you have a better way to fix this.

Tuesday, July 9, 2013

Using ActiveMQ - "Master/Slave" configuration with failover protocol

Objective:

This blog is to share how to run highly available ActiveMQ cluster in Master/Slave configuration.

Introduction:

ActiveMQ broker(s) tends to be a core piece of messaging infrastructure in an enterprise. It is vital for this messaging infrastructure to be highly available and scalable. Please read this link in order to learn more about creating network of brokers to support various use cases. One of the popular use case for ActiveMQ is the Master/Slave configuration with shared database. When this configuration is used, the message consumers and producers can operate without interruptions as they use ActiveMQ's connection factory with failover protocol. The failover protocol insulates the consumers and producers from having to deal with any potential downtime or application level reconnection logic when slave ActiveMQ node takes over to become the master which happens if the current master node goes down for any reason. I must caution that this configuration must not be used to mask any issues which take out the master node. We should iron out any causes that result in an unplanned master node outage.

Overview:

In this blog, I will demonstrate the following:

Run 2 ActiveMQ nodes in a Master/Slave configuration with a shared KahaDB file based database.
Configure ActiveMQ web console hosted in a Tomcat instance to point to whichever node is Master node in the cluster.
Failover scenarios.
Message publisher and consumer behavior oblivious to the failover.

Prerequisites:

Install ActiveMQ version 5.8.0
Download ActiveMQ version 5.8.0 web console war from here
Install Tomcat version 7.0.35 - To host the ActiveMQ Web Console application

Deep Dive:

We will create a 2 node ActiveMQ cluster on the same Windows 7 machine. We will need to configure the ports (TCP & JMX) such that there is no conflict. ActiveMQ provides a script called "activemq-admin.bat" which helps in running multiple instances of ActiveMQ using the same binaries, very much like Tomcat. Let's go ahead and create broker-1 and broker-2 instances.

command: activemq-admin create <DIR-for-ActiveMQ-node>

C:\apache-activemq-5.8.0\bin>activemq-admin create ..\cluster\broker-1

Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
  Heap sizes: current=125632k  free=124976k  max=1864192k
    JVM args: -Dactivemq.classpath=C:\apache-activemq-5.8.0\bin\..\conf;C:\apache-activemq-5.8.0\bin\..\data; -Dactivemq.home=C:\apache-activemq-5.8.0
\bin\.. -Dactivemq.base=C:\apache-activemq-5.8.0\bin\.. -Dactivemq.data=C:\apache-activemq-5.8.0\bin\..\data -Djava.io.tmpdir=C:\apache-activemq-5.8.0
\bin\..\data\tmp -Dactivemq.conf=C:\apache-activemq-5.8.0\bin\..\conf
Extensions classpath:
  [C:\apache-activemq-5.8.0\bin\..\lib,C:\apache-activemq-5.8.0\bin\..\lib\camel,C:\apache-activemq-5.8.0\bin\..\lib\optional,C:\apache-activemq-5.8.0
\bin\..\lib\web,C:\apache-activemq-5.8.0\bin\..\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\bin\..\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\bin\..\data
Running create broker task...
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-1
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-1\bin
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-1\conf
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-1\bin\broker-1.bat
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-1\bin\broker-1
Copying from: C:\apache-activemq-5.8.0\conf\activemq.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-command.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-command.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-demo.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker1.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-dynamic-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker2.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-dynamic-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-jdbc.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-jdbc.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-scalability.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-scalability.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-security.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-security.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-specjms.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-specjms.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker1.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-static-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker2.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-static-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-stomp.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-stomp.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-throughput.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-throughput.xml
Copying from: C:\apache-activemq-5.8.0\conf\broker-localhost.cert
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\broker-localhost.cert
Copying from: C:\apache-activemq-5.8.0\conf\broker.ks
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\broker.ks
Copying from: C:\apache-activemq-5.8.0\conf\broker.ts
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\broker.ts
Copying from: C:\apache-activemq-5.8.0\conf\camel.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\camel.xml
Copying from: C:\apache-activemq-5.8.0\conf\client.ks
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\client.ks
Copying from: C:\apache-activemq-5.8.0\conf\client.ts
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\client.ts
Copying from: C:\apache-activemq-5.8.0\conf\credentials-enc.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\credentials-enc.properties
Copying from: C:\apache-activemq-5.8.0\conf\credentials.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\credentials.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty-demo.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jetty-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\jetty-realm.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jetty-realm.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jetty.xml
Copying from: C:\apache-activemq-5.8.0\conf\jmx.access
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jmx.access
Copying from: C:\apache-activemq-5.8.0\conf\jmx.password
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jmx.password
Copying from: C:\apache-activemq-5.8.0\conf\log4j.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\log4j.properties
Copying from: C:\apache-activemq-5.8.0\conf\logging.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\logging.properties

Now let's create broker-2 instance...

C:\apache-activemq-5.8.0\bin>activemq-admin create ..\cluster\broker-2

Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
  Heap sizes: current=125632k  free=124976k  max=1864192k
    JVM args: -Dactivemq.classpath=C:\apache-activemq-5.8.0\bin\..\conf;C:\apache-activemq-5.8.0\bin\..\data; -Dactivemq.home=C:\apache-activemq-5.8.0
\bin\.. -Dactivemq.base=C:\apache-activemq-5.8.0\bin\.. -Dactivemq.data=C:\apache-activemq-5.8.0\bin\..\data -Djava.io.tmpdir=C:\apache-activemq-5.8.0
\bin\..\data\tmp -Dactivemq.conf=C:\apache-activemq-5.8.0\bin\..\conf
Extensions classpath:
  [C:\apache-activemq-5.8.0\bin\..\lib,C:\apache-activemq-5.8.0\bin\..\lib\camel,C:\apache-activemq-5.8.0\bin\..\lib\optional,C:\apache-activemq-5.8.0
\bin\..\lib\web,C:\apache-activemq-5.8.0\bin\..\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\bin\..\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\bin\..\data
Running create broker task...
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-2
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-2\bin
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-2\conf
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-2\bin\broker-2.bat
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-2\bin\broker-2
Copying from: C:\apache-activemq-5.8.0\conf\activemq.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-command.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-command.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-demo.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker1.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-dynamic-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker2.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-dynamic-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-jdbc.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-jdbc.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-scalability.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-scalability.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-security.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-security.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-specjms.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-specjms.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker1.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-static-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker2.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-static-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-stomp.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-stomp.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-throughput.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-throughput.xml
Copying from: C:\apache-activemq-5.8.0\conf\broker-localhost.cert
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\broker-localhost.cert
Copying from: C:\apache-activemq-5.8.0\conf\broker.ks
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\broker.ks
Copying from: C:\apache-activemq-5.8.0\conf\broker.ts
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\broker.ts
Copying from: C:\apache-activemq-5.8.0\conf\camel.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\camel.xml
Copying from: C:\apache-activemq-5.8.0\conf\client.ks
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\client.ks
Copying from: C:\apache-activemq-5.8.0\conf\client.ts
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\client.ts
Copying from: C:\apache-activemq-5.8.0\conf\credentials-enc.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\credentials-enc.properties
Copying from: C:\apache-activemq-5.8.0\conf\credentials.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\credentials.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty-demo.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jetty-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\jetty-realm.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jetty-realm.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty.xml
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jetty.xml
Copying from: C:\apache-activemq-5.8.0\conf\jmx.access
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jmx.access
Copying from: C:\apache-activemq-5.8.0\conf\jmx.password
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jmx.password
Copying from: C:\apache-activemq-5.8.0\conf\log4j.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\log4j.properties
Copying from: C:\apache-activemq-5.8.0\conf\logging.properties
          to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\logging.properties

You may have noticed the following properties above:

ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\bin\..\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\bin\..\data

These properties need to be fixed as we'd like ACTIVEMQ_BASE and ACTIVEMQ_CONF to be different for both broker-1 and broker-2.

We need to edit the following files:

broker-1.bat in C:\apache-activemq-5.8.0\cluster\broker-1\bin directory

set ACTIVEMQ_HOME="C:/apache-activemq-5.8.0"
set ACTIVEMQ_BASE="C:/apache-activemq-5.8.0/cluster/broker-1"
set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf

broker-2.bat in C:\apache-activemq-5.8.0\cluster\broker-2\bin directory

set ACTIVEMQ_HOME="C:/apache-activemq-5.8.0"
set ACTIVEMQ_BASE="C:/apache-activemq-5.8.0/cluster/broker-2"
set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf

You may observe that both broker-1 and broker-2 nodes share the same ACTIVEMQ_DATA folder. Since we are using the in-built KahaDB for persistence, both broker-1 and broker-2 will share this.

We need to differentiate the tcp ports for broker-1 and broker-2 and also enable JMX and configure JMX ports for remote monitoring.

Let's edit the activemq.xml for broker-1 to fix tcp port:

        <transportConnectors>
            <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
            <!--<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>-->
        </transportConnectors>

Let's edit the activemq.xml for broker-1 to enable JMX monitoring, notice useJMX="true" attribute below.

        <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker-1" dataDirectory="${activemq.data}" useJmx="true">

Let's configure the JMX port

        <managementContext>
            <managementContext createConnector="true" connectorPort="1099"/>
        </managementContext>

Repeat the same for broker-2. set TCP port to 61626 and JMX port to 2099.

Start broker-1.

C:\apache-activemq-5.8.0\cluster\broker-1\bin>broker-1.bat
Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
  Heap sizes: current=1004928k  free=994439k  max=1004928k
    JVM args: -Dcom.sun.management.jmxremote -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dactivemq.classpath=C:/apache-activemq-
5.8.0/cluster/broker-1/conf;C:/apache-activemq-5.8.0/cluster/broker-1/conf;C:/apache-activemq-5.8.0/conf; -Dactivemq.home=C:/apache-activemq-5.8.0 -Da
ctivemq.base=C:/apache-activemq-5.8.0/cluster/broker-1 -Dactivemq.conf=C:/apache-activemq-5.8.0/cluster/broker-1/conf -Dactivemq.data=C:/apache-active
mq-5.8.0\data -Djava.io.tmpdir=C:/apache-activemq-5.8.0\data\tmp
Extensions classpath:
  [C:\apache-activemq-5.8.0\cluster\broker-1\lib,C:\apache-activemq-5.8.0\lib,C:\apache-activemq-5.8.0\cluster\broker-1\lib\camel,C:\apache-activemq-5
.8.0\cluster\broker-1\lib\optional,C:\apache-activemq-5.8.0\cluster\broker-1\lib\web,C:\apache-activemq-5.8.0\cluster\broker-1\lib\extra,C:\apache-act
ivemq-5.8.0\lib\camel,C:\apache-activemq-5.8.0\lib\optional,C:\apache-activemq-5.8.0\lib\web,C:\apache-activemq-5.8.0\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\cluster\broker-1
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\cluster\broker-1\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\data
Loading message broker from: xbean:activemq.xml
 INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@71060478: startup date [Tue Jul 09 16:59:15 CDT 2013]; root of context hierarchy
 INFO | PListStore:[C:\apache-activemq-5.8.0\data\broker-1\tmp_storage] started
 INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\apache-activemq-5.8.0\data\kahadb]
 INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
 INFO | KahaDB is version 4
 INFO | Recovering from the journal ...
 INFO | Recovery replayed 1 operations from the journal in 0.028 seconds.
 INFO | Apache ActiveMQ 5.8.0 (broker-1, ID:AKUNTAMU-1-27777-1373407157813-0:1) is starting
 INFO | Listening for connections at: tcp://AKUNTAMU-1:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
 INFO | Connector openwire Started
 INFO | Apache ActiveMQ 5.8.0 (broker-1, ID:AKUNTAMU-1-27777-1373407157813-0:1) started
 INFO | For help or more information please see: http://activemq.apache.org
 WARN | Store limit is 102400 mb, whilst the data directory: C:\apache-activemq-5.8.0\data\kahadb only has 38889 mb of usable space
ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: C:\apache-activemq-5.8.0\data\broker-1\tmp_storage only has 38889 mb o
f usable space
 INFO | Web console type: embedded
 INFO | ActiveMQ WebConsole initialized.
 INFO | Initializing Spring FrameworkServlet 'dispatcher'
 INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed

Start broker-2.

You will observe that broker-2 is unable to acquire the lock since broker-1 already grabbed it. broker-2 will keep retrying to acquire the lock every 10 seconds.

C:\apache-activemq-5.8.0\cluster\broker-2\bin>broker-2.bat
Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
  Heap sizes: current=1004928k  free=994439k  max=1004928k
    JVM args: -Dcom.sun.management.jmxremote -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dactivemq.classpath=C:/apache-activemq-
5.8.0/cluster/broker-2/conf;C:/apache-activemq-5.8.0/cluster/broker-2/conf;C:/apache-activemq-5.8.0/conf; -Dactivemq.home=C:/apache-activemq-5.8.0 -Da
ctivemq.base=C:/apache-activemq-5.8.0/cluster/broker-2 -Dactivemq.conf=C:/apache-activemq-5.8.0/cluster/broker-2/conf -Dactivemq.data=C:/apache-active
mq-5.8.0\data -Djava.io.tmpdir=C:/apache-activemq-5.8.0\data\tmp
Extensions classpath:
  [C:\apache-activemq-5.8.0\cluster\broker-2\lib,C:\apache-activemq-5.8.0\lib,C:\apache-activemq-5.8.0\cluster\broker-2\lib\camel,C:\apache-activemq-5
.8.0\cluster\broker-2\lib\optional,C:\apache-activemq-5.8.0\cluster\broker-2\lib\web,C:\apache-activemq-5.8.0\cluster\broker-2\lib\extra,C:\apache-act
ivemq-5.8.0\lib\camel,C:\apache-activemq-5.8.0\lib\optional,C:\apache-activemq-5.8.0\lib\web,C:\apache-activemq-5.8.0\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\cluster\broker-2
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\cluster\broker-2\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\data
Loading message broker from: xbean:activemq.xml
 INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@420f9c40: startup date [Tue Jul 09 17:02:55 CDT 2013]; root of context hierarchy
 INFO | PListStore:[C:\apache-activemq-5.8.0\data\broker-2\tmp_storage] started
 INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\apache-activemq-5.8.0\data\kahadb]
 INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
 INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:2099/jmxrmi
 INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
 INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.

Now let's configure the ActiveMQ web console.
By default, ActiveMQ distribution contains admin web console but in the Master/Slave configuration, it is unknown which node is the master. So it is doesn't make sense to use the embedded web console. Hence it is best to have ActiveMQ web console outside of the ActiveMQ nodes.

You can disable the embedded ActiveMQ Web Console in each node by commenting the following line in activemq.xml in conf directory of each ActiveMQ node.



For our example, we will deploy ActiveMQ Web Console web application in Tomcat container and then configure ActiveMQ web console application to intelligently point to the master node in ActiveMQ cluster.

so let's copy the activemq-web-console-5.8.0.war to webapps directory of Tomcat. Add the following line to catalina.bat

set JAVA_OPTS=-Dwebconsole.type=properties -Dwebconsole.jms.url=failover:(tcp://localhost:61616,tcp://localhost:61626) -Dwebconsole.jmx.url=service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi,service:jmx:rmi:///jndi/rmi://localhost:2099/jmxrmi

Now let's start Tomcat.

C:\apache-tomcat-7.0.35\bin>.\catalina.bat run
Using CATALINA_BASE:   "C:\apache-tomcat-7.0.35"
Using CATALINA_HOME:   "C:\apache-tomcat-7.0.35"
Using CATALINA_TMPDIR: "C:\apache-tomcat-7.0.35\temp"
Using JRE_HOME:        "C:\Program Files\Java\jdk1.6.0_31"
Using CLASSPATH:       "C:\apache-tomcat-7.0.35\bin\bootstrap.jar;C:\apache-tomcat-7.0.35\bin\tomcat-juli.jar"
Jul 9, 2013 5:28:08 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jul 9, 2013 5:28:08 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jul 9, 2013 5:28:08 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 635 ms
Jul 9, 2013 5:28:08 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jul 9, 2013 5:28:08 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.35
Jul 9, 2013 5:28:08 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive C:\apache-tomcat-7.0.35\webapps\activemq-web-console-5.8.0.war
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apache-tomcat-7.0.35/webapps/activemq-web-console-5.8.0/WEB-INF/lib/activemq-all-5.8.0.jar!/org/slf4j/impl/Stati
cLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/apache-tomcat-7.0.35/webapps/activemq-web-console-5.8.0/WEB-INF/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/Stat
icLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2013-07-09 17:28:13,389 [ost-startStop-1] INFO  WebConsoleStarter              - Web console type: properties
2013-07-09 17:28:13,960 [ost-startStop-1] INFO  WebConsoleStarter              - ActiveMQ WebConsole initialized.
2013-07-09 17:28:14,095 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/createDestination.action] onto handler '/createDest
ination.action'
2013-07-09 17:28:14,096 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteDestination.action] onto handler '/deleteDest
ination.action'
2013-07-09 17:28:14,097 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/createSubscriber.action] onto handler '/createSubsc
riber.action'
2013-07-09 17:28:14,098 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteSubscriber.action] onto handler '/deleteSubsc
riber.action'
2013-07-09 17:28:14,099 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/sendMessage.action] onto handler '/sendMessage.acti
on'
2013-07-09 17:28:14,100 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/purgeDestination.action] onto handler '/purgeDestin
ation.action'
2013-07-09 17:28:14,101 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteMessage.action] onto handler '/deleteMessage.
action'
2013-07-09 17:28:14,103 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/copyMessage.action] onto handler '/copyMessage.acti
on'
2013-07-09 17:28:14,104 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/moveMessage.action] onto handler '/moveMessage.acti
on'
2013-07-09 17:28:14,105 [ost-startStop-1] INFO  ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteJob.action] onto handler '/deleteJob.action'
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\docs
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\examples
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\host-manager
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\manager
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\ROOT
Jul 9, 2013 5:28:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Jul 9, 2013 5:28:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6642 ms

Let's access the web console: http://localhost:8080/activemq-web-console-5.8.0
Use admin/admin for username/password if prompted. These are default security settings. Refer jetty.xml and jetty-realm.properties in conf directory.

As shown above, "broker-1" is the current master broker.

Now, if you were to shutdown broker-1. Hit Control+C on Terminal window running broker-1, you'd notice that broker-2 acquires the lock and becomes the master.

INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | KahaDB is version 4
INFO | Recovering from the journal ...
INFO | Recovery replayed 2 operations from the journal in 0.022 seconds.
INFO | Apache ActiveMQ 5.8.0 (broker-2, ID:AKUNTAMU-1-28147-1373409767675-0:1) is starting
INFO | Listening for connections at: tcp://AKUNTAMU-1:61626?maximumConnections=1000&wireformat.maxFrameSize=104857600
INFO | Connector openwire Started
INFO | Apache ActiveMQ 5.8.0 (broker-2, ID:AKUNTAMU-1-28147-1373409767675-0:1) started
INFO | For help or more information please see: http://activemq.apache.org
WARN | Store limit is 102400 mb, whilst the data directory: C:\apache-activemq-5.8.0\data\kahadb only has 38888 mb of usable space
ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: C:\apache-activemq-5.8.0\data\broker-2\tmp_storage only has 38888 mb o
f usable space
INFO | Web console type: embedded
INFO | ActiveMQ WebConsole initialized.
INFO | Initializing Spring FrameworkServlet 'dispatcher'
INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed

Let's refresh the web console

As you may observe, now the master node is broker-2.

Now that we have seen seamless switch over between the nodes during a failover scenario using an external web console, let's see the same from the perspective of a message producer and consumer.

I will publish 50 persistent messages to a queue using a simple message publisher and have an asynchronous consumer receive those 50 messages. I will introduce a bit of a delay in sending the messages so we can take the master down a few times and create a few failover scenarios. The objective is to see how the failover protocol makes the node failover completely transparent and thus shields the application from having to deal with any reconnection logic.

Here is a simple producer using ActiveMQConnectionFactory with failover protocol. Notice the highlighted failover protocol URL in the code snippet below.

package com.akuntamukkala.amqms;

import javax.jms.Connection;
import javax.jms.DeliveryMode;
import javax.jms.Destination;
import javax.jms.MessageProducer;
import javax.jms.Session;
import javax.jms.TextMessage;

import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.log4j.Logger;


public class Producer {

	private static final Logger	log = Logger.getLogger(Producer.class);
	
	public static void main(String[] args) throws Exception {
		// Create a ConnectionFactory
		ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
				"failover:(tcp://localhost:61616,tcp://localhost:61626)");

		for (int i = 0; i < 50; i++) {
			
			log.info("Establishing connection");
			// Create a Connection
			Connection connection = connectionFactory.createConnection();
			connection.start();
			
			log.info("Connection established");
			
			// Create a Session
			Session session = connection.createSession(false,
					Session.AUTO_ACKNOWLEDGE);

			// Create the destination (Topic or Queue)
			Destination destination = session.createQueue("TEST.FOO");

			// Create a MessageProducer from the Session to the Topic or Queue
			MessageProducer producer = session.createProducer(destination);
			producer.setDeliveryMode(DeliveryMode.PERSISTENT);

			// Create a messages
			String text = "Message Counter : " + i;
			TextMessage message = session.createTextMessage(text);

			log.info("Sending message : " + text);
			producer.send(message);
			log.info("Sent message : " + text);
			// Clean up
			session.close();
			connection.close();

			Thread.sleep(1000);
		}
	}
}

Here are the logs from execution while I shutdown broker-1 and broker-2 alternatively a few times.

2013-07-10 11:26:32 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:33 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:33 INFO  Producer:30 - Connection established
2013-07-10 11:26:33 INFO  Producer:47 - Sending message : Message Counter : 0
2013-07-10 11:26:33 INFO  Producer:49 - Sent message : Message Counter : 0
2013-07-10 11:26:34 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:35 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:35 INFO  Producer:30 - Connection established
2013-07-10 11:26:35 INFO  Producer:47 - Sending message : Message Counter : 1
2013-07-10 11:26:35 INFO  Producer:49 - Sent message : Message Counter : 1
2013-07-10 11:26:35 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:36 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:36 INFO  Producer:30 - Connection established
2013-07-10 11:26:36 INFO  Producer:47 - Sending message : Message Counter : 2
2013-07-10 11:26:36 INFO  Producer:49 - Sent message : Message Counter : 2
2013-07-10 11:26:37 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:37 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:37 INFO  Producer:30 - Connection established
2013-07-10 11:26:37 INFO  Producer:47 - Sending message : Message Counter : 3
2013-07-10 11:26:37 INFO  Producer:49 - Sent message : Message Counter : 3
2013-07-10 11:26:37 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:38 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:38 INFO  Producer:30 - Connection established
2013-07-10 11:26:38 INFO  Producer:47 - Sending message : Message Counter : 4
2013-07-10 11:26:38 INFO  Producer:49 - Sent message : Message Counter : 4
2013-07-10 11:26:39 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:39 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:39 INFO  Producer:30 - Connection established
2013-07-10 11:26:39 INFO  Producer:47 - Sending message : Message Counter : 5
2013-07-10 11:26:39 INFO  Producer:49 - Sent message : Message Counter : 5
2013-07-10 11:26:39 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:39 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:39 INFO  Producer:30 - Connection established
2013-07-10 11:26:39 INFO  Producer:47 - Sending message : Message Counter : 6
2013-07-10 11:26:39 INFO  Producer:49 - Sent message : Message Counter : 6
2013-07-10 11:26:40 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:40 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:40 INFO  Producer:30 - Connection established
2013-07-10 11:26:40 INFO  Producer:47 - Sending message : Message Counter : 7
2013-07-10 11:26:40 INFO  Producer:49 - Sent message : Message Counter : 7
2013-07-10 11:26:40 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:40 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:40 INFO  Producer:30 - Connection established
2013-07-10 11:26:40 INFO  Producer:47 - Sending message : Message Counter : 8
2013-07-10 11:26:40 INFO  Producer:49 - Sent message : Message Counter : 8
2013-07-10 11:26:41 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:41 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:41 INFO  Producer:30 - Connection established
2013-07-10 11:26:41 INFO  Producer:47 - Sending message : Message Counter : 9
2013-07-10 11:26:41 INFO  Producer:49 - Sent message : Message Counter : 9
2013-07-10 11:26:41 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:41 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:41 INFO  Producer:30 - Connection established
2013-07-10 11:26:41 INFO  Producer:47 - Sending message : Message Counter : 10
2013-07-10 11:26:41 INFO  Producer:49 - Sent message : Message Counter : 10
2013-07-10 11:26:42 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:43 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:43 INFO  Producer:30 - Connection established
2013-07-10 11:26:43 INFO  Producer:47 - Sending message : Message Counter : 11
2013-07-10 11:26:43 INFO  Producer:49 - Sent message : Message Counter : 11
2013-07-10 11:26:43 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:43 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:43 WARN  FailoverTransport:255 - Transport (tcp://127.0.0.1:61616) failed, reason:  java.net.SocketException: Software caused connection abort: recv failed, attempting to automatically reconnect
2013-07-10 11:26:50 INFO  FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61626
2013-07-10 11:26:50 INFO  Producer:30 - Connection established
2013-07-10 11:26:50 INFO  Producer:47 - Sending message : Message Counter : 12
2013-07-10 11:26:50 INFO  Producer:49 - Sent message : Message Counter : 12
2013-07-10 11:26:51 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:52 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:52 INFO  Producer:30 - Connection established
2013-07-10 11:26:52 INFO  Producer:47 - Sending message : Message Counter : 13
2013-07-10 11:26:52 INFO  Producer:49 - Sent message : Message Counter : 13
2013-07-10 11:26:52 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:53 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:53 INFO  Producer:30 - Connection established
2013-07-10 11:26:53 INFO  Producer:47 - Sending message : Message Counter : 14
2013-07-10 11:26:53 INFO  Producer:49 - Sent message : Message Counter : 14
2013-07-10 11:26:54 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:54 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:54 INFO  Producer:30 - Connection established
2013-07-10 11:26:54 INFO  Producer:47 - Sending message : Message Counter : 15
2013-07-10 11:26:54 INFO  Producer:49 - Sent message : Message Counter : 15
2013-07-10 11:26:54 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:55 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:55 INFO  Producer:30 - Connection established
2013-07-10 11:26:55 INFO  Producer:47 - Sending message : Message Counter : 16
2013-07-10 11:26:55 INFO  Producer:49 - Sent message : Message Counter : 16
2013-07-10 11:26:56 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:57 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:57 INFO  Producer:30 - Connection established
2013-07-10 11:26:57 INFO  Producer:47 - Sending message : Message Counter : 17
2013-07-10 11:26:57 INFO  Producer:49 - Sent message : Message Counter : 17
2013-07-10 11:26:57 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:58 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:58 INFO  Producer:30 - Connection established
2013-07-10 11:26:58 INFO  Producer:47 - Sending message : Message Counter : 18
2013-07-10 11:26:58 INFO  Producer:49 - Sent message : Message Counter : 18
2013-07-10 11:26:59 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:59 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:59 INFO  Producer:30 - Connection established
2013-07-10 11:26:59 INFO  Producer:47 - Sending message : Message Counter : 19
2013-07-10 11:26:59 INFO  Producer:49 - Sent message : Message Counter : 19
2013-07-10 11:26:59 INFO  Producer:25 - Establishing connection
2013-07-10 11:26:59 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:59 INFO  Producer:30 - Connection established
2013-07-10 11:26:59 INFO  Producer:47 - Sending message : Message Counter : 20
2013-07-10 11:26:59 INFO  Producer:49 - Sent message : Message Counter : 20
2013-07-10 11:27:00 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:01 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:01 INFO  Producer:30 - Connection established
2013-07-10 11:27:01 INFO  Producer:47 - Sending message : Message Counter : 21
2013-07-10 11:27:01 INFO  Producer:49 - Sent message : Message Counter : 21
2013-07-10 11:27:01 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:01 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:01 INFO  Producer:30 - Connection established
2013-07-10 11:27:01 INFO  Producer:47 - Sending message : Message Counter : 22
2013-07-10 11:27:01 INFO  Producer:49 - Sent message : Message Counter : 22
2013-07-10 11:27:02 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:02 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:02 INFO  Producer:30 - Connection established
2013-07-10 11:27:02 INFO  Producer:47 - Sending message : Message Counter : 23
2013-07-10 11:27:02 INFO  Producer:49 - Sent message : Message Counter : 23
2013-07-10 11:27:02 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:02 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:02 INFO  Producer:30 - Connection established
2013-07-10 11:27:02 INFO  Producer:47 - Sending message : Message Counter : 24
2013-07-10 11:27:02 INFO  Producer:49 - Sent message : Message Counter : 24
2013-07-10 11:27:03 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:03 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:03 INFO  Producer:30 - Connection established
2013-07-10 11:27:03 INFO  Producer:47 - Sending message : Message Counter : 25
2013-07-10 11:27:03 INFO  Producer:49 - Sent message : Message Counter : 25
2013-07-10 11:27:03 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:03 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:03 INFO  Producer:30 - Connection established
2013-07-10 11:27:03 INFO  Producer:47 - Sending message : Message Counter : 26
2013-07-10 11:27:03 INFO  Producer:49 - Sent message : Message Counter : 26
2013-07-10 11:27:04 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:04 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:06 WARN  FailoverTransport:255 - Transport (tcp://127.0.0.1:61626) failed, reason:  java.net.SocketException: Software caused connection abort: recv failed, attempting to automatically reconnect
2013-07-10 11:27:15 INFO  FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61616
2013-07-10 11:27:15 INFO  Producer:30 - Connection established
2013-07-10 11:27:15 INFO  Producer:47 - Sending message : Message Counter : 27
2013-07-10 11:27:15 INFO  Producer:49 - Sent message : Message Counter : 27
2013-07-10 11:27:16 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:17 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:17 INFO  Producer:30 - Connection established
2013-07-10 11:27:17 INFO  Producer:47 - Sending message : Message Counter : 28
2013-07-10 11:27:17 INFO  Producer:49 - Sent message : Message Counter : 28
2013-07-10 11:27:17 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:18 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:18 INFO  Producer:30 - Connection established
2013-07-10 11:27:18 INFO  Producer:47 - Sending message : Message Counter : 29
2013-07-10 11:27:18 INFO  Producer:49 - Sent message : Message Counter : 29
2013-07-10 11:27:19 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:20 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:20 INFO  Producer:30 - Connection established
2013-07-10 11:27:20 INFO  Producer:47 - Sending message : Message Counter : 30
2013-07-10 11:27:20 INFO  Producer:49 - Sent message : Message Counter : 30
2013-07-10 11:27:20 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:20 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:20 INFO  Producer:30 - Connection established
2013-07-10 11:27:20 INFO  Producer:47 - Sending message : Message Counter : 31
2013-07-10 11:27:20 INFO  Producer:49 - Sent message : Message Counter : 31
2013-07-10 11:27:21 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:22 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:22 INFO  Producer:30 - Connection established
2013-07-10 11:27:22 INFO  Producer:47 - Sending message : Message Counter : 32
2013-07-10 11:27:22 INFO  Producer:49 - Sent message : Message Counter : 32
2013-07-10 11:27:22 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:22 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:22 INFO  Producer:30 - Connection established
2013-07-10 11:27:22 INFO  Producer:47 - Sending message : Message Counter : 33
2013-07-10 11:27:22 INFO  Producer:49 - Sent message : Message Counter : 33
2013-07-10 11:27:23 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:23 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:23 INFO  Producer:30 - Connection established
2013-07-10 11:27:23 INFO  Producer:47 - Sending message : Message Counter : 34
2013-07-10 11:27:23 INFO  Producer:49 - Sent message : Message Counter : 34
2013-07-10 11:27:23 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:24 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:24 INFO  Producer:30 - Connection established
2013-07-10 11:27:24 INFO  Producer:47 - Sending message : Message Counter : 35
2013-07-10 11:27:24 INFO  Producer:49 - Sent message : Message Counter : 35
2013-07-10 11:27:25 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:27 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:27 INFO  Producer:30 - Connection established
2013-07-10 11:27:27 INFO  Producer:47 - Sending message : Message Counter : 36
2013-07-10 11:27:27 INFO  Producer:49 - Sent message : Message Counter : 36
2013-07-10 11:27:27 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:28 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:28 INFO  Producer:30 - Connection established
2013-07-10 11:27:28 INFO  Producer:47 - Sending message : Message Counter : 37
2013-07-10 11:27:28 INFO  Producer:49 - Sent message : Message Counter : 37
2013-07-10 11:27:29 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:30 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:30 INFO  Producer:30 - Connection established
2013-07-10 11:27:30 INFO  Producer:47 - Sending message : Message Counter : 38
2013-07-10 11:27:30 INFO  Producer:49 - Sent message : Message Counter : 38
2013-07-10 11:27:30 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:31 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:33 WARN  FailoverTransport:255 - Transport (tcp://127.0.0.1:61616) failed, reason:  java.net.SocketException: Software caused connection abort: recv failed, attempting to automatically reconnect
2013-07-10 11:27:38 INFO  FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61626
2013-07-10 11:27:38 INFO  Producer:30 - Connection established
2013-07-10 11:27:38 INFO  Producer:47 - Sending message : Message Counter : 39
2013-07-10 11:27:38 INFO  Producer:49 - Sent message : Message Counter : 39
2013-07-10 11:27:39 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:39 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:39 INFO  Producer:30 - Connection established
2013-07-10 11:27:39 INFO  Producer:47 - Sending message : Message Counter : 40
2013-07-10 11:27:39 INFO  Producer:49 - Sent message : Message Counter : 40
2013-07-10 11:27:39 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:40 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:40 INFO  Producer:30 - Connection established
2013-07-10 11:27:40 INFO  Producer:47 - Sending message : Message Counter : 41
2013-07-10 11:27:40 INFO  Producer:49 - Sent message : Message Counter : 41
2013-07-10 11:27:41 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:42 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:42 INFO  Producer:30 - Connection established
2013-07-10 11:27:42 INFO  Producer:47 - Sending message : Message Counter : 42
2013-07-10 11:27:42 INFO  Producer:49 - Sent message : Message Counter : 42
2013-07-10 11:27:42 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:42 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:42 INFO  Producer:30 - Connection established
2013-07-10 11:27:42 INFO  Producer:47 - Sending message : Message Counter : 43
2013-07-10 11:27:42 INFO  Producer:49 - Sent message : Message Counter : 43
2013-07-10 11:27:43 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:44 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:44 INFO  Producer:30 - Connection established
2013-07-10 11:27:44 INFO  Producer:47 - Sending message : Message Counter : 44
2013-07-10 11:27:44 INFO  Producer:49 - Sent message : Message Counter : 44
2013-07-10 11:27:44 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:45 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:45 INFO  Producer:30 - Connection established
2013-07-10 11:27:45 INFO  Producer:47 - Sending message : Message Counter : 45
2013-07-10 11:27:45 INFO  Producer:49 - Sent message : Message Counter : 45
2013-07-10 11:27:46 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:47 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:47 INFO  Producer:30 - Connection established
2013-07-10 11:27:47 INFO  Producer:47 - Sending message : Message Counter : 46
2013-07-10 11:27:47 INFO  Producer:49 - Sent message : Message Counter : 46
2013-07-10 11:27:48 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:48 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:48 INFO  Producer:30 - Connection established
2013-07-10 11:27:48 INFO  Producer:47 - Sending message : Message Counter : 47
2013-07-10 11:27:48 INFO  Producer:49 - Sent message : Message Counter : 47
2013-07-10 11:27:48 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:49 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:49 INFO  Producer:30 - Connection established
2013-07-10 11:27:49 INFO  Producer:47 - Sending message : Message Counter : 48
2013-07-10 11:27:49 INFO  Producer:49 - Sent message : Message Counter : 48
2013-07-10 11:27:50 INFO  Producer:25 - Establishing connection
2013-07-10 11:27:51 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:51 INFO  Producer:30 - Connection established
2013-07-10 11:27:51 INFO  Producer:47 - Sending message : Message Counter : 49
2013-07-10 11:27:51 INFO  Producer:49 - Sent message : Message Counter : 49

Here is the web console showing 50 messages enqueued.

Don't bother the #Messages Enqueued = 0 in the above screenshot. This count represents the number of messages enqueued since this node started. Since I had restarted the node after the 50 messages were already enqueued, the count shows as 0.

Now lets try to consume these messages using an asynchronous consumer using ActiveMQ connection factory with failover protocol.

Here is simple asynchronous consumer with ActiveMQConnectionFactory using failover protocol.

package com.akuntamukkala.amqms;

import javax.jms.Connection;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;

import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.log4j.Logger;

public class Consumer implements MessageListener {

	private static final Logger log = Logger.getLogger(Consumer.class);

	public static void main(String[] args) throws Exception {

		ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
				"failover:(tcp://localhost:61616,tcp://localhost:61626)");

		// Create a Connection
		Connection connection = connectionFactory.createConnection();
		connection.start();

		// Create a Session
		Session session = connection.createSession(false,
				Session.AUTO_ACKNOWLEDGE);

		// Create the destination (Topic or Queue)
		Destination destination = session.createQueue("TEST.FOO");

		// Create a MessageConsumer from the Session to the Queue
		MessageConsumer consumer = session.createConsumer(destination);
		consumer.setMessageListener(new Consumer()); // asynchronous listener
		Thread.sleep(120000); // long wait to keep program running
		consumer.close();
		session.close();
		connection.close();

	}

	/**
	 * asynchronous message listener
	 */
	public void onMessage(Message message) {
		try {
			log.info(((TextMessage) message).getText());
			Thread.sleep(500);
		} catch (JMSException e) {
			log.error(e);
		} catch (InterruptedException e) {
			log.error(e);
		}
	}

}

Here are the logs from the execution. I switched the master nodes a few times as evident in the logs.

2013-07-10 11:46:01 INFO  FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:46:01 INFO  Consumer:49 - Message Counter : 0
2013-07-10 11:46:02 INFO  Consumer:49 - Message Counter : 1
2013-07-10 11:46:02 INFO  Consumer:49 - Message Counter : 2
2013-07-10 11:46:03 INFO  Consumer:49 - Message Counter : 3
2013-07-10 11:46:03 INFO  Consumer:49 - Message Counter : 4
2013-07-10 11:46:04 INFO  Consumer:49 - Message Counter : 5
2013-07-10 11:46:04 INFO  Consumer:49 - Message Counter : 6
2013-07-10 11:46:05 INFO  Consumer:49 - Message Counter : 7
2013-07-10 11:46:05 INFO  Consumer:49 - Message Counter : 8
2013-07-10 11:46:06 INFO  Consumer:49 - Message Counter : 9
2013-07-10 11:46:06 INFO  Consumer:49 - Message Counter : 10
2013-07-10 11:46:07 INFO  Consumer:49 - Message Counter : 11
2013-07-10 11:46:07 WARN  FailoverTransport:255 - Transport (tcp://127.0.0.1:61616) failed, reason:  java.io.EOFException, attempting to automatically reconnect
2013-07-10 11:46:18 INFO  FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61626
2013-07-10 11:46:18 WARN  ActiveMQMessageConsumer:1348 - Duplicate dispatch on connection: ID:AKUNTAMU-1-2141-1373474760280-1:1 to consumer: ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, ignoring (auto acking) duplicate: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, destination = queue://TEST.FOO, message = ActiveMQTextMessage {commandId = 5, responseRequired = true, messageId = ID:AKUNTAMU-1-1739-1373473592152-1:12:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:AKUNTAMU-1-1739-1373473592152-1:12:1:1, destination = queue://TEST.FOO, transactionId = null, expiration = 0, timestamp = 1373473603281, arrival = 0, brokerInTime = 1373473603281, brokerOutTime = 1373474778676, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false, text = Message Counter : 11}, redeliveryCounter = 0}
2013-07-10 11:46:18 INFO  Consumer:49 - Message Counter : 12
2013-07-10 11:46:19 INFO  Consumer:49 - Message Counter : 13
2013-07-10 11:46:19 INFO  Consumer:49 - Message Counter : 14
2013-07-10 11:46:20 INFO  Consumer:49 - Message Counter : 15
2013-07-10 11:46:20 INFO  Consumer:49 - Message Counter : 16
2013-07-10 11:46:21 INFO  Consumer:49 - Message Counter : 17
2013-07-10 11:46:21 INFO  Consumer:49 - Message Counter : 18
2013-07-10 11:46:22 INFO  Consumer:49 - Message Counter : 19
2013-07-10 11:46:22 INFO  Consumer:49 - Message Counter : 20
2013-07-10 11:46:23 INFO  Consumer:49 - Message Counter : 21
2013-07-10 11:46:23 INFO  Consumer:49 - Message Counter : 22
2013-07-10 11:46:24 INFO  Consumer:49 - Message Counter : 23
2013-07-10 11:46:24 INFO  Consumer:49 - Message Counter : 24
2013-07-10 11:46:25 INFO  Consumer:49 - Message Counter : 25
2013-07-10 11:46:25 INFO  Consumer:49 - Message Counter : 26
2013-07-10 11:46:26 INFO  Consumer:49 - Message Counter : 27
2013-07-10 11:46:26 INFO  Consumer:49 - Message Counter : 28
2013-07-10 11:46:27 INFO  Consumer:49 - Message Counter : 29
2013-07-10 11:46:27 INFO  Consumer:49 - Message Counter : 30
2013-07-10 11:46:28 INFO  Consumer:49 - Message Counter : 31
2013-07-10 11:46:28 INFO  Consumer:49 - Message Counter : 32
2013-07-10 11:46:29 INFO  Consumer:49 - Message Counter : 33
2013-07-10 11:46:29 INFO  Consumer:49 - Message Counter : 34
2013-07-10 11:46:30 INFO  Consumer:49 - Message Counter : 35
2013-07-10 11:46:30 INFO  Consumer:49 - Message Counter : 36
2013-07-10 11:46:31 INFO  Consumer:49 - Message Counter : 37
2013-07-10 11:46:31 INFO  Consumer:49 - Message Counter : 38
2013-07-10 11:46:32 INFO  Consumer:49 - Message Counter : 39
2013-07-10 11:46:32 WARN  FailoverTransport:255 - Transport (tcp://127.0.0.1:61626) failed, reason:  java.io.EOFException, attempting to automatically reconnect
2013-07-10 11:46:43 INFO  FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61616
2013-07-10 11:46:43 WARN  ActiveMQMessageConsumer:1348 - Duplicate dispatch on connection: ID:AKUNTAMU-1-2141-1373474760280-1:1 to consumer: ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, ignoring (auto acking) duplicate: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, destination = queue://TEST.FOO, message = ActiveMQTextMessage {commandId = 5, responseRequired = true, messageId = ID:AKUNTAMU-1-1739-1373473592152-1:40:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:AKUNTAMU-1-1739-1373473592152-1:40:1:1, destination = queue://TEST.FOO, transactionId = null, expiration = 0, timestamp = 1373473658595, arrival = 0, brokerInTime = 1373473658599, brokerOutTime = 1373474803745, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false, text = Message Counter : 39}, redeliveryCounter = 0}
2013-07-10 11:46:43 INFO  Consumer:49 - Message Counter : 40
2013-07-10 11:46:44 INFO  Consumer:49 - Message Counter : 41
2013-07-10 11:46:44 INFO  Consumer:49 - Message Counter : 42
2013-07-10 11:46:45 INFO  Consumer:49 - Message Counter : 43
2013-07-10 11:46:45 INFO  Consumer:49 - Message Counter : 44
2013-07-10 11:46:46 INFO  Consumer:49 - Message Counter : 45
2013-07-10 11:46:46 INFO  Consumer:49 - Message Counter : 46
2013-07-10 11:46:47 INFO  Consumer:49 - Message Counter : 47
2013-07-10 11:46:47 INFO  Consumer:49 - Message Counter : 48
2013-07-10 11:46:48 INFO  Consumer:49 - Message Counter : 49

Let's check out the ActiveMQ web console:

You may find the # Messages Dequeued = 11 very interesting. This is the number of messages dequeued from the current ActiveMQ master node since it started.

Conclusion:

We have thus seen the following in action:

Run 2 ActiveMQ nodes in a Master/Slave configuration with a shared KahaDB file based database.
Configure ActiveMQ web console hosted in a Tomcat instance to point to whichever node is Master node in the cluster
Failover scenario
Message publisher and consumer behavior oblivious to the failover

In future blogs, I will post some other interesting ActiveMQ configurations. Stay tuned.

Happy ActiveMQ'ing!

References: