Wednesday, March 26, 2014

ActiveMQ - Network of Brokers Explained - Part 4

In the previous part 3 , we have seen how ActiveMQ helps distinguish remote consumers from local consumers which helps in determining shorter routes from message producers to consumers.

In this part 4,  we will look into how to load balance concurrent consumers on remote brokers.

Let’s consider a bit more advanced configuration to load balance concurrent message consumers on a queue in remote brokers as shown below. 

Part 4 - Network of brokers 

In the above configuration, we have a message producer sending messages into a queue moo.bar on broker-1. Broker-1 establishes network connectors to broker-2 and broker-3. Consumer C1 consumes messages from queue moo.bar on broker-2 while consumers C2 and C3 are concurrent consumers on queue moo.bar on broker-3. 

Let's see this in action


Let's create three brokers instances...
  1. Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
    /Users/akuntamukkala/apache-activemq-5.8.0/bin

  2. Ashwinis-MacBook-Pro:bin akuntamukkala$./activemq-admin create ../cluster/broker-1

  3. Ashwinis-MacBook-Pro:bin akuntamukkala$./activemq-admin create ../cluster/broker-2

  4. Ashwinis-MacBook-Pro:bin akuntamukkala$./activemq-admin create ../cluster/broker-3

  5. Fix the broker-2 and broker-3 transport, amqp connectors and jetty http port by modifying the corresponding conf/activemq.xml and conf/jetty.xml as follows:

    BrokerOpenwire PortJetty HTTP PortAMQP Port
    broker-1
    61616
    8161
    5672
    broker-2
    61626
    9161
    5682
    broker-3
    61636
    10161
    5692


  6. Fix network connector on broker-1 such that messages on queues can be forwarded dynamically to consumers on broker-2 and broker-3. This can be done by adding the following XML snippet into broker-1's conf/activemq.xml

    <networkConnectors>

        
    <networkConnector
    1.   name="Q:broker1->broker2"
        uri="static:(tcp://localhost:61626)"
        duplex="false"
        decreaseNetworkConsumerPriority="true"
        networkTTL="2"
        dynamicOnly="true">
        <excludedDestinations>
           <topic physicalName="&gt;" />
        </excludedDestinations>
      </networkConnector>

      <networkConnector
         name="Q:broker1->broker3"
         uri="static:(tcp://localhost:61636)"
         duplex="false"
         decreaseNetworkConsumerPriority="true"
         networkTTL="2"
         dynamicOnly="true">
         <excludedDestinations>
              <topic physicalName="&gt;" />
         </excludedDestinations>
      </networkConnector>
    </networkConnectors>

  7.  Start broker-2, broker-3 and broker-1. We can start these in any order.
    1. /apache-activemq-5.8.0/cluster/broker-3/bin$ ./broker-3 console
    2. /apache-activemq-5.8.0/cluster/broker-2/bin$ ./broker-2 console
    3. /apache-activemq-5.8.0/cluster/broker-1/bin$ ./broker-1 console
  8. Let's start the consumers C1 on broker-2 and C2, C3 on broker-3 but on the same queue called "moo.bar"
    1. /apache-activemq-5.8.0/example$ ant consumer -Durl=tcp://localhost:61626 -Dsubject=moo.bar
    2. /apache-activemq-5.8.0/example$ ant consumer -Durl=tcp://localhost:61636 -Dsubject=moo.bar -DparallelThreads=2

      The consumer subscriptions are forwarded by broker-2 and broker-3 to their neighboring broker-1 which has a network connector established to both broker-2 and broker-3 by the use of advisory messages. 
  9. Let's review the broker web consoles to see the queues and corresponding consumers. 
    1. We find that broker-2's web console shows one queue "moo.bar" having 1 consumer, broker-3's web console shows one queue "moo.bar" having 2 concurrent consumers
    2. Though there are three consumers (C1 on broker-2 and C2,C3 on broker-3), broker-1 sees only two consumers (representing broker-2 and broker-3).
    3. http://localhost:8161/admin/queues.jsp










    4. This is because the network connector from broker-1 to broker-2 and to broker-3 by default has a property "conduitSubscriptions" which is true. 
      Due to which broker-3's C2 and C3 which consume messages from the same queue "moo.bar" are treated as one consumer in broker-1.
  10. Let's produce 30 messages into broker-1's queue moo.bar and see how the messages are divvied among the consumers C1, C2 and C3
Shows how the messages were propagated from producer to consumers C1, C2, C3
As seen above, even though there were three consumers and 30 messages, they didn't get to process 10 messages each as C2, C3 subscriptions were consolidated into one consumer at broker-1. 

conduitSubscriptions="true" is a useful setting if we were creating subscribers on topics as that would prevent duplicate messages. More on this in part 5.

So, in order to make C2 and C3 subscriptions on queue moo.bar propagate to broker-1, let's redo the same steps 6, 7, 8, 9 and 10 after setting conduitSubscriptions="false" in broker-1's network connector configuration in conf/activemq.xml 

Here is the new network connector configuration snippet for broker-1:

<networkConnectors>
  <networkConnector
    name="Q:broker1->broker2"
    uri="static:(tcp://localhost:61626)"
    duplex="false"
    decreaseNetworkConsumerPriority="true"
    networkTTL="2"
    conduitSubscriptions="false"
    dynamicOnly="true">
    <excludedDestinations>
       <topic physicalName="&gt;" />
    </excludedDestinations>
  </networkConnector>
  <networkConnector
    name="Q:broker1->broker3"
    uri="static:(tcp://localhost:61636)"
    duplex="false"
    decreaseNetworkConsumerPriority="true"
    networkTTL="2"
    conduitSubscriptions="false"
    dynamicOnly="true">
    <excludedDestinations>
       <topic physicalName="&gt;" />
    </excludedDestinations>
  </networkConnector>
</networkConnectors>

Upon restarting the brokers and consumers C1, C2 and C3 and producing 30 messages into broker-1's moo.bar queue, we find that all of the three consumer subscriptions are visible at broker-1. As a result broker-1 dispatches 10 messages to each of the consumers in a round-robin fashion to load balance. This is depicted pictorially below. 

Shows how the messages were propagated from producer to consumers C1, C2, C3
Broker-1's web console @ http://localhost:8161/admin/queueConsumers.jsp?JMSDestination=moo.bar shows that broker-1 now sees 3 consumers and dispatches 10 messages to each consumer



Thus in this part 4 of the blog series, we have seen how we can load balance remote concurrent consumers which are consuming messages from a queue. 

As always, your comments and feedback is appreciated!

In the next part 5, we will explore how the same scenario will play out if we were to use a topic instead of a queue. Stay tuned...

References

Resources

  • The configuration files (activemq.xml and jetty.xml) of all the brokers used in this blog are available here.

Tuesday, March 4, 2014

ActiveMQ - Network of Brokers Explained - Part 3

Now that we have understood the basics of ActiveMQ network connector in part 1 and part 2 of this blog series, in this part 3, we will examine how ActiveMQ load balances consumers which connect to a network of brokers.

Introduction

Concurrent consumers are used when messages in a queue can be processed out of order and usually to improve message throughput. ActiveMQ broker dispatches messages in a round-robin fashion among the consumers in order to load balance message consumption across concurrent consumers unless the consumer is specified as exclusive

Let's see the following example where three consumers are concurrently processing messages from queue foo.bar. A producer enqueues 60 messages which are processed by three consumers (20 each) in a round robin fashion.

Start three concurrent consumers on queue foo.bar

Ashwinis-MacBook-Pro:example akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/example


Ashwinis-MacBook-Pro:example akuntamukkala$ ant consumer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -DparallelThreads=3 -Dmax=20

Produce 60 messages 

Ashwinis-MacBook-Pro:example akuntamukkala$ ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Dmax=60

The following screenshot shows 3 consumers processing messages from queue foo.bar. 60 messages were enqueued and dequeued. 


As shown below 20 messages were processed by each of the consumers. 


The following excerpt from log shows that messages are divvied out among three consumers...

[java] [Thread-3] Received: 'Message: 1 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-2] Received: 'Message: 0 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-1] Received: 'Message: 2 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-3] Received: 'Message: 4 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-2] Received: 'Message: 3 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-1] Received: 'Message: 5 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-3] Received: 'Message: 7 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-2] Received: 'Message: 6 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)
[java] [Thread-1] Received: 'Message: 8 sent at: Tue Mar 04 13:46:53 IST 2014  ...' (length 1000)

[java] [Thread-3] Received: 'Message: 10 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)

Now that we have seen how concurrent consumers work on a single broker, we will now examine how they work when consumers are spread across network of brokers. 

Local Vs Remote Consumers

Let's explore how ActiveMQ handles local and remote consumers with the help of a configuration shown in the figure below. 



Consumer-1 and Consumer-2 consume messages from queue foo.bar on Broker-1 and Broker-2 respectively. Broker-1 established a network connector to Broker-2 to forward queue messages. Producer enqueues messages into queue foo.bar on Broker-1

Let's see this in action

  • Edit Broker-1's configuration /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml and open a network connector to Broker-2 and restart Broker-1 and Broker-2

        <networkConnectors>
                <networkConnector
                        name="T:broker1->broker2"
                        uri="static:(tcp://localhost:61626)"
                        duplex="false"
                        decreaseNetworkConsumerPriority="false"
                        networkTTL="2"
                        dynamicOnly="true">
                        <excludedDestinations>
                                <queue physicalName="&gt;" />
                        </excludedDestinations>
                </networkConnector>
                <networkConnector
                        name="Q:broker1->broker2"
                        uri="static:(tcp://localhost:61626)"
                        duplex="false"
                        decreaseNetworkConsumerPriority="false"
                        networkTTL="2"
                        dynamicOnly="true">
                        <excludedDestinations>
                                <topic physicalName="&gt;" />
                        </excludedDestinations>
                </networkConnector>
        </networkConnectors>
  • Start local consumer, Consumer-1

Ashwinis-MacBook-Pro:example akuntamukkala$ ant consumer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar 
  • Start remote consumer, Consumer-2

Ashwinis-MacBook-Pro:example akuntamukkala$ ant consumer -Durl=tcp://localhost:61626 -Dtopic=false -Dsubject=foo.bar
  • Start producer on Broker-1 to enqueue 100 messages
Ashwinis-MacBook-Pro:example akuntamukkala$ ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Dmax=100

Screenshot showing Broker-1's queues:


Let's look at the consumers to see how the messages have been divvied out.


As you may notice, ActiveMQ broker dispatches the messages equally to local consumer over the remote consumer giving them the same priority. 

The remote consumer, Consumer-2 is only broker 1 hop away which is less than configured networkTTL value of 2.

This leads to suboptimal routes especially when brokers are connected such that multiple routes are possible between producers and consumers. It is preferable to dispatch to local consumers over remote consumers in order to ensure shortest path between producers and consumers. 

ActiveMQ provides a way to configure the priority between local consumer and remote consumer using the property decreaseNetworkConsumerPriority on the network connector.

By default, this value is false and hence the local and remote brokers were treated alike. 

If we repeat the above steps after changing the decreaseNetworkConsumerPriority="true" 
then we find that local consumer, Consumer-1 is given preference over remote consumer, Consumer-2 which is 1 broker hop away. 


ActiveMQ intelligently figures out shortest path in a network of brokers between message producers and consumers. 

Please read the following link to gain further understanding of optimal routing by ActiveMQ.
This concludes part 3 of this series where we saw how to differentiate local and remote consumers to assist ActiveMQ determine most optimal path between message producers and consumers. 

As always your comments are very welcome. 

Stay tuned for part 4 where we will go over load balancing remote concurrent consumers...