Monday, February 24, 2014

ActiveMQ - Network of Brokers Explained

Objective

This 7 part blog series is to share about how to create network of ActiveMQ brokers in order to achieve high availability and scalability.

Why network of brokers?


ActiveMQ message broker is a core component of messaging infrastructure in an enterprise. It needs to be highly available and dynamically scalable to facilitate communication between dynamic heterogeneous distributed applications which have varying capacity needs. 

Scaling enterprise applications on commodity hardware is a rage nowadays. ActiveMQ caters to that very well by being able to create a network of brokers to share the load. 

Many times applications running across geographically distributed data centers need to coordinate messages. Running message producers and consumers across geographic regions/data centers can be architected better using network of brokers.  

ActiveMQ uses transport connectors over which it communicates with message producers and consumers. However, in order to facilitate broker to broker communication, ActiveMQ uses network connectors

A network connector is a bridge between two brokers which allows on-demand message forwarding. 

In other words, if Broker B1 initiates a network connector to Broker B2 then the messages on a channel (queue/topic) on B1 get forwarded to B2 if there is at least one consumer on B2 for the same channel. If the network connector was configured to be duplex, the messages get forwarded from B2 to B1 on demand. 

This is very interesting because it is now possible for brokers to communicate with each other dynamically. 
In this 7 part blog series, we will look into the following topics to gain understanding of this very powerful ActiveMQ feature:
    1. Network Connector Basics - Part 1
    2. Duplex network connectors - Part 2 
    3. Load balancing consumers on local/remote brokers - Part 3
    4. Load-balance consumers/subscribers on remote brokers  
      1. Queue: Load balance remote concurrent consumers - Part 4
      2. Topic: Load Balance Durable Subscriptions on Remote Brokers - Part 5
    5. Store/Forward messages and consumer failover  - Part 6
      1. How to prevent stuck  messages
    6. Virtual Destinations - Part 7
To give credit where it is due, the following URLs have helped me in creating this blog post series. 
  1. Advanced Messaging with ActiveMQ by Dejan Bosanac [Slides 32-36] 
  2. Understanding ActiveMQ Broker Networks by Jakub Korab 

Prerequisites

  1. ActiveMQ 5.8.0 – To create broker instances
  2. Apache Ant – To run ActiveMQ sample producer and consumers for demo.
We will use multiple ActiveMQ broker instances on the same machine for the ease of demonstration.

Network Connector Basics - Part 1

The following diagram shows how a network connector functions. It bridges two brokers and is used to forward messages from Broker-1 to Broker-2 on demand if established by Broker-1 to Broker-2. 

A network connector can be duplex so messages could be forwarded in the opposite direction; from Broker-2 to Broker-1, once there is a consumer on Broker-1 for a channel which exists in Broker-2. More on this in Part 2

Setup network connector between broker-1 and broker-2

  • Create two broker instances, say broker-1 and broker-2
Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/bin
Ashwinis-MacBook-Pro:bin akuntamukkala$ ./activemq-admin create ../bridge-demo/broker-1

Ashwinis-MacBook-Pro:bin akuntamukkala$ ./activemq-admin create ../bridge-demo/broker-2

Since we will be running both brokers on the same machine, let's configure broker-2 such that there are no port conflicts. 

  • Edit /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/activemq.xml

    1. Change transport connector to 61626 from 61616
    2. Change AMQP port from 5672 to 6672 (won't be using it for this blog)
  • Edit /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/jetty.xml

    1. Change web console port to 9161 from 8161
  • Configure Network Connector from broker-1 to broker-2
    Add the following XML snippet to 
    /Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml
     <networkConnectors>
         <networkConnector 
            name="T:broker1->broker2" 
            uri="static:(tcp://localhost:61626)" 
            duplex="false" 
            decreaseNetworkConsumerPriority="true" 
            networkTTL="2" 
            dynamicOnly="true">
            <excludedDestinations>
                  <queue physicalName="&gt;" />
            </excludedDestinations>
         </networkConnector>
         <networkConnector 
            name="Q:broker1->broker2
            uri="static:(tcp://localhost:61626)" 
            duplex="false" 
            decreaseNetworkConsumerPriority="true" 
            networkTTL="2" 
            dynamicOnly="true">
            <excludedDestinations>
                  <topic physicalName="&gt;" />
            </excludedDestinations>
         </networkConnector>
     </networkConnectors>

The above XML snippet configures two network connectors "T:broker1->broker2" (only topics as queues are excluded) and "Q:broker1->broker2"  (only queues as topics are excluded). This allows for nice separation between network connectors used for topics and queues. 

The name can be arbitrary although I prefer to specify the [type]:[source broker]->[destination broker].

The URI attribute specifies how to connect to broker-2
  • Start broker-2
Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/bin

Ashwinis-MacBook-Pro:bin akuntamukkala$ ./broker-2 console
  • Start broker-1
Ashwinis-MacBook-Pro:bin akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/bin

Ashwinis-MacBook-Pro:bin akuntamukkala$ ./broker-1 console

Logs on broker-1 show 2 network connectors being established with broker-2

 INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626
 INFO | Connector vm://broker-1 Started
 INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626
 INFO | Network connection between vm://broker-1#24 and tcp://localhost/127.0.0.1:61626@52132(broker-2) has been established.
 INFO | Network connection between vm://broker-1#26 and tcp://localhost/127.0.0.1:61626@52133(broker-2) has been established.

Web Console on broker-1 @ http://localhost:8161/admin/connections.jsp shows the two network connectors established to broker-2



The same on broker-2 does not show any network connectors since no network connectors were initiated by broker-2

Let's see this in action

Let's produce 100 persistent messages on a queue called "foo.bar" on broker-1. 

Ashwinis-MacBook-Pro:example akuntamukkala$ pwd
/Users/akuntamukkala/apache-activemq-5.8.0/example
Ashwinis-MacBook-Pro:example akuntamukkala$ ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100

broker-1 web console shows that 100 messages have been enqueued in queue "foo.bar"

http://localhost:8161/admin/queues.jsp

Let's start a consumer on a queue called "foo.bar" on broker-2. The important thing to note here is that the destination name "foo.bar" should match exactly. 

Ashwinis-MacBook-Pro:example akuntamukkala$ ant consumer -Durl=tcp://localhost:61626 -Dtopic=false -Dsubject=foo.bar 

We find that all the 100 messages from broker-1's foo.bar queue get forwarded to broker-2's foo.bar queue consumer. 

broker-1 admin console at http://localhost:8161/admin/queues.jsp


broker-2 admin console @ http://localhost:9161/admin/queues.jsp shows that the consumer we had started has consumed all 100 messages which were forwarded on-demand from broker-1 

broker-2 consumer details on foo.bar queue


broker-1 admin console shows that all 100 messages have been dequeued [forwarded to broker-2 via the network connector]. 


broker-1 consumer details on "foo.bar" queue shows that the consumer is created on demand: [name of connector]_[destination broker]_inbound_[source broker]


Thus we have seen the basics of network connector in ActiveMQ. 

As always, please feel to comment about anything that can be improved. Your inputs are welcome!

Stay tuned for Part 2...

16 comments:

  1. I have created both brokers,and even after including the changes to activemq.xml and jetty.xml i dont get two different admin pages for each broker what could be the reaseon?

    ReplyDelete
    Replies
    1. Suja:
      I have uploaded activemq.xml and jetty.xml of both broker-1 and broker-2 here.
      https://drive.google.com/folderview?id=0B6UoSlNMfxQOalNrQ1pQc3Raa1k&usp=sharing
      Could you please see if this works for you?
      ~Ashwin

      Delete
    2. Thank you for the files but it still dint work can please review the file at https://drive.google.com/file/d/0B75D5rY8V8pnZlBRUVpHMUh6RlE/edit?usp=sharing
      And let know your thoughts.

      Delete
    3. Suja:
      Could you please post the activemq.log file from both brokers when you start them. I think you may have an issue with path where activemq.xml is being loaded from.

      ACTIVEMQ_HOME: ...
      ACTIVEMQ_BASE: ...
      ACTIVEMQ_CONF: ...
      ACTIVEMQ_DATA: ...
      Loading message broker from: xbean:activemq.xml
      ...

      Those lines may need not be right...If so then fix broker-1.bat and broker-2.bat as shown below.

      broker-1.bat in C:\apache-activemq-5.8.0\cluster\broker-1\bin directory
      set ACTIVEMQ_HOME="C:/apache-activemq-5.8.0"
      set ACTIVEMQ_BASE="C:/apache-activemq-5.8.0/cluster/broker-1"
      set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf
      broker-2.bat in C:\apache-activemq-5.8.0\cluster\broker-2\bin directory
      set ACTIVEMQ_HOME="C:/apache-activemq-5.8.0"
      set ACTIVEMQ_BASE="C:/apache-activemq-5.8.0/cluster/broker-2"
      set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf

      If you still have issue, please post your broker-1.bat and broker-2.bat

      Delete
  2. nice job with the article. Could you increase the font size for the "code" part and also the images? (just a tip )

    ReplyDelete
  3. Could you tell me where I can run such "ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100" command. Built source of ActiveMQ/sample does not include such ant build file.

    ReplyDelete
    Replies
    1. From http://activemq.apache.org/activemq-580-release.html, download the binary zip file. In my case I unzipped this file in /Users/akuntamukkala directory.
      So I can run the ant commands from /Users/akuntamukkala/apache-activemq-5.8.0/example

      You will need to have ant installed on your machine and configure the path variable of your machine in order to be able to run this command.

      Delete
    2. Thanks! I found it. Because I was using 5.9.0, which is different structure in sample directory.

      Delete
  4. Let's say I have 500,000 messages in Broker 1 with 25 consumers connecting to it and another 25 consumers connecting to Broker 2. I didn't see messages moving from Broker 1 to Broker 2 for a quicker consumption so that both brokers having 25 consumers each help the client system draining messages quickly. I believe it works only when Broker 1 has no consumer and Broker 2 having some. What are the gotchas for using Network of Brokers? When exactly it kicks in?

    ReplyDelete
  5. Could you elaborate on the "nice separation between network connectors used for topics and queues"? Is there a performance or monitoring benefit? The TCP endpoint is the same and the traffic should be the same, isn't it?

    ReplyDelete
  6. Hi Ashwini, thats a nice article on network of brokers , I have a question ..do we have any mechanism to get the region broker instance , i am aware that it is possible with the broker service instance and i am trying to get the broker service instance with the help of some parameters ...but its not working to fit my use case ? my use case is getting the session count across the broker , to get the session count i am getting the region broker and region broker is giving the map of connection states , then connection states are giving the session size. so i got stuck into the problem where i am not getting the broker service instance ..i am currently trying to get the broker service instance for one of the dev broker from the local machine ? please post some snippets if you come across any such scenario.

    ReplyDelete
  7. when I ran the ant consumer -Durl=tcp://localhost:61626 on broker 2.
    I see messages moving from Broke-1 , but I don't see the queue consumer on Broker-2.

    ReplyDelete
  8. working 5.13.1 version those ant commands are not working .
    using activemq producer -Durl=tcp://localhost:61616 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100
    however queue is created with TEST rather foo.bar
    Any idea what abt commands related to 5.13.1

    ReplyDelete
  9. Very good. one updates
    add below to broker1.bat and broker2.bat
    set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf
    set ACTIVEMQ_DATA=%ACTIVEMQ_BASE%/data

    ReplyDelete
  10. HI Ashwini very nice Article, what changes need to be done to connect to remote broker.i tried to put remotehostname in tcp://localhost:61616 but it dint work , any more changes required or what , it is working on a single machine though

    ReplyDelete