tag:blogger.com,1999:blog-85588992959691645712024-03-12T21:36:47.604-07:00Ashwini Kuntamukkala - Technology Enthusiast Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-8558899295969164571.post-12294472597336076832017-04-19T00:29:00.000-07:002017-04-19T00:31:01.893-07:00Pros And Cons Of MicroServices Architecture<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">I will be speaking at Global Big Data Conference at Dallas on May 7th 2017 on the following topic. </span></div>
<div style="text-align: left;">
</div>
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Pros And Cons Of MicroServices Architecture</span></li>
</ul>
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Hands On Lab Using Pivotal CloudFoundry And RedHat OpenShift Origin</span></li>
</ul>
<br />
<h3 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Abstract</b></span></h3>
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">For most enterprises, it is not a question of if but when they will execute a viable cloud strategy. When that happens, IT operations teams will have their task cut out but what about application development and delivery teams? </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">The common patterns of on-premise old-school application development and practices become anti-patterns when faced with challenge of developing applications that will run on infrastructure provisioned as code in the cloud. Along with that comes the risk of cloud vendor lock-in. While the popular open source PaaS solutions such as Cloud Foundry and OpenShift prevent IaaS vendor lock-in, solution architects and developers need to up their game to build cloud native applications. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">In this talk, we will go over important considerations from a mindset point of view when building a cloud native applications. We will cover pros and cons of choices you'll make and most importantly, we will do hands on exercises using Pivotal Cloud Foundry and RedHat OpenShift Origin PaaS and how to build cloud native applications!</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Here is the link to this conference schedule</span><br />
<a href="http://globalbigdataconference.com/dallas/big-data-bootcamp/schedule-84.html" target="_blank">http://globalbigdataconference.com/dallas/big-data-bootcamp/schedule-84.html</a><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">See you there!</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<br />
<br /></div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-37767795428678131672016-05-13T22:24:00.002-07:002016-05-13T22:24:59.648-07:00Data Wrangling Examples<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Hands-On Data Wrangling: What, How, and Why</span></h2>
<h3 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Objective:</span></h3>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Companies are finding that data can be a powerful differentiator and are investing heavily in infrastructure, tools and personnel to ingest and curate raw data to be "analyzable". This process of data curation is called "Data Wrangling"</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">This task can be very cumbersome and requires trained personnel. However with the advances in open source and commercial tooling, this process has gotten a lot easier and the technical expertise required to do this effectively has dropped several notches.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">In this tutorial, we will get a feel for what data wranglers do and use the following tools with some </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">hands-on exercises. Given the introductory nature of these examples, the size of data sets is small and does not necessitate HDFS and cluster computing. In the next few installments of this blog series, I intend to add such examples. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Here are the tools you will need to install on your computer:</span><br />
<br />
<table border="1" bordercolor="#0033FF" cellpadding="2" cellspacing="2">
<tbody>
<tr>
<th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Tool</span></th>
<th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Version</span></th>
<th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Download & Install Instructions</span></th>
<th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Type</span></th>
</tr>
<tr>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">R language</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">3.2.4</span></td>
<td><a href="https://cran.rstudio.com/"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">https://cran.rstudio.com/</span></a></td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Open Source</span></td>
</tr>
<tr>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">R Studio</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">0.99.887</span></td>
<td><a href="https://www.rstudio.com/products/rstudio/download/"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">https://www.rstudio.com/products/rstudio/download/</span></a></td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Open Source</span></td>
</tr>
<tr>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">OpenRefine</span></td>
<td><h3 id="openrefine-26" style="color: #333333; line-height: 1.28205em; margin: 12px 0px;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; font-size: small; font-weight: normal;">2.6</span></h3>
</td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="http://openrefine.org/download.html">http://openrefine.org/download.html</a><span id="goog_1467884001"></span><span id="goog_1467884002"></span><a href="https://www.blogger.com/"></a></span></td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Open Source</span></td>
</tr>
<tr>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Trifacta Wrangler</span></td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">3.0.1-client1</span></td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="https://www.trifacta.com/trifacta-wrangler">https://www.trifacta.com/trifacta-wrangler</a><span id="goog_1467884001"></span><span id="goog_1467884002"></span><a href="https://www.blogger.com/"></a></span></td>
<td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Commercial but free offering with limited functionality</span></td>
</tr>
</tbody></table>
<a href="https://2.bp.blogspot.com/-mJbWluNL6BU/VyURr61bQpI/AAAAAAAAAbs/ufMXprl0DUc_toiosCTlmILJyMXvd8QqwCLcB/s1600/caution.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a>
<a href="https://2.bp.blogspot.com/-mJbWluNL6BU/VyURr61bQpI/AAAAAAAAAbs/ufMXprl0DUc_toiosCTlmILJyMXvd8QqwCLcB/s1600/caution.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="50" src="https://2.bp.blogspot.com/-mJbWluNL6BU/VyURr61bQpI/AAAAAAAAAbs/ufMXprl0DUc_toiosCTlmILJyMXvd8QqwCLcB/s200/caution.jpg" width="50" /></a><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i style="color: #0b5394; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: small;"><i style="color: #0b5394; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: small;"><br />By no stretch of imagination is this tutorial supposed to be the end-all and all-inclusive learning experience of data wrangling tools and strategies but it merely scratches the scratch of the plethora of tools we have at our disposal to wrangle data.</i></i></span><br />
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Download Resources</b></span><br />
<br />
<ol style="text-align: left;">
<li><span style="font-family: helvetica neue, arial, helvetica, sans-serif;"><a href="https://docs.google.com/uc?authuser=0&id=0B6UoSlNMfxQOUTRBQkpPUVdaMkU&export=download" target="_blank">Data Wrangling - Slide Deck</a></span></li>
<li><span style="font-family: helvetica neue, arial, helvetica, sans-serif;"><a href="https://docs.google.com/uc?authuser=0&id=0B6UoSlNMfxQON3Q1NXV2QXJmQmc&export=download" target="_blank">Data Wrangling - Exercise Code</a></span></li>
<li><span style="font-family: helvetica neue, arial, helvetica, sans-serif;"><a href="https://docs.google.com/uc?authuser=0&id=0B6UoSlNMfxQOb2ZrOEZ5M1VOZE0&export=download" target="_blank">Transform Legacy Enterprise to Data Driven Business - Slide Deck</a></span></li>
</ol>
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b><br /></b></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Exercise 1: Fix date formats</b></span><br />
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="https://drive.google.com/open?id=0B6UoSlNMfxQOQ09oSjVMc3kwajQ" target="_blank">Raw data file</a></span></li>
</ul>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Solution:</b></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Using R & R Studio</u></i></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 1 Preparation:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Install the following packages</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("stringr")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("dplyr")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("lubridate")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("tidyr")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 1a Solution:</span><br />
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Load the previously installed packages</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(stringr)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(dplyr)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(lubridate)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(tidyr)</span><br />
<div>
<br /></div>
<div class="p1">
<br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Replace this with file name where you want to download the raw file</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">rawfile <- "ReplaceWithFileNameIncludingPath"</span><br />
<br /></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># download the file containing dates </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">download.file("https://docs.google.com/uc?authuser=0&id=0B6UoSlNMfxQOQ09oSjVMc3kwajQ&export=download", destfile = rawfile)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># read the file, since the file does not contain a header, it is false</span></span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">dates <- read.csv(rawfile, header = FALSE, col.names = c("date"))</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># there should be 22</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">length(dates$date)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># use mutate to transform content from dates, ignore the warnings. </span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># the intelligence is embedded in useful functions in lubridate package</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># create three separate columns based on date formats and then merge them all </span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># into one column </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed <- mutate(dates, yyyymmdd = ymd(date), mmddyyyy = mdy(date), ddmmyyyy = dmy(date)) %>% unite("fixedup", yyyymmdd, ddmmyyyy, mmddyyyy, sep ="")</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span></span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span></div>
<div class="p1">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># replace NAs with blank strings</span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed$fixedup <- gsub("NA","", fixed$fixedup)</span></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span>
<span class="s1"><span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># View fixed</span></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed</span></span></div>
<div class="p1">
<br /></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># identify those dates where month and day are both less than equal to 12</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed_md_dm <- fixed %>% mutate(fixedup, month = month(fixedup), day = day(fixedup), to_be_reviewed = day(fixedup) <= 12 & month(fixedup) <= 12)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed_md_dm</span></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">---------</span></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 1b Solution:</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># Continuing from previous example</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># Alternative approach highlighting functions in R</span></span></div>
<div>
<br /></div>
<div>
<div class="p1">
<span class="s1"><span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># read the file, since the file does not contain a header, it is false</span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">dates <- read.csv(rawfile, header = FALSE, col.names = c("date"))</span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># there should be 22</span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">length(dates$date)</span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></span></div>
<span class="s1">
</span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></span></span><br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;">#Define function</span></span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixdates <- function(date) {</span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: #6aa84f;"># empty vector</span></span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> fixed <- c()</span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: #6aa84f;"># iterate over the vector containing raw dates</span></span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> for(i in 1:length(date)) {</span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: #6aa84f;"># is it yyyymmdd?</span></span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if(is.na(ymd(date[i]))) {</span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: #6aa84f;"># is it mmddyyyy</span></span></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if(is.na(mdy(date[i]))) {</span></span><br />
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: #6aa84f;"># is it ddmmyyyy</span></span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if(is.na(dmy(date[i]))) {</span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<br />
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <span style="color: #6aa84f;"># none, then undertermined</span></span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
</span></span>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> fixed[i] <- NA</span></span></div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">
<div>
} else {</div>
<div>
<span style="color: #6aa84f;"># assign ddmmyyyy</span><br />
fixed[i] <- dmy(date[i])</div>
<div>
}</div>
<div>
} else {</div>
<div>
<span style="color: #6aa84f;"># assign mmddyyyy</span></div>
</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><div>
fixed[i] <- mdy(date[i])</div>
<div>
}</div>
<div>
} else {</div>
<div>
<span style="color: #6aa84f;"># assign yyyymmdd</span></div>
</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><div>
fixed[i] <- ymd(date[i])</div>
<div>
}</div>
<div>
}</div>
<div>
<span style="color: #6aa84f;"># convert to date</span></div>
<div>
class(fixed) <- "Date"</div>
<div>
<span style="color: #6aa84f;"># return vector containing converted dates</span></div>
<div>
fixed</div>
<div>
}</div>
<div>
<br /></div>
<div>
<span style="color: #6aa84f;"># invoke function to convert raw dates</span></div>
<div>
dates$FixedDate <- fixdates(dates$date)<br />
<br />
<span style="color: #38761d;"># view contents</span><br />
dates</div>
</span></span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">---------</span></span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 1c Solution:</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Continuing from previous example</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"><br /></span></span></div>
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Extension exercise: add three different dates</span></span></div>
<div>
<br /></div>
<div>
<span class="s1"></span><br />
<div class="p1">
<span class="s1"><span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># read the file, since the file does not contain a header, it is false</span></span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">dates <- read.csv(rawfile, header = FALSE, stringsAsFactors = FALSE, col.names = c("date"))</span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># there should be 22</span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">length(dates$date)</span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></span></div>
<span class="s1">
</span>
<br />
<div class="p1">
<span class="s1"><span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># adding three more dates</span></span></span></div>
<span class="s1">
</span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">dates[nrow(dates)+1,] <- c("01/02/2013")</span></span></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">dates[nrow(dates)+1,] <- c("12021010")</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">dates[nrow(dates)+1,] <- c("20021010")</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># three turn in valid responses for more than one of yyyymmdd, ddmmyyyy, mmddyyyy!</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># should be reviewed by someone who has domain knowledge about this data and </span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># hence we need to flag such data accordingly</span></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<div>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># use mutate to transform content from dates, ignore the warnings. </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed <- mutate(dates, yyyymmdd = ymd(date), mmddyyyy = mdy(date), ddmmyyyy = dmy(date))</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># add a new column to know if a certain date observation has to be reviewed</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed %>% mutate(to_be_reviewed = ifelse((!is.na(fixed$yyyymmdd)) + (!is.na(fixed$mmddyyyy)) + (!is.na(fixed$ddmmyyyy)) == 1, FALSE, TRUE))</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># extension to above, here we find those which seem to be valid and convert them </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">fixed %>% mutate(to_be_reviewed = ifelse((!is.na(fixed$yyyymmdd)) + (!is.na(fixed$mmddyyyy)) + (!is.na(fixed$ddmmyyyy)) == 1, FALSE, TRUE)) %>% filter(to_be_reviewed==FALSE) %>%unite("fixedup", yyyymmdd, ddmmyyyy, mmddyyyy, sep ="") %>% mutate(fixedup = str_replace_all(fixedup, "NA","")) %>% select(date, fixedup)</span></div>
<div>
<br /></div>
<div>
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div>
<span class="s1"></span><br />
<div class="p1">
<span class="s1"><span class="s1"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Trifacta recipe using "Wrangle"</u></i></span></span></span></div>
<span class="s1">
</span>
<br />
<div>
<span class="s1"><span class="s1"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></span></span></div>
<span class="s1">
</span>
<br />
<div>
<span class="s1"><span class="s1"></span><br /></span>
<br />
<ul class="transform-list" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Proxima Nova', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 12px; line-height: 17.1429px; list-style: none; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; white-space: nowrap; widows: auto;"><div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="s1"><span class="s1"><span class="operator" style="box-sizing: border-box; color: #0277bd;">splitrows</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'\n'</span></span></span></span></div>
</div>
</div>
<span class="s1">
<span class="s1">
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column1</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`(th)|(st)`</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">derive</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">value</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="operator" style="box-sizing: border-box; color: #01579b;">dateformat</span>(<span class="column" style="box-sizing: border-box; color: black;">column1</span>, <span class="string" style="box-sizing: border-box; color: #43a047;">'yyyy-MM-dd'</span>) <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">as</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'column2'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">derive</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">value</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span>(<span class="operator" style="box-sizing: border-box; color: #01579b;">day</span>(<span class="column" style="box-sizing: border-box; color: black;">column2</span>) <= 12) && (<span class="operator" style="box-sizing: border-box; color: #01579b;">month</span>(<span class="column" style="box-sizing: border-box; color: black;">column2</span>) <= 12) <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">as</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'column3'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column2</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Fixed_Dates'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column3</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'To_Be_Reviewed'</span></div>
</div>
</div>
</span></span></ul>
<span class="s1"><span class="s1">
</span></span></div>
<span class="s1">
</span></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Exercise 2: Fix currency values</b></span><br />
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="https://drive.google.com/open?id=0B6UoSlNMfxQONW01LXpUWTdmdXM" target="_blank">Raw data file</a></span></li>
</ul>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Solution:</b></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Using R and R Studio</u></i></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Replace this with file name where you want to download the raw file</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">rawfile <- "ReplaceWithFileNameIncludingPath"</span><br />
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># download sales file</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">download.file("https://docs.google.com/uc?authuser=0&</span></span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">id=0B6UoSlNMfxQONW01LXpUWTdmdXM</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">&export=download", rawfile)</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># read the file</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sales <- read.csv(rawfile, stringsAsFactors = FALSE, header = T, sep = "\n")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># There is a lot going on here. </span><br />
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Create a separate column extracting the currency symbol</span><br />
<span style="color: #6aa84f;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Create another column containing numeric content by replacing comma, $ and </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">£</span></span><br />
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Convert to numeric values</span><br />
<span style="color: #6aa84f;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Normalize to USD by converting </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">£ to USD using 1.44 conversion ratio</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sales %>% mutate(currency = str_extract(sales,"\\$|£") ) %>% mutate(raw_amount = str_replace_all(sales, ",|\\$|£", "") ) %>% mutate(amount = as.numeric(raw_amount)) %>%mutate(convertedAmountInUSD = ifelse(currency == '$', amount, ifelse(currency == '£', round(amount*1.44,2), NA)))</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div class="p1">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Trifacta recipe using "Wrangle"</u></i></span></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ul class="transform-list" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Proxima Nova', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 12px; line-height: 17.1429px; list-style: none; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; white-space: nowrap; widows: auto;"><div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">splitrows</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'\n'</span></span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">header</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">extract</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">sales</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`$|£`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">sales</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`$|£`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">merge</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">sales2</span>,<span class="column" style="box-sizing: border-box; color: black;">sales3</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">as</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'column1'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'amounts'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">settype</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">amounts</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">type</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'Float'</span></span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">amounts</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`,`</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">drop</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">sales2</span>,<span class="column" style="box-sizing: border-box; color: black;">sales3</span></span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">derive</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">value</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span>(<span class="column" style="box-sizing: border-box; color: black;">sales1</span> == <span class="string" style="box-sizing: border-box; color: #43a047;">'$'</span>) ? <span class="column" style="box-sizing: border-box; color: black;">amounts</span> : ((<span class="column" style="box-sizing: border-box; color: black;">sales1</span> == <span class="string" style="box-sizing: border-box; color: #43a047;">'£'</span>) ? (<span class="column" style="box-sizing: border-box; color: black;">amounts</span> * 1.44) : <span class="string" style="box-sizing: border-box; color: #43a047;">'test'</span>) <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">as</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'column1'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">settype</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">type</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'Float'</span></span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">set</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">value</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="operator" style="box-sizing: border-box; color: #01579b;">numformat</span>(<span class="column" style="box-sizing: border-box; color: black;">column1</span>, <span class="string" style="box-sizing: border-box; color: #43a047;">'$##,###.##'</span>)</div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="text" style="box-sizing: border-box;">
<br /></div>
</div>
</ul>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Exercise 3: Fix Java Log4j log file with exception stack traces</b></span><br />
<ul style="text-align: left;">
<li><a href="https://drive.google.com/open?id=0B6UoSlNMfxQOQWlGQzdvbzhNRG8" target="_blank"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Raw data file</span></a></li>
</ul>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Solution:</b></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"></span><br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
</div>
<br />
<div style="color: black; font-family: times; font-size: medium; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<div style="margin: 0px;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Using R & R Studio</u></i></span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; font-size: small;">Exercise 3 Preparation:</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; font-size: small;"><br /></span></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># in addition to the previously loaded libraries</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># install another package to deal with Excel using cpp instead of </span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># error prone java dependency from R</span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("openxlsx")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; font-size: small;">Exercise 3 Solution:</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># if you have loaded plyr, there are conflict with dplyr. </span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># hence detaching it</span><br />
<span style="color: #134f5c; font-family: "courier new" , "courier" , monospace; font-size: x-small;">detach("package:plyr", unload=TR</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">UE)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(openxlsx)</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># specify input and output file</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">rawfile <- </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"ReplaceWithFileNameIncludingPath"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputFile <- </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"ReplaceWithFileNameIncludingPath"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div class="p1">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># download log file from log file</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">download.file("https://docs.google.com/uc?authuser=0&id=0B6UoSlNMfxQOQWlGQzdvbzhNRG8&export=download", rawfile)</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div class="p1">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># read log file, each line is a row</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">logs <- readLines(rawfile)</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># read 311 lines</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">length(logs)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># vector of characters</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">class(logs)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># convert into a data frame</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">logsdf <- as.data.frame(logs, stringsAsFactors = FALSE)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># find structure of the data frame</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">str(logsdf)</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># add a new column with line number</span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">logsdf$ID <- seq.int(nrow(logsdf))</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># peek at first 6 lines </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">head(logsdf)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># look at the class of columns of data frame</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sapply(logsdf, class)</span></span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span class="s1"></span><br /></span></div>
<div class="p2">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># function hacked to group exception lines together </span></div>
<div class="p2">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># feel silly hacking such a contrived function...bear with me here</span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">groupLogs <- function(ID) {</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> log_group <- c()</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i <- 1</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> while (i <= length(ID)) {</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> log_group[i] = ID[i]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> j <- i+1</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> while(j <=length(ID)) {</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if(ID[j] - ID[j-1] == 1) {</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> log_group[j] <- ID[i]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> } else {</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> log_group[j] <- ID[j]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i <- j</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> break</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> j <- j+1</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if(j > length(ID))</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i <- j</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> log_group</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span></span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span class="s1"></span><br /></span></div>
<div class="p2">
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span class="s1"></span># find lines in log file that represent exceptions </span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">exception_logs <- logsdf[str_detect(logsdf$logs, "^[^\\d]"),]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># invoke function to group together exception lines part of the same stack trace</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">exception_logs$log_grp <- groupLogs(exception_logs$ID)</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #6aa84f;"># group all the lines in the same exception into a single collapsed row separated # </span>by carriage return</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">exception_logs <- exception_logs %>% select(logs, log_grp) %>% group_by(log_grp) %>% summarise(stacktrace = paste(logs, collapse = "\r"))</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># little math trick to help merge the exception into a the log line previous to</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># this exception</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">exception_logs$log_grp <- ifelse(exception_logs$log_grp == 1, </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">exception_logs$log_grp, exception_logs$log_grp - 1)</span></span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># log lines that begin with timestamp</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">proper_logs <- logsdf[str_detect(logsdf$logs, "^[\\d]"),]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># split the log message as a separate column using " - " separator </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">proper_logs <- proper_logs %>% separate(logs, c("logs", "message" ), sep = "\\s-\\s") </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># split the log text into time, thread, level and class</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">proper_logs <- proper_logs %>% separate(logs, c("time", "thread", "level", "class" ), sep = "\\s+")</span></span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># left outer join to include exceptions by joining the two data frames created</span></span></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># earlier</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">logs <- merge(proper_logs, exception_logs, by.x = c("ID"), by.y = c("log_grp"), all.x = TRUE)</span></span></div>
<div class="p2">
<br /></div>
<div class="p1">
<span class="s1"><span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># write the excel file, example </span></span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="color: #38761d;">/Users/akuntamu/temp/text.xls</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">write.xlsx(logs,</span></span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> outputFile</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">)</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Trifacta recipe using "Wrangle"</u></i></span></span></div>
<div class="p1">
<br /></div>
<div class="p1">
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Requires a two step approach. </span></li>
</ul>
</div>
<ul style="text-align: left;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">
<li>Create an intermediate data set that distinguishes the regular log statements which begin with a timestamp from the exception stack traces</li>
</span></ul>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">
</span>
<br />
<div>
<div class="transform" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Monaco, Menlo, Consolas, monospace; font-size: 12px; line-height: 17.1429px; overflow: hidden; position: relative; text-overflow: ellipsis; white-space: nowrap; widows: auto;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">splitrows</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'\n'</span></span></div>
</div>
</div>
<div class="transform" style="background-color: white; box-sizing: border-box; color: #333333; font-family: Monaco, Menlo, Consolas, monospace; font-size: 12px; line-height: 17.1429px; overflow: hidden; position: relative; text-overflow: ellipsis; white-space: nowrap; widows: auto;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">set</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">value</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="operator" style="box-sizing: border-box; color: #01579b;">merge</span>([<span class="array" style="box-sizing: border-box;"><span class="operator" style="box-sizing: border-box; color: #0277bd;">matches</span>([<span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column1</span></span>], <span class="pattern" style="box-sizing: border-box; color: #43a047;">`{start}{digit}`</span>) ? <span class="string" style="box-sizing: border-box; color: #43a047;">'Q'</span> : <span class="string" style="box-sizing: border-box; color: #43a047;">''</span>,<span class="column" style="box-sizing: border-box; color: black;">column1</span></span>])</div>
</div>
</div>
</div>
<div style="text-align: left;">
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Split rows based on the dummy identifier Q added in the previous step. This technique can help span multi row observations/records.</span></li>
</ul>
</div>
<br />
<ul class="transform-list" style="background-color: white; box-sizing: border-box; color: #333333; font-family: 'Proxima Nova', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 12px; line-height: 17.1429px; list-style: none; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px; white-space: nowrap; widows: auto;"><div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">splitrows</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'Q'</span></span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`\n`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column2</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`{delim-ws}`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">' '</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">limit</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span>3</div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column7</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`{start}{delim}`</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column2</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Time'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column5</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Thread'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column6</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Level'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column7</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Class'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column4</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Message'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column3</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'Exception'</span></div>
</div>
</div>
</ul>
<div class="p1">
<span style="color: #333333; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; font-size: 12px; line-height: 17.1429px; white-space: nowrap;"><br /></span></div>
<div style="text-align: left;">
<span style="color: #333333; line-height: 17.1429px; white-space: nowrap;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></span></div>
<div style="text-align: left;">
<span style="color: #333333; line-height: 17.1429px; white-space: nowrap;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></span></div>
<div style="text-align: left;">
<span style="color: #333333; line-height: 17.1429px; white-space: nowrap;"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Exercise 4: Web Scraping Top 50 pop songs in last decade</b></span></span></div>
<ul style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="http://www.pastemagazine.com/blogs/lists/2009/11/the-best-albums-of-the-decade.html?a=1" target="_blank">Web Page</a></span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If the above page is unavailable, use this cached <a href="https://drive.google.com/open?id=0B6UoSlNMfxQOTXlKUWpraHMyQ0k" target="_blank">page</a></span></li>
</ul>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Solution:</b></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><u><i>Using R & R Studio</i></u></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 4 Preparation:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Install the following packages</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("gtools")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("rvest")</span><br />
<div>
<br /></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 4a Solution:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(rvest)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(tidyr)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(stringr)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(gtools)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Scrape the page listing top 50 pop hits in the last decade</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">scraping_top50_pop <- read_html("http://www.pastemagazine.com/blogs/lists/2009/11/the-best-albums-of-the-decade.html?a=1")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># If the above does not work, I have a cached copy of this page here.</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># scraping_top50_pop <- read_html("")</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Scraping using rvest page (which was inspired by BeautifulSoup, python library</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop <- html_nodes(scraping_top50_pop, ".big") %>% html_text %>% trimws(which ="both")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># split the rank and rest of the text</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">matrix_top_50_pop <- str_split_fixed(top_50_pop, "\\.", 2)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># supply column names</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">colnames(matrix_top_50_pop) <- c("rank", "text")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># convert to a data frame</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_df <- as.data.frame(matrix_top_50_pop, stringsAsFactors = FALSE)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># convert rank from character to numeric</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_df$rank <- as.numeric(top_50_pop_df$rank)</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># split the artist based on : delimiter</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_df <- top_50_pop_df %>% separate(col = text, into = c("artist", "text"), sep = ":")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># split lines that contains all three fields: title, company, year</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_df1 <- top_50_pop_df[which(top_50_pop_df$text %>% str_detect("\\[.*\\]")),] %>% separate(col = text, into = c("title", "company", "year"), sep = "\\[|\\]")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># split lines that contain only title and year</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_df2 <- top_50_pop_df[which(!top_50_pop_df$text %>% str_detect("\\[.*\\]")),] %>% separate(col = text, into = c("title", "year"), sep = "\\s\\(")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># smartly combine the above data frames since company is not common column in both</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_combined <- smartbind(top_50_pop_df1,top_50_pop_df2)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># edit the year field to trim and replace parantheses</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_combined$year <- apply(top_50_pop_combined[,"year", drop = FALSE], 2, function(year) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">str_replace_all(trimws(year, which = c("both")), "\\(|\\)", "")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">)[,"year"]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># trim artist</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_combined$artist <- trimws(top_50_pop_combined$artist, which = "both")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># trim title</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_combined$title <- trimws(top_50_pop_combined$title, which = "both")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># sort the final data frame using rank in the descending order</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_final <- arrange(top_50_pop_combined, -rank)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># print</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop_final</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Provide output CSV file name</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Example outputCSV <- "/Users/akuntamu/test/top-50-pop.csv"</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputCSV <- </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"ReplaceWithFileNameIncludingPath"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># export to a CSV </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">write.table(top_50_pop_final, outputCSV, row.names = FALSE, fileEncoding = "UTF-8", sep = "|")</span></div>
<div style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
<br /></div>
</div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Trifacta recipe using "Wrangle":</u></i></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Since Trifacta currently does not have data ingestion capability at the time of this writing, let's perform the first few steps of data scraping in R and create raw data <a href="https://drive.google.com/open?id=0B6UoSlNMfxQOSkhlc3B6Q1F3aTA" target="_blank">file</a> using the following commands.</span></div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 4b Solution:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(dplyr)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(rvest)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(tidyr)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(stringr)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">scraping_top50_pop <- read_html("http://www.pastemagazine.com/blogs/lists/2009/11/the-best-albums-of-the-decade.html?a=1")</span></div>
<div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">top_50_pop <- html_nodes(scraping_top50_pop, ".big") %>% html_text %>% trimws(which ="both")</span></div>
<div>
<br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># replace outputRawCSV </span><br />
<span style="color: #38761d;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Example: </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputRawCSV <- </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"/Users/akuntamu/test/top-50-pop-raw.csv"</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputRawCSV <- </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"ReplaceWithFileNameIncludingPath"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">write.table(as.data.frame(top_50_pop), </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputRawCSV</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">, row.names = FALSE, fileEncoding = "UTF-8", sep = "|")</span></div>
<div style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
<br />
Now upload this file as an input data set into Trifacta Wrangler, create a project and use the following script<br />
<br /></div>
</div>
<div>
<table class="table table-condensed" style="background-color: white; border-collapse: collapse; border-spacing: 0px; box-sizing: border-box; color: #333333; font-family: 'Proxima Nova', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 12px; line-height: 17.1429px; margin-bottom: 17px; max-width: 100%; widows: auto; width: 538px;"><tbody style="box-sizing: border-box;">
<tr style="box-sizing: border-box;"><td class="script-body" style="border: 0px; box-sizing: border-box; line-height: 1.42857143; padding: 5px; text-overflow: ellipsis; vertical-align: top; white-space: nowrap;"><ul class="transform-list" style="box-sizing: border-box; list-style: none; margin: 0px; min-height: 0px; min-width: 0px; padding: 0px;"><div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">splitrows</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="string" style="box-sizing: border-box; color: #43a047;">'\n'</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">quote</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'\"'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column1</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="regexp" style="box-sizing: border-box; color: #43a047;">/^"/</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column1</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="regexp" style="box-sizing: border-box; color: #43a047;">/"$/</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">delete</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">row</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="operator" style="box-sizing: border-box; color: #01579b;">matches</span>([<span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column1</span></span>], <span class="pattern" style="box-sizing: border-box; color: #43a047;">`top_50_pop`</span>)</div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`{delim-ws}`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column3</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`{delim-ws}`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column4</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">` [`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">split</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column5</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`] `</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">extract</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column3</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">` \({digit}+\)`</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column3</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">` \({digit}+\)`</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">merge</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column5</span>,<span class="column" style="box-sizing: border-box; color: black;">column6</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">as</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'column7'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">drop</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column5</span>,<span class="column" style="box-sizing: border-box; color: black;">column6</span></span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">set</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column7</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">value</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="operator" style="box-sizing: border-box; color: #01579b;">trim</span>(<span class="column" style="box-sizing: border-box; color: black;">column7</span>)</div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column7</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`\(`</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">replace</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;"><span class="column" style="box-sizing: border-box; color: black;">column7</span></span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">on</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="pattern" style="box-sizing: border-box; color: #43a047;">`\)`</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">with</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">''</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column2</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'rank'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column1</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'artist'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column3</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'title'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column4</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'company'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">rename</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">col</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="column" style="box-sizing: border-box; color: black;">column7</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">to</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="string" style="box-sizing: border-box; color: #43a047;">'year'</span></div>
</div>
</div>
<div class="transform" style="box-sizing: border-box; font-family: Monaco, Menlo, Consolas, monospace; overflow: hidden; position: relative; text-overflow: ellipsis;">
<div class="content flex" data-html="true" style="box-sizing: border-box; display: flex;">
<div class="text" style="box-sizing: border-box;">
<span class="operator" style="box-sizing: border-box; color: #0277bd;">sort</span> <span class="param" style="box-sizing: border-box; color: #b71c1c; font-style: italic;">order</span><span class="symbol" style="box-sizing: border-box; color: #9c27b0;">: </span><span class="array" style="box-sizing: border-box;">-<span class="column" style="box-sizing: border-box; color: black;">rank</span></span></div>
</div>
</div>
</ul>
</td></tr>
</tbody></table>
</div>
<div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Exercise 5: Web Scrape Using OpenRefine, R</b></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">This is a simple concocted exercise to consolidate entire 3 day agenda on this <a href="http://globalbigdataconference.com/64/dallas/big-data-bootcamp/schedule.html" target="_blank">page</a> into a simple table showing time slot and session information.</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><b>Solution: </b></span><br />
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Using Open Refine</u></i></span><br />
<ol style="text-align: left;">
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Launch openrefine application</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">This starts a jetty web server on your machine which hosts the openrefine web app</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Go to http://localhost:3333</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Create Project</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Using Clipboard</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Paste the following link: </span></li>
<ol>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="http://globalbigdataconference.com/64/dallas/big-data-bootcamp/schedule.html">http://globalbigdataconference.com/64/dallas/big-data-bootcamp/schedule.html</a></span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If not available, cached <a href="https://docs.google.com/uc?export=download&id=0B6UoSlNMfxQOVkFRZzB5M0xCczg" target="_blank">copy</a></span></li>
</ol>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Follow the prompts and select UTF-8 character encoding</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Create project </span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">The recipe for data scraping and transformation can be downloaded from <a href="https://docs.google.com/uc?export=download&id=0B6UoSlNMfxQOUWZYZGNPV0w2NjA" target="_blank">here</a>. </span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Once the project is created successfully, you will find Undo/Redo link </span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Click on that link to find Apply button</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Paste the content of downloaded file into textbox in the dialog box and click perform operations</span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">It will take a few seconds and run through the 14 steps. </span></li>
<li><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">We will review these 14 steps and walk through these individually</span></li>
</ol>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u>Using R and R Studio</u></i></span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i><u><br /></u></i></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 5 Preparation:</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Install the following packages</span><br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># you may be prompted to restart R session. accept it</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">install.packages("xml2")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Exercise 5 Solution:</span><br />
<br />
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># reloading several packages since R session restarted</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(rvest)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(xml2)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(dplyr)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(gtools)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(openxlsx)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">library(xml2)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># read html content</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">webpage <- read_html("http://globalbigdataconference.com/64/dallas/big-data-bootcamp/schedule.html")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># extract all tables</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">tbls <- html_nodes(webpage, "table")</span><br />
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #6aa84f; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># inspect tables</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">head(tbls)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># fetch only 1, 3, 4 tables since 2nd and 5th are not useful</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">tbls_list <- tbls %>% .[c(1,3,4)] %>% html_table(fill=TRUE)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># structure of returned object</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">str(tbls_list)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># set up column names for each of the data frames representing tables</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">colnames(tbls_list[[1]]) <- c("time-slot", "session-info")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">colnames(tbls_list[[2]]) <- c("time-slot", "session-info")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">colnames(tbls_list[[3]]) <- c("time-slot", "session-info")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># smartly append three tables</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">agenda <- smartbind(tbls_list[[1]],tbls_list[[2]], tbls_list[[3]])</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># inspect structure again</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">str(agenda)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># there is a row with NA, the following command shows how to filter complete rows</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">agenda <- agenda[complete.cases(agenda),]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<div>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># replace outputXLS </span><br />
<span style="color: #38761d;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Example: </span></span><span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputXLS</span><span style="color: #38761d;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <- </span></span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"/Users/akuntamu/test/agenda.xls"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">outputXLS</span> <- </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"ReplaceWithFileNameIncludingPath"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
</div>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></span>
<span style="color: #38761d; font-family: "courier new" , "courier" , monospace; font-size: x-small;"># write it out</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">write.xlsx(agenda, </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">outputXLS</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">)</span><br />
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">This concludes this installment of data wrangling exercises using R, R Studio and Trifacta. Hope this wets your appetite for learning more about data wrangling. In future installments, we will look into dealing with missing values, identifying outliers etc using other technologies such as python pandas, open refine and/or any freeware offering by a commercial vendor to get more hands-on expertise </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Feel free to post questions, comments, suggestions for improvement on Twitter. I am reachable @akuntamukkala. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Look forward to hearing from you!</span></div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com1tag:blogger.com,1999:blog-8558899295969164571.post-38604239192984763492016-05-08T01:12:00.000-07:002016-05-09T07:48:30.828-07:00Webinar: Data Wrangling - Why, What and How?<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">I had the pleasure of sharing my insights on a </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">very </span><span style="background-color: white; color: #222222; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">exciting and dynamic industry of data preparation called "Data Wrangling" in a 45 minutes </span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">complimentary webinar which was arranged by Global Big Data Conference on May 7th 2016. </span></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">This is a precursor to the 2-1/2 hour hands-on session I will be doing on May 14th 2016. You can find additional details <a href="http://globalbigdataconference.com/64/dallas/big-data-bootcamp/event.html" target="_blank">here</a>. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">In this webinar, I discussed the overarching objective and basics of data wrangling. I reviewed the open source and commercial tools such as Trifacta, Paxata, DataWatch, Datameer etc we can potentially use to curate data. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">The best way in my opinion to learn data wrangling is by getting familiar with the process of data wrangling and doing hands-on exercises by using applicable tools on small "raw" data sets and eventually perform data wrangling at scale by leveraging Big Data infrastructure powered by Hadoop ecosystem. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">If you couldn't attend the webinar yesterday, here is the recording that you can watch at your leisure. </span><br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/k3EZxLBOsi0/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/k3EZxLBOsi0?feature=player_embedded" width="320"></iframe></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Please feel free to post comments or share it. I would love to know your feedback and any suggestions to improve the content presented. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Also, if you'd want me to focus on any particular use cases during the hands-on session, please let me know!</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Please find the following section for instructions on what tools you need to have installed on your laptop before the hands-on session on May 14th 2016</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Hands-On Data Wrangling: What, How, and Why</span></h3>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Here are the tools you will need to install on your computer before we engage in the hands-on session:</span><br />
<br />
<table border="1" bordercolor="#0033FF" cellpadding="2" cellspacing="2"><tbody>
<tr><th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Tool</span></th><th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Version</span></th><th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Download & Install Instructions</span></th><th><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Type</span></th></tr>
<tr><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">R language</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">3.2.4</span></td><td><a href="https://cran.rstudio.com/"><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">https://cran.rstudio.com/</span></a></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Open Source</span></td></tr>
<tr><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">R Studio</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">0.99.887</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="https://www.rstudio.com/products/rstudio/download/">https://www.rstudio.com/products/rstudio/download/</a></span><br />
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Please install and load the following R packages</span><br />
<ul>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">s</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">tringr</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">d</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">plyr</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">t</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">idyr</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">r</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">eadxl</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">xlsx</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">l</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">ubridate</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">g</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">tools</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">p</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">lyr</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">r</span><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">vest</span></li>
<li><span style="color: #262626; font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">stringdist </span></li>
</ul>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">You can use the following commands to install and load a package called "plyr" in R Studio. </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="color: #073763; font-family: "courier new" , "courier" , monospace;">>install.packages("plyr")</span><br />
<span style="color: #073763; font-family: "courier new" , "courier" , monospace;">>library(plyr)</span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Similarly, use the above mentioned commands and replace "plyr" with other package names.</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Open Source</span></td></tr>
<tr><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">OpenRefine</span></td><td><h3 id="openrefine-26" style="color: #333333; line-height: 1.28205em; margin: 12px 0px;">
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif; font-size: small; font-weight: normal;">2.6</span></h3>
</td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="http://openrefine.org/download.html">http://openrefine.org/download.html</a><span id="goog_1467884001"></span><span id="goog_1467884002"></span><a href="https://www.blogger.com/"></a></span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Open Source</span></td></tr>
<tr><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Trifacta Wrangler</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">3.0.1-client1</span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><a href="https://www.trifacta.com/trifacta-wrangler">https://www.trifacta.com/trifacta-wrangler</a><span id="goog_1467884001"></span><span id="goog_1467884002"></span><a href="https://www.blogger.com/"></a></span></td><td><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Commercial but free offering with limited functionality</span></td></tr>
</tbody></table>
<a href="https://2.bp.blogspot.com/-mJbWluNL6BU/VyURr61bQpI/AAAAAAAAAbs/ufMXprl0DUc_toiosCTlmILJyMXvd8QqwCLcB/s1600/caution.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-left: 1em;"><br /></a><a href="https://2.bp.blogspot.com/-mJbWluNL6BU/VyURr61bQpI/AAAAAAAAAbs/ufMXprl0DUc_toiosCTlmILJyMXvd8QqwCLcB/s1600/caution.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="50" src="https://2.bp.blogspot.com/-mJbWluNL6BU/VyURr61bQpI/AAAAAAAAAbs/ufMXprl0DUc_toiosCTlmILJyMXvd8QqwCLcB/s200/caution.jpg" width="50" /></a><span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><i style="color: #0b5394; font-size: small;"><br />By no stretch of imagination is this tutorial supposed to be the end-all and all-inclusive learning experience of data wrangling tools and strategies but it merely scratches the scratch of the plethora of tools we have at our disposal to wrangle data.</i></span><br />
<br />
<br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">You will need to have Java installed. You can install JDK 1.7+ </span><br />
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div style="direction: ltr; margin-bottom: 0pt; margin-left: .5in; margin-top: 0pt; mso-line-break-override: none; punctuation-wrap: hanging; text-align: left; unicode-bidi: embed; word-break: normal;">
<br /></div>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;">Thank you!</span>
<span style="font-family: "helvetica neue" , "arial" , "helvetica" , sans-serif;"><br /></span></div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-12631266708201807302016-04-14T08:59:00.004-07:002016-04-14T08:59:56.449-07:00My Interview @ Global Big Data Conference, Dallas <div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">One more month to go for the conference and I am busy preparing slide content and hands-on data wrangling exercises. It is an awesome experience to learn how to teach :)</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Anyways, here is the link to my interview that got published a couple of days ago to give the audience a feel for what they can expect. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://globalbigdataconference.com/news/129000/interview-with-ashwini-kuntamukkala-software-architect-vizient.html">http://globalbigdataconference.com/news/129000/interview-with-ashwini-kuntamukkala-software-architect-vizient.html</a></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-32630774446759478052016-03-21T12:46:00.001-07:002016-03-21T12:46:08.454-07:00Upcoming talks at Global Big Data Convention in Dallas in May 13/14/15 2016<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I am excited to share that I will be giving two talks at the <a href="http://globalbigdataconference.com/64/dallas/big-data-bootcamp/schedule.html" target="_blank">Dallas Big Data Convention</a> on the weekend of May 13th 2016 as <a href="http://www.irvingconventioncenter.com/about/contact-us/" target="_blank">Las Colinas Convention Center</a>. Here are the topics and corresponding abstracts. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Data Wrangling - What, Why and How? [Industry state, business applicability]</b></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><i>Abstract:</i></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">"Garbage in -> Garbage out (GIGO)" is a popular quote in the field of computer science. But is that really true? </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">We are now nearing post "Big Data” era where scaling data storage and compute capacity is almost as easy as pushing a button. The ecosystem of data processing tools is getting richer by the day. In such a thriving environment, data is not the new “oil" but the new “soil” where companies can grow several data driven business models. Many "forward-looking" companies are already unlocking the hidden insights in treasure troves of data they already have along with publicly available data. </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Just as gold is extracted from ore through a very rigorous refinement process, insights from data in the crudest form have to be discovered through rigorous process. This raw data is typically locked up in spread sheets, web pages, web/machine logs, PDFs, CSVs, TSVs, XML, JSON, Word, images, videos, hand written notes, audio, RDFs, sensor signals, databases etc. </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">One can’t perform accurate statistical analysis on inaccurate data. So, in order to facilitate effective use of raw data, there is an upsurge in the market place for plethora of tools that ease converting raw data into usable form. This process of converting raw data into usable form is called "<b>Data Wrangling</b>”. </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">After this iterative step of curated datasets, we can unleash rich analytics, visualizations etc to drive the end business objectives. </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this talk, I will cover the essentials of data wrangling, the necessary workflow, open source tools available at our disposal and also provide comparison among popular commercial vendors in this field based on my own experience and use cases. </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Transform your Enterprise into Data-Driven Digital Business</b></span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><i>Abstract: </i></span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Is your company in on “Digital Transformation”? </span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This phenomenon is causing enterprises to rethink their strategy as they modernize themselves to stay relevant in an ultra competitive world. Darwin's quote “The fittest survive” is more relevant in business today than ever before. The enterprises that adapt and adopt this fundamentally a new “culture" will disrupt their own business and stand the test of time. </span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The reason I say “culture” instead of “strategy” is because Peter Drucker, father of modern management, has said that “Culture eats strategy for breakfast”. So in a company if the spirit of innovation, agility are missing, no disruptive strategy has a chance to work.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">So if you are someone who wants to be a change agent, a catalyst or a visionary for your company to create more value or enter a new growth spurt or perhaps even a new market but are running into walls because the culture does not facilitate that, you can be frustrated </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">like the driver trying to steer a parked car. </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this talk, I will share my insights from my own experience and much of others that I have had the good fortune to collaborate with. This will help you carve a successful digital transformation roadmap for your company. </span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span class="im" style="background-color: white; color: #500050;"></span><span class="im" style="background-color: white; color: #500050;"></span></span><br />
<div dir="ltr">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">You will be empowered with practical insights, tidbits, and result oriented successful practices that you can take back and start your company’s "digital transformation".</span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">--------</span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">I look forward to hearing your thoughts as I put together content for my talk. Please feel free to reach on <a href="https://www.linkedin.com/in/akuntamukkala" target="_blank">LinkedIn</a> or <a href="https://twitter.com/akuntamukkala" target="_blank">Twitter</a> [@akuntamukkala] if you have any questions or comments. </span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">See you there!</span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-91172107464719069692014-11-18T22:16:00.002-08:002014-11-18T22:16:39.227-08:00DZone Refcard on Apache Spark<div dir="ltr" style="text-align: left;" trbidi="on">
Glad to share that DZone Refcard for Apache Spark is now available for download at <a href="http://t.co/s3tNmWPqcr">http://t.co/s3tNmWPqcr</a><div>
<br /></div>
<div>
It is a short digest of what Apache Spark is about and capabilities it enables for engineers and data scientists. </div>
<div>
<div>
<br /></div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-78677010135929274742014-11-10T07:26:00.000-08:002014-11-10T07:26:38.843-08:00DZone - Developer of the Week<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Over the years I have benefited from the curated content published by DZone. I am amazed by the way the content writers at DZone publish intriguing and engaging content and especially the useful developer friendly Refcards.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Recently I got an opportunity to work with DZone content creators to write a Refcard on Apache Spark. It was a fantastic experience and one that I highly recommend for anyone who intends to work with the best in the industry.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As a preview to the Refcard, I was honored to be featured as the developer of the week on DZone. My interview is published </span><a href="http://java.dzone.com/articles/dev-week-ashwini-kuntamukkala" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;" target="_blank">here</a><span style="color: black; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">.</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> </span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I am very excited about the future of Apache Spark in the Big Data ecosystem.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Per the industry trends, many IT professionals are transitioning their careers to Big Data as companies are realizing that data is their new currency as it can potentially unlock the door to new revenue streams.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Herein lies a challenge where many struggle with "separating noise from the voice". Since Hadoop ecosystem has grown over the last 10 years, it can be a daunting task for anyone who wants to get into this space because there are so many tools and solutions to solve plethora of Big Data use cases.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This is exactly the reason I am excited about Apache Spark. It is compelling platform that provides a unified approach to solve most common Big Data use cases classified into batch, interactive and real time data processing.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In the DZone Refcard on Apache Spark, I have catered to new or moderately experienced IT professionals who want to discover the capabilities of Apache Spark. I have included simple hands on examples and techniques that demonstrate how easily can one become productive using Apache Spark and start solving Big Data use cases.</span></div>
<div dir="ltr" style="background-color: white; color: #222222;">
<br /></div>
<span style="background-color: white; color: #222222; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">At SciSpike, we are excited about helping our clients adopt Apache Spark in their Big Data infrastructure and prove its merits and become a platform of choice for Big Data applications. </span><br />
<span style="background-color: white; color: #222222; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div dir="ltr" style="background-color: white; color: #222222;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I look forward to hearing your thoughts on the Refcard. It should be out in Nov 2014.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="background-color: white; color: #222222;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I am reachable via Twitter @akuntamukkala</span></span><br />
<span style="background-color: white; color: #222222;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></span></div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-12117977914670306262014-05-29T01:03:00.002-07:002014-05-29T01:32:54.708-07:00ActiveMQ - Network of Brokers Explained - Part 5<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In the previous <a href="http://akuntamukkala.blogspot.com/2014/03/activemq-network-of-brokers-explained_26.html" target="_blank">part 4</a> we have seen how to load balance remote consumers on a queue using network connectors.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this part 5, we will see how the same configuration would work if we had concurrent remote durable subscribers on a topic. Consider the following configuration.... </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-rrXgILisSdo/U2FQlKuq5ZI/AAAAAAAAAUA/uDEPnl072rk/s1600/part-5-lb-rcc+-+New+Page.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-rrXgILisSdo/U2FQlKuq5ZI/AAAAAAAAAUA/uDEPnl072rk/s1600/part-5-lb-rcc+-+New+Page.png" height="211" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Fig 1: Network of Brokers - Load balance subscribers on a topic</span></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As shown above, we have Broker-1 which initiates two network connectors to Broker-2 and Broker-3. A producer sends messages to a topic "moo.bar" on Broker-1 while Broker-2 has subscriber C1 and Broker-3 has two subscribers C2 and C3 on the same topic "moo.bar". </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">You may observe that this set up is very similar to <a href="http://akuntamukkala.blogspot.com/2014/03/activemq-network-of-brokers-explained_26.html" target="_blank">part 4</a>. The only difference is that here we are dealing with topics while in<a href="http://akuntamukkala.blogspot.com/2014/03/activemq-network-of-brokers-explained_26.html" target="_blank"> part 4</a>, we were dealing with queues. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see this in action</span></h3>
<ol style="text-align: left;"><a href="http://4.bp.blogspot.com/-_mYscArIkMI/U4a9WEu77MI/AAAAAAAAAUc/DmZh61vD8Mc/s1600/broker-1-connections.tiff" imageanchor="1" style="clear: left; float: left; font-family: Times; margin-bottom: 1em; margin-left: 1em;"><br /></a>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Add the following network connector configuration in Broker-1's activemq.xml configuration file</span><br /><br /><div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"> <networkConnectors></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><networkConnector</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>name="T:broker1->broker2"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>uri="static:(tcp://localhost:61626)"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>duplex="false"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>decreaseNetworkConsumerPriority="false"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>networkTTL="2"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>conduitSubscriptions="false"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>dynamicOnly="true"></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><excludedDestinations></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><queue physicalName="&gt;" /></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></excludedDestinations></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></networkConnector></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><networkConnector</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>name="T:broker1->broker3"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>uri="static:(tcp://localhost:61636)"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>duplex="false"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>decreaseNetworkConsumerPriority="false"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>networkTTL="2"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>conduitSubscriptions="false"</span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>dynamicOnly="true"></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><excludedDestinations></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><queue physicalName="&gt;" /></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></excludedDestinations></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></networkConnector></span></div>
<div class="separator" style="clear: both; font-family: Times;">
<span style="font-family: Courier New, Courier, monospace;"></networkConnectors></span></div>
<br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></li>
<li><div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's start broker-2, broker-3 and broker-1 in that order.</span></div>
</li>
<li><div>
<span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/cluster/broker-2/bin$<b> ./broker-2 console</b></span></div>
</li>
<li><div>
<span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/cluster/broker-3/bin$ <b>./broker-3 console</b></span></div>
</li>
<li><div>
<span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/cluster/broker-1/bin$ <b>./broker-1 console</b></span></div>
</li>
<li><a href="http://4.bp.blogspot.com/-_mYscArIkMI/U4a9WEu77MI/AAAAAAAAAUc/DmZh61vD8Mc/s1600/broker-1-connections.tiff" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Broker-1's admin console connections show that two network connectors have been established as configured from Broker-1 to Broker-2 and Broker-3 respectively</span></li>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-_mYscArIkMI/U4a9WEu77MI/AAAAAAAAAUc/DmZh61vD8Mc/s1600/broker-1-connections.tiff" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-_mYscArIkMI/U4a9WEu77MI/AAAAAAAAAUc/DmZh61vD8Mc/s1600/broker-1-connections.tiff" height="53" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Broker-1's Connections @ http://localhost:8161/admin/connections.jsp</span></td></tr>
</tbody></table>
<br /><br /><br /><br /><br /><br /><br />
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's start the subscriber C1 on Broker-2 subscribing to messages to topic "moo.bar" and subscribers C2 and C3 on Broker-3 subscribing to messages on same topic "moo.bar"</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Durable Subscribers require unique combination of client id and subscriber name. In order for us to create durable subscribers C2 and C3 we need to enhance the functionality provided in /Users/akuntamukkala/apache-activemq-5.8.0/example/src/ConsumerTool.java where /Users/akuntamukkala/apache-activemq-5.8.0 is the directory where ActiveMQ is installed.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The modified code consists of editing build.xml and ConsumerTool.java to add a new parameter "subscriberName". The edited files build.xml and ConsumerTool.java can be obtained from <a href="https://drive.google.com/file/d/0B6UoSlNMfxQOdmpCVXhxT0dNaGc/edit?usp=sharing" target="_blank">here</a> and <a href="https://drive.google.com/file/d/0B6UoSlNMfxQOMDc5R3V5VzRJS3M/edit?usp=sharing" target="_blank">here</a> respectively.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's start the subscribers now.</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/example$ <b>ant consumer -Durl=tcp://localhost:61626 -Dtopic=true -Dsubject=moo.bar -DclientId=C1 -Ddurable=true -DsubscriberName=mb.C1</b></span></li>
<li><span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/example$ <b>ant consumer -Durl=tcp://localhost:61636 -Dtopic=true -Dsubject=moo.bar -DclientId=C2 -Ddurable=true -DsubscriberName=mb.C2</b></span></li>
<li><span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/example$ <b>ant consumer -Durl=tcp://localhost:61636 -Dtopic=true -Dsubject=moo.bar -DclientId=C3 -Ddurable=true -DsubscriberName=mb.C3</b></span><br /><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Durable subscriber on Broker-2<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td><a href="http://3.bp.blogspot.com/-uk33mzQ2Xxc/U4bShOuTwJI/AAAAAAAAAUo/pUT44b_3NsE/s1600/broker-2-durable-subscribers.tiff" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-uk33mzQ2Xxc/U4bShOuTwJI/AAAAAAAAAUo/pUT44b_3NsE/s1600/broker-2-durable-subscribers.tiff" height="84" width="640" /></a></td></tr>
<tr><td class="tr-caption"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">http://localhost:9161/admin/subscribers.jsp</span></td></tr>
</tbody></table>
</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Durable subscribers on Broker-3</span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-yrppS2KekMI/U4bS_0aUq-I/AAAAAAAAAUw/vgqkTUIua2Q/s1600/broker-3-durable-subscribers.tiff" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="http://3.bp.blogspot.com/-yrppS2KekMI/U4bS_0aUq-I/AAAAAAAAAUw/vgqkTUIua2Q/s1600/broker-3-durable-subscribers.tiff" height="107" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">http://localhost:10161/admin/subscribers.jsp</span></td></tr>
</tbody></table>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Durable subscribers on Broker-1 (because of network connectors)</span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-vnJcgh5YfVQ/U4bTNSiLMqI/AAAAAAAAAU4/K5g0RYgYzSU/s1600/broker-1-durable-subscribers.tiff" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="http://3.bp.blogspot.com/-vnJcgh5YfVQ/U4bTNSiLMqI/AAAAAAAAAU4/K5g0RYgYzSU/s1600/broker-1-durable-subscribers.tiff" height="120" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">http://localhost:8161/admin/subscribers.jsp</span></td></tr>
</tbody></table>
</li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> Now let's send 10 durable messages to topic moo.bar on Broker-1</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">akuntamukkala@localhost~/apache-activemq-5.8.0/example$ <b>ant producer -Durl=tcp://localhost:61616 -Dtopic=true -Dsubject=moo.bar -Dmax=10 -Ddurable=true</b></span></li>
<li style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">See the console on Broker-3<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-S-pYLyDuGZU/U4bbdSoHznI/AAAAAAAAAVI/GjemEV26Ams/s1600/broker-3-duplicate-msg-warning.tiff" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="http://3.bp.blogspot.com/-S-pYLyDuGZU/U4bbdSoHznI/AAAAAAAAAVI/GjemEV26Ams/s1600/broker-3-duplicate-msg-warning.tiff" height="99" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Log file output on Broker-3</td></tr>
</tbody></table>
</li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As you may observe, Broker-3 receives the same message twice, once per each subscription C2 and C3. <b>ActiveMQ by default does not permit processing of duplicate messages.</b></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><i><span style="color: blue;">This happens because both the subscriptions mb.C2 and mb.C3 on Broker-3 are propagated to Broker-1. So when 10 messages are published to moo.bar on Broker-1, those messages are sent over to subscribers mb.C2 and mb.C3 on the same broker: Broker-3. Since the messages have the same ID, duplicate messages are discarded and hence the warning shown in the log messages....(shown in step 19)</span></i></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here is the console showing statistics on Broker-1</span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-p2PuBMyUxh8/U4beWLCArwI/AAAAAAAAAVU/VbZb2ZE-TVg/s1600/broker-1-stats.tiff" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="http://4.bp.blogspot.com/-p2PuBMyUxh8/U4beWLCArwI/AAAAAAAAAVU/VbZb2ZE-TVg/s1600/broker-1-stats.tiff" height="122" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">http://localhost:8161/admin/subscribers.jsp</td></tr>
</tbody></table>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here is the console showing statistics on Broker-3</span><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-tnjwMX_S8oA/U4be4IxGaZI/AAAAAAAAAVc/zSDcp5hBEVI/s1600/broker-3-stats.tiff" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="http://3.bp.blogspot.com/-tnjwMX_S8oA/U4be4IxGaZI/AAAAAAAAAVc/zSDcp5hBEVI/s1600/broker-3-stats.tiff" height="106" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">http://localhost:10161/admin/subscribers.jsp</td></tr>
</tbody></table>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As you can see even though the enqueue counter shows 20, the dequeue counter shows only 10, since the other 10 messages were discarded by the Broker-3. <i>This is a useful feature which helps to ensure that a message gets processed at most once by a broker.</i></span></li>
</ol>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The reason why this is occurring is because both subscriptions C2 and C3 are propagated to upstream broker Broker-1 </span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-tOduYj7TC80/U4buS8VeTAI/AAAAAAAAAWY/_-gHSUYFvJU/s1600/broker-3-duplicate-msg-warning.tiff" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-tOduYj7TC80/U4buS8VeTAI/AAAAAAAAAWY/_-gHSUYFvJU/s1600/broker-3-duplicate-msg-warning.tiff" height="348" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Duplicate Messages on Broker-3</span></td></tr>
</tbody></table>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's retry the same scenario using a minor tweak in the network connector settings by making </span><span style="font-family: Courier New, Courier, monospace;"><b>conduitSubscriptions="true"</b></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> on both network connectors from Broker-1 to Broker-2 and Broker-3 respectively. After restarting the brokers, delete the inactive durable subscribers and then repeat the above steps. </span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> </span><span style="font-family: Courier New, Courier, monospace;"><networkConnectors></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><networkConnector</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>name="T:broker1->broker2"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>uri="static:(tcp://localhost:61626)"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>duplex="false"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>decreaseNetworkConsumerPriority="false"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>networkTTL="2"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><b>conduitSubscriptions="true"</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>dynamicOnly="true"></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><excludedDestinations></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><queue physicalName="&gt;" /></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></excludedDestinations></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></networkConnector></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><networkConnector</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>name="T:broker1->broker3"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>uri="static:(tcp://localhost:61636)"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>duplex="false"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>decreaseNetworkConsumerPriority="false"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>networkTTL="2"</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><b>conduitSubscriptions="true"</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>dynamicOnly="true"></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><excludedDestinations></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span><queue physicalName="&gt;" /></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></excludedDestinations></span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></networkConnector></span><br />
<span style="font-family: Courier New, Courier, monospace;"></networkConnectors></span><br />
<div>
<br /></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The following screenshot shows that Broker-1 now sees only two durable subscribers, one from each broker, Broker-2 and Broker-3. </span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-5_GXT1vjjaA/U4bjM3YdE3I/AAAAAAAAAV0/QRBVbsvnl8I/s1600/broker-1-cs-true.tiff" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-5_GXT1vjjaA/U4bjM3YdE3I/AAAAAAAAAV0/QRBVbsvnl8I/s1600/broker-1-cs-true.tiff" height="99" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Durable Subscribers in Broker-1 when conduitSubscriptions="true"</span></td></tr>
</tbody></table>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><b>Upon publishing 10 durable messages on Broker-1, we find that we don't have the same issue of duplicate messages this time. </b></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As expected all the 10 messages are processed by C1, C2 and C3 as shown by screenshots below. </span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-cU3KjUdpEdw/U4bkeg55gpI/AAAAAAAAAWA/YH_J0PWSaU0/s1600/broker-1-msg-processed.tiff" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-cU3KjUdpEdw/U4bkeg55gpI/AAAAAAAAAWA/YH_J0PWSaU0/s1600/broker-1-msg-processed.tiff" height="104" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Broker-1's Durable Topic Subscribers</span></td></tr>
</tbody></table>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-_rQLvnjWVEs/U4bk9u3UC9I/AAAAAAAAAWI/QgNdXdWaKTo/s1600/broker-3-msg-processed.tiff" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-_rQLvnjWVEs/U4bk9u3UC9I/AAAAAAAAAWI/QgNdXdWaKTo/s1600/broker-3-msg-processed.tiff" height="104" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Broker-3's Durable Topic Subscribers C2 and C3 receive and process 10 messages each</span></td></tr>
</tbody></table>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Hence we have seen how </span><span style="font-family: Courier New, Courier, monospace;">conduitSubscriptions</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> attribute can help in reducing message traffic by avoiding duplicate messages in a network of brokers.</span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In the next part 6, we will see how ActiveMQ provides "message replay" capabilities in order to prevent stuck message scenarios. </span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com4tag:blogger.com,1999:blog-8558899295969164571.post-37790988686861181212014-05-09T13:44:00.001-07:002014-05-09T15:44:12.151-07:00Speaking at Global Big Data Conference in Dallas May 11, 2014<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I am going to be speaking about <a href="http://spark.apache.org/" target="_blank">Apache Spark</a> in <a href="http://globalbigdataconference.com/" target="_blank">Global Big Data Conference</a> on May 11th 2014 from 11.00am to 12.00pm @ Irving Convention Center, 500 W Las Colinas Blvd, Irving, TX 75039</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> </span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here is the abstract of the presentation: </span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://globalbigdataconference.com/31/dallas/big-data-bootcamp/speaker-details/364/ashwini-kuntamukkala.html">http://globalbigdataconference.com/31/dallas/big-data-bootcamp/speaker-details/364/ashwini-kuntamukkala.html</a></span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I am impressed with the capabilities Apache Spark enables to unify batch, streaming and interactive big data use cases. The brilliant folks at AMPLabs @ UC Berkeley have created a tremendous solution that takes big data processing to the next level! </span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see some lightning fast big data analytics powered by Apache Spark!</span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Look forward to seeing you there!</span></div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-69486469548161542732014-03-26T22:12:00.004-07:002014-03-27T08:45:10.793-07:00ActiveMQ - Network of Brokers Explained - Part 4<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
In the previous
<a href="http://akuntamukkala.blogspot.com/2014/03/activemq-network-of-brokers-explained.html" target="_blank">part 3</a>
, we have seen how ActiveMQ helps distinguish remote consumers from
local consumers which helps in determining shorter routes from
message producers to consumers.
</span>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
<br />
</span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In
this part 4,
</span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">we will look into how to load balance concurrent
consumers on remote brokers.
</span>
<br />
<br />
<div class="p1">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let’s
consider a bit more advanced configuration to load balance
concurrent message consumers on a queue in remote brokers as shown
below.
</span>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><img alt="" border="0" src="http://4.bp.blogspot.com/-hGiiLwAZ5So/UzJ9PribFXI/AAAAAAAAARQ/Z1wNLWSkwRg/s3200/part-4-lb-rcc+-+New+Page+(4).png" height="214" style="margin-left: auto; margin-right: auto;" title="Network of brokers" width="400" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Part 4 - Network of brokers </span></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-hGiiLwAZ5So/UzJ9PribFXI/AAAAAAAAARQ/Z1wNLWSkwRg/s3200/part-4-lb-rcc+-+New+Page+(4).png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
</a>
</div>
<div class="p1">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In
the above configuration, we have a message producer sending messages
into a queue moo.bar on broker-1. Broker-1 establishes network
connectors to broker-2 and broker-3. Consumer C1
consumes messages from queue moo.bar on broker-2 while consumers C2
and C3 are concurrent consumers on queue moo.bar on broker-3.
</span>
</div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see this in action</span></h3>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="p1">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's create three brokers instances...</span></div>
<div class="p1">
<ol style="text-align: left;">
<li>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ashwinis-MacBook-Pro:bin akuntamukkala$ </span>
<span style="color: blue; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">pwd</span>
<br /><span style="color: blue; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">/Users/akuntamukkala/apache-activemq-5.8.0/bin</span></li>
<br />
<li>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ashwinis-MacBook-Pro:bin akuntamukkala$</span><span style="color: blue; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">./activemq-admin create ../cluster/broker-1</span>
</li>
<br />
<li>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ashwinis-MacBook-Pro:bin akuntamukkala$</span><span style="color: blue; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">./activemq-admin create ../cluster/broker-2</span>
</li>
<br />
<li>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Ashwinis-MacBook-Pro:bin akuntamukkala$</span><span style="color: blue; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">./activemq-admin create ../cluster/broker-3</span>
</li>
<br />
<li>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
Fix the broker-2 and broker-3 transport, amqp connectors and jetty http port by modifying the corresponding conf/activemq.xml and
conf/jetty.xml as follows:<br /><br /><table border="1"><tbody>
<tr><th>Broker</th><th>Openwire Port</th><th>Jetty HTTP Port</th><th>AMQP Port</th></tr>
<tr><td><center>
broker-1</center>
</td><td><center>
61616</center>
</td><td><center>
8161</center>
</td><td><center>
5672</center>
</td></tr>
<tr><td><center>
broker-2</center>
</td><td><center>
61626</center>
</td><td><center>
9161</center>
</td><td><center>
5682</center>
</td></tr>
<tr><td><center>
broker-3</center>
</td><td><center>
61636</center>
</td><td><center>
10161</center>
</td><td><center>
5692</center>
</td></tr>
</tbody></table>
<br />
</span></li>
<br />
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Fix network connector on broker-1 such that
messages on queues can be forwarded dynamically to consumers on
broker-2 and broker-3. This can be done by adding the following XML snippet into
broker-1's conf/activemq.xml<br /><b style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><br /><networkConnectors></span></b><br /><span style="font-family: 'Courier New', Courier, monospace;"><b><span style="color: #990000;"> </span></b></span></span><b style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><networkConnector</span></b></li>
<ol><span style="font-family: Courier New, Courier, monospace;"><b><span style="color: #990000;"> name="</span><span style="color: blue;">Q:broker1->broker2</span><span style="color: #990000;">"</span></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> uri="static:(tcp://localhost:61626)"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> duplex="false"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> decreaseNetworkConsumerPriority="true"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> networkTTL="2"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> dynamicOnly="true"></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <excludedDestinations></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <topic physicalName="&gt;" /></b></span></ol>
<ol><b><span style="color: #990000;"><span style="font-family: Courier New, Courier, monospace;"> </excludedDestinations></span></span></b><br /><b><span style="color: #990000;"><span style="font-family: 'Courier New', Courier, monospace;"></networkConnector></span></span></b></ol>
<br />
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b><networkConnector</b></span></ol>
<ol><span style="font-family: Courier New, Courier, monospace;"><b><span style="color: #990000;"> name="</span><span style="color: blue;">Q:broker1->broker3</span><span style="color: #990000;">"</span></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> uri="static:(tcp://localhost:61636)"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> duplex="false"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> decreaseNetworkConsumerPriority="true"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> networkTTL="2"</b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> dynamicOnly="true"></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <excludedDestinations></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <topic physicalName="&gt;" /></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> </excludedDestinations></b></span></ol>
<ol><span style="color: #990000; font-family: Courier New, Courier, monospace;"><b></networkConnector></b></span></ol>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"><b><span style="color: #990000;"></networkConnectors></span></b></span></div>
<br />
<li>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> Start broker-2, broker-3 and broker-1. We can start these in any order.</span></li>
<ol>
<li><span style="font-family: Courier New, Courier, monospace;">/apache-activemq-5.8.0/cluster/broker-3/bin$ <span style="color: blue;"><b>./broker-3 console</b></span></span></li>
<li><span style="font-family: Courier New, Courier, monospace;">/apache-activemq-5.8.0/cluster/broker-2/bin$ <b><span style="color: blue;">./broker-2 console</span></b></span></li>
<li><span style="font-family: Courier New, Courier, monospace;">/apache-activemq-5.8.0/cluster/broker-1/bin$ <b><span style="color: blue;">./broker-1 console</span></b></span></li>
</ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's start the consumers C1 on broker-2 and C2, C3 on broker-3 but on the same queue called "moo.bar"</span></li>
<ol>
<li><span style="font-family: Courier New, Courier, monospace;">/apache-activemq-5.8.0/example$ <b><span style="color: blue;">ant consumer -Durl=tcp://localhost:61626 -Dsubject=moo.bar</span></b></span></li>
<li><span style="font-family: Courier New, Courier, monospace;">/apache-activemq-5.8.0/example$ </span><span style="color: blue; font-family: 'Courier New', Courier, monospace; font-weight: bold;">ant consumer -Durl=tcp://localhost:61636 -Dsubject=moo.bar -DparallelThreads=2</span><br /><br /><i><span style="color: #660000;"><span style="background-color: yellow;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The consumer subscriptions are forwarded by broker-2 and broker-3 to their neighboring broker-1 which has a network connector established to both broker-2 and broker-3 by the use of advisory messages. </span></span></span></i></li>
</ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's review the broker web consoles to see the queues and corresponding consumers. </span></li>
<ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">We find that broker-2's web console shows one queue "moo.bar" having 1 consumer, broker-3's web console shows one queue "moo.bar" having 2 concurrent consumers</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Though there are three consumers (C1 on broker-2 and C2,C3 on broker-3), broker-1 sees only two consumers (representing broker-2 and broker-3).</span></li>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-left: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-PdsusOP4JCo/UzOwIpAciJI/AAAAAAAAARk/ArW20RSF3sU/s3200/part4-broker1-wc-cf.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img alt="" border="0" src="http://3.bp.blogspot.com/-PdsusOP4JCo/UzOwIpAciJI/AAAAAAAAARk/ArW20RSF3sU/s3200/part4-broker1-wc-cf.png" height="110" title="http://localhost:8161/admin/queues.jsp" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">http://localhost:8161/admin/queues.jsp</span></td></tr>
</tbody></table>
<br /><br /><br /><br /><br /><br /><br /><br />
<li><a href="http://1.bp.blogspot.com/-TtslRrZzFoA/UzOyHRgd9EI/AAAAAAAAARw/JdDlztLashk/s3200/part4-broker1-consumers.png" imageanchor="1" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://1.bp.blogspot.com/-TtslRrZzFoA/UzOyHRgd9EI/AAAAAAAAARw/JdDlztLashk/s3200/part4-broker1-consumers.png" height="101" width="400" /></a><br /><br /><i style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span style="color: #660000;"><span style="background-color: yellow;">This is because the network connector from broker-1 to broker-2 and to broker-3 by default has a property "conduitSubscriptions" which is true. </span></span></i><i style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span style="color: #660000;"><span style="background-color: yellow;"><br />Due to which broker-3's C2 and C3 which consume messages from the same queue "moo.bar" are treated as one consumer in broker-1.</span></span></i></li>
</ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's produce 30 messages into broker-1's queue moo.bar and see how the messages are divvied among the consumers C1, C2 and C3</span></li>
</ol>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-hjD1C6Vh7aI/UzOy8LUZNXI/AAAAAAAAAR8/1587GKoQA-U/s3200/part4-rcc-cSt.png" imageanchor="1" style="font-family: Times; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" src="http://3.bp.blogspot.com/-hjD1C6Vh7aI/UzOy8LUZNXI/AAAAAAAAAR8/1587GKoQA-U/s3200/part4-rcc-cSt.png" height="320" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Shows how the messages were propagated from producer to consumers C1, C2, C3</span></td></tr>
</tbody></table>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As seen above, even though there were three consumers and 30 messages, they didn't get to process 10 messages each as C2, C3 subscriptions were consolidated into one consumer at broker-1. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="color: #660000; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><i style="background-color: yellow;">conduitSubscriptions="true" is a useful setting if we were creating subscribers on topics as that would prevent duplicate messages. More on this in part 5.</i></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">So, in order to make C2 and C3 subscriptions on queue moo.bar propagate to broker-1, let's redo the same steps 6, 7, 8, 9 and 10 after setting </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">conduitSubscriptions="false" </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">in broker-1's network connector configuration in conf/activemq.xml </span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Here is the new network connector configuration snippet for broker-1:</span></div>
<div>
<br /></div>
<div>
<div>
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b><networkConnectors></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <networkConnector</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> name="Q:broker1->broker2"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> uri="static:(tcp://localhost:61626)"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> duplex="false"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> decreaseNetworkConsumerPriority="true"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> networkTTL="2"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <span style="background-color: yellow;">conduitSubscriptions="false"</span></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> dynamicOnly="true"></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <excludedDestinations></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <topic physicalName="&gt;" /></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> </excludedDestinations></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> </networkConnector></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <networkConnector</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> name="Q:broker1->broker3"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> uri="static:(tcp://localhost:61636)"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> duplex="false"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> decreaseNetworkConsumerPriority="true"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> networkTTL="2"</b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <span style="background-color: yellow;">conduitSubscriptions="false"</span></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> dynamicOnly="true"></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <excludedDestinations></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> <topic physicalName="&gt;" /></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> </excludedDestinations></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b> </networkConnector></b></span><br />
<span style="color: #660000; font-family: Courier New, Courier, monospace;"><b></networkConnectors></b></span></div>
<b><span style="color: #660000; font-family: Courier New, Courier, monospace;">
</span></b></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Upon restarting the brokers and consumers C1, C2 and C3 and producing 30 messages into broker-1's moo.bar queue, we find that all of the three consumer subscriptions are visible at broker-1. As a result broker-1 dispatches 10 messages to each of the consumers in a round-robin fashion to load balance. This is depicted pictorially below. </span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-QHVmlQETZG4/UzPLLNibYEI/AAAAAAAAASI/aJd-5vwQWrI/s3200/part4-broker-1-cc.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-QHVmlQETZG4/UzPLLNibYEI/AAAAAAAAASI/aJd-5vwQWrI/s3200/part4-broker-1-cc.png" height="310" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Shows how the messages were propagated from producer to consumers C1, C2, C3</span></td></tr>
</tbody></table>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Broker-1's web console @ </span><span style="font-family: Courier New, Courier, monospace;">http://localhost:8161/admin/queueConsumers.jsp?JMSDestination=moo.bar</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> shows that broker-1 now sees 3 consumers and dispatches 10 messages to each consumer</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-EVTeRxS5oDI/UzPL5TsbxbI/AAAAAAAAASQ/i3urc2487wk/s3200/part4-broker-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-EVTeRxS5oDI/UzPL5TsbxbI/AAAAAAAAASQ/i3urc2487wk/s3200/part4-broker-1.png" height="248" width="640" /></a></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="p1">
<center>
<div style="text-align: left;">
<br /></div>
</center>
</div>
<div class="p1">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Thus in this part 4 of the blog series, we have seen how we can load balance remote concurrent consumers which are consuming messages from a queue. </span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As always, your comments and feedback is appreciated!</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In the next part 5, we will explore how the same scenario will play out if we were to use a topic instead of a queue. Stay tuned...</span><br />
<br />
<h3 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">References</span></h3>
<ul style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://fusesource.com/docs/esb/4.3/amq_clustering/Networks-Connectors.html">http://fusesource.com/docs/esb/4.3/amq_clustering/Networks-Connectors.html</a></span></li>
</ul>
<h3 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Resources</span></h3>
</div>
<div class="p1">
<ul style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The configuration files (activemq.xml and jetty.xml) of all the brokers used in this blog are available <a href="https://drive.google.com/folderview?id=0B6UoSlNMfxQOWDdwVWp2SEJlSmc&usp=sharing" target="_blank">here</a>.</span></li>
</ul>
</div>
</div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com2tag:blogger.com,1999:blog-8558899295969164571.post-72568495244685324982014-03-04T11:12:00.000-08:002014-03-04T11:20:59.874-08:00ActiveMQ - Network of Brokers Explained - Part 3<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that we have understood the basics of ActiveMQ network connector in <a href="http://akuntamukkala.blogspot.in/2014/02/activemq-network-of-brokers-explained.html" target="_blank">part 1</a> and <a href="http://akuntamukkala.blogspot.in/2014/02/activemq-network-of-brokers-explained_28.html" target="_blank">part 2</a> of this blog series, in this part 3, we will examine </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">how ActiveMQ load balances consumers which connect to a network of brokers.</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: large;">Introduction</span></h2>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Concurrent consumers are used when messages in a queue can be processed out of order and usually to improve message throughput. ActiveMQ broker dispatches messages in a round-robin fashion among the consumers in order to load balance message consumption across concurrent consumers unless the consumer is specified as <a href="http://activemq.apache.org/exclusive-consumer" target="_blank">exclusive</a>. </span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's see the following example where three consumers are concurrently processing messages from queue foo.bar. A producer enqueues 60 messages which are processed by three consumers (20 each) in a round robin fashion.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="http://3.bp.blogspot.com/-DL-vIfgJkI4/UxWJ2fLsdnI/AAAAAAAAAN8/Qx5gayQpo54/s1600/Concurrent+Consumers+-+New+Page.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-DL-vIfgJkI4/UxWJ2fLsdnI/AAAAAAAAAN8/Qx5gayQpo54/s1600/Concurrent+Consumers+-+New+Page.png" height="154" width="400" /></a></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Start three concurrent consumers on queue foo.bar</span></div>
<div style="text-align: left;">
<br /></div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <span style="color: blue;"><b>pwd</b></span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">/Users/akuntamukkala/apache-activemq-5.8.0/example</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<br />
<div style="text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b><span style="color: blue;">ant consumer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -DparallelThreads=3 -Dmax=20</span></b></span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Produce 60 messages </span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <span style="color: blue;"><b>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Dmax=60</b></span></span></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following screenshot shows 3 consumers processing messages from queue foo.bar. 60 messages were enqueued and dequeued.</span> </div>
<div style="text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="http://1.bp.blogspot.com/-XTpRZuxe-iY/UxWM_cH1zBI/AAAAAAAAAOI/1n1H8ZJrn1o/s1600/broker-1-60msgs-concurrent-consumers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-XTpRZuxe-iY/UxWM_cH1zBI/AAAAAAAAAOI/1n1H8ZJrn1o/s1600/broker-1-60msgs-concurrent-consumers.png" height="41" width="400" /></a></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As shown below 20 messages were processed by each of the consumers. </span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="http://3.bp.blogspot.com/-D8cvPCn-060/UxWQpp-fYbI/AAAAAAAAAOU/izzPmSuRxLE/s1600/broker-1-cc-60msgs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-D8cvPCn-060/UxWQpp-fYbI/AAAAAAAAAOU/izzPmSuRxLE/s1600/broker-1-cc-60msgs.png" height="65" width="400" /></a></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The following excerpt from log shows that messages are divvied out among three consumers...</span></div>
<span style="background-color: yellow; font-family: 'Courier New', Courier, monospace; font-size: xx-small;"><br /></span>
<span style="background-color: yellow; font-family: 'Courier New', Courier, monospace; font-size: xx-small;">[java] [Thread-3] Received: 'Message: 1 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: #ea9999;">[java] [Thread-2] Received: 'Message: 0 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: lime;">[java] [Thread-1] Received: 'Message: 2 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: yellow;">[java] [Thread-3] Received: 'Message: 4 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: #ea9999;">[java] [Thread-2] Received: 'Message: 3 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: lime;">[java] [Thread-1] Received: 'Message: 5 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: yellow;">[java] [Thread-3] Received: 'Message: 7 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: #ea9999;">[java] [Thread-2] Received: 'Message: 6 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: lime;">[java] [Thread-1] Received: 'Message: 8 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span style="background-color: yellow;">[java] [Thread-3] Received: 'Message: 10 sent at: Tue Mar 04 13:46:53 IST 2014 ...' (length 1000)</span></span><br />
<div>
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that we have seen how concurrent consumers work on a single broker, we will now examine how they work when consumers are spread across network of brokers. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: large;">Local Vs Remote Consumers</span></h2>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's explore how ActiveMQ handles local and remote consumers with the help of a configuration shown in the figure below. </span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-qLpjRawfKDs/UxVwW-3DGRI/AAAAAAAAANs/37CcP0WmSLU/s1600/local-remote-consumers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-qLpjRawfKDs/UxVwW-3DGRI/AAAAAAAAANs/37CcP0WmSLU/s1600/local-remote-consumers.png" height="233" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Consumer-1 and Consumer-2 consume messages from queue foo.bar on Broker-1 and Broker-2 respectively. Broker-1 established a network connector to Broker-2 to forward queue messages. Producer enqueues messages into queue foo.bar on Broker-1</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: large;">Let's see this in action</span></h2>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Edit Broker-1's configuration </span><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">and open a network connector to Broker-2 and restart Broker-1 and Broker-2</span></li>
</ul>
<br />
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <networkConnectors></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <networkConnector</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> name="T:broker1->broker2"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> uri="static:(tcp://localhost:61626)"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> duplex="false"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> decreaseNetworkConsumerPriority="false"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> networkTTL="2"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> dynamicOnly="true"></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <excludedDestinations></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <queue physicalName="&gt;" /></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> </excludedDestinations></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> </networkConnector></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <networkConnector</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> name="Q:broker1->broker2"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> uri="static:(tcp://localhost:61626)"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> duplex="false"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> decreaseNetworkConsumerPriority="false"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> networkTTL="2"</span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> dynamicOnly="true"></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <excludedDestinations></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> <topic physicalName="&gt;" /></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> </excludedDestinations></span></div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> </networkConnector></span></div>
<div class="separator" style="clear: both;">
</div>
<div class="separator" style="clear: both;">
<span style="color: purple; font-family: Courier New, Courier, monospace; font-size: x-small;"> </networkConnectors></span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Start local consumer, Consumer-1</span></li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b><span style="color: blue;">ant consumer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar</span></b></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> </span></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Start remote consumer, Consumer-2</span></li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <span style="color: blue;"><b>ant consumer -Durl=tcp://localhost:61626 -Dtopic=false -Dsubject=foo.bar</b></span></span></div>
<div>
<ul style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Start producer on Broker-1 to enqueue 100 messages</span></li>
</ul>
</div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <span style="color: blue;"><b>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Dmax=100</b></span></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span style="color: blue;"><b><br /></b></span></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Screenshot showing Broker-1's queues:</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-l6j0PuCu4eQ/UxWeQubKO-I/AAAAAAAAAOk/f9mhzMYIre0/s1600/broker-1-cc-100ed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-l6j0PuCu4eQ/UxWeQubKO-I/AAAAAAAAAOk/f9mhzMYIre0/s1600/broker-1-cc-100ed.png" height="64" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's look at the consumers to see how the messages have been divvied out.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Viqo9QFYBRk/UxYZmaygEBI/AAAAAAAAAPE/fKj3DWtp3_w/s1600/broker-1-localvsremote-equal.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-Viqo9QFYBRk/UxYZmaygEBI/AAAAAAAAAPE/fKj3DWtp3_w/s1600/broker-1-localvsremote-equal.png" height="92" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As you may notice, ActiveMQ broker dispatches the messages equally to local consumer over the remote consumer giving them the same priority. </span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The remote consumer, Consumer-2 is only broker 1 hop away which is less than configured </span><span style="color: #990000; font-family: Courier New, Courier, monospace; font-size: x-small;">networkTTL</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> value of 2.</span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This leads to suboptimal routes especially when brokers are connected such that multiple routes are possible between producers and consumers. It is preferable to dispatch to local consumers over remote consumers in order to ensure shortest path between producers and consumers. </span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">ActiveMQ provides a way to configure the priority between local consumer and remote consumer using the property </span><span style="color: purple; font-family: 'Courier New', Courier, monospace; font-size: x-small;">decreaseNetworkConsumerPriority </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">on the network connector.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">By default, this value is false and hence the local and remote brokers were treated alike. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">If we repeat the above steps after changing the </span><span style="color: purple; font-family: 'Courier New', Courier, monospace; font-size: x-small;">decreaseNetworkConsumerPriority="true" </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">then we find that local consumer, Consumer-1 is given preference over remote consumer, Consumer-2 which is 1 broker hop away. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-n6MCl7LBS08/UxYNS_zlOII/AAAAAAAAAO0/prnaIov46P4/s1600/broker-1-localvsremote.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-n6MCl7LBS08/UxYNS_zlOII/AAAAAAAAAO0/prnaIov46P4/s1600/broker-1-localvsremote.png" height="91" width="640" /></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">ActiveMQ intelligently figures out shortest path in a network of brokers between message producers and consumers. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Please read the following link to gain further understanding of optimal routing by ActiveMQ.</span><br />
<ul style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><a href="http://fusesource.com/docs/esb/4.3/amq_clustering/Networks-OptimizingRoutes.htm">http://fusesource.com/docs/esb/4.3/amq_clustering/Networks-OptimizingRoutes.htm</a>l</span></li>
</ul>
<span style="background-color: white; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18.479999542236328px;">This concludes part 3 of this series where we saw how to differentiate local and remote consumers to assist ActiveMQ determine most optimal path between message producers and consumers. </span><br />
<div class="separator" style="background-color: white; clear: both; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="background-color: white; clear: both; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As always your comments are very welcome. </span></div>
<div class="separator" style="background-color: white; clear: both; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="background-color: white; clear: both; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Stay tuned for part 4 where we will go over l</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">oad balancing remote concurrent consumers...</span></div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com2tag:blogger.com,1999:blog-8558899295969164571.post-37527173639341535292014-02-28T10:20:00.003-08:002014-02-28T10:34:47.019-08:00ActiveMQ - Network of Brokers Explained - Part 2<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<br /></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In this blog we will see how <b>duplex network connectors</b> work.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In the previous <a href="http://akuntamukkala.blogspot.com/2014/02/activemq-network-of-brokers-explained.html" target="_blank">part 1</a> we created a network connector from broker-1 and broker-2. We were able to see how messages for queue "foo.bar" on broker-1 were forwarded queue "foo.bar" on broker-2 when there was a consumer on broker-2 for queue "foo.bar"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's try doing the reverse by producing messages into broker-2's queue foo.bar and consume from broker-1's queue "foo.bar"</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>ant producer -Durl=tcp://localhost:61626 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100</b></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>ant consumer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar</b></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-9gXrBD4J4Rk/Uw9fj53WVxI/AAAAAAAAALw/VBE3xX0QYH0/s1600/broker-2-100-msg-enqueue.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-9gXrBD4J4Rk/Uw9fj53WVxI/AAAAAAAAALw/VBE3xX0QYH0/s1600/broker-2-100-msg-enqueue.png" height="52" width="400" /></a></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In the previous blog post, we had enqueued/dequeued 100 messages. Hence the #messages enqueued now shows as 200 here. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As shown above, 100 new messages are enqueued on foo.bar queue on broker-2 but there are no consumers though there is a network connector for all queues from broker-1 to broker-2. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="color: purple; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><i style="background-color: yellow;">The reason is that a network connector unless specified as "duplex" is unidirectional from the source to the destination broker. </i></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's change the following attribute highlighted in yellow in </span><span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> configuration file for broker-1.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b><span style="font-size: xx-small;"> </span><span style="font-size: x-small;"> <networkConnectors></span></b></span><span style="font-size: x-small;"><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> <networkConnector </b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><b style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;"><span style="color: #990000; font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;">name="</span><span style="color: blue;">T:broker1->broker2</span><span style="color: #990000;">"</span></span><span style="color: #990000; font-family: 'Courier New', Courier, monospace;"> </span></b><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> uri="static:(tcp://localhost:61626)" </b></span></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> duplex="false" </b></span></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> decreaseNetworkConsumerPriority="true" </b></span></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> networkTTL="2" </b></span></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> dynamicOnly="true"></b></span></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> <excludedDestinations></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> <queue physicalName="&gt;" /></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> </excludedDestinations></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> </networkConnector></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> <networkConnector </b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><b style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;"><span style="color: #990000; font-family: 'Courier New', Courier, monospace;"> name="</span><span style="font-family: 'Courier New', Courier, monospace;"><span style="background-color: #f3f3f3;"><span style="color: blue;">Q:broker1->broker2</span></span><span style="color: #990000;">" </span></span></b><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> uri="static:(tcp://localhost:61626)" </b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="color: #990000; font-family: 'Courier New', Courier, monospace;"><b><span style="background-color: white;"> </span><span style="background-color: yellow;">duplex="true" </span></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> decreaseNetworkConsumerPriority="true" </b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> networkTTL="2" </b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> dynamicOnly="true"></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> <excludedDestinations></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> <topic physicalName="&gt;" /></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> </excludedDestinations></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> </networkConnector></b></span><br style="background-color: white; font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; line-height: 18.479999542236328px;" /><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b> </networkConnectors></b></span></span></div>
<div>
<span style="font-size: x-small;"><span style="background-color: white; color: #990000; font-family: 'Courier New', Courier, monospace;"><b><br /></b></span></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's restart the brokers and connect to the brokers using jConsole.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here is broker-1 jConsole MBean tab screenshot which shows the following:</span></div>
<div>
<ol>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Q:broker1->broker2 network connector is duplex.</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">There is now a dynamic producer into broker-1 from broker-2 because the<br />Q:broker1->broker2 network connector is "duplex".</span></li>
</ol>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/--_RrI8cnwBE/UxDEPOsl96I/AAAAAAAAAMA/-16Cu0drbEs/s1600/broker-1-jconsole.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/--_RrI8cnwBE/UxDEPOsl96I/AAAAAAAAAMA/-16Cu0drbEs/s1600/broker-1-jconsole.png" height="363" width="640" /></a></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Here is broker-2 jConsole MBean tab screenshot which shows the following:</span></div>
<div>
<ol>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Duplex network connector from broker-2 to broker-1</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Two dynamic message producers from broker-1 to broker-2</span></li>
<ol>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Please note that "Q:broker1->broker2" network connector shows as duplex as configured in activemq.xml </span></li>
</ol>
</ol>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-0AzE6EYN74s/UxDF6PVbntI/AAAAAAAAAMM/qwBTjNq7-Bk/s1600/broker-2-jconsole.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-0AzE6EYN74s/UxDF6PVbntI/AAAAAAAAAMM/qwBTjNq7-Bk/s1600/broker-2-jconsole.png" height="368" width="640" /></a></div>
<h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see this in action</span></h2>
<div class="separator" style="clear: both;">
</div>
<ol>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Producer 100 messages into broker-2</span></li>
</ol>
<div class="separator" style="clear: both;">
<span style="color: blue; font-family: 'Courier New', Courier, monospace;"><span style="font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>ant producer -Durl=tcp://localhost:61626 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100</b></span></span></div>
<div class="separator" style="clear: both;">
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Screenshot of queues in broker-2:</span><span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> http://localhost:9161/admin/queues.jsp</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-3GR-Px1qTyY/UxDJNaLyBkI/AAAAAAAAAMY/_zb2Y-zJGgg/s1600/broker-2-duplex-noconsumers.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-3GR-Px1qTyY/UxDJNaLyBkI/AAAAAAAAAMY/_zb2Y-zJGgg/s1600/broker-2-duplex-noconsumers.png" height="126" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; text-align: left;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; text-align: left;"></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> 2. Create a consumer on foo.bar on broker-1</span></div>
<div>
<br /></div>
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>ant consumer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar</b></span><br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following screenshot from broker-2 shows that all the 100 messages have been dequeued by a consumer (dynamically forwarded to broker-1).</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">http://localhost:9161/admin/queues.jsp</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-cU-rL6X4Pc4/UxDKaUKpOmI/AAAAAAAAAMk/96iDzegLiu8/s1600/broker-2-duplex-msg-dequeued.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-cU-rL6X4Pc4/UxDKaUKpOmI/AAAAAAAAAMk/96iDzegLiu8/s1600/broker-2-duplex-msg-dequeued.png" height="128" width="640" /></a></div>
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following screenshot shows the details of this dynamic consumer on broker-2's foo.bar queue.</span></div>
<div class="separator" style="clear: both;">
<br /></div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">http://localhost:9161/admin/queueConsumers.jsp?JMSDestination=foo.bar</span><br />
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-ZzaJ45Z4pP0/UxDMD79sgII/AAAAAAAAAMw/PZpFHwP9ukE/s1600/broker-2-duplex-dynamicconsumer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-ZzaJ45Z4pP0/UxDMD79sgII/AAAAAAAAAMw/PZpFHwP9ukE/s1600/broker-2-duplex-dynamicconsumer.png" height="102" width="640" /></a></div>
<div>
<br /></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The following screenshot shows that the 100 messages which were dynamically moved from broker-2's foo.bar queue to broker-1's foo.bar queue have been successfully consumed by the consumer which we created in step #2</span></div>
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-f66hLC4SMg0/UxDOcV16YPI/AAAAAAAAAM8/jGXctbLsV3M/s1600/broker-1-duplex-msg-consumed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-f66hLC4SMg0/UxDOcV16YPI/AAAAAAAAAM8/jGXctbLsV3M/s1600/broker-1-duplex-msg-consumed.png" height="122" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This concludes part 2 of this series where we saw how duplex network connectors work. </span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">As always your comments are very welcome. </span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Stay tuned for part 3 where we will go over l</span><span style="background-color: white; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18.479999542236328px;">oad balancing consumers on local/remote brokers...</span></div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"></span></div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-41768406686002954702014-02-24T07:48:00.000-08:002014-03-26T07:33:03.420-07:00ActiveMQ - Network of Brokers Explained<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="p1">
<h2 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-large;"><b>Objective</b></span></h2>
</div>
<div class="p1">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">This 7 part blog series is to share about how to create network of ActiveMQ brokers in order to achieve high availability and scalability.</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><br />
<h3 style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: large;"><b>Why network of brokers?</b></span></h3>
</div>
<div class="p2">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">ActiveMQ message broker is a core component of messaging infrastructure in an enterprise. It needs to be highly available and dynamically scalable to facilitate communication between dynamic heterogeneous distributed applications which have varying capacity needs. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Scaling enterprise applications on commodity hardware is a rage nowadays. ActiveMQ caters to that very well by being able to create a network of brokers to share the load. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Many times applications running across geographically distributed data centers need to coordinate messages. Running message producers and consumers across geographic regions/data centers can be architected better using network of brokers. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="p2">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">ActiveMQ uses transport connectors over which it communicates with message producers and consumers. However, in order to facilitate broker to broker communication, ActiveMQ uses <b>network connectors</b>. </span></div>
<div class="p2">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="p2">
<span style="color: blue; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><i>A network connector is a bridge between two brokers which allows on-demand message forwarding. </i></span></div>
<div class="p2">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="p2">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In other words, if Broker B1 initiates a network connector to Broker B2 then the messages on a channel (queue/topic) on B1 get forwarded to B2 if there is at least one consumer on B2 for the same channel. If the network connector was configured to be duplex, the messages get forwarded from B2 to B1 on demand. </span></div>
<div class="p2">
<br /></div>
<div class="p2">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This is very interesting because it is now possible for brokers to communicate with each other dynamically. </span><br />
<ol style="text-align: left;">
</ol>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">In this 7 part blog series, we will look into the following topics to gain understanding of this very powerful ActiveMQ feature:</span></div>
<ol class="ol1" style="text-align: left;">
<ol class="ol2">
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Network Connector Basics - Part 1</span></li>
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Duplex network connectors - Part 2 </span></li>
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Load balancing consumers on local/remote brokers - Part 3</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Load-balance consumers/subscribers on remote brokers </span></li>
<ol class="ol3">
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Queue: Load balance remote concurrent consumers </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">- Part 4</span></li>
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Topic: Load Balance Durable Subscriptions on Remote Brokers - Part 5</span></li>
</ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Store/Forward messages and consumer failover</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> - Part 6</span></li>
<ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">How to prevent stuck</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">messages</span></span></li>
</ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Virtual Destinations - Part 7</span></li>
<ol class="ol3">
</ol>
</ol>
</ol>
<div class="p2">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">To give credit where it is due, t</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">he following URLs have helped me in creating this blog post series. </span><br />
<ol>
<li><a href="http://www.slideshare.net/dejanb/advanced-messaging-with-apache-activemq" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;" target="_blank">Advanced Messaging with ActiveMQ</a><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> by </span><a href="https://twitter.com/dejanb" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;" target="_blank">Dejan Bosanac</a><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"> [Slides 32-36] </span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://www.jakubkorab.net/2011/11/understanding-activemq-broker-networks.html" target="_blank">Understanding ActiveMQ Broker Networks</a> by </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="https://twitter.com/jakekorab" target="_blank">Jakub Korab</a> </span></li>
</ol>
<h2 style="text-align: left;">
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Prerequisites</span></b></h2>
</div>
<div class="p2">
<ol class="ol1">
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">ActiveMQ 5.8.0 – To create broker instances</span></li>
<li class="li2"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Apache Ant – To run ActiveMQ sample producer and consumers for demo.</span></li>
</ol>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">We will use multiple ActiveMQ broker instances on the same machine for the ease of demonstration.</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="p2">
<h2 style="text-align: left;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: x-large;">Network Connector Basics - Part 1</span></h2>
<div>
<div class="p2">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The following diagram shows how a network connector functions. It bridges two brokers and is used to forward messages from Broker-1 to Broker-2 on demand if established by Broker-1 to Broker-2. </span></div>
<div class="p3">
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-zlxQBnsaE-U/UwuLpKOQDeI/AAAAAAAAAKI/2Z9zX6ETXTs/s1600/NC.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-zlxQBnsaE-U/UwuLpKOQDeI/AAAAAAAAAKI/2Z9zX6ETXTs/s1600/NC.png" height="146" width="400" /></a></div>
<br /></div>
<div class="p3">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">A network connector can be duplex so messages could be forwarded in the opposite direction; from Broker-2 to Broker-1, once there is a consumer on Broker-1 for a channel </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">which exists in Broker-2. More on this in Part 2</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<h3 style="text-align: left;">
<div class="separator" style="clear: both; font-size: medium; font-weight: normal;">
</div>
</h3>
<h3 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Setup network connector between broker-1 and broker-2</b></span></h3>
<ul style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Create two broker instances, say broker-1 and broker-2</span></li>
</ul>
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: blue;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>pwd</b></span></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">/Users/akuntamukkala/apache-activemq-5.8.0/bin</span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>./activemq-admin create ../bridge-demo/broker-1</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;"><br /></span>
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>./activemq-admin create ../bridge-demo/broker-2</b></span><br />
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Since we will be running both brokers on the same machine, let's configure broker-2 such that there are no port conflicts. </span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"></span>
<br />
<ul style="text-align: left;"><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Edit </span><span style="color: blue; font-family: Courier New, Courier, monospace;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/activemq.xml</span></li>
</span></ul>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">
</span>
<br />
<ol style="text-align: left;"><ol>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Change transport connector to 61626 from 61616</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Change AMQP port from 5672 to 6672 (won't be using it for this blog)</span></li>
</ol>
</ol>
<ul style="text-align: left;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Edit </span><span style="color: blue; font-family: Courier New, Courier, monospace;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/conf/jetty.xml</span></li>
</span></ul>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
</span>
<br />
<ol style="text-align: left;">
<ol>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Change web console port </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">to 9161 from 8161</span></li>
</ol>
</ol>
<ul style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Configure Network Connector from broker-1 to broker-2</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br />Add the following XML snippet to<span style="color: blue;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;"><span style="color: blue;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/conf/activemq.xml</span></span></li>
</ul>
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b><span style="font-size: x-small;"> </span> <networkConnectors></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <networkConnector </b></span><br />
<b><span style="color: #990000; font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: Courier New, Courier, monospace;"><span style="color: #990000;">name="</span><span style="background-color: white;"><span style="color: blue;">T:broker1->broker2</span></span><span style="color: #990000;">"</span></span><span style="color: #990000; font-family: 'Courier New', Courier, monospace;"> </span></b><br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> uri="static:(tcp://localhost:61626)" </b></span></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> duplex="false" </b></span></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> decreaseNetworkConsumerPriority="true" </b></span></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> networkTTL="2" </b></span></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: #990000;"><b> dynamicOnly="true"></b></span></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <excludedDestinations></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <queue physicalName="&gt;" /></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> </excludedDestinations></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> </networkConnector></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <networkConnector </b></span><br />
<b><span style="color: #990000; font-family: Courier New, Courier, monospace;"> name="</span><span style="font-family: Courier New, Courier, monospace;"><span style="background-color: #f3f3f3;"><span style="color: blue;">Q:broker1->broker2</span></span><span style="color: #990000;">" </span></span></b><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> uri="static:(tcp://localhost:61626)" </b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> duplex="false" </b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> decreaseNetworkConsumerPriority="true" </b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> networkTTL="2" </b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> dynamicOnly="true"></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <excludedDestinations></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> <topic physicalName="&gt;" /></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> </excludedDestinations></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> </networkConnector></b></span><br />
<span style="color: #990000; font-family: Courier New, Courier, monospace;"><b> </networkConnectors></b></span><br />
<div class="p3">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The above XML snippet configures two network connectors "T:broker1->broker2" (only topics as queues are excluded) and "Q:broker1->broker2" (only queues as topics are excluded). </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">This allows for nice separation between network connectors used for topics and queues. </span><br />
<div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The name can be arbitrary although I prefer to specify the [type]:[source broker]->[destination broker].</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">The URI attribute specifies how to connect to broker-2</span><br />
<ul style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Start broker-2</span></li>
</ul>
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>pwd</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-2/bin</span><br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace; font-size: x-small;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>./broker-2 console</b></span><br />
<ul style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Start broker-1</span></li>
</ul>
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>pwd</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">/Users/akuntamukkala/apache-activemq-5.8.0/bridge-demo/broker-1/bin</span><br />
<br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:bin akuntamukkala$ <b>./broker-1 console</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Logs on broker-1 show 2 network connectors being established with broker-2</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="color: #38761d; font-family: Courier New, Courier, monospace;"> INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626</span><br />
<span style="color: #38761d; font-family: Courier New, Courier, monospace;"> INFO | Connector vm://broker-1 Started</span><br />
<span style="color: #38761d; font-family: Courier New, Courier, monospace;"> INFO | Establishing network connection from vm://broker-1?async=false&network=true to tcp://localhost:61626</span><br />
<span style="color: #38761d;"><span style="font-family: Courier New, Courier, monospace;"> </span><span style="font-family: Courier New, Courier, monospace;">INFO | Network connection between vm://broker-1#24 and tcp://localhost/127.0.0.1:61626@52132(broker-2) has been established.</span></span><br />
<span style="color: #38761d; font-family: 'Courier New', Courier, monospace;"> INFO | Network connection between vm://broker-1#26 and tcp://localhost/127.0.0.1:61626@52133(broker-2) has been established.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Web Console on broker-1 @ </span><span style="font-family: Courier New, Courier, monospace;">http://localhost:8161/admin/connections.jsp</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> shows the two network connectors established to broker-2</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-g0rsnqWTDpc/Uw105S7KF-I/AAAAAAAAAK0/ZfPaw788yzE/s1600/broker-1-NCs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-g0rsnqWTDpc/Uw105S7KF-I/AAAAAAAAAK0/ZfPaw788yzE/s1600/broker-1-NCs.png" height="76" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The same on broker-2 does not show any network connectors since no network connectors were initiated by broker-2</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<h2 style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: x-large;"><b>Let's see this in action</b></span></h2>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's produce 100 persistent messages on a queue called "foo.bar" on broker-1. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>pwd</b></span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">/Users/akuntamukkala/apache-activemq-5.8.0/example</span><br />
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Ddurable=true -Dsubject=foo.bar -Dmax=100</b></span><br />
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">broker-1 web console shows that 100 messages have been enqueued in queue "foo.bar"</span></div>
<div class="p1">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span></div>
<div class="p1">
<span style="font-family: 'Courier New', Courier, monospace;">http://localhost:8161/admin/queues.jsp</span></div>
<div class="p1">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-rlQh8aHdhwc/Uw1AETQejeI/AAAAAAAAAKY/nvvplH8g_hk/s1600/broker-1-100msgs-produced.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-rlQh8aHdhwc/Uw1AETQejeI/AAAAAAAAAKY/nvvplH8g_hk/s1600/broker-1-100msgs-produced.png" height="169" width="640" /></a></div>
<div class="p1">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Let's start a consumer on a queue called "foo.bar" on broker-2. The important thing to note here is that the destination name "foo.bar" should match exactly. </span></div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="color: blue; font-family: Courier New, Courier, monospace;">Ashwinis-MacBook-Pro:example akuntamukkala$ <b>ant consumer -Durl=tcp://localhost:61626 -Dtopic=false -Dsubject=foo.bar </b></span></div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">We find that all the 100 messages from broker-1's foo.bar queue get forwarded to broker-2's foo.bar queue consumer.</span> </div>
<div class="p1">
<br /></div>
<div class="p1">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">broker-1 admin console at</span> <span style="font-family: Courier New, Courier, monospace;">http://localhost:8161/admin/queues.jsp</span></div>
<div class="p1">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-BePy4d-3hy0/Uw1EWtXdYOI/AAAAAAAAAKk/AtFcjSGL8T8/s1600/broker-1-100msgs-consumed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-BePy4d-3hy0/Uw1EWtXdYOI/AAAAAAAAAKk/AtFcjSGL8T8/s1600/broker-1-100msgs-consumed.png" height="140" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">broker-2 admin console @</span> <span style="font-family: Courier New, Courier, monospace;">http://localhost:9161/admin/queues.jsp</span><span style="font-family: Courier New, Courier, monospace;"> </span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">shows that the consumer we had started has consumed all 100 messages which were forwarded on-demand from broker-1 </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-vpE1ObOZA8A/Uw18jEa2FNI/AAAAAAAAALE/LEZkGBFAI70/s1600/broker-2-dequeue.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-vpE1ObOZA8A/Uw18jEa2FNI/AAAAAAAAALE/LEZkGBFAI70/s1600/broker-2-dequeue.png" height="78" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">broker-2 consumer details on foo.bar queue</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-WkSJSFRdzSM/Uw1851If9DI/AAAAAAAAALM/zp1ubzC18CU/s1600/broker-2-consumer-details.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-WkSJSFRdzSM/Uw1851If9DI/AAAAAAAAALM/zp1ubzC18CU/s1600/broker-2-consumer-details.png" height="64" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">broker-1 admin console shows that all 100 messages have been dequeued [forwarded to broker-2 via the network connector]. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-sWOHiBGaHdk/Uw19asTiEEI/AAAAAAAAALY/ys5bvHViWh0/s1600/broker-1-nc-queue-consumer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-sWOHiBGaHdk/Uw19asTiEEI/AAAAAAAAALY/ys5bvHViWh0/s1600/broker-1-nc-queue-consumer.png" height="70" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="p1">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">broker-1 consumer details on "foo.bar" queue shows that the consumer is created on demand: [name of connector]_[destination broker]_inbound_[source broker]</span></div>
<div class="p1">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ZSjWIXJxK_w/Uw195ItFSyI/AAAAAAAAALg/JJZWB4H70xY/s1600/broker-1-nc-consumer-details.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-ZSjWIXJxK_w/Uw195ItFSyI/AAAAAAAAALg/JJZWB4H70xY/s1600/broker-1-nc-consumer-details.png" height="66" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Thus we have seen the basics of network connector in ActiveMQ. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As always, please feel to comment about anything that can be improved. Your inputs are welcome!</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Stay tuned for Part 2...</span></div>
<div class="p1">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
</div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com16tag:blogger.com,1999:blog-8558899295969164571.post-39550941352147897512014-01-01T02:58:00.002-08:002014-01-01T11:36:02.796-08:00Temp, Store and Memory Percent Usage in ActiveMQ<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">In order to effectively use ActiveMQ, it is very important to understand how ActiveMQ manages memory and disk resources to handle non-persistent and persistent messages. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">ActiveMQ has three key parameters which need to be kept under check. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-r0YVkOvT9DQ/UsM5NBJWZkI/AAAAAAAAAIc/21nj12cByAo/s1600/parameters.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="273" src="http://3.bp.blogspot.com/-r0YVkOvT9DQ/UsM5NBJWZkI/AAAAAAAAAIc/21nj12cByAo/s1600/parameters.png" width="400" /></a></div>
<br />
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;"><b>Temp Percent Usage</b></span></li>
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is the % of assigned disk storage that has been used up to spool non-persistent messages</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Non persistent messages are those that don't survive broker restart </span></li>
</ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;"><b>Store Percent Usage</b></span></li>
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is the % of assigned disk space that has been used up to store persistent messages</span></li>
</ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;"><b>Memory Percent Usage</b></span></li>
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is the % of assigned memory of the broker that has been used up to keep track of destinations, cache messages etc. This value needs to be lesser than -Xmx (Max JVM heap size)</span></li>
</ol>
</ol>
<span style="font-family: Arial, Helvetica, sans-serif;">This blog attempts to clarify how store,temp and memory % usage of a single node ActiveMQ broker instance are calculated. We are using ActiveMQ version 5.8.0 for this explanation. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Once we gain clarity on how ActiveMQ operates these values, we can fine tune ActiveMQ using key configuration settings in order to handle the following use cases. </span><br />
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Large number of destinations (queues/topics)</span></li>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The destinations could be created/deleted as needed</span></li>
</ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Slow consumers</span></li>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is huge issue when consumers are unable to keep up with the rate at which messages are being produced. </span></li>
</ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Message Burst</span></li>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Rapid influx of large number of messages with huge payload size for a brief period of time</span></li>
</ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Inappropriate Resource utilization</span></li>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Few destinations chewing up resources causing others to starve</span></li>
</ul>
</ol>
<div>
<h4 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Scaling Strategies</span></h4>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">If you are interested to know how ActiveMQ can be scaled horizontally, please refer to a slide deck created by Bosanac Dejan. You can find it </span><span style="font-family: Arial, Helvetica, sans-serif;"><a href="http://www.slideshare.net/dejanb/advanced-messaging-with-apache-activemq" target="_blank">here</a> </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">It contains different ActiveMQ topologies which can be used effectively to meet volume throughput in addition to various parameters to tune ActiveMQ. I found it extremely useful.</span></div>
<div>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><br /></span></h3>
<h3 style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: small;">Let's dig right in...</span></h3>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The following XML snippet is taken from configuration activemq.xml. The values specified for memoryUsage, storeUsage and tempUsage are for discussion purposes only. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <memoryUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <memoryUsage limit="256 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </memoryUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <storeUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <storeUsage limit="512 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </storeUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <tempUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <tempUsage limit="256 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </tempUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </systemUsage></span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> </systemUsage></span><br />
<br />
<ol style="text-align: left;">
<li><span style="font-family: Arial, Helvetica, sans-serif;">Memory Usage </span></li>
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">256MB of JVM memory is available for the broker. Not to be confused with -Xmx parameter.</span></li>
</ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Store Usage: </span></li>
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is the disk space used by persistent messages (using KahaDB)</span></li>
</ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Temp Usage: </span></li>
<ol>
<li><span style="font-family: Arial, Helvetica, sans-serif;">This is the disk space used by non-persistent message, assuming we are using default KahaDB. ActiveMQ spools non-persistent messages to disk in order to prevent broker running out of memory</span></li>
</ol>
</ol>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><b>Understanding Temp Usage</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />Broker availability is critical for message infrastructure. Hence <a href="http://java.dzone.com/articles/activemq-understanding-memory" target="_blank">producer flow control</a> is a protection mechanism that prevents a runaway producer from pumping non-persistent messages into a destination when there are no consumers or when consumer(s) is unable to keep up with the rate at which messages are being produced into the destination.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />Let’s take an example of producing non-persistent messages having 1MB payload size into a destination “foo.bar” in a local broker instance</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">C:\apache-activemq-5.8.0\example>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Ddurable=false -DmessageSize=1048576</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />The producer eventually hangs as temp % usage hits 100%</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-V2qFDN7p6b4/UsNIRY7GB3I/AAAAAAAAAIs/caewBFEq2bI/s1600/temp-100.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="152" src="http://3.bp.blogspot.com/-V2qFDN7p6b4/UsNIRY7GB3I/AAAAAAAAAIs/caewBFEq2bI/s1600/temp-100.png" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Since the messages are non-persistent, they are going to be stored in tmp_storage on the disk</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-bPL3Rr1D90E/UsNIW5MGCII/AAAAAAAAAI0/Htf3gHSp8C4/s1600/temp_storage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-bPL3Rr1D90E/UsNIW5MGCII/AAAAAAAAAI0/Htf3gHSp8C4/s1600/temp_storage.png" width="244" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">ActiveMQ provides mechanism to tune memory usage per destination. Here we have a generic policy for all queues where producer flow control is enabled and destination memory limit is 100MB (again this is only for illustration purposes).</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><policyEntry queue=">" optimizedDispatch="true" producerFlowControl="true" cursorMemoryHighWaterMark="30" memoryLimit="100 mb" > </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />The temp % usage is calculated as follows:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">(Size of the tmp_storage folder / temp usage memory limit ) * 100 </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">So in our case: </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">265,025,856/(256*1024*1024) * 100 = 99.8 ~ 100% as shown in the broker console. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The following log message shows up in activemq.log</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> INFO | Usage(default:temp:queue://foo.bar:temp) percentUsage=99%, usage=268535808, limit=268435456, percentUsageMinDelta=1%;Parent:Usage(default:temp</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">) percentUsage=100%, usage=268535808, limit=268435456, percentUsageMinDelta=1%: Temp Store is Full (99% of 268435456). Stopping producer (ID:AKUNTAMU-</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">1-61270-1388528939599-1:1:1:1) to prevent flooding queue://foo.bar. See http://activemq.apache.org/producer-flow-control.html for more info (blocking</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">for: 1421s)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Let's take another example...</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Consider the following system usage configuration. We have reduced tempUsage to 50MB while keeping the same destination level policy.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <memoryUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <memoryUsage limit="256 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </memoryUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <storeUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <storeUsage limit="512 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </storeUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <tempUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <tempUsage limit="50 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </tempUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </systemUsage></span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </systemUsage></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"> </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">In this case we find that temp usage balloons to 191%</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-m4-GtEFzExk/UsNOUzLkQNI/AAAAAAAAAJE/snIjl4kYLdA/s1600/temp-191.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="149" src="http://1.bp.blogspot.com/-m4-GtEFzExk/UsNOUzLkQNI/AAAAAAAAAJE/snIjl4kYLdA/s1600/temp-191.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">temp_storage stops growing at close to 96MB and producer hangs..</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-UUErfABNODE/UsNOpiGP18I/AAAAAAAAAJM/YmggChdoAmQ/s1600/temp_storage-191.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-UUErfABNODE/UsNOpiGP18I/AAAAAAAAAJM/YmggChdoAmQ/s1600/temp_storage-191.png" width="248" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Temp percent usage is 191% because (95.5MB / 50 MB)*100 where 95.5 MB is size of the folder and 50MB is temp usage limit. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">The destination has a limit of 100MB so the temp_storage didn't grow past it. It is sort of confusing which is caused by the fact that temp usage limit is less that per destination memory limit. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Store Usage</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />Let’s repeat the same test with persistent messages.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">The system usage is configured as follows:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <memoryUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <memoryUsage limit="256 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </memoryUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <storeUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <storeUsage limit="512 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </storeUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <tempUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> <tempUsage limit="256 mb"/></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </tempUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </systemUsage></span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> </systemUsage></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Per destination policy is as follows:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><policyEntry queue=">" optimizedDispatch="true" producerFlowControl="true" cursorMemoryHighWaterMark="30" memoryLimit="100 mb" > </span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Let's produce 1MB persistent messages into a queue named "foo.bar"</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">C:\apache-activemq-5.8.0\example>ant producer -Durl=tcp://localhost:61616 -Dtopic=false -Dsubject=foo.bar -Ddurable=true -DmessageSize=1048576</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Producer hangs after 512 messages<br /><br />The following log statement appears in broker log file</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> INFO | Usage(default:store:queue://foo.bar:store) percentUsage=99%, usage=537210471, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:st</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ore) percentUsage=100%, usage=537210471, limit=536870912,percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (ID: </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">AKUNTAMU-1-31754-1388571228628-1:1:1:1) to prevent flooding queue://foo.bar. See http://activemq.apache.org/producer-flow-control.html for more info (</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">blocking for: 155s)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Broker store usage is 100% as shown below.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-Ofqn-6YJJR4/UsPsEWm1OcI/AAAAAAAAAJc/tcUe4oCLqJ0/s1600/store-100.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="145" src="http://1.bp.blogspot.com/-Ofqn-6YJJR4/UsPsEWm1OcI/AAAAAAAAAJc/tcUe4oCLqJ0/s1600/store-100.png" width="320" /></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;">Since the messages are persistent, they need to be saved onto the file system. Store usage limit is 512MB. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="text-align: center;">
<a href="http://3.bp.blogspot.com/-R1hTub1iBhk/UsPs6tGgCKI/AAAAAAAAAJo/yehc7NXWPh4/s1600/store-disk-100.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: right;"><img border="0" height="320" src="http://3.bp.blogspot.com/-R1hTub1iBhk/UsPs6tGgCKI/AAAAAAAAAJo/yehc7NXWPh4/s1600/store-disk-100.png" width="246" /></a><br />
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;">The above screenshot shows the kahadb folder where persistent messages is 543 MB (512MB for the messages and other database related files)</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><b>Memory Usage</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">In the above example, the memory usage percentage is 11. How did that come about?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">As per the destination policy, the memory allocated per destination is 100MB and the </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">cursorMemoryHighWaterMark </span><span style="font-family: Arial, Helvetica, sans-serif;">is specified to be 30. So 30% of 100MB is 30MB. Hence 30MB is used to store messages in memory for faster processing in addition to be being stored in the KahaDB. . </span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">The memory usage limit is configured to be 256MB. So 30MB is ~ 11% of 256</span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">(30/256) * 100 ~ 11%</span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">So if we were to have 9 such queues where similar situation was to occur then we would have exhausted broker memory usage as 11 % * 9 = 99% ~ 100% </span></div>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">Memory usage is the amount of memory used by the broker for storing messages. Many a times, this can become a bottleneck as once this space is full, the broker will stall the producers. There are trade-offs between fast processing and effective memory management. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">If we keep more messages in memory, the processing is faster. However the memory consumption will be very high. On the contrary, if messages are kept on the disk then processing will become slow.</span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Conclusion</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">We have seen in this blog how store, temp and memory usage work in ActiveMQ. % of store and temp usage cannot be configured per destination while % of memory usage can be because of </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">cursorMemoryHighWaterMark.</span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Hope you found this information useful. The examples given here are for explanation purposes only and not meant to be production ready. You will need to do proper capacity planning and determine your broker topology for optimal configuration. Feel free to reach out if any comments! </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>References</b></span><br />
<ul style="text-align: left;"><span style="font-family: Arial, Helvetica, sans-serif;"><b>
</b>
<li><a href="http://blog.raulkr.net/2012/08/demystifying-producer-flow-control-and.html">http://blog.raulkr.net/2012/08/demystifying-producer-flow-control-and.html</a></li>
<li><a href="http://tmielke.blogspot.com/2011/02/observations-on-activemqs-temp-storage.html">http://tmielke.blogspot.com/2011/02/observations-on-activemqs-temp-storage.html</a></li>
<li><a href="http://activemq.apache.org/javalangoutofmemory.html">http://activemq.apache.org/javalangoutofmemory.html</a></li>
<li><a href="http://www.slideshare.net/dejanb/advanced-messaging-with-apache-activemq">http://www.slideshare.net/dejanb/advanced-messaging-with-apache-activemq</a> -Bosanac Dejan</li>
<li><a href="http://www.pepperdust.org/?p=150">http://www.pepperdust.org/?p=150</a></li>
<li><a href="http://stackoverflow.com/questions/2361541/how-do-you-scale-your-activemq-vertically">http://stackoverflow.com/questions/2361541/how-do-you-scale-your-activemq-vertically</a></li>
</span></ul>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<div style="text-align: left;">
<br /></div>
</div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com1tag:blogger.com,1999:blog-8558899295969164571.post-10216224192689174862013-08-19T14:31:00.001-07:002013-08-19T15:00:12.362-07:00Active 5.8.0 Master Slave Web Console Fix<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css"></link>
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js" type="text/javascript"></script>
<script language="javascript">
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = 'http://alexgorbatchev.com/pub/sh/current/scripts/clipboard.swf';
SyntaxHighlighter.all();
</script>
</div>
Recently I ran into an issue when I attempted to use ActiveMQ 5.8.0 Web Console to monitor a two node Master/Slave ActiveMQ cluster.<br />
<div>
<br /></div>
<div>
The ActiveMQ 5.8.0 Web Console (hosted in a Tomcat container) connects to remote ActiveMQ instance over JMX. In ActiveMQ 5.8.0, the JMX connector is enabled for both Master and Slave nodes. Due to which the Web Console is unable to determine which node is the master node. Hence the issue. </div>
<div>
<br /></div>
<div>
In order to get around this problem, I created a class which extends the RemoteJMXBrokerFacade class by adding intelligence to an overridden findBrokers(...) method such that the Web Console considers an ActiveMQ node as a master node if and only if it is able to fetch a reference to the BrokerViewMBean. </div>
<div>
<br /></div>
<div>
The above operation throws an IllegalStateException if the broker was to be slave node. </div>
<div>
<br />
The Web Console gets the list of JMX Connectors for all the ActiveMQ Brokers in a cluster via the following property<br />
<br />
<br />
<div class="p1">
<span class="s1">-Dwebconsole.jmx.url=</span><span class="s2">service:jmx:rmi:///jndi/rmi://amq-node-1-machine:12099/jmxrmi</span><span class="s1">,</span>service:jmx:rmi:///jndi/rmi://amq-node-2-machine:12099/jmxrmi</div>
<div class="p1">
<br /></div>
</div>
<div>
The following is the source code:</div>
<div>
<br /></div>
<pre class="brush: xml;overflow-x:scroll !important;highlight: [49,50,51,52]">package org.apache.activemq.web;
import java.io.IOException;
import java.util.Set;
import javax.management.MBeanServerConnection;
import javax.management.MBeanServerInvocationHandler;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import org.apache.activemq.broker.jmx.BrokerViewMBean;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* This class helps ActiveMQ Web Console determine if the remote broker instance is master or slave
*
* In the webconsole-properties.xml, replace the <code>org.apache.activemq.web.RemoteJMXBrokerFacade</code> class
* with <code>org.apache.activemq.web.MasterSlaveAwareJMXBrokerFacade</code>
* <pre>
* {@code
* <bean id="brokerQuery" class="org.apache.activemq.web.MasterSlaveAwareJMXBrokerFacade" autowire="constructor" destroy-method="shutdown">
<property name="configuration" ref="configuration"/>
<property name="brokerName"><null/></property>
</bean>
* }
* </pre>
* @author AKUNTAMUKKALA
*
*/
public class MasterSlaveAwareJMXBrokerFacade extends RemoteJMXBrokerFacade {
private static final transient Logger LOG = LoggerFactory
.getLogger(MasterSlaveAwareJMXBrokerFacade.class);
@Override
protected Set<ObjectName> findBrokers(MBeanServerConnection connection)
throws IOException, MalformedObjectNameException {
LOG.debug("Finding brokers");
ObjectName name;
name = new ObjectName("org.apache.activemq:type=Broker,brokerName=*");
Set<ObjectName> brokers = connection.queryNames(name, null);
try {
ObjectName brokerObjectName = (ObjectName) brokers.iterator().next();
BrokerViewMBean mbean = (BrokerViewMBean) MBeanServerInvocationHandler
.newProxyInstance(connection, brokerObjectName,
BrokerViewMBean.class, true);
LOG.debug("Broker Id : " + mbean.getBrokerId());
} catch (Throwable t) {
// TODO Auto-generated catch block
LOG.error("Exception occurred while trying to retrieve BrokerViewMBean", t);
throw new RuntimeException(t);
}
return brokers;
}
}
</pre>
If the Web Console attempts to connect to slave broker, the following exception gets thrown.
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: [1,2]">
SEVERE: Servlet.service() for servlet [jsp] in context with path [/activemq-web-console-5.8.0] threw exception [javax.el.ELException: Error reading 'brokerName' on type org.apache.activemq.web.RemoteJMXBrokerFacade] with root cause
java.lang.IllegalStateException: Broker is not yet started.
at org.apache.activemq.broker.jmx.BrokerView.safeGetBroker(BrokerView.java:459)
at org.apache.activemq.broker.jmx.BrokerView.getBrokerName(BrokerView.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
at javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
</pre>
<br />
This solution has worked for me. I would like to know if you have a better way to fix this.
<br /></div>Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com3tag:blogger.com,1999:blog-8558899295969164571.post-52701528701157284112013-07-09T09:57:00.000-07:002013-07-10T12:11:08.405-07:00Using ActiveMQ - "Master/Slave" configuration with failover protocol <div dir="ltr" style="text-align: left;" trbidi="on">
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css"></link>
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushVb.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js" type="text/javascript"></script>
<script language="javascript">
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = 'http://alexgorbatchev.com/pub/sh/current/scripts/clipboard.swf';
SyntaxHighlighter.all();
</script>
<br />
<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Objective:</b><br />
<br />
This blog is to share how to run highly available ActiveMQ cluster in Master/Slave configuration.<br />
<br />
<b>Introduction:</b><br />
<br />
ActiveMQ broker(s) tends to be a core piece of messaging infrastructure in an enterprise. It is vital for this messaging infrastructure to be highly available and scalable. Please read this <a href="http://activemq.apache.org/networks-of-brokers.html" target="_blank">link</a> in order to learn more about creating network of brokers to support various use cases. One of the popular use case for ActiveMQ is the Master/Slave configuration with shared database. When this configuration is used, the message consumers and producers can operate without interruptions as they use ActiveMQ's connection factory with failover protocol. The failover protocol insulates the consumers and producers from having to deal with any potential downtime or application level reconnection logic when slave ActiveMQ node takes over to become the master which happens if the current master node goes down for any reason. I must caution that this configuration must not be used to mask any issues which take out the master node. We should iron out any causes that result in an unplanned master node outage.<br />
<br />
<b>Overview:</b><br />
<br />
In this blog, I will demonstrate the following:<br />
<ol style="text-align: left;">
<li>Run 2 ActiveMQ nodes in a Master/Slave configuration with a shared KahaDB file based database. </li>
<li>Configure ActiveMQ web console hosted in a Tomcat instance to point to whichever node is Master node in the cluster.</li>
<li>Failover scenarios.</li>
<li>Message publisher and consumer behavior oblivious to the failover.</li>
</ol>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-ybVvJZE-NN0/UdxYw-EjitI/AAAAAAAAAGA/r1GDHNLTnr8/s1600/Screen+Shot+2013-07-09+at+11.59.29+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="http://4.bp.blogspot.com/-ybVvJZE-NN0/UdxYw-EjitI/AAAAAAAAAGA/r1GDHNLTnr8/s400/Screen+Shot+2013-07-09+at+11.59.29+PM.png" width="400" /></a></div>
<b>Prerequisites:</b><br />
<ol style="text-align: left;">
<li>Install ActiveMQ version 5.8.0 </li>
<li>Download ActiveMQ version 5.8.0 web console war from <a href="http://repo1.maven.org/maven2/org/apache/activemq/activemq-web-console/5.8.0/" target="_blank">here</a> </li>
<li>Install Tomcat version 7.0.35 - To host the ActiveMQ Web Console application</li>
</ol>
<b>Deep Dive:</b><br />
<br />
We will create a 2 node ActiveMQ cluster on the same Windows 7 machine. We will need to configure the ports (TCP & JMX) such that there is no conflict. ActiveMQ provides a script called "activemq-admin.bat" which helps in running multiple instances of ActiveMQ using the same binaries, very much like Tomcat. Let's go ahead and create broker-1 and broker-2 instances.<br />
<br />
command: <b><span style="color: blue;">activemq-admin create <DIR-for-ActiveMQ-node></span></b><br />
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: 1">C:\apache-activemq-5.8.0\bin>activemq-admin create ..\cluster\broker-1
Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
Heap sizes: current=125632k free=124976k max=1864192k
JVM args: -Dactivemq.classpath=C:\apache-activemq-5.8.0\bin\..\conf;C:\apache-activemq-5.8.0\bin\..\data; -Dactivemq.home=C:\apache-activemq-5.8.0
\bin\.. -Dactivemq.base=C:\apache-activemq-5.8.0\bin\.. -Dactivemq.data=C:\apache-activemq-5.8.0\bin\..\data -Djava.io.tmpdir=C:\apache-activemq-5.8.0
\bin\..\data\tmp -Dactivemq.conf=C:\apache-activemq-5.8.0\bin\..\conf
Extensions classpath:
[C:\apache-activemq-5.8.0\bin\..\lib,C:\apache-activemq-5.8.0\bin\..\lib\camel,C:\apache-activemq-5.8.0\bin\..\lib\optional,C:\apache-activemq-5.8.0
\bin\..\lib\web,C:\apache-activemq-5.8.0\bin\..\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\bin\..\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\bin\..\data
Running create broker task...
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-1
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-1\bin
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-1\conf
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-1\bin\broker-1.bat
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-1\bin\broker-1
Copying from: C:\apache-activemq-5.8.0\conf\activemq.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-command.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-command.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-demo.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker1.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-dynamic-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker2.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-dynamic-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-jdbc.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-jdbc.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-scalability.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-scalability.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-security.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-security.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-specjms.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-specjms.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker1.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-static-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker2.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-static-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-stomp.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-stomp.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-throughput.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\activemq-throughput.xml
Copying from: C:\apache-activemq-5.8.0\conf\broker-localhost.cert
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\broker-localhost.cert
Copying from: C:\apache-activemq-5.8.0\conf\broker.ks
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\broker.ks
Copying from: C:\apache-activemq-5.8.0\conf\broker.ts
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\broker.ts
Copying from: C:\apache-activemq-5.8.0\conf\camel.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\camel.xml
Copying from: C:\apache-activemq-5.8.0\conf\client.ks
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\client.ks
Copying from: C:\apache-activemq-5.8.0\conf\client.ts
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\client.ts
Copying from: C:\apache-activemq-5.8.0\conf\credentials-enc.properties
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\credentials-enc.properties
Copying from: C:\apache-activemq-5.8.0\conf\credentials.properties
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\credentials.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty-demo.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jetty-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\jetty-realm.properties
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jetty-realm.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty.xml
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jetty.xml
Copying from: C:\apache-activemq-5.8.0\conf\jmx.access
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jmx.access
Copying from: C:\apache-activemq-5.8.0\conf\jmx.password
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\jmx.password
Copying from: C:\apache-activemq-5.8.0\conf\log4j.properties
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\log4j.properties
Copying from: C:\apache-activemq-5.8.0\conf\logging.properties
to: C:\apache-activemq-5.8.0\cluster\broker-1\conf\logging.properties
</pre>
<br />
Now let's create broker-2 instance...<br />
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: 1">C:\apache-activemq-5.8.0\bin>activemq-admin create ..\cluster\broker-2
Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
Heap sizes: current=125632k free=124976k max=1864192k
JVM args: -Dactivemq.classpath=C:\apache-activemq-5.8.0\bin\..\conf;C:\apache-activemq-5.8.0\bin\..\data; -Dactivemq.home=C:\apache-activemq-5.8.0
\bin\.. -Dactivemq.base=C:\apache-activemq-5.8.0\bin\.. -Dactivemq.data=C:\apache-activemq-5.8.0\bin\..\data -Djava.io.tmpdir=C:\apache-activemq-5.8.0
\bin\..\data\tmp -Dactivemq.conf=C:\apache-activemq-5.8.0\bin\..\conf
Extensions classpath:
[C:\apache-activemq-5.8.0\bin\..\lib,C:\apache-activemq-5.8.0\bin\..\lib\camel,C:\apache-activemq-5.8.0\bin\..\lib\optional,C:\apache-activemq-5.8.0
\bin\..\lib\web,C:\apache-activemq-5.8.0\bin\..\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\bin\..
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\bin\..\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\bin\..\data
Running create broker task...
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-2
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-2\bin
Creating directory: C:\apache-activemq-5.8.0\cluster\broker-2\conf
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-2\bin\broker-2.bat
Creating new file: C:\apache-activemq-5.8.0\cluster\broker-2\bin\broker-2
Copying from: C:\apache-activemq-5.8.0\conf\activemq.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-command.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-command.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-demo.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker1.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-dynamic-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-dynamic-network-broker2.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-dynamic-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-jdbc.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-jdbc.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-scalability.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-scalability.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-security.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-security.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-specjms.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-specjms.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker1.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-static-network-broker1.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-static-network-broker2.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-static-network-broker2.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-stomp.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-stomp.xml
Copying from: C:\apache-activemq-5.8.0\conf\activemq-throughput.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\activemq-throughput.xml
Copying from: C:\apache-activemq-5.8.0\conf\broker-localhost.cert
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\broker-localhost.cert
Copying from: C:\apache-activemq-5.8.0\conf\broker.ks
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\broker.ks
Copying from: C:\apache-activemq-5.8.0\conf\broker.ts
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\broker.ts
Copying from: C:\apache-activemq-5.8.0\conf\camel.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\camel.xml
Copying from: C:\apache-activemq-5.8.0\conf\client.ks
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\client.ks
Copying from: C:\apache-activemq-5.8.0\conf\client.ts
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\client.ts
Copying from: C:\apache-activemq-5.8.0\conf\credentials-enc.properties
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\credentials-enc.properties
Copying from: C:\apache-activemq-5.8.0\conf\credentials.properties
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\credentials.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty-demo.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jetty-demo.xml
Copying from: C:\apache-activemq-5.8.0\conf\jetty-realm.properties
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jetty-realm.properties
Copying from: C:\apache-activemq-5.8.0\conf\jetty.xml
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jetty.xml
Copying from: C:\apache-activemq-5.8.0\conf\jmx.access
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jmx.access
Copying from: C:\apache-activemq-5.8.0\conf\jmx.password
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\jmx.password
Copying from: C:\apache-activemq-5.8.0\conf\log4j.properties
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\log4j.properties
Copying from: C:\apache-activemq-5.8.0\conf\logging.properties
to: C:\apache-activemq-5.8.0\cluster\broker-2\conf\logging.properties
</pre>
<br />
You may have noticed the following properties above: <br />
<br />
<span style="color: blue;">ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\bin\..</span><br />
<span style="color: blue;">ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\bin\..\conf</span><br />
<span style="color: blue;">ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\bin\..\data</span><br />
<br />
These properties need to be fixed as we'd like ACTIVEMQ_BASE and ACTIVEMQ_CONF to be different for both broker-1 and broker-2.<br />
<br />
We need to edit the following files:<br />
<ul style="text-align: left;">
<li><span style="color: blue;">broker-1.bat</span> in C:\apache-activemq-5.8.0\cluster\broker-1\bin directory</li>
<ul>
<li>set ACTIVEMQ_HOME="C:/apache-activemq-5.8.0" </li>
<li><span style="color: #cc0000;">set ACTIVEMQ_BASE="C:/apache-activemq-5.8.0/cluster/broker-1"</span></li>
<li>set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf </li>
</ul>
</ul>
<ul style="text-align: left;">
<li><span style="color: blue;">broker-2.bat</span> in C:\apache-activemq-5.8.0\cluster\broker-2\bin directory </li>
<ul>
<li>set ACTIVEMQ_HOME="C:/apache-activemq-5.8.0"</li>
<li><span style="color: #cc0000;">set ACTIVEMQ_BASE="C:/apache-activemq-5.8.0/cluster/broker-2"</span></li>
<li>set ACTIVEMQ_CONF=%ACTIVEMQ_BASE%/conf</li>
</ul>
</ul>
You may observe that both broker-1 and broker-2 nodes share the same ACTIVEMQ_DATA folder. Since we are using the in-built KahaDB for persistence, both broker-1 and broker-2 will share this.<br />
<br />
We need to differentiate the tcp ports for broker-1 and broker-2 and also enable JMX and configure JMX ports for remote monitoring.<br />
<br />
Let's edit the activemq.xml for broker-1 to fix tcp port:<br />
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: 3"> <transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
<!--<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>-->
</transportConnectors>
</pre>
Let's edit the activemq.xml for broker-1 to enable JMX monitoring, notice useJMX="true" attribute below.<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: 1"> <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker-1" dataDirectory="${activemq.data}" useJmx="true">
</pre>
Let's configure the JMX port
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: 2"> <managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
</pre>
<br />
Repeat the same for broker-2. set TCP port to 61626 and JMX port to 2099.<br />
<br />
Start broker-1.<br />
<pre class="brush: xml;overflow-x:scroll !important;"></pre>
<pre class="brush: xml;overflow-x:scroll !important;">C:\apache-activemq-5.8.0\cluster\broker-1\bin>broker-1.bat
Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
Heap sizes: current=1004928k free=994439k max=1004928k
JVM args: -Dcom.sun.management.jmxremote -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dactivemq.classpath=C:/apache-activemq-
5.8.0/cluster/broker-1/conf;C:/apache-activemq-5.8.0/cluster/broker-1/conf;C:/apache-activemq-5.8.0/conf; -Dactivemq.home=C:/apache-activemq-5.8.0 -Da
ctivemq.base=C:/apache-activemq-5.8.0/cluster/broker-1 -Dactivemq.conf=C:/apache-activemq-5.8.0/cluster/broker-1/conf -Dactivemq.data=C:/apache-active
mq-5.8.0\data -Djava.io.tmpdir=C:/apache-activemq-5.8.0\data\tmp
Extensions classpath:
[C:\apache-activemq-5.8.0\cluster\broker-1\lib,C:\apache-activemq-5.8.0\lib,C:\apache-activemq-5.8.0\cluster\broker-1\lib\camel,C:\apache-activemq-5
.8.0\cluster\broker-1\lib\optional,C:\apache-activemq-5.8.0\cluster\broker-1\lib\web,C:\apache-activemq-5.8.0\cluster\broker-1\lib\extra,C:\apache-act
ivemq-5.8.0\lib\camel,C:\apache-activemq-5.8.0\lib\optional,C:\apache-activemq-5.8.0\lib\web,C:\apache-activemq-5.8.0\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\cluster\broker-1
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\cluster\broker-1\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\data
Loading message broker from: xbean:activemq.xml
INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@71060478: startup date [Tue Jul 09 16:59:15 CDT 2013]; root of context hierarchy
INFO | PListStore:[C:\apache-activemq-5.8.0\data\broker-1\tmp_storage] started
INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\apache-activemq-5.8.0\data\kahadb]
INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
INFO | KahaDB is version 4
INFO | Recovering from the journal ...
INFO | Recovery replayed 1 operations from the journal in 0.028 seconds.
INFO | Apache ActiveMQ 5.8.0 (broker-1, ID:AKUNTAMU-1-27777-1373407157813-0:1) is starting
INFO | Listening for connections at: tcp://AKUNTAMU-1:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
INFO | Connector openwire Started
INFO | Apache ActiveMQ 5.8.0 (broker-1, ID:AKUNTAMU-1-27777-1373407157813-0:1) started
INFO | For help or more information please see: http://activemq.apache.org
WARN | Store limit is 102400 mb, whilst the data directory: C:\apache-activemq-5.8.0\data\kahadb only has 38889 mb of usable space
ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: C:\apache-activemq-5.8.0\data\broker-1\tmp_storage only has 38889 mb o
f usable space
INFO | Web console type: embedded
INFO | ActiveMQ WebConsole initialized.
INFO | Initializing Spring FrameworkServlet 'dispatcher'
INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed
</pre>
Start broker-2.<br />
<br />
You will observe that broker-2 is unable to acquire the lock since broker-1 already grabbed it. broker-2 will keep retrying to acquire the lock every 10 seconds.
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: [19,20,23,24]"></pre>
<pre class="brush: xml;overflow-x:scroll !important;highlight: [19,20,23,24]">C:\apache-activemq-5.8.0\cluster\broker-2\bin>broker-2.bat
Java Runtime: Sun Microsystems Inc. 1.6.0_31 C:\Program Files\Java\jdk1.6.0_31\jre
Heap sizes: current=1004928k free=994439k max=1004928k
JVM args: -Dcom.sun.management.jmxremote -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dactivemq.classpath=C:/apache-activemq-
5.8.0/cluster/broker-2/conf;C:/apache-activemq-5.8.0/cluster/broker-2/conf;C:/apache-activemq-5.8.0/conf; -Dactivemq.home=C:/apache-activemq-5.8.0 -Da
ctivemq.base=C:/apache-activemq-5.8.0/cluster/broker-2 -Dactivemq.conf=C:/apache-activemq-5.8.0/cluster/broker-2/conf -Dactivemq.data=C:/apache-active
mq-5.8.0\data -Djava.io.tmpdir=C:/apache-activemq-5.8.0\data\tmp
Extensions classpath:
[C:\apache-activemq-5.8.0\cluster\broker-2\lib,C:\apache-activemq-5.8.0\lib,C:\apache-activemq-5.8.0\cluster\broker-2\lib\camel,C:\apache-activemq-5
.8.0\cluster\broker-2\lib\optional,C:\apache-activemq-5.8.0\cluster\broker-2\lib\web,C:\apache-activemq-5.8.0\cluster\broker-2\lib\extra,C:\apache-act
ivemq-5.8.0\lib\camel,C:\apache-activemq-5.8.0\lib\optional,C:\apache-activemq-5.8.0\lib\web,C:\apache-activemq-5.8.0\lib\extra]
ACTIVEMQ_HOME: C:\apache-activemq-5.8.0
ACTIVEMQ_BASE: C:\apache-activemq-5.8.0\cluster\broker-2
ACTIVEMQ_CONF: C:\apache-activemq-5.8.0\cluster\broker-2\conf
ACTIVEMQ_DATA: C:\apache-activemq-5.8.0\data
Loading message broker from: xbean:activemq.xml
INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@420f9c40: startup date [Tue Jul 09 17:02:55 CDT 2013]; root of context hierarchy
INFO | PListStore:[C:\apache-activemq-5.8.0\data\broker-2\tmp_storage] started
INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\apache-activemq-5.8.0\data\kahadb]
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:2099/jmxrmi
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
</pre>
<br />
Now let's configure the ActiveMQ web console.<br />
By default, ActiveMQ distribution contains admin web console but in the Master/Slave configuration, it is unknown which node is the master. So it is doesn't make sense to use the embedded web console. Hence it is best to have ActiveMQ web console outside of the ActiveMQ nodes.<br />
<br />
You can disable the embedded ActiveMQ Web Console in each node by commenting the following line in activemq.xml in conf directory of each ActiveMQ node.<br />
<br />
<span style="color: #38761d;"><!--<import resource="jetty.xml"/>--></span><br />
<br />
For our example, we will deploy ActiveMQ Web Console web application in Tomcat container and then configure ActiveMQ web console application to intelligently point to the master node in ActiveMQ cluster.<br />
<br />
so let's copy the activemq-web-console-5.8.0.war to webapps directory of Tomcat.
Add the following line to catalina.bat
<br />
<br />
set JAVA_OPTS=-Dwebconsole.type=properties -Dwebconsole.jms.url=<b><span style="color: blue;">failover:(tcp://localhost:61616,tcp://localhost:61626)</span></b> -Dwebconsole.jmx.url=<b><span style="color: blue;">service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi,service:jmx:rmi:///jndi/rmi://localhost:2099/jmxrmi
</span></b><br />
<br />
Now let's start Tomcat.<br />
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: 17">C:\apache-tomcat-7.0.35\bin>.\catalina.bat run
Using CATALINA_BASE: "C:\apache-tomcat-7.0.35"
Using CATALINA_HOME: "C:\apache-tomcat-7.0.35"
Using CATALINA_TMPDIR: "C:\apache-tomcat-7.0.35\temp"
Using JRE_HOME: "C:\Program Files\Java\jdk1.6.0_31"
Using CLASSPATH: "C:\apache-tomcat-7.0.35\bin\bootstrap.jar;C:\apache-tomcat-7.0.35\bin\tomcat-juli.jar"
Jul 9, 2013 5:28:08 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jul 9, 2013 5:28:08 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Jul 9, 2013 5:28:08 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 635 ms
Jul 9, 2013 5:28:08 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jul 9, 2013 5:28:08 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.35
Jul 9, 2013 5:28:08 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive C:\apache-tomcat-7.0.35\webapps\activemq-web-console-5.8.0.war
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apache-tomcat-7.0.35/webapps/activemq-web-console-5.8.0/WEB-INF/lib/activemq-all-5.8.0.jar!/org/slf4j/impl/Stati
cLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/apache-tomcat-7.0.35/webapps/activemq-web-console-5.8.0/WEB-INF/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/Stat
icLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2013-07-09 17:28:13,389 [ost-startStop-1] INFO WebConsoleStarter - Web console type: properties
2013-07-09 17:28:13,960 [ost-startStop-1] INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
2013-07-09 17:28:14,095 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/createDestination.action] onto handler '/createDest
ination.action'
2013-07-09 17:28:14,096 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteDestination.action] onto handler '/deleteDest
ination.action'
2013-07-09 17:28:14,097 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/createSubscriber.action] onto handler '/createSubsc
riber.action'
2013-07-09 17:28:14,098 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteSubscriber.action] onto handler '/deleteSubsc
riber.action'
2013-07-09 17:28:14,099 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/sendMessage.action] onto handler '/sendMessage.acti
on'
2013-07-09 17:28:14,100 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/purgeDestination.action] onto handler '/purgeDestin
ation.action'
2013-07-09 17:28:14,101 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteMessage.action] onto handler '/deleteMessage.
action'
2013-07-09 17:28:14,103 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/copyMessage.action] onto handler '/copyMessage.acti
on'
2013-07-09 17:28:14,104 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/moveMessage.action] onto handler '/moveMessage.acti
on'
2013-07-09 17:28:14,105 [ost-startStop-1] INFO ndingBeanNameUrlHandlerMapping - Mapped URL path [/deleteJob.action] onto handler '/deleteJob.action'
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\docs
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\examples
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\host-manager
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\manager
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory C:\apache-tomcat-7.0.35\webapps\ROOT
Jul 9, 2013 5:28:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Jul 9, 2013 5:28:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Jul 9, 2013 5:28:14 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6642 ms
</pre>
<br />
Let's access the web console: http://localhost:8080/activemq-web-console-5.8.0<br />
Use admin/admin for username/password if prompted. These are default security settings. Refer jetty.xml and jetty-realm.properties in conf directory.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-xAcap_9aBzY/UdyQv23aSOI/AAAAAAAAAGQ/UU0TKCZ4lhY/s1600/broker-1+-+ActiveMQ+Console.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="377" src="http://1.bp.blogspot.com/-xAcap_9aBzY/UdyQv23aSOI/AAAAAAAAAGQ/UU0TKCZ4lhY/s400/broker-1+-+ActiveMQ+Console.png" width="400" /></a></div>
<br />
As shown above, "broker-1" is the current master broker.<br />
<br />
Now, if you were to shutdown broker-1. Hit Control+C on Terminal window running broker-1, you'd notice that broker-2 acquires the lock and becomes the master.<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: [10,11,12,13,14]"> INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | Database C:\apache-activemq-5.8.0\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOExceptio
n: File 'C:\apache-activemq-5.8.0\data\kahadb\lock' could not be locked.
INFO | KahaDB is version 4
INFO | Recovering from the journal ...
INFO | Recovery replayed 2 operations from the journal in 0.022 seconds.
INFO | Apache ActiveMQ 5.8.0 (broker-2, ID:AKUNTAMU-1-28147-1373409767675-0:1) is starting
INFO | Listening for connections at: tcp://AKUNTAMU-1:61626?maximumConnections=1000&wireformat.maxFrameSize=104857600
INFO | Connector openwire Started
INFO | Apache ActiveMQ 5.8.0 (broker-2, ID:AKUNTAMU-1-28147-1373409767675-0:1) started
INFO | For help or more information please see: http://activemq.apache.org
WARN | Store limit is 102400 mb, whilst the data directory: C:\apache-activemq-5.8.0\data\kahadb only has 38888 mb of usable space
ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: C:\apache-activemq-5.8.0\data\broker-2\tmp_storage only has 38888 mb o
f usable space
INFO | Web console type: embedded
INFO | ActiveMQ WebConsole initialized.
INFO | Initializing Spring FrameworkServlet 'dispatcher'
INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed
</pre>
<br />
Let's refresh the web console<br />
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
<a href="http://4.bp.blogspot.com/-aD2zr7rEKMs/UdyTPxGZRjI/AAAAAAAAAGg/nn8ev2J2HYk/s1600/broker-2+-+ActiveMQ+Console.png" imageanchor="1"><img border="0" height="244" src="http://4.bp.blogspot.com/-aD2zr7rEKMs/UdyTPxGZRjI/AAAAAAAAAGg/nn8ev2J2HYk/s320/broker-2+-+ActiveMQ+Console.png" width="320" /></a></div>
<div style="text-align: left;">
<br />
As you may observe, now the master node is broker-2. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Now that we have seen seamless switch over between the nodes during a failover scenario using an external web console, let's see the same from the perspective of a message producer and consumer.<br />
<br />
I will publish 50 persistent messages to a queue using a simple message publisher and have an asynchronous consumer receive those 50 messages. I will introduce a bit of a delay in sending the messages so we can take the master down a few times and create a few failover scenarios. The objective is to see how the failover protocol makes the node failover completely transparent and thus shields the application from having to deal with any reconnection logic.<br />
<br />
Here is a simple producer using ActiveMQConnectionFactory with failover protocol. Notice the highlighted failover protocol URL in the code snippet below.<br />
<br />
<pre class="brush: java;overflow-x:scroll !important;highlight: [20,21]">package com.akuntamukkala.amqms;
import javax.jms.Connection;
import javax.jms.DeliveryMode;
import javax.jms.Destination;
import javax.jms.MessageProducer;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.log4j.Logger;
public class Producer {
private static final Logger log = Logger.getLogger(Producer.class);
public static void main(String[] args) throws Exception {
// Create a ConnectionFactory
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
"failover:(tcp://localhost:61616,tcp://localhost:61626)");
for (int i = 0; i < 50; i++) {
log.info("Establishing connection");
// Create a Connection
Connection connection = connectionFactory.createConnection();
connection.start();
log.info("Connection established");
// Create a Session
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
// Create the destination (Topic or Queue)
Destination destination = session.createQueue("TEST.FOO");
// Create a MessageProducer from the Session to the Topic or Queue
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
// Create a messages
String text = "Message Counter : " + i;
TextMessage message = session.createTextMessage(text);
log.info("Sending message : " + text);
producer.send(message);
log.info("Sent message : " + text);
// Clean up
session.close();
connection.close();
Thread.sleep(1000);
}
}
}
</pre>
Here are the logs from execution while I shutdown broker-1 and broker-2 alternatively a few times.
<br />
<br />
<pre class="brush: xml;overflow-x:scroll !important;highlight: [2,63,64,140,141,202,203]">2013-07-10 11:26:32 INFO Producer:25 - Establishing connection
2013-07-10 11:26:33 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:33 INFO Producer:30 - Connection established
2013-07-10 11:26:33 INFO Producer:47 - Sending message : Message Counter : 0
2013-07-10 11:26:33 INFO Producer:49 - Sent message : Message Counter : 0
2013-07-10 11:26:34 INFO Producer:25 - Establishing connection
2013-07-10 11:26:35 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:35 INFO Producer:30 - Connection established
2013-07-10 11:26:35 INFO Producer:47 - Sending message : Message Counter : 1
2013-07-10 11:26:35 INFO Producer:49 - Sent message : Message Counter : 1
2013-07-10 11:26:35 INFO Producer:25 - Establishing connection
2013-07-10 11:26:36 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:36 INFO Producer:30 - Connection established
2013-07-10 11:26:36 INFO Producer:47 - Sending message : Message Counter : 2
2013-07-10 11:26:36 INFO Producer:49 - Sent message : Message Counter : 2
2013-07-10 11:26:37 INFO Producer:25 - Establishing connection
2013-07-10 11:26:37 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:37 INFO Producer:30 - Connection established
2013-07-10 11:26:37 INFO Producer:47 - Sending message : Message Counter : 3
2013-07-10 11:26:37 INFO Producer:49 - Sent message : Message Counter : 3
2013-07-10 11:26:37 INFO Producer:25 - Establishing connection
2013-07-10 11:26:38 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:38 INFO Producer:30 - Connection established
2013-07-10 11:26:38 INFO Producer:47 - Sending message : Message Counter : 4
2013-07-10 11:26:38 INFO Producer:49 - Sent message : Message Counter : 4
2013-07-10 11:26:39 INFO Producer:25 - Establishing connection
2013-07-10 11:26:39 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:39 INFO Producer:30 - Connection established
2013-07-10 11:26:39 INFO Producer:47 - Sending message : Message Counter : 5
2013-07-10 11:26:39 INFO Producer:49 - Sent message : Message Counter : 5
2013-07-10 11:26:39 INFO Producer:25 - Establishing connection
2013-07-10 11:26:39 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:39 INFO Producer:30 - Connection established
2013-07-10 11:26:39 INFO Producer:47 - Sending message : Message Counter : 6
2013-07-10 11:26:39 INFO Producer:49 - Sent message : Message Counter : 6
2013-07-10 11:26:40 INFO Producer:25 - Establishing connection
2013-07-10 11:26:40 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:40 INFO Producer:30 - Connection established
2013-07-10 11:26:40 INFO Producer:47 - Sending message : Message Counter : 7
2013-07-10 11:26:40 INFO Producer:49 - Sent message : Message Counter : 7
2013-07-10 11:26:40 INFO Producer:25 - Establishing connection
2013-07-10 11:26:40 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:40 INFO Producer:30 - Connection established
2013-07-10 11:26:40 INFO Producer:47 - Sending message : Message Counter : 8
2013-07-10 11:26:40 INFO Producer:49 - Sent message : Message Counter : 8
2013-07-10 11:26:41 INFO Producer:25 - Establishing connection
2013-07-10 11:26:41 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:41 INFO Producer:30 - Connection established
2013-07-10 11:26:41 INFO Producer:47 - Sending message : Message Counter : 9
2013-07-10 11:26:41 INFO Producer:49 - Sent message : Message Counter : 9
2013-07-10 11:26:41 INFO Producer:25 - Establishing connection
2013-07-10 11:26:41 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:41 INFO Producer:30 - Connection established
2013-07-10 11:26:41 INFO Producer:47 - Sending message : Message Counter : 10
2013-07-10 11:26:41 INFO Producer:49 - Sent message : Message Counter : 10
2013-07-10 11:26:42 INFO Producer:25 - Establishing connection
2013-07-10 11:26:43 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:43 INFO Producer:30 - Connection established
2013-07-10 11:26:43 INFO Producer:47 - Sending message : Message Counter : 11
2013-07-10 11:26:43 INFO Producer:49 - Sent message : Message Counter : 11
2013-07-10 11:26:43 INFO Producer:25 - Establishing connection
2013-07-10 11:26:43 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:26:43 WARN FailoverTransport:255 - Transport (tcp://127.0.0.1:61616) failed, reason: java.net.SocketException: Software caused connection abort: recv failed, attempting to automatically reconnect
2013-07-10 11:26:50 INFO FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61626
2013-07-10 11:26:50 INFO Producer:30 - Connection established
2013-07-10 11:26:50 INFO Producer:47 - Sending message : Message Counter : 12
2013-07-10 11:26:50 INFO Producer:49 - Sent message : Message Counter : 12
2013-07-10 11:26:51 INFO Producer:25 - Establishing connection
2013-07-10 11:26:52 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:52 INFO Producer:30 - Connection established
2013-07-10 11:26:52 INFO Producer:47 - Sending message : Message Counter : 13
2013-07-10 11:26:52 INFO Producer:49 - Sent message : Message Counter : 13
2013-07-10 11:26:52 INFO Producer:25 - Establishing connection
2013-07-10 11:26:53 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:53 INFO Producer:30 - Connection established
2013-07-10 11:26:53 INFO Producer:47 - Sending message : Message Counter : 14
2013-07-10 11:26:53 INFO Producer:49 - Sent message : Message Counter : 14
2013-07-10 11:26:54 INFO Producer:25 - Establishing connection
2013-07-10 11:26:54 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:54 INFO Producer:30 - Connection established
2013-07-10 11:26:54 INFO Producer:47 - Sending message : Message Counter : 15
2013-07-10 11:26:54 INFO Producer:49 - Sent message : Message Counter : 15
2013-07-10 11:26:54 INFO Producer:25 - Establishing connection
2013-07-10 11:26:55 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:55 INFO Producer:30 - Connection established
2013-07-10 11:26:55 INFO Producer:47 - Sending message : Message Counter : 16
2013-07-10 11:26:55 INFO Producer:49 - Sent message : Message Counter : 16
2013-07-10 11:26:56 INFO Producer:25 - Establishing connection
2013-07-10 11:26:57 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:57 INFO Producer:30 - Connection established
2013-07-10 11:26:57 INFO Producer:47 - Sending message : Message Counter : 17
2013-07-10 11:26:57 INFO Producer:49 - Sent message : Message Counter : 17
2013-07-10 11:26:57 INFO Producer:25 - Establishing connection
2013-07-10 11:26:58 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:58 INFO Producer:30 - Connection established
2013-07-10 11:26:58 INFO Producer:47 - Sending message : Message Counter : 18
2013-07-10 11:26:58 INFO Producer:49 - Sent message : Message Counter : 18
2013-07-10 11:26:59 INFO Producer:25 - Establishing connection
2013-07-10 11:26:59 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:59 INFO Producer:30 - Connection established
2013-07-10 11:26:59 INFO Producer:47 - Sending message : Message Counter : 19
2013-07-10 11:26:59 INFO Producer:49 - Sent message : Message Counter : 19
2013-07-10 11:26:59 INFO Producer:25 - Establishing connection
2013-07-10 11:26:59 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:26:59 INFO Producer:30 - Connection established
2013-07-10 11:26:59 INFO Producer:47 - Sending message : Message Counter : 20
2013-07-10 11:26:59 INFO Producer:49 - Sent message : Message Counter : 20
2013-07-10 11:27:00 INFO Producer:25 - Establishing connection
2013-07-10 11:27:01 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:01 INFO Producer:30 - Connection established
2013-07-10 11:27:01 INFO Producer:47 - Sending message : Message Counter : 21
2013-07-10 11:27:01 INFO Producer:49 - Sent message : Message Counter : 21
2013-07-10 11:27:01 INFO Producer:25 - Establishing connection
2013-07-10 11:27:01 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:01 INFO Producer:30 - Connection established
2013-07-10 11:27:01 INFO Producer:47 - Sending message : Message Counter : 22
2013-07-10 11:27:01 INFO Producer:49 - Sent message : Message Counter : 22
2013-07-10 11:27:02 INFO Producer:25 - Establishing connection
2013-07-10 11:27:02 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:02 INFO Producer:30 - Connection established
2013-07-10 11:27:02 INFO Producer:47 - Sending message : Message Counter : 23
2013-07-10 11:27:02 INFO Producer:49 - Sent message : Message Counter : 23
2013-07-10 11:27:02 INFO Producer:25 - Establishing connection
2013-07-10 11:27:02 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:02 INFO Producer:30 - Connection established
2013-07-10 11:27:02 INFO Producer:47 - Sending message : Message Counter : 24
2013-07-10 11:27:02 INFO Producer:49 - Sent message : Message Counter : 24
2013-07-10 11:27:03 INFO Producer:25 - Establishing connection
2013-07-10 11:27:03 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:03 INFO Producer:30 - Connection established
2013-07-10 11:27:03 INFO Producer:47 - Sending message : Message Counter : 25
2013-07-10 11:27:03 INFO Producer:49 - Sent message : Message Counter : 25
2013-07-10 11:27:03 INFO Producer:25 - Establishing connection
2013-07-10 11:27:03 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:03 INFO Producer:30 - Connection established
2013-07-10 11:27:03 INFO Producer:47 - Sending message : Message Counter : 26
2013-07-10 11:27:03 INFO Producer:49 - Sent message : Message Counter : 26
2013-07-10 11:27:04 INFO Producer:25 - Establishing connection
2013-07-10 11:27:04 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:06 WARN FailoverTransport:255 - Transport (tcp://127.0.0.1:61626) failed, reason: java.net.SocketException: Software caused connection abort: recv failed, attempting to automatically reconnect
2013-07-10 11:27:15 INFO FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61616
2013-07-10 11:27:15 INFO Producer:30 - Connection established
2013-07-10 11:27:15 INFO Producer:47 - Sending message : Message Counter : 27
2013-07-10 11:27:15 INFO Producer:49 - Sent message : Message Counter : 27
2013-07-10 11:27:16 INFO Producer:25 - Establishing connection
2013-07-10 11:27:17 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:17 INFO Producer:30 - Connection established
2013-07-10 11:27:17 INFO Producer:47 - Sending message : Message Counter : 28
2013-07-10 11:27:17 INFO Producer:49 - Sent message : Message Counter : 28
2013-07-10 11:27:17 INFO Producer:25 - Establishing connection
2013-07-10 11:27:18 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:18 INFO Producer:30 - Connection established
2013-07-10 11:27:18 INFO Producer:47 - Sending message : Message Counter : 29
2013-07-10 11:27:18 INFO Producer:49 - Sent message : Message Counter : 29
2013-07-10 11:27:19 INFO Producer:25 - Establishing connection
2013-07-10 11:27:20 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:20 INFO Producer:30 - Connection established
2013-07-10 11:27:20 INFO Producer:47 - Sending message : Message Counter : 30
2013-07-10 11:27:20 INFO Producer:49 - Sent message : Message Counter : 30
2013-07-10 11:27:20 INFO Producer:25 - Establishing connection
2013-07-10 11:27:20 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:20 INFO Producer:30 - Connection established
2013-07-10 11:27:20 INFO Producer:47 - Sending message : Message Counter : 31
2013-07-10 11:27:20 INFO Producer:49 - Sent message : Message Counter : 31
2013-07-10 11:27:21 INFO Producer:25 - Establishing connection
2013-07-10 11:27:22 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:22 INFO Producer:30 - Connection established
2013-07-10 11:27:22 INFO Producer:47 - Sending message : Message Counter : 32
2013-07-10 11:27:22 INFO Producer:49 - Sent message : Message Counter : 32
2013-07-10 11:27:22 INFO Producer:25 - Establishing connection
2013-07-10 11:27:22 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:22 INFO Producer:30 - Connection established
2013-07-10 11:27:22 INFO Producer:47 - Sending message : Message Counter : 33
2013-07-10 11:27:22 INFO Producer:49 - Sent message : Message Counter : 33
2013-07-10 11:27:23 INFO Producer:25 - Establishing connection
2013-07-10 11:27:23 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:23 INFO Producer:30 - Connection established
2013-07-10 11:27:23 INFO Producer:47 - Sending message : Message Counter : 34
2013-07-10 11:27:23 INFO Producer:49 - Sent message : Message Counter : 34
2013-07-10 11:27:23 INFO Producer:25 - Establishing connection
2013-07-10 11:27:24 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:24 INFO Producer:30 - Connection established
2013-07-10 11:27:24 INFO Producer:47 - Sending message : Message Counter : 35
2013-07-10 11:27:24 INFO Producer:49 - Sent message : Message Counter : 35
2013-07-10 11:27:25 INFO Producer:25 - Establishing connection
2013-07-10 11:27:27 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:27 INFO Producer:30 - Connection established
2013-07-10 11:27:27 INFO Producer:47 - Sending message : Message Counter : 36
2013-07-10 11:27:27 INFO Producer:49 - Sent message : Message Counter : 36
2013-07-10 11:27:27 INFO Producer:25 - Establishing connection
2013-07-10 11:27:28 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:28 INFO Producer:30 - Connection established
2013-07-10 11:27:28 INFO Producer:47 - Sending message : Message Counter : 37
2013-07-10 11:27:28 INFO Producer:49 - Sent message : Message Counter : 37
2013-07-10 11:27:29 INFO Producer:25 - Establishing connection
2013-07-10 11:27:30 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:30 INFO Producer:30 - Connection established
2013-07-10 11:27:30 INFO Producer:47 - Sending message : Message Counter : 38
2013-07-10 11:27:30 INFO Producer:49 - Sent message : Message Counter : 38
2013-07-10 11:27:30 INFO Producer:25 - Establishing connection
2013-07-10 11:27:31 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:27:33 WARN FailoverTransport:255 - Transport (tcp://127.0.0.1:61616) failed, reason: java.net.SocketException: Software caused connection abort: recv failed, attempting to automatically reconnect
2013-07-10 11:27:38 INFO FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61626
2013-07-10 11:27:38 INFO Producer:30 - Connection established
2013-07-10 11:27:38 INFO Producer:47 - Sending message : Message Counter : 39
2013-07-10 11:27:38 INFO Producer:49 - Sent message : Message Counter : 39
2013-07-10 11:27:39 INFO Producer:25 - Establishing connection
2013-07-10 11:27:39 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:39 INFO Producer:30 - Connection established
2013-07-10 11:27:39 INFO Producer:47 - Sending message : Message Counter : 40
2013-07-10 11:27:39 INFO Producer:49 - Sent message : Message Counter : 40
2013-07-10 11:27:39 INFO Producer:25 - Establishing connection
2013-07-10 11:27:40 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:40 INFO Producer:30 - Connection established
2013-07-10 11:27:40 INFO Producer:47 - Sending message : Message Counter : 41
2013-07-10 11:27:40 INFO Producer:49 - Sent message : Message Counter : 41
2013-07-10 11:27:41 INFO Producer:25 - Establishing connection
2013-07-10 11:27:42 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:42 INFO Producer:30 - Connection established
2013-07-10 11:27:42 INFO Producer:47 - Sending message : Message Counter : 42
2013-07-10 11:27:42 INFO Producer:49 - Sent message : Message Counter : 42
2013-07-10 11:27:42 INFO Producer:25 - Establishing connection
2013-07-10 11:27:42 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:42 INFO Producer:30 - Connection established
2013-07-10 11:27:42 INFO Producer:47 - Sending message : Message Counter : 43
2013-07-10 11:27:42 INFO Producer:49 - Sent message : Message Counter : 43
2013-07-10 11:27:43 INFO Producer:25 - Establishing connection
2013-07-10 11:27:44 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:44 INFO Producer:30 - Connection established
2013-07-10 11:27:44 INFO Producer:47 - Sending message : Message Counter : 44
2013-07-10 11:27:44 INFO Producer:49 - Sent message : Message Counter : 44
2013-07-10 11:27:44 INFO Producer:25 - Establishing connection
2013-07-10 11:27:45 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:45 INFO Producer:30 - Connection established
2013-07-10 11:27:45 INFO Producer:47 - Sending message : Message Counter : 45
2013-07-10 11:27:45 INFO Producer:49 - Sent message : Message Counter : 45
2013-07-10 11:27:46 INFO Producer:25 - Establishing connection
2013-07-10 11:27:47 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:47 INFO Producer:30 - Connection established
2013-07-10 11:27:47 INFO Producer:47 - Sending message : Message Counter : 46
2013-07-10 11:27:47 INFO Producer:49 - Sent message : Message Counter : 46
2013-07-10 11:27:48 INFO Producer:25 - Establishing connection
2013-07-10 11:27:48 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:48 INFO Producer:30 - Connection established
2013-07-10 11:27:48 INFO Producer:47 - Sending message : Message Counter : 47
2013-07-10 11:27:48 INFO Producer:49 - Sent message : Message Counter : 47
2013-07-10 11:27:48 INFO Producer:25 - Establishing connection
2013-07-10 11:27:49 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:49 INFO Producer:30 - Connection established
2013-07-10 11:27:49 INFO Producer:47 - Sending message : Message Counter : 48
2013-07-10 11:27:49 INFO Producer:49 - Sent message : Message Counter : 48
2013-07-10 11:27:50 INFO Producer:25 - Establishing connection
2013-07-10 11:27:51 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61626
2013-07-10 11:27:51 INFO Producer:30 - Connection established
2013-07-10 11:27:51 INFO Producer:47 - Sending message : Message Counter : 49
2013-07-10 11:27:51 INFO Producer:49 - Sent message : Message Counter : 49
</pre>
<br />
Here is the web console showing 50 messages enqueued.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-_0T_y5tswfc/Ud2Ng15SCfI/AAAAAAAAAGw/mYdfDOUc6rY/s1600/broker-1+-+Queues.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="203" src="http://2.bp.blogspot.com/-_0T_y5tswfc/Ud2Ng15SCfI/AAAAAAAAAGw/mYdfDOUc6rY/s400/broker-1+-+Queues.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Don't bother the #Messages Enqueued = 0 in the above screenshot. This count represents the number of messages enqueued since this node started. Since I had restarted the node after the 50 messages were already enqueued, the count shows as 0.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
Now lets try to consume these messages using an asynchronous consumer using ActiveMQ connection factory with failover protocol.<br />
<br />
Here is simple asynchronous consumer with ActiveMQConnectionFactory using failover protocol.
<br />
<br />
<pre class="brush: java;overflow-x:scroll !important;highlight: [22,23]">package com.akuntamukkala.amqms;
import javax.jms.Connection;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.log4j.Logger;
public class Consumer implements MessageListener {
private static final Logger log = Logger.getLogger(Consumer.class);
public static void main(String[] args) throws Exception {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
"failover:(tcp://localhost:61616,tcp://localhost:61626)");
// Create a Connection
Connection connection = connectionFactory.createConnection();
connection.start();
// Create a Session
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
// Create the destination (Topic or Queue)
Destination destination = session.createQueue("TEST.FOO");
// Create a MessageConsumer from the Session to the Queue
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(new Consumer()); // asynchronous listener
Thread.sleep(120000); // long wait to keep program running
consumer.close();
session.close();
connection.close();
}
/**
* asynchronous message listener
*/
public void onMessage(Message message) {
try {
log.info(((TextMessage) message).getText());
Thread.sleep(500);
} catch (JMSException e) {
log.error(e);
} catch (InterruptedException e) {
log.error(e);
}
}
}
</pre>
<br />
Here are the logs from the execution. I switched the master nodes a few times as evident in the logs. <br />
<br />
<pre class="brush: java;overflow-x:scroll !important;highlight: [1,14,15,16,45,46,47]">2013-07-10 11:46:01 INFO FailoverTransport:1030 - Successfully connected to tcp://localhost:61616
2013-07-10 11:46:01 INFO Consumer:49 - Message Counter : 0
2013-07-10 11:46:02 INFO Consumer:49 - Message Counter : 1
2013-07-10 11:46:02 INFO Consumer:49 - Message Counter : 2
2013-07-10 11:46:03 INFO Consumer:49 - Message Counter : 3
2013-07-10 11:46:03 INFO Consumer:49 - Message Counter : 4
2013-07-10 11:46:04 INFO Consumer:49 - Message Counter : 5
2013-07-10 11:46:04 INFO Consumer:49 - Message Counter : 6
2013-07-10 11:46:05 INFO Consumer:49 - Message Counter : 7
2013-07-10 11:46:05 INFO Consumer:49 - Message Counter : 8
2013-07-10 11:46:06 INFO Consumer:49 - Message Counter : 9
2013-07-10 11:46:06 INFO Consumer:49 - Message Counter : 10
2013-07-10 11:46:07 INFO Consumer:49 - Message Counter : 11
2013-07-10 11:46:07 WARN FailoverTransport:255 - Transport (tcp://127.0.0.1:61616) failed, reason: java.io.EOFException, attempting to automatically reconnect
2013-07-10 11:46:18 INFO FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61626
2013-07-10 11:46:18 WARN ActiveMQMessageConsumer:1348 - Duplicate dispatch on connection: ID:AKUNTAMU-1-2141-1373474760280-1:1 to consumer: ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, ignoring (auto acking) duplicate: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, destination = queue://TEST.FOO, message = ActiveMQTextMessage {commandId = 5, responseRequired = true, messageId = ID:AKUNTAMU-1-1739-1373473592152-1:12:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:AKUNTAMU-1-1739-1373473592152-1:12:1:1, destination = queue://TEST.FOO, transactionId = null, expiration = 0, timestamp = 1373473603281, arrival = 0, brokerInTime = 1373473603281, brokerOutTime = 1373474778676, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false, text = Message Counter : 11}, redeliveryCounter = 0}
2013-07-10 11:46:18 INFO Consumer:49 - Message Counter : 12
2013-07-10 11:46:19 INFO Consumer:49 - Message Counter : 13
2013-07-10 11:46:19 INFO Consumer:49 - Message Counter : 14
2013-07-10 11:46:20 INFO Consumer:49 - Message Counter : 15
2013-07-10 11:46:20 INFO Consumer:49 - Message Counter : 16
2013-07-10 11:46:21 INFO Consumer:49 - Message Counter : 17
2013-07-10 11:46:21 INFO Consumer:49 - Message Counter : 18
2013-07-10 11:46:22 INFO Consumer:49 - Message Counter : 19
2013-07-10 11:46:22 INFO Consumer:49 - Message Counter : 20
2013-07-10 11:46:23 INFO Consumer:49 - Message Counter : 21
2013-07-10 11:46:23 INFO Consumer:49 - Message Counter : 22
2013-07-10 11:46:24 INFO Consumer:49 - Message Counter : 23
2013-07-10 11:46:24 INFO Consumer:49 - Message Counter : 24
2013-07-10 11:46:25 INFO Consumer:49 - Message Counter : 25
2013-07-10 11:46:25 INFO Consumer:49 - Message Counter : 26
2013-07-10 11:46:26 INFO Consumer:49 - Message Counter : 27
2013-07-10 11:46:26 INFO Consumer:49 - Message Counter : 28
2013-07-10 11:46:27 INFO Consumer:49 - Message Counter : 29
2013-07-10 11:46:27 INFO Consumer:49 - Message Counter : 30
2013-07-10 11:46:28 INFO Consumer:49 - Message Counter : 31
2013-07-10 11:46:28 INFO Consumer:49 - Message Counter : 32
2013-07-10 11:46:29 INFO Consumer:49 - Message Counter : 33
2013-07-10 11:46:29 INFO Consumer:49 - Message Counter : 34
2013-07-10 11:46:30 INFO Consumer:49 - Message Counter : 35
2013-07-10 11:46:30 INFO Consumer:49 - Message Counter : 36
2013-07-10 11:46:31 INFO Consumer:49 - Message Counter : 37
2013-07-10 11:46:31 INFO Consumer:49 - Message Counter : 38
2013-07-10 11:46:32 INFO Consumer:49 - Message Counter : 39
2013-07-10 11:46:32 WARN FailoverTransport:255 - Transport (tcp://127.0.0.1:61626) failed, reason: java.io.EOFException, attempting to automatically reconnect
2013-07-10 11:46:43 INFO FailoverTransport:1032 - Successfully reconnected to tcp://localhost:61616
2013-07-10 11:46:43 WARN ActiveMQMessageConsumer:1348 - Duplicate dispatch on connection: ID:AKUNTAMU-1-2141-1373474760280-1:1 to consumer: ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, ignoring (auto acking) duplicate: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:AKUNTAMU-1-2141-1373474760280-1:1:1:1, destination = queue://TEST.FOO, message = ActiveMQTextMessage {commandId = 5, responseRequired = true, messageId = ID:AKUNTAMU-1-1739-1373473592152-1:40:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:AKUNTAMU-1-1739-1373473592152-1:40:1:1, destination = queue://TEST.FOO, transactionId = null, expiration = 0, timestamp = 1373473658595, arrival = 0, brokerInTime = 1373473658599, brokerOutTime = 1373474803745, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = null, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false, text = Message Counter : 39}, redeliveryCounter = 0}
2013-07-10 11:46:43 INFO Consumer:49 - Message Counter : 40
2013-07-10 11:46:44 INFO Consumer:49 - Message Counter : 41
2013-07-10 11:46:44 INFO Consumer:49 - Message Counter : 42
2013-07-10 11:46:45 INFO Consumer:49 - Message Counter : 43
2013-07-10 11:46:45 INFO Consumer:49 - Message Counter : 44
2013-07-10 11:46:46 INFO Consumer:49 - Message Counter : 45
2013-07-10 11:46:46 INFO Consumer:49 - Message Counter : 46
2013-07-10 11:46:47 INFO Consumer:49 - Message Counter : 47
2013-07-10 11:46:47 INFO Consumer:49 - Message Counter : 48
2013-07-10 11:46:48 INFO Consumer:49 - Message Counter : 49
</pre>
<br />
Let's check out the ActiveMQ web console:
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-XBvEHEskk-c/Ud2WJepcvtI/AAAAAAAAAHA/PWk23angasU/s1600/broker-1+-+Queues-Failover.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="191" src="http://4.bp.blogspot.com/-XBvEHEskk-c/Ud2WJepcvtI/AAAAAAAAAHA/PWk23angasU/s400/broker-1+-+Queues-Failover.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
You may find the # Messages Dequeued = 11 very interesting. This is the number of messages dequeued from the current ActiveMQ master node since it started. </div>
<br />
<b>Conclusion:</b><br />
<br />
We have thus seen the following in action:<br />
<ol>
<li>Run 2 ActiveMQ nodes in a Master/Slave configuration with a shared KahaDB file based database. </li>
<li>Configure ActiveMQ web console hosted in a Tomcat instance to point to whichever node is Master node in the cluster</li>
<li>Failover scenario</li>
<li>Message publisher and consumer behavior oblivious to the failover</li>
</ol>
In future blogs, I will post some other interesting ActiveMQ configurations. Stay tuned.<br />
<br />
Happy ActiveMQ'ing!<br />
<br />
References:<br />
<br />
<ol style="text-align: left;">
<li><a href="http://activemq.apache.org/">http://activemq.apache.org/</a></li>
<li><a href="http://www.jakubkorab.net/category/technology/activemq">http://www.jakubkorab.net/category/technology/activemq</a></li>
</ol>
<br />
<br /></div>
</div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com14tag:blogger.com,1999:blog-8558899295969164571.post-43897719976013192512013-06-29T08:35:00.001-07:002013-06-29T09:59:32.103-07:00ActiveMQ as a Windows Service using YAJSW<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<b>Objective:</b><br />
<br />
This blog is to demonstrate how to run <a href="http://activemq.apache.org/" target="_blank">ActiveMQ</a> JMS Broker as a Windows Service using <a href="http://yajsw.sourceforge.net/" target="_blank">YAJSW</a>.<br />
<br />
<b>Introduction:</b><br />
<br />
ActiveMQ JMS Broker(s) is a vital part of messaging infrastructure in an enterprise. In order to ensure that the ActiveMQ process stays up and running, it is best to run it as a service using a service wrapper such as Tanuki or YAJSW. Running ActiveMQ as a service helps in automatically restarting ActiveMQ process when the host machine restarts (planned or unplanned).<br />
Service Wrapper ensures that the ActiveMQ process remains healthy. It can be configured to restart ActiveMQ process if it was to determine that the process is hung or not responding to ping. However, I must say that this is not a substitute for fixing issue(s) which actually cause ActiveMQ to hang or shut down. Service Wrapper may cure the symptom with a restart but it will not fix the root cause. It is best to troubleshoot to determine what is troubling the ActiveMQ process in order to prevent production nightmares :) Some of the common issues that I have seen relate to memory & connection leaks and inadequate capacity configuration.<br />
<div>
<br /></div>
<div>
<a href="http://wrapper.tanukisoftware.com/" target="_blank">Tanuki Wrapper</a> is one the most popular open source Service Wrappers but it does not offer community edition for 64bit Windows OS as of this writing :)</div>
<div>
<br /></div>
<div>
Please read the Tanuki Service Wrapper release notes here:</div>
<div>
<a href="http://wrapper.tanukisoftware.com/doc/english/download.jsp">http://wrapper.tanukisoftware.com/doc/english/download.jsp</a><br />
<br />
There is a viable open source alternative in YAJSW. If you are interested in running ActiveMQ as a Windows Service on a 64bit Windows OS, you may want to consider YAJSW. There may be other alternatives but I have found YAJSW sufficient. </div>
<div>
<br />
<b>Overview:</b></div>
<div>
<br />
In this article, I will show how to use YAJSW to run ActiveMQ as a Windows Service. We will use ActiveMQ version 5.7.0, YAJSW version 11.03 on 64bit Windows 7 OS. Usually when you register a process to run as a service, there are additional parameters to think about such as security - which user or user groups can administer this service, determine which interested parties receive process health status notifications, etc. In this example, we will consider a plain vanilla use case with out-of-the-box settings of ActiveMQ and YAJSW.</div>
<div>
<br /></div>
<div>
The actual process can be summarized in 3 easy steps. </div>
<div>
<ol style="text-align: left;">
<li>Run YAJSW "getConfig.bat" script against a live ActiveMQ process to generate YAJSW configuration file</li>
<li>Edit the configuration file to change a few settings</li>
<li>Run YAJSW "installService.bat" script to register ActiveMQ process to run as a Windows Service. </li>
</ol>
<div>
<b>Prerequisites:</b></div>
<div>
<ol style="text-align: left;">
<li>Install ActiveMQ 5.7.0</li>
<li>Install YAJSW 11.03</li>
<li>Windows 7</li>
<li>JDK 1.6+</li>
</ol>
</div>
<div>
<b>Deep Dive:</b></div>
</div>
<div>
<br /></div>
<div>
Launch ActiveMQ using the default activemq.bat script<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-nYrsW6mDwEs/Uc5qyHcJ92I/AAAAAAAAAEY/HBalNG3zoWs/s1600/Screen+Shot+2013-06-29+at+10.27.54+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="268" src="http://4.bp.blogspot.com/-nYrsW6mDwEs/Uc5qyHcJ92I/AAAAAAAAAEY/HBalNG3zoWs/s640/Screen+Shot+2013-06-29+at+10.27.54+AM.png" width="640" /></a></div>
</div>
<div>
<br />
Find the process ID using jps -v<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-wHAmcn-Shc0/Uc5rvm1w4fI/AAAAAAAAAEo/R2dhNR_CrEM/s1600/Screen+Shot+2013-06-29+at+10.37.34+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="52" src="http://1.bp.blogspot.com/-wHAmcn-Shc0/Uc5rvm1w4fI/AAAAAAAAAEo/R2dhNR_CrEM/s640/Screen+Shot+2013-06-29+at+10.37.34+AM.png" width="640" /></a></div>
<br />
The ActiveMQ process ID is 3024<br />
<br />
Let's generate YAJSW configuration now. We will use genConfig.bat script and pass the ActiveMQ process ID as an argument.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-sYnLxSpiqQc/Uc5xIVLvGdI/AAAAAAAAAE4/itR4Dd4SBNg/s1600/Screen+Shot+2013-06-29+at+10.58.17+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="236" src="http://4.bp.blogspot.com/-sYnLxSpiqQc/Uc5xIVLvGdI/AAAAAAAAAE4/itR4Dd4SBNg/s640/Screen+Shot+2013-06-29+at+10.58.17+AM.png" width="640" /></a></div>
<br />
The generated configuration is a good start but is not good enough. We still need to edit wrapper.conf file in c:\yajsw-stable-11.03\conf directory and provide the settings listed below:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">wrapper.working.dir=C:\\apache-activemq-5.7.0\\bin\\</span><br />
<br />
ActiveMQ uses a runnable jar "run.jar" so we need to specify that accordingly<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"># Java Application main class</span><br />
<span style="font-family: Courier New, Courier, monospace;">#wrapper.java.app.mainclass=</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"># or jar file</span><br />
<span style="font-family: Courier New, Courier, monospace;">wrapper.java.app.jar=C:\\apache-activemq-5.7.0\\bin\\run.jar</span></div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Name of the service</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">wrapper.ntservice.name=ActiveMQ</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Display name of the service</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">wrapper.ntservice.displayname=ActiveMQ-Service</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Description of the service</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">wrapper.ntservice.description=ActiveMQ Broker Service</span></div>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># Log file to use for wrapper output logging.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">wrapper.logfile=C:\\yajsw-stable-11.03\\log\\wrapper.log</span></div>
</div>
<div>
<br /></div>
<div>
Now let's install ActiveMQ as a Windows Service</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-0nQI9MMH6Fk/Uc551mbbgRI/AAAAAAAAAFI/LDA98QfvXdg/s1600/Screen+Shot+2013-06-29+at+11.19.30+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="92" src="http://2.bp.blogspot.com/-0nQI9MMH6Fk/Uc551mbbgRI/AAAAAAAAAFI/LDA98QfvXdg/s640/Screen+Shot+2013-06-29+at+11.19.30+AM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Let's confirm if service got registered correctly</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
start -> run -> <b>services.msc</b></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-w9hdIrTdZf0/Uc564Z991VI/AAAAAAAAAFY/tSTAZDRZh6A/s1600/Screen+Shot+2013-06-29+at+11.41.52+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="114" src="http://3.bp.blogspot.com/-w9hdIrTdZf0/Uc564Z991VI/AAAAAAAAAFY/tSTAZDRZh6A/s640/Screen+Shot+2013-06-29+at+11.41.52+AM.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Details:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-pLy4GVI-ksg/Uc57roFx5pI/AAAAAAAAAFo/NepdDR5dtqE/s950/Screen+Shot+2013-06-29+at+11.45.11+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-pLy4GVI-ksg/Uc57roFx5pI/AAAAAAAAAFo/NepdDR5dtqE/s320/Screen+Shot+2013-06-29+at+11.45.11+AM.png" width="280" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
In order to uninstall, run the uninstallService.bat</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-zLwV4aloGv4/Uc58YnogMII/AAAAAAAAAFw/WxJr8XxFiB4/s1600/Screen+Shot+2013-06-29+at+11.48.22+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="96" src="http://3.bp.blogspot.com/-zLwV4aloGv4/Uc58YnogMII/AAAAAAAAAFw/WxJr8XxFiB4/s640/Screen+Shot+2013-06-29+at+11.48.22+AM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
<b>Conclusion:</b></div>
<div>
<br /></div>
<div>
We have just seen how to install ActiveMQ as a Windows Service using YAJSW on 64bit Windows. </div>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-33576164735311864252013-06-16T22:27:00.000-07:002013-06-24T13:37:22.846-07:00Maven Release Plugin with Git, GitHub and CloudBees<div dir="ltr" style="text-align: left;" trbidi="on">
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css"></link>
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushVb.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js" type="text/javascript"></script>
<script language="javascript">
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = 'http://alexgorbatchev.com/pub/sh/current/scripts/clipboard.swf';
SyntaxHighlighter.all();
</script>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Objective:</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This blog is to demonstrate the use of maven release plugin using Git, GitHub and CloudBees repository. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Introduction:</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Typical software release process entails many repetitive and mundane tasks which can be automated. These tasks include building, tagging, running automated tests, generating documentation, re-versioning, deploying generated artifact(s), rollback etc. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Maven is an opinionated software project management tool. It has a plugin architecture which facilitates automating software project lifecycle management. It provides a release-plugin which automates the software release in two steps: "prepare" and "perform" (more on this later) </span><br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">However, in most companies, software release process varies depending on technology, product and process maturity. The software release process doesn't usually lend itself to be a "one-size-fits-all" process. That is why t</span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">here is a huge debate in the community on merits and demerits of this plugin. There is a very good article by Lieven Doclo: <span style="color: blue;"> </span></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: small; letter-spacing: -0.04em;"><a href="http://www.insaneprogramming.be/?p=362" style="letter-spacing: -0.04em;"><span style="color: blue;">Why I never use the Maven release plugin</span></a>. I highly suggest to read that blog & relevant comments. </span><br />
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Overview:</b></span><br />
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: small; letter-spacing: -0.04em;">In this article I will be sharing how to use Maven release plugin "only" for small shared components or projects where out of the box maven release plugin is sufficient. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: small; letter-spacing: -0.04em;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: small; letter-spacing: -0.04em;">I will be using GitHub as SCM and CloudBees as artifact repository where release artifacts will be deployed. Both GitHub and CloudBees offer free usage tier which is just awesome! </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: small; letter-spacing: -0.04em;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; font-size: small; letter-spacing: -0.04em;">Here is the outline of the topics we will cover in this blog:</span><br />
<ol style="text-align: left;">
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Understand what happens in "prepare" goal</span></li>
<ul>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Release tagging</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">POM Re-versioning</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Meta data to assist with "perform" or "rollback" goal</span></li>
</ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Limitation of "rollback" goal</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Understand what happens in "perform" goal</span></li>
<li><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Browse CloudBees artifact repository to find deployed release artifact</span></li>
</ol>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Prerequisites:</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><br /></b></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">You may find familiarity with the following useful as you read this article :)</span><br />
<ol style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Maven</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Git</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">GitHub</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">CloudBees</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">WebDAV (not essential but recommended)</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Configure SSH keys so local machine can access GitHub repository over ssh</span></li>
</ol>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Deep Dive:</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Creating a project repository: "mrp-git-cb-demo"</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-DtqCv_fzuTw/Ub4CplHuhJI/AAAAAAAAADw/42pI_SHHNDk/s1600/Screen+Shot+2013-06-15+at+10.11.09+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="353" src="http://1.bp.blogspot.com/-DtqCv_fzuTw/Ub4CplHuhJI/AAAAAAAAADw/42pI_SHHNDk/s640/Screen+Shot+2013-06-15+at+10.11.09+PM.png" width="640" /></a></div>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Clone the remote repo from GitHub.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GIT akuntamukkala$ <b><span style="color: blue;">git clone git@github.com:akuntamukkala/mrp-git-cb-demo.git</span></b>
Cloning into 'mrp-git-cb-demo'...<br />
remote: Counting objects: 3, done.<br />
remote: Compressing objects: 100% (2/2), done.<br />
remote: Total 3 (delta 0), reused 0 (delta 0)<br />
Receiving objects: 100% (3/3), done.</code></div>
<div>
<br /></div>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Add .gitignore file with file types which needed not be checked in and commit the changes to local and push to remote repository</span><br />
<br /></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Creating a sample maven project, say "Utility"using maven quickstart archetype</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br />
Ashwinis-MacBook-Pro:mrp-git-cb-demo akuntamukkala$ <b><span style="color: blue;">mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.1</span></b><br />
[INFO] Scanning for projects...<br />
[INFO] <br />
[INFO] ------------------------------------------------------------------------<br />
[INFO] Building Maven Stub Project (No POM) 1<br />
[INFO] ------------------------------------------------------------------------<br />
[INFO] <br />
[INFO] >>> maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom >>><br />
[INFO] <br />
[INFO] <<< maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom <<<<br />
[INFO] <br />
[INFO] --- maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom ---<br /><span style="color: blue;">
[INFO] Generating project in Interactive mode<br />
[INFO] Archetype repository missing. Using the one from [org.apache.maven.archetypes:maven-archetype-quickstart:1.1] found in catalog remote<br />
Define value for property 'groupId': : com.app<br />
Define value for property 'artifactId': : utility<br />
Define value for property 'version': 1.0-SNAPSHOT: : <br />
Define value for property 'package': com.app: : <br />
Confirm properties configuration:<br />
groupId: com.app<br />
artifactId: utility<br />
version: 1.0-SNAPSHOT<br />
package: com.app<br />
Y: : Y</span><br />
[INFO] ----------------------------------------------------------------------------<br />
[INFO] Using following parameters for creating project from Old (1.x) Archetype: maven-archetype-quickstart:1.1<br />
[INFO] ----------------------------------------------------------------------------<br />
[INFO] Parameter: groupId, Value: com.app<br />
[INFO] Parameter: packageName, Value: com.app<br />
[INFO] Parameter: package, Value: com.app<br />
[INFO] Parameter: artifactId, Value: utility<br />
[INFO] Parameter: basedir, Value: /Users/akuntamukkala/GIT/mrp-git-cb-demo<br />
[INFO] Parameter: version, Value: 1.0-SNAPSHOT<br />
[INFO] project created from Old (1.x) Archetype in dir: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility<br />
[INFO] ------------------------------------------------------------------------<br />
[INFO] BUILD SUCCESS<br />
[INFO] ------------------------------------------------------------------------<br />
[INFO] Total time: 26.656s<br />
[INFO] Finished at: Sun Jun 16 14:55:37 GMT+05:30 2013<br />
[INFO] Final Memory: 9M/81M<br />
[INFO] ------------------------------------------------------------------------
</code></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Commit the "Utility" project to GitHub</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:mrp-git-cb-demo akuntamukkala$ <b><span style="color: blue;">git status</span></b>
</code><br />
<div>
<code class="brush: core"># On branch master</code></div>
<div>
<code class="brush: core"># Untracked files:</code></div>
<div>
<code class="brush: core"># (use "git add <file>..." to include in what will be committed)</code></div>
<div>
<code class="brush: core">#</code></div>
<div>
<code class="brush: core">#<span class="Apple-tab-span" style="white-space: pre;"> </span>.gitignore</code></div>
<div>
<code class="brush: core">#<span class="Apple-tab-span" style="white-space: pre;"> </span>utility/</code></div>
<div>
<code class="brush: core">nothing added to commit but untracked files present (use "git add" to track)</code><br />
<code class="brush: core"><br /></code></div>
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:mrp-git-cb-demo akuntamukkala$ <b><span style="color: blue;">git add .</span></b></code><br />
<code class="brush: core"><br /></code></div>
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:mrp-git-cb-demo akuntamukkala$ <b><span style="color: blue;">git commit -m "first commit"</span></b></code></div>
<div>
<code class="brush: core">[master 8a70309] first commit</code></div>
<div>
<code class="brush: core"> 4 files changed, 92 insertions(+)</code></div>
<div>
<code class="brush: core"> create mode 100644 .gitignore</code></div>
<div>
<code class="brush: core"> create mode 100644 utility/pom.xml</code></div>
<div>
<code class="brush: core"> create mode 100644 utility/src/main/java/com/app/App.java</code></div>
<div>
<code class="brush: core"> create mode 100644 utility/src/test/java/com/app/AppTest.java</code></div>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:mrp-git-cb-demo akuntamukkala$ <b><span style="color: blue;">git push</span></b></code></div>
<div>
<code class="brush: core">Counting objects: 17, done.</code></div>
<div>
<code class="brush: core">Delta compression using up to 8 threads.</code></div>
<div>
<code class="brush: core">Compressing objects: 100% (8/8), done.</code></div>
<div>
<code class="brush: core">Writing objects: 100% (16/16), 1.63 KiB, done.</code></div>
<div>
<code class="brush: core">Total 16 (delta 0), reused 0 (delta 0)</code></div>
<div>
<code class="brush: core">To git@github.com:akuntamukkala/mrp-git-cb-demo.git</code></div>
<code class="brush: core">
421475b..8a70309 master -> master<br />
</code>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The last commit is now visible in master branch at mrp-git-cb-demo repo at GitHub as shown in the following screenshot. </span> </div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-9cnJIAygACY/Ub2LgMmfVWI/AAAAAAAAACM/8rg0zV7rqq4/s1600/Screen+Shot+2013-06-16+at+3.24.08+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="179" src="http://4.bp.blogspot.com/-9cnJIAygACY/Ub2LgMmfVWI/AAAAAAAAACM/8rg0zV7rqq4/s640/Screen+Shot+2013-06-16+at+3.24.08+PM.png" width="640" /></a></div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Next we will make the following changes:</span><br />
<ol style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Edit Maven's settings.xml so that Maven is aware of CloudBees release and snapshot repository credentials. </span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Edit Utility project's maven pom.xml to add the following:</span></li>
<ol>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">GitHub SCM configuration</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Distribution Management of generated artifacts</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Maven Release Plugin (GAV and any additional parameters)</span></li>
</ol>
</ol>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><b>CloudBees online Artifact Repository for release and snapshots:</b></span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">When you create an account at CloudBees, you get 4 free online instances JFrog Artifact Repositories (backup, private, release, snapshot).</span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Please note URL of following artifact repositories:</span><br />
<br />
<span style="background-color: #f3f6f9; border: 0px; font-family: Helvetica Neue, Arial, Helvetica, sans-serif; line-height: 14.6875px; margin: 0px; padding: 0px; vertical-align: baseline;">Release: <a class="normal" href="https://repository-akuntamukkala.forge.cloudbees.com/release/" style="background-color: #f3f6f9; border: 0px; color: #3e85cb; line-height: 14.6875px; margin: 0px; padding: 0px; vertical-align: baseline;">https://repository-akuntamukkala.forge.cloudbees.com/release/</a></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Snapshot:</span> <a class="normal" href="https://repository-akuntamukkala.forge.cloudbees.com/release/" style="background-color: #f3f6f9; border: 0px; color: #3e85cb; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 14.6875px; margin: 0px; padding: 0px; vertical-align: baseline;">https://repository-akuntamukkala.forge.cloudbees.com/snapshot/</a><br />
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The repository access is restricted by your CloudBees account user name and password. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In my case it is "akuntamukkala" and "canyouguess?!" respectively.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following is a screen shot of CloudBees account showing release repository details. Please note that this repository is accessible using WebDAV. </span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-eBwZ3BaKDJc/Ub4GmTny0VI/AAAAAAAAAD8/H8r0BNIRW80/s1600/Screen+Shot+2013-06-17+at+12.09.53+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="212" src="http://2.bp.blogspot.com/-eBwZ3BaKDJc/Ub4GmTny0VI/AAAAAAAAAD8/H8r0BNIRW80/s640/Screen+Shot+2013-06-17+at+12.09.53+AM.png" width="640" /></a></div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">We need to now configure Maven installation to be aware of the CloudBees release and snapshot repository access credentials. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Locate the settings.xml in Maven in MVN_HOME/conf directory and add the following XML snippet in the "servers" node.</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<server><br />
<id>akuntamukkala-release</id><br />
<username>akuntamukkala</username><br />
<password>canyouguess?!</password><br />
</server><br />
<server><br />
<id>akuntamukkala-snapshot</id><br />
<username>akuntamukkala</username><br />
<password>canyouguess?!</password><br />
</server></code><br />
<code class="brush: core">
</code>
</div>
<div>
<div style="text-align: left;">
<br />
<span style="background-color: white; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Edit Utility project's maven pom.xml to add the following XML snippet:</span><br />
<span style="background-color: azure; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core"><span style="font-size: x-small;">
</span></code>
<br />
<div style="text-align: left;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><!-- SCM configuration to GitHub <span class="s2">Repo</span> --></span><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"><</span>scm<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">connection</span><span class="s3">></span>scm:git:git<span class="s2">@github.com:akuntamukkala</span>/<span class="s2">mrp</span>-<span class="s2">git</span>-<span class="s2">cb</span>-demo.git<span class="s3"></</span><span class="s4">connection</span><span class="s3">> </span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </span><span class="s3"><</span><span class="s4">url</span><span class="s3">></span>scm:git:git<span class="s2">@github.com:akuntamukkala</span>/<span class="s2">mrp</span>-<span class="s2">git</span>-<span class="s2">cb</span>-demo.git<span class="s3"></</span><span class="s4">url</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">developerConnection</span><span class="s3">></span>scm:git:git<span class="s2">@github.com:akuntamukkala</span>/<span class="s2">mrp</span>-<span class="s2">git</span>-<span class="s2">cb</span>-demo.git<span class="s3"></</span><span class="s4">developerConnection</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </span><span class="s3"><</span><span class="s4">tag</span><span class="s3">></span>HEAD<span class="s3"></</span><span class="s4">tag</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"></</span>scm<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><br /></span></code>
<code class="brush: core"><span style="font-size: x-small;"><!-- CloudBees Artifact Repository for release and snapshot builds --></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><!-- The repositories are accessible using WebDAV --></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"><</span>distributionManagement<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>repository<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">id</span><span class="s3">></span><span class="s2">akuntamukkala</span>-release<span class="s3"></</span><span class="s4">id</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">name</span><span class="s3">></span>Internal Release Repository<span class="s3"></</span><span class="s4">name</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">url</span><span class="s3">></span><span style="color: blue;">dav:https://repository-akuntamukkala.forge.cloudbees.com/release</span><span class="s3"></</span><span class="s4">url</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </</span>repository<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>snapshotRepository<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">id</span><span class="s3">></span><span class="s2">akuntamukkala</span>-snapshot<span class="s3"></</span><span class="s4">id</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">name</span><span class="s3">></span>Internal Snapshot Repository<span class="s3"></</span><span class="s4">name</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">url</span><span class="s3">></span><span style="color: blue;">dav:https://repository-akuntamukkala.forge.cloudbees.com/snapshot</span><span class="s3"></</span><span class="s4">url</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </</span>snapshotRepository<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"></</span>distributionManagement<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><br /></span></code>
<code class="brush: core"><span style="font-size: x-small;"><!-- The <span class="s2">maven</span> release <span class="s2">plugin</span> configuration along with WebDAV extension to</span></code><br />
<code class="brush: core"><span style="font-size: x-small;">help deploy the build artifacts into remote artifact repository --></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"><br /></span></span></code>
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"><</span>build<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>extensions<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"> <!-- Extension required to deploy a snapshot or a release to the CloudBees</span></code><br />
<code class="brush: core"><span style="font-size: x-small;"> remote <span class="s2">maven</span> repository using <span class="s2">Webdav</span> --></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>extension<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">groupId</span><span class="s3">></span>org.apache.maven.wagon<span class="s3"></</span><span class="s4">groupId</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>artifactId<span class="s3">></span><span class="s1">wagon-</span><span class="s5">webdav</span><span class="s3"></</span>artifactId<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">version</span><span class="s3">></span>1.0-beta-2<span class="s3"></</span><span class="s4">version</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </</span>extension<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </</span>extensions<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>plugins<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>plugin<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">groupId</span><span class="s3">></span>org.apache.maven.plugins<span class="s3"></</span><span class="s4">groupId</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span><span class="s4">artifactId</span><span class="s3">></span><span class="s2">maven</span>-release-<span class="s2">plugin</span><span class="s3"></</span><span class="s4">artifactId</span><span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> <</span>version<span class="s3">></span><span class="s1">2.4.1</span><span class="s3"></</span>version<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </</span>plugin<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"> </</span>plugins<span class="s3">></span></span></code><br />
<code class="brush: core"><span style="font-size: x-small;"><span class="s3"></</span>build<span class="s3">></span></span></code></div>
<code class="brush: core">
</code>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now let's attempt to run maven release plugin's prepare goal. The prepare goal performs the following tasks:</span></div>
<div style="text-align: left;">
</div>
<ol style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Increments project's pom.xml version</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Makes a release tag with pom.xml containing release version</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Creates meta data files such as release.properties and pom.xml.releaseBackup (useful if we need to rollback)</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Propagate these changes to SCM </span></li>
</ol>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see this in action by running the maven release plugin prepare goal.</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ pwd</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">/Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$<span style="color: blue;"><b> mvn release:prepare -DdryRun=true</b></span></code></div>
<code class="brush: core">
<br /><span style="color: red;">
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.4.1:prepare (default-cli) on project utility: Cannot prepare the release because you have local modifications :<br />
[ERROR] [pom.xml:modified]
</span></code></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Oops! As you just noticed, an error occurred. This is due to the fact that there are uncommitted changes to pom.xml file. We need to ensure that there are no pending changes and all changes are pushed to GitHub repository before running maven release plugin's prepare goal . </span></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">So let's commit all changes to the local git cloned repo and push to GitHub</span> </div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git status </span></b><br />
# On branch master<br />
# Changes not staged for commit:<br />
# (use "git add <file>..." to update what will be committed)<br />
# (use "git checkout -- <file>..." to discard changes in working directory)<br />
#<br />
# modified: pom.xml<br />
#<br />
# Untracked files:<br />
# (use "git add <file>..." to include in what will be committed)<br />
#<br />
# release.properties<br />
no changes added to commit (use "git add" and/or "git commit -a")</code></div>
</div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Hmm..there is release.properties file generated by the previous execution of maven release plugin's prepare goal. We need to clean up to go back to a state prior to running prepare goal. Here is where maven release plugin's "clean" goal comes into play. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">mvn release:clean</span></b></code><br />
<code class="brush: core">[INFO] Scanning for projects...</code><br />
<code class="brush: core">[INFO] </code><br />
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code><br />
<code class="brush: core">[INFO] Building utility 1.0-SNAPSHOT</code><br />
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code><br />
<code class="brush: core">[INFO] </code><br />
<code class="brush: core">[INFO] --- maven-release-plugin:2.4.1:clean (default-cli) @ utility ---</code><br />
<code class="brush: core">[INFO] Cleaning up after release...</code><br />
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code><br />
<code class="brush: core">[INFO] BUILD SUCCESS</code><br />
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code><br />
<code class="brush: core">[INFO] Total time: 1.233s</code><br />
<code class="brush: core">[INFO] Finished at: Sun Jun 16 15:38:39 GMT+05:30 2013</code><br />
<code class="brush: core">[INFO] Final Memory: 5M/81M</code><br />
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<code class="brush: core">
</code>
</div>
<div>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now you can see that only uncommitted change is our changes to pom.xml where we added SCM, distribution management and release plugin's GAV </span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
Ashwinis-MacBook-Pro:utility akuntamukkala$<b><span style="color: blue;"> git status</span></b><br />
# On branch master<br />
# Changes not staged for commit:<br />
# (use "git add <file>..." to update what will be committed)<br />
# (use "git checkout -- <file>..." to discard changes in working directory)<br />
#<br />
# modified: pom.xml<br />
#<br />
no changes added to commit (use "git add" and/or "git commit -a")<br />
<br />
</code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ git add .</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ git commit -m "scm, distribution management, release plugin config"</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[master e8422bb] scm, distribution management, release plugin config</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core"> 1 file changed, 67 insertions(+), 25 deletions(-)</code></div>
<code class="brush: core">
rewrite utility/pom.xml (83%)<br />
</code>
</div>
<br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Push the change to GitHub</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git push</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Counting objects: 7, done.</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Delta compression using up to 8 threads.</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Compressing objects: 100% (4/4), done.</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Writing objects: 100% (4/4), 1.17 KiB, done.</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Total 4 (delta 1), reused 0 (delta 0)</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">To git@github.com:akuntamukkala/mrp-git-cb-demo.git</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core"> 8a70309..e8422bb master -> master</code></div>
<code class="brush: core">
</code>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that we have ensured that there are no more pending commits and both local and GitHub repo are in sync, let's run maven release:prepare command</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$<b><span style="color: blue;"> mvn release:prepare</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Scanning for projects...</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] </code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Building utility 1.0-SNAPSHOT</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] </code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] --- maven-release-plugin:2.4.1:prepare (default-cli) @ utility ---</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Verifying that there are no local modifications...</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] ignoring changes on: **/pom.xml.backup, **/release.properties, **/pom.xml.branch, **/pom.xml.next, **/pom.xml.releaseBackup, **/pom.xml.tag</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Checking dependencies and plugins for snapshots ...</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">What is the release version for "utility"? (com.app:utility) 1.0: : </code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">What is SCM release tag or label for "utility"? (com.app:utility) utility-1.0: : </code></div>
<code class="brush: core">
</code>
<div>
<code class="brush: core">What is the new development version for "utility"? (com.app:utility) 1.1-SNAPSHOT: : </code></div>
<code class="brush: core">
<div>
[INFO] Transforming 'utility'...</div>
<div>
[INFO] Not generating release POMs</div>
<div>
[INFO] Executing goals 'clean verify'...</div>
<div>
[WARNING] Maven will be executed in interactive mode, but no input stream has been configured for this MavenInvoker instance.</div>
<div>
[INFO] [INFO] Scanning for projects...</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] [INFO] Building utility 1.0</div>
<div>
[INFO] [INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ utility ---</div>
<div>
[INFO] [INFO] Deleting /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ utility ---</div>
<div>
[INFO] [debug] execute contextualize</div>
<div>
[INFO] [INFO] Using 'UTF-8' encoding to copy filtered resources.</div>
<div>
[INFO] [INFO] skip non existing resourceDirectory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/src/main/resources</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ utility ---</div>
<div>
[INFO] [INFO] Compiling 1 source file to /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/classes</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ utility ---</div>
<div>
[INFO] [debug] execute contextualize</div>
<div>
[INFO] [INFO] Using 'UTF-8' encoding to copy filtered resources.</div>
<div>
[INFO] [INFO] skip non existing resourceDirectory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/src/test/resources</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ utility ---</div>
<div>
[INFO] [INFO] Compiling 1 source file to /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/test-classes</div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-surefire-plugin:2.10:test (default-test) @ utility ---</div>
<div>
[INFO] [INFO] Surefire report directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/surefire-reports</div>
<div>
[INFO] </div>
<div>
[INFO] -------------------------------------------------------</div>
<div>
[INFO] T E S T S</div>
<div>
[INFO] -------------------------------------------------------</div>
<div>
[INFO] Running com.app.AppTest</div>
<div>
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec</div>
<div>
[INFO] </div>
<div>
[INFO] Results :</div>
<div>
[INFO] </div>
<div>
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0</div>
<div>
[INFO] </div>
<div>
[INFO] [INFO] </div>
<div>
[INFO] [INFO] --- maven-jar-plugin:2.3.2:jar (default-jar) @ utility ---</div>
<div>
[INFO] [INFO] Building jar: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/utility-1.0.jar</div>
<div>
[INFO] [INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] [INFO] BUILD SUCCESS</div>
<div>
[INFO] [INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] [INFO] Total time: 1.755s</div>
<div>
[INFO] [INFO] Finished at: Sun Jun 16 16:16:57 GMT+05:30 2013</div>
<div>
[INFO] [INFO] Final Memory: 9M/81M</div>
<div>
[INFO] [INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] Checking in modified POMs...</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git add -- pom.xml</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git commit --verbose -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-1228315049.commit pom.xml</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git symbolic-ref HEAD</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git master:master</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Tagging release with the label utility-1.0...</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git tag -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-1142006148.commit utility-1.0</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git utility-1.0</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git ls-files</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Transforming 'utility'...</div>
<div>
[INFO] Not removing release POMs</div>
<div>
[INFO] Checking in modified POMs...</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git add -- pom.xml</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git commit --verbose -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-1383546415.commit pom.xml</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git symbolic-ref HEAD</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git master:master</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Release preparation complete.</div>
<div>
[INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] BUILD SUCCESS</div>
<div>
[INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] Total time: 30.308s</div>
<div>
[INFO] Finished at: Sun Jun 16 16:17:20 GMT+05:30 2013</div>
<div>
[INFO] Final Memory: 5M/81M</div>
<div>
[INFO] ------------------------------------------------------------------------</div>
</code>
</div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>What just happened?</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Looking at the above logs from release:prepare execution, we can see that project version has been updated to 1.1-SNAPSHOT and release pom with version 1.0 has been tagged "utility-1.0". All of these changes have been made on local repo and pushed to GitHub.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following GitHub screenshot shows tag "utility-1.0"</span></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-5rhO60agVWA/Ub2brvy5TZI/AAAAAAAAACc/vauUHRY6U9M/s1600/Screen+Shot+2013-06-16+at+4.33.40+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="177" src="http://1.bp.blogspot.com/-5rhO60agVWA/Ub2brvy5TZI/AAAAAAAAACc/vauUHRY6U9M/s640/Screen+Shot+2013-06-16+at+4.33.40+PM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see pom.xml in master branch in the following screenshot. Please observe that version has been changed to 1.1-SNAPSHOT</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-lI-c7O8PfQg/Ub2cKr7aCqI/AAAAAAAAACk/mOgEI-4J2jk/s1600/Screen+Shot+2013-06-16+at+4.35.45+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-lI-c7O8PfQg/Ub2cKr7aCqI/AAAAAAAAACk/mOgEI-4J2jk/s640/Screen+Shot+2013-06-16+at+4.35.45+PM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see the pom.xml in the utility-1.0 tag in the following screenshot. Please observe that release version is 1.0. This is going to be used when maven release plugin's "perform" goal is executed. More on this later.</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-xxWc1MzUUN0/Ub2cmD_D0UI/AAAAAAAAACs/t4SclAyLXeU/s1600/Screen+Shot+2013-06-16+at+4.37.44+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://4.bp.blogspot.com/-xxWc1MzUUN0/Ub2cmD_D0UI/AAAAAAAAACs/t4SclAyLXeU/s640/Screen+Shot+2013-06-16+at+4.37.44+PM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git status</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"># On branch master</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"># Your branch is ahead of 'origin/master' by 2 commits.</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">#</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"># Untracked files:</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"># (use "git add <file>..." to include in what will be committed)</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">#</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">#<span class="Apple-tab-span" style="white-space: pre;"> </span>pom.xml.releaseBackup</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">#<span class="Apple-tab-span" style="white-space: pre;"> </span>release.properties</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">nothing added to commit but untracked files present (use "git add" to track)</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"><br /></code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git pull</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">From github.com:akuntamukkala/mrp-git-cb-demo</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"> e8422bb..fda4cdd master -> origin/master</code></div>
<code class="brush: core">
</code>
<div class="separator" style="clear: both;">
<code class="brush: core">Already up-to-date.</code></div>
<code class="brush: core">
<div class="separator" style="clear: both;">
Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git status</span></b></div>
<div class="separator" style="clear: both;">
# On branch master</div>
<div class="separator" style="clear: both;">
# Untracked files:</div>
<div class="separator" style="clear: both;">
# (use "git add <file>..." to include in what will be committed)</div>
<div class="separator" style="clear: both;">
#</div>
<div class="separator" style="clear: both;">
#<span class="Apple-tab-span" style="white-space: pre;"> </span>pom.xml.releaseBackup</div>
<div class="separator" style="clear: both;">
#<span class="Apple-tab-span" style="white-space: pre;"> </span>release.properties</div>
</code>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Rollback after release prepare:</b></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">What if we need to rollback the changes made during "prepare" goal execution?</span></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This could happen because of many reasons: there is always that one more bug that needs to go in this release, release gets pushed by business folks or QA reveals a critical bug etc. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I mean we have all seen these situations at some point. So we decide to rollback using maven release plugin's rollback goal. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This works fine...except for the fact that it does not delete the "utility-1.0" tag in the repo at least as of this writing. </span></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see this in action...</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">mvn release:rollback</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Scanning for projects...</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] </code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Building utility 1.1-SNAPSHOT</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] </code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] --- maven-release-plugin:2.4.1:rollback (default-cli) @ utility ---</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Checking in modified POMs...</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git add -- pom.xml</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</code></div>
<code class="brush: core">
</code>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<code class="brush: core">
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git commit --verbose -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-1207924889.commit pom.xml</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git symbolic-ref HEAD</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git master:master</div>
<div>
[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</div>
<div>
[INFO] Cleaning up after release...</div>
<div>
[INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] BUILD SUCCESS</div>
<div>
[INFO] ------------------------------------------------------------------------</div>
<div>
[INFO] Total time: 8.952s</div>
<div>
[INFO] Finished at: Sun Jun 16 16:42:04 GMT+05:30 2013</div>
<div>
[INFO] Final Memory: 5M/81M</div>
<div>
[INFO] ------------------------------------------------------------------------</div>
</code>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Rollback reverted the changes so master branch's pom.xml contains version 1.0-SNAPSHOT. For the rollback to have executed, it is vital to have release.properties file. If that was missing, then this operation would have failed. </span></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's inspect the changes committed by rollback operation</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git status</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core"># On branch master</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core"># Your branch is ahead of 'origin/master' by 1 commit.</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">#</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">nothing to commit (working directory clean)</code></div>
<code class="brush: core">
</code>
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">pom.xml in master branch reveals that version is reset to 1.0-SNAPSHOT as shown in the screenshot below. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-JYU19eza4UI/Ub2e6V_DX6I/AAAAAAAAAC8/p2C0ybOtXEs/s1600/Screen+Shot+2013-06-16+at+4.47.40+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="252" src="http://4.bp.blogspot.com/-JYU19eza4UI/Ub2e6V_DX6I/AAAAAAAAAC8/p2C0ybOtXEs/s640/Screen+Shot+2013-06-16+at+4.47.40+PM.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here you may notice that the "utility-1.0" tag still remains in GitHub repo. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This feature of rollback where the release tag needs to be deleted has not been implemented as of this writing. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Refer <a href="http://maven.apache.org/maven-release/maven-release-plugin/examples/rollback-release.html">http://maven.apache.org/maven-release/maven-release-plugin/examples/rollback-release.html</a> for more details.</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-GIUz66C-9Uk/Ub2fn5qFaZI/AAAAAAAAADE/CAxxQ6_Q2hk/s1600/Screen+Shot+2013-06-16+at+4.50.26+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="288" src="http://4.bp.blogspot.com/-GIUz66C-9Uk/Ub2fn5qFaZI/AAAAAAAAADE/CAxxQ6_Q2hk/s640/Screen+Shot+2013-06-16+at+4.50.26+PM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's go ahead and delete this manually for now</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git pull</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">From github.com:akuntamukkala/mrp-git-cb-demo</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core"> fda4cdd..d280671 master -> origin/master</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Already up-to-date.</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git tag -n</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">utility-1.0 [maven-release-plugin] copy for tag utility-1.0</code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git tag -d utility-1.0</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div class="separator" style="clear: both;">
<code class="brush: core">Deleted tag 'utility-1.0' (was d5f3fc4)</code></div>
<code class="brush: core">
</code>
<div class="separator" style="clear: both;">
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">git push origin :refs/tags/utility-1.0</span></b></code></div>
<code class="brush: core">
<div class="separator" style="clear: both;">
To git@github.com:akuntamukkala/mrp-git-cb-demo.git</div>
<div class="separator" style="clear: both;">
- [deleted] utility-1.0</div>
</code>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's resume the release process again by first executing the prepare goal again.</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code><br />
<div>
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$ <b><span style="color: blue;">mvn release:prepare</span></b></code></div>
<div>
<code class="brush: core">[INFO] Scanning for projects...</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Building utility 1.0-SNAPSHOT</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] --- maven-release-plugin:2.4.1:prepare (default-cli) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] Verifying that there are no local modifications...</code></div>
<div>
<code class="brush: core">[INFO] ignoring changes on: **/pom.xml.backup, **/release.properties, **/pom.xml.branch, **/pom.xml.next, **/pom.xml.releaseBackup, **/pom.xml.tag</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Checking dependencies and plugins for snapshots ...</code></div>
<div>
<code class="brush: core"><span style="color: blue;">What is the release version for "utility"? (com.app:utility) 1.0: : </span></code></div>
<div>
<code class="brush: core"><span style="color: blue;">What is SCM release tag or label for "utility"? (com.app:utility) utility-1.0: : </span></code></div>
<div>
<code class="brush: core"><span style="color: blue;">What is the new development version for "utility"? (com.app:utility) 1.1-SNAPSHOT: : </span></code></div>
<div>
<code class="brush: core">[INFO] Transforming 'utility'...</code></div>
<div>
<code class="brush: core">[INFO] Not generating release POMs</code></div>
<div>
<code class="brush: core">[INFO] Executing goals 'clean verify'...</code></div>
<div>
<code class="brush: core">[WARNING] Maven will be executed in interactive mode, but no input stream has been configured for this MavenInvoker instance.</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Scanning for projects...</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Building utility 1.0</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Deleting /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [debug] execute contextualize</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Using 'UTF-8' encoding to copy filtered resources.</code></div>
<div>
<code class="brush: core">[INFO] [INFO] skip non existing resourceDirectory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/src/main/resources</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Compiling 1 source file to /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/classes</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [debug] execute contextualize</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Using 'UTF-8' encoding to copy filtered resources.</code></div>
<div>
<code class="brush: core">[INFO] [INFO] skip non existing resourceDirectory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/src/test/resources</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Compiling 1 source file to /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/test-classes</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-surefire-plugin:2.10:test (default-test) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Surefire report directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/surefire-reports</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] -------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] T E S T S</code></div>
<div>
<code class="brush: core">[INFO] -------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Running com.app.AppTest</code></div>
<div>
<code class="brush: core">[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Results :</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-jar-plugin:2.3.2:jar (default-jar) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Building jar: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/utility-1.0.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] BUILD SUCCESS</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Total time: 1.813s</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Finished at: Sun Jun 16 17:40:01 GMT+05:30 2013</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Final Memory: 9M/81M</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Checking in modified POMs...</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git add -- pom.xml</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git commit --verbose -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-1892728079.commit pom.xml</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git symbolic-ref HEAD</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git master:master</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Tagging release with the label utility-1.0...</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git tag -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-1866393319.commit utility-1.0</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git utility-1.0</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git ls-files</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Transforming 'utility'...</code></div>
<div>
<code class="brush: core">[INFO] Not removing release POMs</code></div>
<div>
<code class="brush: core">[INFO] Checking in modified POMs...</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git add -- pom.xml</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git status</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git commit --verbose -F /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/maven-scm-347133367.commit pom.xml</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git symbolic-ref HEAD</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility && git push git@github.com:akuntamukkala/mrp-git-cb-demo.git master:master</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility</code></div>
<div>
<code class="brush: core">[INFO] Release preparation complete.</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] BUILD SUCCESS</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Total time: 1:19.582s</code></div>
<div>
<code class="brush: core">[INFO] Finished at: Sun Jun 16 17:40:22 GMT+05:30 2013</code></div>
<div>
<code class="brush: core">[INFO] Final Memory: 5M/81M</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
</div>
<code class="brush: core">
</code>
</div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now let's execute the perform goal of release plugin to see the following tasks being performed.</span></div>
<div>
<ol style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Clone the utility-1.0 tag </span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Execute maven build cycle to build release artifact</span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Upload the release artifact into CloudBees release artifact repository using WebDAV</span></li>
</ol>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's see this in action</span></div>
<div>
<br /></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
</code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:utility akuntamukkala$<b><span style="color: blue;"> mvn release:perform</span></b></code></div>
<code class="brush: core">
</code>
<br />
<div>
<div>
<code class="brush: core">[INFO] Scanning for projects...</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Building utility 1.1-SNAPSHOT</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] --- maven-release-plugin:2.4.1:perform (default-cli) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] Checking out the project to perform the release ...</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target && git clone --branch utility-1.0 git@github.com:akuntamukkala/mrp-git-cb-demo.git /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T/ && git ls-remote git@github.com:akuntamukkala/mrp-git-cb-demo.git</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /var/folders/zs/9w9zbpnx4db89z8yrll8301r0000gn/T</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout && git fetch git@github.com:akuntamukkala/mrp-git-cb-demo.git</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout && git checkout utility-1.0</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout</code></div>
<div>
<code class="brush: core">[INFO] Executing: /bin/sh -c cd /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout && git ls-files</code></div>
<div>
<code class="brush: core">[INFO] Working directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout</code></div>
<div>
<code class="brush: core">[INFO] Invoking perform goals in directory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility</code></div>
<div>
<code class="brush: core">[INFO] Executing goals 'deploy'...</code></div>
<div>
<code class="brush: core">[WARNING] Maven will be executed in interactive mode, but no input stream has been configured for this MavenInvoker instance.</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Scanning for projects...</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] </code></div>
<div>
<code class="brush: core">[INFO] [WARNING] Some problems were encountered while building the effective model for com.app:utility:jar:1.0</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-javadoc-plugin is missing.</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-deploy-plugin is missing.</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing.</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] </code></div>
<div>
<code class="brush: core">[INFO] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] </code></div>
<div>
<code class="brush: core">[INFO] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.</code></div>
<div>
<code class="brush: core">[INFO] [WARNING] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Building utility 1.0</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [debug] execute contextualize</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Using 'UTF-8' encoding to copy filtered resources.</code></div>
<div>
<code class="brush: core">[INFO] [INFO] skip non existing resourceDirectory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/src/main/resources</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Compiling 1 source file to /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/classes</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [debug] execute contextualize</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Using 'UTF-8' encoding to copy filtered resources.</code></div>
<div>
<code class="brush: core">[INFO] [INFO] skip non existing resourceDirectory /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/src/test/resources</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Compiling 1 source file to /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/test-classes</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-surefire-plugin:2.10:test (default-test) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Surefire report directory: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/surefire-reports</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] -------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] T E S T S</code></div>
<div>
<code class="brush: core">[INFO] -------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Running com.app.AppTest</code></div>
<div>
<code class="brush: core">[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Results :</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-jar-plugin:2.3.2:jar (default-jar) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Building jar: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/utility-1.0.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] >>> maven-source-plugin:2.2.1:jar (attach-sources) @ utility >>></code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] <<< maven-source-plugin:2.2.1:jar (attach-sources) @ utility <<<</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-source-plugin:2.2.1:jar (attach-sources) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Building jar: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/utility-1.0-sources.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-javadoc-plugin:2.9:jar (attach-javadocs) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] Loading source files for package com.app...</code></div>
<div>
<code class="brush: core">[INFO] Constructing Javadoc information...</code></div>
<div>
<code class="brush: core">[INFO] Standard Doclet version 1.6.0_45</code></div>
<div>
<code class="brush: core">[INFO] Building tree for all the packages and classes...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/com/app//App.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/com/app//package-frame.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/com/app//package-summary.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/com/app//package-tree.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/constant-values.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/com/app//class-use/App.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/com/app//package-use.html...</code></div>
<div>
<code class="brush: core">[INFO] Building index for all the packages and classes...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/overview-tree.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/index-all.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/deprecated-list.html...</code></div>
<div>
<code class="brush: core">[INFO] Building index for all classes...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/allclasses-frame.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/allclasses-noframe.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/index.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/help-doc.html...</code></div>
<div>
<code class="brush: core">[INFO] Generating /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/apidocs/stylesheet.css...</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Building jar: /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/utility-1.0-javadoc.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Installing /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/utility-1.0.jar to /Users/akuntamukkala/.m2/repository/com/app/utility/1.0/utility-1.0.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Installing /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/pom.xml to /Users/akuntamukkala/.m2/repository/com/app/utility/1.0/utility-1.0.pom</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Installing /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/utility-1.0-sources.jar to /Users/akuntamukkala/.m2/repository/com/app/utility/1.0/utility-1.0-sources.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Installing /Users/akuntamukkala/GIT/mrp-git-cb-demo/utility/target/checkout/utility/target/utility-1.0-javadoc.jar to /Users/akuntamukkala/.m2/repository/com/app/utility/1.0/utility-1.0-javadoc.jar</code></div>
<div>
<code class="brush: core">[INFO] [INFO] </code></div>
<div>
<code class="brush: core">[INFO] [INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ utility ---</code></div>
<div>
<code class="brush: core">[INFO] WAGON_VERSION: 1.0-beta-2</code></div>
<div>
<code class="brush: core">[INFO] Uploading: dav:https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0.jar</code></div>
<div>
<code class="brush: core">[INFO] 3 KB </code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Uploaded: dav:https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0.jar (3 KB at 0.2 KB/sec)</code></div>
<div>
<code class="brush: core">[INFO] Uploading: https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0.pom</code></div>
<div>
<code class="brush: core">[INFO] 3 KB </code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Uploaded: https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0.pom (3 KB at 0.3 KB/sec)</code></div>
<div>
<code class="brush: core">[INFO] Downloading: https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/maven-metadata.xml</code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Uploading: https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/maven-metadata.xml</code></div>
<div>
<code class="brush: core">[INFO] 290 B </code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Uploaded: https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/maven-metadata.xml (290 B at 0.1 KB/sec)</code></div>
<div>
<code class="brush: core">[INFO] Uploading: dav:https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0-sources.jar</code></div>
<div>
<code class="brush: core">[INFO] 761 B </code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Uploaded: dav:https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0-sources.jar (761 B at 0.1 KB/sec)</code></div>
<div>
<code class="brush: core">[INFO] Uploading: dav:https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0-javadoc.jar</code></div>
<div>
<code class="brush: core">[INFO] 4 KB </code></div>
<div>
<code class="brush: core">[INFO] 8 KB </code></div>
<div>
<code class="brush: core">[INFO] 12 KB </code></div>
<div>
<code class="brush: core">[INFO] 16 KB </code></div>
<div>
<code class="brush: core">[INFO] 20 KB </code></div>
<div>
<code class="brush: core">[INFO] 21 KB </code></div>
<div>
<code class="brush: core">[INFO] </code></div>
<div>
<code class="brush: core">[INFO] Uploaded: dav:https://repository-akuntamukkala.forge.cloudbees.com/release/com/app/utility/1.0/utility-1.0-javadoc.jar (21 KB at 2.5 KB/sec)</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] BUILD SUCCESS</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Total time: 42.872s</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Finished at: Sun Jun 16 17:58:53 GMT+05:30 2013</code></div>
<div>
<code class="brush: core">[INFO] [INFO] Final Memory: 16M/81M</code></div>
<div>
<code class="brush: core">[INFO] [INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Cleaning up after release...</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] BUILD SUCCESS</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
<div>
<code class="brush: core">[INFO] Total time: 1:03.465s</code></div>
<div>
<code class="brush: core">[INFO] Finished at: Sun Jun 16 17:58:53 GMT+05:30 2013</code></div>
<div>
<code class="brush: core">[INFO] Final Memory: 5M/81M</code></div>
<div>
<code class="brush: core">[INFO] ------------------------------------------------------------------------</code></div>
</div>
<code class="brush: core">
</code>
</div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Viewing the remote repository using a WebDAV client:</b></span></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Download and configure CyberDuck WebDAV Client to browse the CloudBees release artifact repository. Use the same credentials which we used in settings.xml</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-R8QwExYJKKw/Ub2xwdG0MHI/AAAAAAAAADY/xa8z_XB0O5E/s1600/Screen+Shot+2013-06-16+at+6.04.37+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="540" src="http://4.bp.blogspot.com/-R8QwExYJKKw/Ub2xwdG0MHI/AAAAAAAAADY/xa8z_XB0O5E/s640/Screen+Shot+2013-06-16+at+6.04.37+PM.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following screenshot shows the contents on release artifact repository. </span></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/--PH0Air0QIk/Ub2x8aQJQfI/AAAAAAAAADg/uP0PPlwh40A/s1600/Screen+Shot+2013-06-16+at+6.05.25+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="460" src="http://4.bp.blogspot.com/--PH0Air0QIk/Ub2x8aQJQfI/AAAAAAAAADg/uP0PPlwh40A/s640/Screen+Shot+2013-06-16+at+6.05.25+PM.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Conclusion:</span></b></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">We have seen how to use maven release plugin's prepare and perform goals in a project using Git, GitHub and CloudBees JFrog Artifact Repository. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Maven release plugin is feature rich and there are so many options that I haven't discovered . For most part we have seen the plain vanilla use case here. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I'd love to hear about your use cases where you found maven release plugin useful and where you did not. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As always, any comments, suggestions to improve content are most welcome. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Thanks again for your interest! Long live open source!</span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Useful References:</span></b></div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ol style="text-align: left;">
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://maven.apache.org/maven-release/maven-release-plugin/index.html">http://maven.apache.org/maven-release/maven-release-plugin/index.html</a></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://www.insaneprogramming.be/?p=362">http://www.insaneprogramming.be/?p=362</a></span></li>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="https://github.com/arey/maven-config-github-cloudbees/blob/master/pom.xml">https://github.com/arey/maven-config-github-cloudbees/blob/master/pom.xml</a></span></li>
</ol>
<br /></div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-83422913700337945942013-06-13T20:38:00.000-07:002013-06-15T06:35:11.696-07:00Git flow in action<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css"></link>
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushVb.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js" type="text/javascript"></script>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js" type="text/javascript"></script>
<script language="javascript">
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = 'http://alexgorbatchev.com/pub/sh/current/scripts/clipboard.swf';
SyntaxHighlighter.all();
</script><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>
Objective:</b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This is a gentle introduction to "git flow" using a hands-on example.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Introduction:</b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In an typical Agile/SCRUM shop, there are several teams constituting of developers and QA personnel who collaborate across several sprint lanes. Their primary objective is to deliver functional user stories at the end of each sprint cycle. In such an environment of rapid iterative development, it is essential to have a very well defined strategy to branch, merge, tag, release, patch the source code repository. This can become problematic real fast especially when using DVCS such as GIT. Here is where "Git Flow" comes into play.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Git Flow is both a methodology and software (underneath the covers runs git commands). It takes the best enterprise practices we are familiar with in a centralized version control system and applies them to GIT.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here are 4 resources that I found really helpful while learning Git Flow.</span><br />
<ol style="text-align: left;">
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://www.youtube.com/watch?v=NdXhz4rt_sQ">2012 UTOSC - Using Git-Flow to Relieve Your Headaches - Tom Doggett</a></span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://nvie.com/posts/a-successful-git-branching-model/">Successful Git Branching Model - Vincent Driessen</a></span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://danielkummer.github.io/git-flow-cheatsheet/">Git flow Cheat Sheet - Daniel Kummer</a></span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://yakiloo.com/getting-started-git-flow/">Getting started with Git Flow</a></span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://scottchacon.com/2011/08/31/github-flow.html">Issues with Git Flow; Intro to GitHub Flow - Scott Chacon</a></span></li>
</ol>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Overview:</b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this post, I'd like to share how two popular fictitious developers (Bob and Alice) use Git Flow to develop software collaboratively. Both have their own git repository and collaborate via central repository hosted on Github.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">We will cover the following scenarios:</span><br />
<ol style="text-align: left;">
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://akuntamukkala.blogspot.in/2013/06/git-flow-in-action.html#IGF">Initializing Git Flow's recommended branching mechanism</a></span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Developing "features"</span></li>
<ul>
<li><a href="http://akuntamukkala.blogspot.in/2013/06/git-flow-in-action.html#WI" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Working in isolation (developer implements by self)</a></li>
<li><a href="http://akuntamukkala.blogspot.in/2013/06/git-flow-in-action.html#WC" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">Working collaboratively via common repository on GitHub</a></li>
</ul>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://akuntamukkala.blogspot.in/2013/06/git-flow-in-action.html#RM">Release Management</a></span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://akuntamukkala.blogspot.in/2013/06/git-flow-in-action.html#HF">Hot Fixes</a></span></li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-JhXOvavvqwo/UbxUzx_gXII/AAAAAAAAAAM/XPaKlcqXNb0/s1600/Screen-Shot-2013-04-26-at-12.20.17-AM-1024x659.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="408" src="http://1.bp.blogspot.com/-JhXOvavvqwo/UbxUzx_gXII/AAAAAAAAAAM/XPaKlcqXNb0/s640/Screen-Shot-2013-04-26-at-12.20.17-AM-1024x659.png" width="640" /></a></div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><br /></b></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Prerequisites:</b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><br /></b></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">If you are reading about git flow, it is fair to assume that you are aware of basics of git and have used GitHub. There are lot of blogs that go over git basics and how to install git and git flow. I'd suggest that you get a bit of hands-on experience with git and GitHub before reading further as it will help you follow along much easily.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this example, both Bob and Alice have read and write access to the same shared GitHub repository.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Deep Dive:</b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><a href="http://www.blogger.com/blogger.g?blogID=8558899295969164571" name="IGF">Initializing Git Flow's recommended branching mechanism</a></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob begins by creating a repo in Github called "gitflow-example"</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-KG7xjrkbq50/UbxW8QVav-I/AAAAAAAAAAc/qWOoGwJsJ1w/s1600/Screen-Shot-2013-04-24-at-1.03.44-AM1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="345" src="http://3.bp.blogspot.com/-KG7xjrkbq50/UbxW8QVav-I/AAAAAAAAAAc/qWOoGwJsJ1w/s640/Screen-Shot-2013-04-24-at-1.03.44-AM1.png" width="640" /></a></div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that Bob has an empty repo on Github, he creates a local Git repo using gitflow and sticks to the recommended branch naming convention suggested by Git Flow.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<span style="background-color: azure; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><code class="brush: core">
Ashwinis-MacBook-Pro:git akuntamukkala$ mkdir GitFlowExample<br />
Ashwinis-MacBook-Pro:git akuntamukkala$ cd GitFlowExample/<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git flow init</b></span><br />
Initialized empty Git repository in /Users/akuntamukkala/GIT/GitFlowExample/.git/<br />
No branches exist yet. Base branches must be created now.<br />
Branch name for production releases: [master]<br />
Branch name for "next release" development: [develop]<br />
<br />
How to name your supporting branch prefixes?<br />
Feature branches? [feature/]<br />
Release branches? [release/]<br />
Hotfix branches? [hotfix/]<br />
Support branches? [support/]<br />
Version tag prefix? []<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch</span></b><br />
* develop<br />
master<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ touch README.md<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ echo "this is to help understand use of git flow" > README.md<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git add .</b></span><br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git commit -m "first commit"</span></b><br />
[develop 4b88910] first commit<br />
1 file changed, 1 insertion(+)<br />
create mode 100644 README.md<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git remote add origin </span></b>git@github.com:akuntamukkala/gitflow-example.git<br />
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git push --all</span></b><br />
Counting objects: 5, done.<br />
Delta compression using up to 8 threads.<br />
Compressing objects: 100% (2/2), done.<br />
Writing objects: 100% (5/5), 430 bytes, done.<br />
Total 5 (delta 0), reused 0 (delta 0)<br />
To git@github.com:akuntamukkala/gitflow-example.git<br />
* [new branch] develop -> develop<br />
* [new branch] master -> master<br />
</code>
</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><code class="brush: core"><br /></code></span></div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Here is the screenshot of repository on Github showing both develop and master branches. Current active branch "develop"</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-1vpH2A3ikTU/UbxXWVVQfRI/AAAAAAAAAAk/c96vycNl3IE/s1600/Screen-Shot-2013-04-24-at-2.45.35-PM-1024x631.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="394" src="http://4.bp.blogspot.com/-1vpH2A3ikTU/UbxXWVVQfRI/AAAAAAAAAAk/c96vycNl3IE/s640/Screen-Shot-2013-04-24-at-2.45.35-PM-1024x631.png" width="640" /></a></div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice clones the gitflow-example.git repo from Github and initializes git flow on her local repository. She also sticks to the same branch naming convention recommended by Git Flow.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core"><br /> C:\GIT\GitFlowExample><span style="color: blue;"><b>git clone git@github.com:akuntamukkala/gitflow-example.git</b></span><br />Cloning into 'gitflow-example'...<br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />remote: Counting objects: 5, done.<br />remote: Compressing objects: 100% (2/2), done.<br />remote: Total 5 (delta 0), reused 5 (delta 0)<br />Receiving objects: 100% (5/5), done.<br /><br /> C:\GIT\GitFlowExample>cd gitflow-example<br /><br /> C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git branch -a</b></span><br />* master<br /> remotes/origin/HEAD -> origin/master<br /> remotes/origin/develop<br /> remotes/origin/master<br /><br /> C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git flow init</b></span><br /> Which branch should be used for bringing forth production releases?<br /> - master<br />Branch name for production releases: [master]<br />Branch name for "next release" development: [develop]<br /> How to name your supporting branch prefixes?<br />Feature branches? [feature/]<br />Release branches? [release/]<br />Hotfix branches? [hotfix/]<br />Support branches? [support/]<br />Version tag prefix? []<br /><br /> C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -a</span></b><br />* develop<br /> master<br /> remotes/origin/HEAD -> origin/master<br /> remotes/origin/develop<br /> remotes/origin/master<br /><br /> C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -vv</span></b><br />* develop 4b88910 [origin/develop] first commit<br />master 078f615 [origin/master] Initial commit</code><br />
<code class="brush: core"><br /> C:\GIT\GitFlowExample\gitflow-example>dir<br />Volume in drive C has no label.<br />Volume Serial Number is 48D7-B5DC<br /><br />Directory of C:\GIT\GitFlowExample\gitflow-example<br />04/24/2013 02:49 PM <DIR> .<br />04/24/2013 02:49 PM <DIR> ..<br />04/24/2013 02:49 PM 44 README.md<br />1 File(s) 44 bytes<br />2 Dir(s) 39,041,245,184 bytes free<br /><br />C:\GIT\GitFlowExample\gitflow-example>type README.md<br />this is to help understand use of git flow</code></div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><a href="http://www.blogger.com/blogger.g?blogID=8558899295969164571" name="WI">Developing "features" in isolation:</a></b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's say Alice is implementing a feature "xyz" by herself (not in collaboration)</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The following commands creates a branch "feature/feature-xyz" off the "develop" branch.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git flow feature start feature-xyz</span></b><br />
Switched to a new branch 'feature/feature-xyz'<br />
Summary of actions:<br />
- A new branch 'feature/feature-xyz' was created, based on 'develop'<br />
- You are now on branch 'feature/feature-xyz'<br />
<br />
Now, start committing on your feature. When done, use:<br />
git flow feature finish feature-xyz<br />
<br />
C:\GIT\GitFlowExample\gitflow-example>dir<br />
Volume in drive C has no label.<br />
Volume Serial Number is 48D7-B5DC
Directory of C:\GIT\GitFlowExample\gitflow-example<br />
04/24/2013 02:49 PM <DIR> .<br />
04/24/2013 02:49 PM <DIR> ..<br />
04/24/2013 02:49 PM 44 README.md<br />
1 File(s) 44 bytes<br />
2 Dir(s) 39,041,175,552 bytes free<br />
<br />
C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -vv</span></b><br />
develop 4b88910 [origin/develop] first commit<br />
* feature/feature-xyz 4b88910 first commit<br />
master 078f615 [origin/master] Initial commit
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice starts implementing the feature xyz...</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core"><br />C:\GIT\GitFlowExample\gitflow-example>echo "feature xyz implemented" > feature-xyz-impl.txt<br /><br />
C:\GIT\GitFlowExample\gitflow-example>echo "feature xyz adds cool capability" >> README.md<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git add .</span></b><br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git commit -m "implemented feature-xyz"</span></b><br />[feature/feature-xyz 02d1a06] implemented feature-xyz<br />2 files changed, 2 insertions(+)<br />create mode 100644 feature-xyz-impl.txt<br /><br />
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that Alice commits the newly implemented feature xyz to her local repository, it is time for her to merge her changes to the "develop" branch and delete "feature/feature-xyz" branch.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git flow feature finish feature-xyz</span></b><br />Switched to branch 'develop'<br />Updating 4b88910..02d1a06<br />Fast-forward<br />README.md | 1 +<br />feature-xyz-impl.txt | 1 +<br />2 files changed, 2 insertions(+)<br />create mode 100644 feature-xyz-impl.txt<br />Deleted branch feature/feature-xyz (was 02d1a06).<br /><br />
Summary of actions:<br />- The feature branch 'feature/feature-xyz' was merged into 'develop'<br />- Feature branch 'feature/feature-xyz' has been removed<br />- You are now on branch 'develop'<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git log --pretty=format:"%h %s" --graph</span></b> * 02d1a06 implemented feature-xyz<br />* 4b88910 first commit<br />* 078f615 Initial commit<br /><br />
</code>
</div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice pushes her changes from "develop" branch from local repository to the "develop" branch on central repository on GitHub.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br />
C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git push</b></span><br />
Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />
Counting objects: 6, done.<br />
Delta compression using up to 2 threads.<br />
Compressing objects: 100% (3/3), done.<br />
Writing objects: 100% (4/4), 414 bytes, done.<br />
Total 4 (delta 0), reused 0 (delta 0)<br />
To git@github.com:akuntamukkala/gitflow-example.git<br />
4b88910..02d1a06 develop -> develop<br />
<br />
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now Bob wants to get the newly added feature xyz</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$<span style="color: blue;"><b> git branch -vv</b></span><br />* develop 4b88910 first commit<br />master 078f615 Initial commit</code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git branch --set-upstream develop origin/develop</b></span></code><br />
<code class="brush: core">Branch develop set up to track remote branch develop from origin.</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch --set-upstream master origin/master</span></b></code><br />
<code class="brush: core">Branch master set up to track remote branch master from origin.</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b></code><br />
<code class="brush: core">* develop 4b88910 [origin/develop: behind 1] first commit</code><br />
<code class="brush: core">master 078f615 [origin/master] Initial commit</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git pull</span></b></code><br />
<code class="brush: core">Updating 4b88910..02d1a06</code><br />
<code class="brush: core">Fast-forward</code><br />
<code class="brush: core">README.md | 1 +</code><br />
<code class="brush: core">feature-xyz-impl.txt | 1 +</code><br />
<code class="brush: core">2 files changed, 2 insertions(+)</code><br />
<code class="brush: core">create mode 100644 feature-xyz-impl.txt</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git branch -vv</b></span></code><br />
<code class="brush: core">* develop 02d1a06 [origin/develop] implemented feature-xyz</code><br />
<code class="brush: core">master 078f615 [origin/master] Initial commit</code></div>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><a href="http://www.blogger.com/blogger.g?blogID=8558899295969164571" name="WC">Developing "features" in collaboration:</a></b></span></div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Say Bob and Alice now want to start developing a feature collaboratively.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob creates a new feature "abc"</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git flow feature start feature-abc</b></span><br />
Switched to a new branch 'feature/feature-abc'<br />
Summary of actions:<br />
</code><br />
<div>
<code class="brush: core">- A new branch 'feature/feature-abc' was created, based on 'develop'</code><br />
<code class="brush: core">- You are now on branch 'feature/feature-abc'</code><br />
<code class="brush: core">Now, start committing on your feature. When done, use:</code></div>
<code class="brush: core">
</code>
<br />
<div>
<code class="brush: core">git flow feature finish feature-abc</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git flow feature list</b></span></code><br />
<code class="brush: core">* feature-abc</code></div>
<code class="brush: core">
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob implements feature "abc", commits changes to local repo's feature/feature-abc branch.</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ touch feature-abc-impl.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ echo "dev 1 implementing feature abc partially" > feature-abc-impl.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git add .</b></span><br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git commit -m "dev 1 - partial impl of feature-abc"</b></span><br />[feature/feature-abc c50551c] dev 1 - partial impl of feature-abc<br />1 file changed, 1 insertion(+)<br />create mode 100644 feature-abc-impl.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git push</span></b><br />Everything up-to-date<br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob now publishes feature-abc to repository on GitHub.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git flow feature publish feature-abc</span></b><br />Counting objects: 4, done.<br />Delta compression using up to 8 threads.<br />Compressing objects: 100% (2/2), done.<br />Writing objects: 100% (3/3), 381 bytes, done.<br />Total 3 (delta 0), reused 0 (delta 0)<br />To git@github.com:akuntamukkala/gitflow-example.git<br />* [new branch] feature/feature-abc -> feature/feature-abc<br />Already on 'feature/feature-abc'<br />Summary of actions:<br />- A new remote branch 'feature/feature-abc' was created<br />- The local branch 'feature/feature-abc' was configured to track the remote branch<br />- You are now on branch 'feature/feature-abc'<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b><br />develop 02d1a06 [origin/develop] implemented feature-xyz<br />* feature/feature-abc c50551c [origin/feature/feature-abc] dev 1 - partial impl of feature-abc<br />master 078f615 [origin/master] Initial commit<br />
</code>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-jx94O8w5hTc/UbxYEWbedII/AAAAAAAAAAs/q-7TzgEwp8Y/s1600/Screen-Shot-2013-04-24-at-4.27.39-PM-1024x692.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="432" src="http://3.bp.blogspot.com/-jx94O8w5hTc/UbxYEWbedII/AAAAAAAAAAs/q-7TzgEwp8Y/s640/Screen-Shot-2013-04-24-at-4.27.39-PM-1024x692.png" width="640" /></a></div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice now attempts to pull the feature-abc published by Bob</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git flow feature pull origin feature-abc</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Created local branch feature/feature-abc based on origin's feature/feature-abc.<br /><br />C:\GIT\GitFlowExample\gitflow-example>git branch -a<br />develop<br />* feature/feature-abc<br />master<br />remotes/origin/HEAD -> origin/master<br />remotes/origin/develop<br />remotes/origin/feature/feature-abc<br />remotes/origin/master<br /><br />C:\GIT\GitFlowExample\gitflow-example>dir<br />Volume in drive C has no label.<br />Volume Serial Number is 48D7-B5DC<br />Directory of C:\GIT\GitFlowExample\gitflow-example<br />04/24/2013 04:25 PM <DIR> .<br />04/24/2013 04:25 PM <DIR> ..<br />04/24/2013 04:25 PM 42 feature-abc-impl.txt<br />04/24/2013 03:04 PM 28 feature-xyz-impl.txt<br />04/24/2013 03:04 PM 81 README.md<br />3 File(s) 151 bytes<br />2 Dir(s) 38,988,984,320 bytes free<br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice continues to implement feature abc.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example>type feature-abc-impl.txt<br />dev 1 implementing feature abc partially<br />
</code>
</div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice finishes the implementation of feature "abc" and commits her changes to her local repository.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example>printf "\ndev2 completed implementation of feature abc" >> feature-abc-impl.txt<br />
<br />
C:\GIT\GitFlowExample\gitflow-example>type feature-abc-impl.txt<br />
dev 1 implementing feature abc partially<br />
dev2 completed implementation of feature abc<br />
<br />
C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git add .</span></b><br />
warning: LF will be replaced by CRLF in feature-abc-impl.txt.<br />
The file will have its original line endings in your working directory.<br />
<br />
C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git commit -m "feature abc implementation completed by dev 2"</span></b><br />
[feature/feature-abc warning: LF will be replaced by CRLF in feature-abc-impl.txt.<br />
The file will have its original line endings in your working directory.<br />
5671d9f] feature abc implementation completed by dev 2<br />
warning: LF will be replaced by CRLF in feature-abc-impl.txt.<br />
The file will have its original line endings in your working directory.<br />
1 file changed, 3 insertions(+), 1 deletion(-)<br />
<br />
C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git flow feature list</span></b><br />
* feature-abc<br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that this new feature abc is completely implemented, Alice figures it is time to merge changes from "feature/feature-abc" branch to the "develop" branch of her local repository.</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git flow feature finish feature-abc</span></b><br />Branches 'feature/feature-abc' and 'origin/feature/feature-abc' have diverged.<br />And local branch 'feature/feature-abc' is ahead of 'origin/feature/feature-abc'.<br />Switched to branch 'develop'<br />Merge made by the 'recursive' strategy.<br />feature-abc-impl.txt | 3 +++<br />1 file changed, 3 insertions(+)<br />create mode 100644 feature-abc-impl.txt<br />Deleted branch feature/feature-abc (was 5671d9f).<br /><br />Summary of actions:<br />- The feature branch 'feature/feature-abc' was merged into 'develop'<br />- Feature branch 'feature/feature-abc' has been removed<br />- You are now on branch 'develop'<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -r</span></b><br />origin/HEAD -> origin/master<br />origin/develop<br />origin/feature/feature-abc<br />origin/master<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git branch -a</b></span><br />* develop<br />master<br />remotes/origin/HEAD -> origin/master<br />remotes/origin/develop<br />remotes/origin/feature/feature-abc<br />remotes/origin/master<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git status</span></b><br /># On branch develop<br /># Your branch is ahead of 'origin/develop' by 3 commits.<br />#<br />nothing to commit, working directory clean<br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice pushes changes from her local repository containing completed feature-abc implementation to the central repository on GitHub.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git push</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Counting objects: 7, done.<br />Delta compression using up to 2 threads.<br />Compressing objects: 100% (4/4), done.<br />Writing objects: 100% (4/4), 573 bytes, done.<br />Total 4 (delta 1), reused 0 (delta 0)<br />To git@github.com:akuntamukkala/gitflow-example.git<br />02d1a06..fdef09a develop -> develop<br />
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-H3vxL4ljNJg/UbxYUPQsWlI/AAAAAAAAAA0/o4dXihArUzo/s1600/Screen-Shot-2013-04-24-at-4.53.30-PM-1024x707.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="440" src="http://1.bp.blogspot.com/-H3vxL4ljNJg/UbxYUPQsWlI/AAAAAAAAAA0/o4dXihArUzo/s640/Screen-Shot-2013-04-24-at-4.53.30-PM-1024x707.png" width="640" /></a></div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">It may be noted that "feature/feature-abc" branch has been deleted in Alice's local repository but still exists in Github repository. Alice deletes feature-abc branch from Github repo.</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -a</span></b><br />* develop<br />master<br />remotes/origin/HEAD -> origin/master<br />remotes/origin/develop<br />remotes/origin/feature/feature-abc<br />remotes/origin/master<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git fetch origin feature/feature-abc</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />From github.com:akuntamukkala/gitflow-example<br />* branch feature/feature-abc -> FETCH_HEAD<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -vv</span></b><br />* develop fdef09a [origin/develop] Merge branch 'feature/feature-abc' into develop<br />master 078f615 [origin/master] Initial commit<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git push origin :feature/feature-abc</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />To git@github.com:akuntamukkala/gitflow-example.git<br />- [deleted] feature/feature-abc<br />
</code>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-VpZOvBwH0hE/UbxbP4C7hWI/AAAAAAAAABE/e2anZUHVpjY/s1600/Screen-Shot-2013-04-24-at-5.16.44-PM-1024x480.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="http://4.bp.blogspot.com/-VpZOvBwH0hE/UbxbP4C7hWI/AAAAAAAAABE/e2anZUHVpjY/s640/Screen-Shot-2013-04-24-at-5.16.44-PM-1024x480.png" width="640" /></a></div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob updates local repo to get completely implemented feature "abc"</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -a</span></b><br />develop<br />* feature/feature-abc<br />master<br />remotes/origin/develop<br />remotes/origin/feature/feature-abc<br />remotes/origin/master</code><br />
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git remote prune origin</span></b><br />Pruning origin<br />URL: git@github.com:akuntamukkala/gitflow-example.git<br />* [pruned] origin/feature/feature-abc</code><br />
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b><br />develop 02d1a06 [origin/develop: behind 3] implemented feature-xyz<br />* feature/feature-abc c50551c [origin/feature/feature-abc] dev 1 - partial impl of feature-abc<br />master 078f615 [origin/master] Initial commit</code><br />
<div>
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git checkout develop</b></span></code><br />
<code class="brush: core">Switched to branch 'develop'</code><br />
<code class="brush: core">Your branch is behind 'origin/develop' by 3 commits, and can be fast-forwarded.</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git pull</span></b></code><br />
<code class="brush: core">Updating 02d1a06..fdef09a</code><br />
<code class="brush: core">Fast-forward</code><br />
<code class="brush: core">feature-abc-impl.txt | 3 +++</code><br />
<code class="brush: core">1 file changed, 3 insertions(+)</code><br />
<code class="brush: core">create mode 100644 feature-abc-impl.txt</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch</span></b></code><br />
<code class="brush: core">* develop</code><br />
<code class="brush: core">feature/feature-abc</code><br />
<code class="brush: core">master</code></div>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob needs to delete feature/feature-abc branch from local repo because his local repository's "develop" branch already contains the merged feature abc implementation.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">"git remote prune origin" command helps identify that remote branch (feature/feature-abc) that Bob was tracking has been deleted.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -d </span></b>feature/feature-abc<br />Deleted branch feature/feature-abc (was c50551c).</code><br />
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b><br />* develop fdef09a [origin/develop] Merge branch 'feature/feature-abc' into develop<br />master 078f615 [origin/master] Initial commit<br />
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><br /></b></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b></b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b></b></span><br />
<a name='more'></a><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><br /></b></span>
<a href="http://www.blogger.com/blogger.g?blogID=8558899295969164571" name="RM">
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Release Management:</b></span>
</a>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now let's talk about making a release as both features "xyz" and "abc" have been implemented and tested.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob is ready to cut a release, say release-1.0 :)</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git flow release start release-1.0</b></span><br />Switched to a new branch 'release/release-1.0'<br /><br />Summary of actions:<br />- A new branch 'release/release-1.0' was created, based on 'develop'<br />- You are now on branch 'release/release-1.0'<br /><br />Follow-up actions:<br />- Bump the version number now!<br />- Start committing last-minute fixes in preparing your release<br />- When done, run:</code><br />
<div>
<code class="brush: core">git flow release finish 'release-1.0'</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b></code><br />
<code class="brush: core">develop fdef09a [origin/develop] Merge branch 'feature/feature-abc' into develop</code><br />
<code class="brush: core">master 078f615 [origin/master] Initial commit</code><br />
<code class="brush: core">* release/release-1.0 fdef09a Merge branch 'feature/feature-abc' into develop</code></div>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob publishes the release-1.0 to remote repo so Alice may start tracking the release-1.0 branch</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br /> Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git flow release publish release-1.0</span></b><br />Total 0 (delta 0), reused 0 (delta 0)<br />To git@github.com:akuntamukkala/gitflow-example.git<br />* [new branch] release/release-1.0 -> release/release-1.0<br />Already on 'release/release-1.0'<br />Summary of actions:<br />- A new remote branch 'release/release-1.0' was created<br />- The local branch 'release/release-1.0' was configured to track the remote branch<br />- You are now on branch 'release/release-1.0'<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b><br />develop fdef09a [origin/develop] Merge branch 'feature/feature-abc' into develop<br />master 078f615 [origin/master] Initial commit<br />* release/release-1.0 fdef09a [origin/release/release-1.0] Merge branch 'feature/feature-abc' into develop<br /><br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice fetches the latest from remote repo and begins tracking release-1.0 branch</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git fetch</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />From github.com:akuntamukkala/gitflow-example<br />* [new branch] release/release-1.0 -> origin/release/release-1.0<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git branch -a</b></span><br />* develop<br />master<br />remotes/origin/HEAD -> origin/master<br />remotes/origin/develop<br />remotes/origin/master<br />remotes/origin/release/release-1.0<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -vv</span></b><br />* develop fdef09a [origin/develop] Merge branch 'feature/feature-abc' into develop<br />master 078f615 [origin/master] Initial commit<br /><br /> C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git flow release track release-1.0</b></span><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Branch release/release-1.0 set up to track remote branch release/release-1.0 from origin.<br />Switched to a new branch 'release/release-1.0'<br /><br />Summary of actions:<br />- A new remote tracking branch 'release/release-1.0' was created<br />- You are now on branch 'release/release-1.0'<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git branch -vv</span></b><br />develop fdef09a [origin/develop] Merge branch 'feature/feature-abc' into develop<br />master 078f615 [origin/master] Initial commit<br />* release/release-1.0 fdef09a [origin/release/release-1.0] Merge branch 'feature/feature-abc' into develop<br />
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob creates release notes for release-1.0 claiming the valuable addition of features abc and xyz</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git status -s</span></b><br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ ls<br />README.md feature-abc-impl.txt feature-xyz-impl.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ touch release-notes.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ echo this new release contains two new features abc and xyz > release-notes.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git status -s</span></b><br />?? release-notes.txt<br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git add .</b></span></code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git commit -m "release notes for release-1.0"</span></b></code><br />
<code class="brush: core">[release/release-1.0 15af591] release notes for release-1.0</code><br />
<code class="brush: core">1 file changed, 1 insertion(+)</code><br />
<code class="brush: core">create mode 100644 release-notes.txt</code><br />
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git push</b></span></code><br />
<code class="brush: core">Counting objects: 4, done.</code><br />
<code class="brush: core">Delta compression using up to 8 threads.</code><br />
<code class="brush: core">Compressing objects: 100% (3/3), done.</code><br />
<code class="brush: core">Writing objects: 100% (3/3), 347 bytes, done.</code><br />
<code class="brush: core">Total 3 (delta 1), reused 0 (delta 0)</code><br />
<code class="brush: core">To git@github.com:akuntamukkala/gitflow-example.git</code><br />
<code class="brush: core">fdef09a..15af591 release/release-1.0 -> release/release-1.0</code></div>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Now that the release has been deemed successful, Bob proceeds to finish release process.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$<span style="color: blue;"><b> git flow release finish release-1.0</b></span><br />Switched to branch 'master'<br />Merge made by the 'recursive' strategy.<br />README.md | 2 ++<br />feature-abc-impl.txt | 3 +++<br />feature-xyz-impl.txt | 1 +<br />release-notes.txt | 1 +<br />4 files changed, 7 insertions(+)<br />create mode 100644 README.md<br />create mode 100644 feature-abc-impl.txt<br />create mode 100644 feature-xyz-impl.txt<br />create mode 100644 release-notes.txt<br />Switched to branch 'develop'<br />Merge made by the 'recursive' strategy.<br />release-notes.txt | 1 +<br />1 file changed, 1 insertion(+)<br />create mode 100644 release-notes.txt<br />Deleted branch release/release-1.0 (was 15af591).<br /><br />Summary of actions:<br />- Latest objects have been fetched from 'origin'<br />- Release branch has been merged into 'master'<br />- The release was tagged 'release-1.0'<br />- Release branch has been back-merged into 'develop'<br />- Release branch 'release/release-1.0' has been deleted<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$<b><span style="color: blue;"> git log --oneline --graph</span></b><br />* 8e36a5e Merge branch 'release/release-1.0' into develop<br />|\<br />| * 15af591 release notes for release-1.0<br />|/<br />* fdef09a Merge branch 'feature/feature-abc' into develop<br />|\<br />| * 5671d9f feature abc implementation completed by dev 2<br />| * c50551c dev 1 - partial impl of feature-abc<br />|/<br />* 02d1a06 implemented feature-xyz<br />* 4b88910 first commit<br />* 078f615 Initial commit<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b><br />* develop 8e36a5e [origin/develop: ahead 2] Merge branch 'release/release-1.0' into develop<br />master cde8e44 [origin/master: ahead 7] Merge branch 'release/release-1.0'<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -a</span></b><br />* develop<br />master<br />remotes/origin/develop<br />remotes/origin/master<br />remotes/origin/release/release-1.0<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git push origin :release/release-1.0</span></b><br />To git@github.com:akuntamukkala/gitflow-example.git<br />- [deleted] release/release-1.0<br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git status -s</span></b><br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git checkout master</span></b><br />Switched to branch 'master'<br />Your branch is ahead of 'origin/master' by 7 commits.<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$<b><span style="color: blue;"> git push</span></b><br />Counting objects: 7, done.<br />Delta compression using up to 8 threads.<br />Compressing objects: 100% (5/5), done.<br />Writing objects: 100% (5/5), 529 bytes, done.<br />Total 5 (delta 3), reused 0 (delta 0)<br />To git@github.com:akuntamukkala/gitflow-example.git<br />fdef09a..8e36a5e develop -> develop<br />078f615..cde8e44 master -> master<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git log --oneline --graph</span></b><br />* cde8e44 Merge branch 'release/release-1.0'<br />|\<br />| * 15af591 release notes for release-1.0<br />| * fdef09a Merge branch 'feature/feature-abc' into develop<br />| |\<br />| | * 5671d9f feature abc implementation completed by dev 2<br />| | * c50551c dev 1 - partial impl of feature-abc<br />| |/<br />| * 02d1a06 implemented feature-xyz<br />| * 4b88910 first commit<br />|/<br />* 078f615 Initial commit<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git checkout develop</span></b><br />Switched to branch 'develop'<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git status</span></b><br /># On branch develop<br />nothing to commit (working directory clean)<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b><br />* develop 8e36a5e [origin/develop] Merge branch 'release/release-1.0' into develop<br />master cde8e44 [origin/master] Merge branch 'release/release-1.0'<br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice updates local repo</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git remote prune origin</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Pruning origin<br />URL: git@github.com:akuntamukkala/gitflow-example.git<br />* [pruned] origin/release/release-1.0<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git checkout master</b></span><br />Switched to branch 'master'<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git branch -a</b></span><br />develop<br />* master<br />release/release-1.0<br />remotes/origin/HEAD -> origin/master<br />remotes/origin/develop<br />remotes/origin/master<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git pull</b></span><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />remote: Counting objects: 2, done.<br />remote: Total 2 (delta 1), reused 2 (delta 1)<br />Unpacking objects: 100% (2/2), done.<br />From github.com:akuntamukkala/gitflow-example<br />078f615..cde8e44 master -> origin/master<br />fdef09a..8e36a5e develop -> origin/develop<br />Updating 078f615..cde8e44<br />Fast-forward<br />README.md | 2 ++<br />feature-abc-impl.txt | 3 +++<br />feature-xyz-impl.txt | 1 +<br />release-notes.txt | 1 +<br />4 files changed, 7 insertions(+)<br />create mode 100644 README.md<br />create mode 100644 feature-abc-impl.txt<br />create mode 100644 feature-xyz-impl.txt<br />create mode 100644 release-notes.txt<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git checkout develop</b></span><br />Switched to branch 'develop'<br />Your branch is behind 'origin/develop' by 2 commits, and can be fast-forwarded.<br /><br />C:\GIT\GitFlowExample\gitflow-example>git pull<br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Updating fdef09a..8e36a5e<br />Fast-forward<br />release-notes.txt | 1 +<br />1 file changed, 1 insertion(+)<br />create mode 100644 release-notes.txt<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git branch -d release/release-1.0</b></span><br />Deleted branch release/release-1.0 (was 15af591).<br />
</code>
</div>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's look at repo history on Bob's local repo</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">gitk --all</span></b><br />
</code>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-Zv2P_n0PHQQ/UbxbshF8VWI/AAAAAAAAABQ/gIrRccIKA5Q/s1600/Screen-Shot-2013-04-24-at-11.51.06-PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="169" src="http://2.bp.blogspot.com/-Zv2P_n0PHQQ/UbxbshF8VWI/AAAAAAAAABQ/gIrRccIKA5Q/s640/Screen-Shot-2013-04-24-at-11.51.06-PM.png" width="640" /></a></div>
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core"> Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$<b><span style="color: blue;"> git branch -a</span></b><br />* develop<br />master<br />remotes/origin/develop<br />remotes/origin/master<br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git tag</b></span><br />release-1.0<br />
</code>
</div>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><a href="http://www.blogger.com/blogger.g?blogID=8558899295969164571" name="HF">Hot Fix:</a></b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Let's say an urgent production issue has been discovered for which a fix needs to be made.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Bob makes a hotfix branch against master (which has the same code merged from release-1.0 branch)</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core"><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git flow hotfix start hotfix-1-release-1.0</b></span><br />Switched to a new branch 'hotfix/hotfix-1-release-1.0'<br /><br />Summary of actions:<br />- A new branch 'hotfix/hotfix-1-release-1.0' was created, based on 'master'<br />- You are now on branch 'hotfix/hotfix-1-release-1.0'<br /><br />Follow-up actions:<br />- Bump the version number now!<br />- Start committing your hot fixes<br />- When done, run:<br /><br />git flow hotfix finish 'hotfix-1-release-1.0'<br /><br />
</code>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-RV225oisVxs/UbxetcDv7mI/AAAAAAAAABc/1xTcwLhR350/s1600/Screen-Shot-2013-04-25-at-12.19.12-AM-1024x206.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="128" src="http://2.bp.blogspot.com/-RV225oisVxs/UbxetcDv7mI/AAAAAAAAABc/1xTcwLhR350/s640/Screen-Shot-2013-04-25-at-12.19.12-AM-1024x206.png" width="640" /></a></div>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git branch -vv</b></span><br />develop 8e36a5e [origin/develop] Merge branch 'release/release-1.0' into develop<br />* hotfix/hotfix-1-release-1.0 cde8e44 Merge branch 'release/release-1.0'<br />master cde8e44 [origin/master] Merge branch 'release/release-1.0'<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ ls<br />README.md feature-abc-impl.txt feature-xyz-impl.txt release-notes.txt<br /><br />Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ printf "\ncritical memory leak fix made to feature abc\n" >> feature-abc-impl.txt</code><br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ printf "\nhotfix #1 - memory leak issue fixed\n" >> release-notes.txt</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git add .</span></b></code><br />
<code class="brush: core"><br /></code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <span style="color: blue;"><b>git commit -m "critical hotfix#1 made"</b></span></code><br />
<code class="brush: core">[hotfix/hotfix-1-release-1.0 3d8b4ec] critical hotfix#1 made</code><br />
<code class="brush: core">2 files changed, 4 insertions(+), 1 deletion(-)</code><br />
<code class="brush: core"><br /></code>
<br />
<div>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">gitk --all</span></b></code></div>
</div>
</div>
</div>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-wBJEQben6ps/UbxgfumIogI/AAAAAAAAABs/vsMDcbNoRfM/s1600/Screen-Shot-2013-04-25-at-12.26.12-AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="170" src="http://4.bp.blogspot.com/-wBJEQben6ps/UbxgfumIogI/AAAAAAAAABs/vsMDcbNoRfM/s640/Screen-Shot-2013-04-25-at-12.26.12-AM.png" width="640" /></a></div>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<br />
<code class="brush: core"><code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git flow hotfix finish hotfix-1-release-1.0</span></b></code><br />
<code class="brush: core">Switched to branch 'master'</code><br />
<code class="brush: core">Merge made by the 'recursive' strategy.</code><br />
<code class="brush: core">feature-abc-impl.txt | 3 ++-</code><br />
<code class="brush: core">release-notes.txt | 2 ++</code><br />
<code class="brush: core">2 files changed, 4 insertions(+), 1 deletion(-)</code><br />
<code class="brush: core">Switched to branch 'develop'</code><br />
<code class="brush: core">Merge made by the 'recursive' strategy.</code><br />
<code class="brush: core">feature-abc-impl.txt | 3 ++-</code><br />
<code class="brush: core">release-notes.txt | 2 ++</code><br />
<code class="brush: core">2 files changed, 4 insertions(+), 1 deletion(-)</code><br />
<code class="brush: core">Deleted branch hotfix/hotfix-1-release-1.0 (was 3d8b4ec).</code><br />
<code class="brush: core"><br /></code>
<code class="brush: core">Summary of actions:</code><br />
<code class="brush: core">- Latest objects have been fetched from 'origin'</code><br />
<code class="brush: core">- Hotfix branch has been merged into 'master'</code><br />
<code class="brush: core">- The hotfix was tagged 'hotfix-1-release-1.0'</code><br />
<code class="brush: core">- Hotfix branch has been back-merged into 'develop'</code><br />
<code class="brush: core">- Hotfix branch 'hotfix/hotfix-1-release-1.0' has been deleted</code><br />
<code class="brush: core"><br /></code>
<br />
</code><br />
<div>
<code class="brush: core"><code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">gitk --all</span></b></code></code></div>
</div>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-DYc7LhtLEDs/UbxhFUQ-2aI/AAAAAAAAAB0/X6PgS7pYSKI/s1600/Screen-Shot-2013-04-25-at-12.33.02-AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="218" src="http://3.bp.blogspot.com/-DYc7LhtLEDs/UbxhFUQ-2aI/AAAAAAAAAB0/X6PgS7pYSKI/s640/Screen-Shot-2013-04-25-at-12.33.02-AM.png" width="640" /></a></div>
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">
<code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git branch -vv</span></b></code></code><br />
<code class="brush: core"><code class="brush: core">* develop 5a7edd4 [origin/develop: ahead 3] Memory leak issue Merge branch 'hotfix/hotfix-1-release-1.0' into develop</code></code><br />
<code class="brush: core"><code class="brush: core">master 4a9a0e3 [origin/master: ahead 2] Merge branch 'hotfix/hotfix-1-release-1.0'</code></code><br />
<code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git tag -n</span></b></code></code><br />
<code class="brush: core"><code class="brush: core">hotfix-1-release-1.0 fixed memory leak issue</code></code><br />
<code class="brush: core"><code class="brush: core">release-1.0 tag for release-1.0 : introducing features abc and xyz</code></code><br />
<code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git fetch origin develop</span></b></code></code><br />
<code class="brush: core"><code class="brush: core">From github.com:akuntamukkala/gitflow-example</code></code><br />
<code class="brush: core"><code class="brush: core">* branch develop -> FETCH_HEAD</code></code><br />
<code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git fetch origin master</span></b></code></code><br />
<code class="brush: core"><code class="brush: core">From github.com:akuntamukkala/gitflow-example</code></code><br />
<code class="brush: core"><code class="brush: core">* branch master -> FETCH_HEAD</code></code><br />
<code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git diff origin/develop develop</span></b></code></code><br />
<code class="brush: core"><code class="brush: core">diff --git a/feature-abc-impl.txt b/feature-abc-impl.txt</code></code><br />
<code class="brush: core"><code class="brush: core">index c1421c6..0cdb22a 100644</code></code><br />
<code class="brush: core"><code class="brush: core">--- a/feature-abc-impl.txt</code></code><br />
<code class="brush: core"><code class="brush: core">+++ b/feature-abc-impl.txt</code></code><br />
<code class="brush: core"><code class="brush: core">@@ -1,3 +1,4 @@</code></code><br />
<code class="brush: core"><code class="brush: core">dev 1 implementing feature abc partially</code></code><br />
</code><br />
<div>
<code class="brush: core"><code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">-dev2 completed implementation of feature abc</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">\ No newline at end of file</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+dev2 completed implementation of feature abc</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+critical memory leak fix made to feature abc</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">diff --git a/release-notes.txt b/release-notes.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">index 0a0b518..6f3c3ff 100644</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">--- a/release-notes.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+++ b/release-notes.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">@@ -1 +1,3 @@</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">this new release contains two new features abc and xyz</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+</code></code></code><br />
<div>
<code class="brush: core"><code class="brush: core"><code class="brush: core">+hotfix #1 - memory leak issue fixed</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core"><br /></code>
</code></code><br />
<div>
<code class="brush: core"><code class="brush: core"><code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git checkout master</span></b></code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Switched to branch 'master'</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Your branch is ahead of 'origin/master' by 2 commits.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git diff origin/master master</span></b></code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">diff --git a/feature-abc-impl.txt b/feature-abc-impl.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">index c1421c6..0cdb22a 100644</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">--- a/feature-abc-impl.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+++ b/feature-abc-impl.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">@@ -1,3 +1,4 @@</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">dev 1 implementing feature abc partially</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">-dev2 completed implementation of feature abc</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">\ No newline at end of file</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+dev2 completed implementation of feature abc</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+critical memory leak fix made to feature abc</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">diff --git a/release-notes.txt b/release-notes.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">index 0a0b518..6f3c3ff 100644</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">--- a/release-notes.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+++ b/release-notes.txt</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">@@ -1 +1,3 @@</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">this new release contains two new features abc and xyz</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">+hotfix #1 - memory leak issue fixed</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core"><br /></code>
<code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git push origin --all</span></b></code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Counting objects: 9, done.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Delta compression using up to 8 threads.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Compressing objects: 100% (6/6), done.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Writing objects: 100% (6/6), 674 bytes, done.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Total 6 (delta 3), reused 0 (delta 0)</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">To git@github.com:akuntamukkala/gitflow-example.git</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">8e36a5e..5a7edd4 develop -> develop</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">cde8e44..4a9a0e3 master -> master</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Ashwinis-MacBook-Pro:GitFlowExample akuntamukkala$ <b><span style="color: blue;">git push origin --tags</span></b></code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Counting objects: 2, done.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Delta compression using up to 8 threads.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Compressing objects: 100% (2/2), done.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Writing objects: 100% (2/2), 350 bytes, done.</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">Total 2 (delta 0), reused 0 (delta 0)</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">To git@github.com:akuntamukkala/gitflow-example.git</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">* [new tag] hotfix-1-release-1.0 -> hotfix-1-release-1.0</code></code></code><br />
<code class="brush: core"><code class="brush: core"><code class="brush: core">* [new tag] release-1.0 -> release-1.0</code></code></code></div>
</div>
</div>
<code class="brush: core">
</code></div>
<code class="brush: core">
</code>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice updates her local repo from remote repo on Github which contains the hotfix #1 merged into "develop" and "master" branches along with "release-1.0" and "hotfix-1-release-1.0" tags</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core">C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git remote update</b></span><br />Fetching origin<br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />remote: Counting objects: 11, done.<br />remote: Compressing objects: 100% (5/5), done.<br />Unpacking objects: 100% (8/8), done.<br />remote: Total 8 (delta 3), reused 8 (delta 3)<br />From github.com:akuntamukkala/gitflow-example<br />8e36a5e..5a7edd4 develop -> origin/develop<br />cde8e44..4a9a0e3 master -> origin/master<br />* [new tag] release-1.0 -> release-1.0<br />From github.com:akuntamukkala/gitflow-example<br />* [new tag] hotfix-1-release-1.0 -> hotfix-1-release-1.0<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git diff develop origin/develop</span></b><br />diff --git a/feature-abc-impl.txt b/feature-abc-impl.txt<br />index c1421c6..0cdb22a 100644<br />--- a/feature-abc-impl.txt<br />+++ b/feature-abc-impl.txt<br />@@ -1,3 +1,4 @@<br />dev 1 implementing feature abc partially<br /><br />-dev2 completed implementation of feature abc<br />\ No newline at end of file<br />+dev2 completed implementation of feature abc<br />+critical memory leak fix made to feature abc<br />diff --git a/release-notes.txt b/release-notes.txt<br />index 0a0b518..6f3c3ff 100644<br />--- a/release-notes.txt<br />+++ b/release-notes.txt<br />@@ -1 +1,3 @@<br />this new release contains two new features abc and xyz<br />+<br />+hotfix #1 - memory leak issue fixed<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git diff master origin/master</b></span><br />diff --git a/feature-abc-impl.txt b/feature-abc-impl.txt<br />index c1421c6..0cdb22a 100644<br />--- a/feature-abc-impl.txt<br />+++ b/feature-abc-impl.txt<br />@@ -1,3 +1,4 @@<br />dev 1 implementing feature abc partially<br /><br />-dev2 completed implementation of feature abc<br />\ No newline at end of file<br />+dev2 completed implementation of feature abc<br />+critical memory leak fix made to feature abc<br />diff --git a/release-notes.txt b/release-notes.txt<br />index 0a0b518..6f3c3ff 100644<br />--- a/release-notes.txt<br />+++ b/release-notes.txt<br />@@ -1 +1,3 @@<br />this new release contains two new features abc and xyz<br />+<br />+hotfix #1 - memory leak issue fixed<br /><br />C:\GIT\GitFlowExample\gitflow-example><span style="color: blue;"><b>git tag -n</b></span><br />hotfix-1-release-1.0 fixed memory leak issue<br />release-1.0 tag for release-1.0 : introducing features abc and xyz<br />
</code>
</div>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Alice pulls in the changes made to origin/master and origin/develop into her local repository</span><br />
<br />
<div style="background-color: azure; border: 1px solid black; padding: 10px;">
<code class="brush: core"><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git checkout master</span></b><br />Switched to branch 'master'<br />Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git pull</span></b><br /><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Updating cde8e44..4a9a0e3<br />Fast-forward<br />feature-abc-impl.txt | 3 ++-<br />release-notes.txt | 2 ++<br />2 files changed, 4 insertions(+), 1 deletion(-)<br /><br /><br /> C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git checkout develop</span></b><br />Switched to branch 'develop'<br />Your branch is behind 'origin/develop' by 3 commits, and can be fast-forwarded.<br /><br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">git pull</span></b><br />Enter passphrase for key '/c/Users/akuntamu/.ssh/id_rsa':<br />Updating 8e36a5e..5a7edd4<br />Fast-forward<br />feature-abc-impl.txt | 3 ++-<br />release-notes.txt | 2 ++<br />2 files changed, 4 insertions(+), 1 deletion(-)<br />C:\GIT\GitFlowExample\gitflow-example><b><span style="color: blue;">gitk --all</span></b><br />
</code>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-I3nrIXBu44M/UbxhTQDP5lI/AAAAAAAAAB8/K8F6Im8b0Ng/s1600/Screen-Shot-2013-04-26-at-10.32.05-AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="232" src="http://2.bp.blogspot.com/-I3nrIXBu44M/UbxhTQDP5lI/AAAAAAAAAB8/K8F6Im8b0Ng/s640/Screen-Shot-2013-04-26-at-10.32.05-AM.png" width="640" /></a></div>
<br />
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Conclusion:</b></span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this blog we have explored a quick hands on introduction to git flow in order to do the following:</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<ol style="text-align: left;">
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Initializing Git Flow's recommended branching mechanism</span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Developing "features"</span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Working In isolation (developer implements by self)</span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Working collaboratively via common repository on GitHub</span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Release Management</span></li>
<li><span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">HotFixes</span></li>
</ol>
<br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I hope that you found this helpful. I am constantly looking for ways to improve my proficiency in git and git flow. If you find something that can be improved, please don't hesitate to drop in a comment.</span><br />
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Thanks again for reading and making it this far ;)</span>
</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0tag:blogger.com,1999:blog-8558899295969164571.post-31765429828748903262013-06-13T18:49:00.000-07:002013-06-13T19:27:22.268-07:00Moving from Blog.com to Blogspot.comI had started my blog at http://technologyenthusiast.blog.com a couple of months ago but given blog.com's challenges with uptime where blog.com is more down than up, I decided to abandon blog.com in favor of blogspot.com.<br />
I will continue my open source technology ramblings here on blogspot.com going forward...Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com1tag:blogger.com,1999:blog-8558899295969164571.post-26763540632171040932013-06-13T18:38:00.002-07:002013-06-13T18:45:04.851-07:00Purpose of this blog<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
This is a blog where I will be sharing some of my hands-on experiences with different open source projects.</div>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
I have keen interest in the following:</div>
<ul style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; list-style: square; margin: 0px 0px 1.625em 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Spring Framework</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">ORM technologies</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Distributed Version Control Systems</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">GIT/GitHub</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Middleware Integration</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">WSO2 product suite</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Apache Camel Integration framework</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Apache ActiveMQ</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">SOAP/REST Web Services</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Spring WS</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Axis</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">CXF</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">OSGI</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Apache Karaf</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Virgo</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Scripting languages</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Test Driven Development</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">JUnit</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">TestNG</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">DBUnit</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">XmlUnit</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Mocking Technologies</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style-image: initial; list-style-position: initial; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Easymock</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Powermock</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Mockito</li>
</ul>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Build Management</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Maven</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Continuous Integration</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Jenkins on Cloudbees</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Code Quality</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Sonar</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">FindBugs</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Artifact Repository</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Artifactory</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Archiva</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Nexus</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Big Data</li>
<ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">DataStax/Cassandra</li>
</ul>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">AWS/Rackspace/Cloudfoundry</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Troubleshooting</li>
<ul style="border: 0px; font-family: inherit; font-style: inherit; list-style: square; margin: 0px 0px 0px 2.5em; outline: 0px; padding: 0px; vertical-align: baseline;">
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">SAP Memory Analyzer for JVM heap dump (surely not fun at all)</li>
<li style="border: 0px; font-family: inherit; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">JVisualVM (remote monitoring through JMX)</li>
</ul>
</ul>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
I like to revisit GoF Patterns, Enterprise Integration Patterns and SOA Design patterns from time to time. In my opinion those are essential books to have.</div>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Over time my interests will evolve as I handle newer projects that will stretch me to handle greater challenges.</div>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Like most software professionals I use google(90%)/bing(10%) to find answers to errors, stacktraces or to learn up and coming technologies. The more equipped we are with varied skills and technologies, the better we can be at finding a suitable solution to a given problem. We have heard that if the only tool you have is a hammer then everything looks like a nail. Off late I find myself doing that because I tend to apply what I already know. But I find that keeping myself open to newer ideas and technologies helps me find better solutions over time.</div>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Best part of working with open source projects is the breadth of choices and depth of content/expertise already available. It is like “standing on shoulders of giants” as my brilliant colleague Matthew says!</div>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
At times I have wondered..hmm..wouldn’t it be nice if a certain open source product or a specific functionality was better documented or explained. It would have saved me considerable amount of time. My goal is to help make time to share some of my practical know-how in certain technologies. I have also realized that through sharing I actually end up learning more! Nice side effect <img alt=":)" class="wp-smiley" src="http://technologyenthusiast.blog.com/wp-includes/images/smilies/icon_smile.gif" style="border: none; margin-bottom: 0px; margin-top: 0px; max-width: 97.5%; padding: 0px;" /></div>
<div style="background-color: white; border: 0px; color: #373737; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 24px; margin-bottom: 1.625em; outline: 0px; padding: 0px; vertical-align: baseline;">
Thanks again for visiting!</div>
Ashwini Kuntamukkalahttp://www.blogger.com/profile/08119557274513301439noreply@blogger.com0