Real World Business Activity Monitoring (BAM) using BizTalk 2010 part 1

In all my years working with BizTalk, my current employer is the first that I've had a real requirement for using Business Activity Monitoring (BAM).  From a high level, we needed to provide the business with a real time view of all the different messages coming into and out of the Enterprise Service Bus (ESB) environment.  Additionally, the business wanted to see some high level statistics of these messages like volume throughput, average size, and average time.

To meet these requirements, we used two different solutions.  BizTalk BAM tracking was used to track the receiving and sending of each different type of message in the ESB.  The second solution was a custom ASP.NET web site to display the information in a consolidated manner for  the business users.  I'll explain why we didn't use the out of the box BAM monitoring portal in a subsequent post.

As I've mentioned in some previous posts, the current BizTalk environment I work in performs a lot of content based routing for messaging.  When looking at a way to monitor all of the messages, it made sense to leverage BAM tracking.  This not only allowed the tracking of all the different individual messages, but it also provided a means of tracking specific information related to each message.   Below is a screen shot of the web site that is displayed to the user:

From looking at the above screen shot, the intent of the web page is to display a high level view of the different messages moving through the ESB.  The information presented to the user on the site is based mostly off of the views created with BAM tracking.  Each row shown above identifies a different subscriber to a particular message publication.  While meeting the requirements of the business, it also gives some indicators to the health of each individual message in the ESB.  *Note: Credit must be given to my then manager Darrick Johnson for creating the original web site.

My next post will walk through the "how to" of creating the tracking for an individual message. In my last post, I will look into more specifics of how the ESB monitor is organized and what it provides for the users.

BizTalk Toolkit 2.1 ESB Add Namespace Pipeline Component and Encoding

Here is something that tripped me up the other week.  I was working on a Proof of Concept (POC) with BizTalk and writing XML messages to a RedHat Linux Network File Share (NFS).  To connect to the NFS, I followed a great blog post How to connect to an NFS with Windows Server 2008 R2 by Randy Aldrich Paulo.  Anyway, what tripped me up was the encoding of the files that were written to the server.

On send ports, I usually use the "Out of the box" XMLTransmit pipeline component.  However, this particular POC had a requirement to strip all namespace prefix tags from the XML elements contained in the message while retaining the target namespace on the root element.  I thought, no problem, the ESB Toolkit has two pipeline components I can use to solve this.  Specifically, I created a pipeline component that used the ESB Remove Namespace Pipeline Component and the ESB Add Namespace Pipeline Component.

So I get the send port created and assign the new pipeline component that satisfies the requirements:
  

I run a test through and everything looks fine from a BizTalk perspective.  I inform the Linux Administrator that a new XML message is available for him to review.  He confirms the receipt of the file and I think life is good.  

I get a call back from the Linux Administrator after about 10 minutes with a problem.  Apparently, the file being sent has some special characters ‘��’ at the start of the XML.  I think to myself, that is strange, maybe it is Byte Order Mark (BOM) related.  I change the property "RemoveBOM" on the ESB Remove Namespace Component and set it to "True", thinking this was the problem:

So I run another test through and inform the Linux Administrator that a new file is available.  He confirms the receipt but on inspection of the file the special character problem persists.  Furthermore, he follows up with the fact that the encoding on the XML file is UTF-16 when they are expecting UTF-8.  Again, I think to myself, shouldn't BizTalk be sending this message in UTF-8?  

I go to the Tracked Message in BizTalk Administrator for the send port and look at the MessagePart:

So the Character set on the message looks to be "utf-8".  I go back to the Linux Administrator and argue a bit with him until he finally sends me the output file from Putty showing the file properties.  One of those properties being the encoding which was set to: Little-endian UTF-16 Unicode text.

So me being my hard headed self set out to prove the Linux Administrator wrong.  I find a cool utility on CodePlex called File Encoding Checker and download the tool.  I create a dummy send port (using the new pipeline component) that outputs the file to a local folder on my development BizTalk server.  I check the encoding and this is what I find:

What???  How could this be?  I check the documentation for the ESB Add Namespace Pipeline Component at msdn.microsoft.com and I can find no information about what encoding is used.  Apparently, the ESB Add Namespace Pipeline Component defaults the encoding to UTF-16.  Additionally, the pipeline component has no property for setting the encoding (although the Remove Namespace Pipeline Component does).

In order to resolve this problem, I had to create a custom pipeline component which adds a namespace to the root element and uses UTF-8 to encode the output.  An interesting lesson on the Add Namespace Pipeline Component and something to keep in mind when leveraging it.  It would be great if the Add Namespace Pipeline Component could be updated with a property to set the encoding, just like the Remove Namespace Pipeline Component.

BizTalk 2010 and integrating with the Java Messaging Service (JMS) SonicMQ Part 3 Creating the Send Port and Wrap Up

Has it really been three months since my last post?  Sorry about the delay in getting this finished, but I just got through a substantial sized project.  On to the goods...

Creating the JMS Adapter Send Port
I'm assuming that creating a send port is already known for someone reading this post.  Setting the send port properties of the JMS Adapter are pretty straightforward and all dependent on the queue you are connecting to:

The first section of properties entitled "Adapter Behavior" are pre configured and I left the default settings alone.  I haven't played around with these settings at all, but they revolve around how the send port handles exceptions and timeouts.

The next section of properties to configure is under the title "Connection Properties".  The Host Name and Port Number coincide with the JMS server you are connecting to and the port in which the JMS server is listening on.

The third section is called "JMS Operation Properties" and have to do with the specific instance of JMS being connected to.  The JMS Object Name is the actual name of the Queue or Topic that messages will be sent to.  The JMS Object Type is a pull down list with either Queue or Topic.  For Message Type, this is another pull down list of the different message types.  In my case, I selected Text, but the options are: Bytes, Map, Text, Text ISO-8859-15, and Text UTF.  Lastly, the Transactions Enabled property tells the port whether or not transactions are enabled.

The last section is called "Overrride Transmit Handler Properties", and although I didn't configure these, I'm assuming you can use this to override what was configured in the JMS send adapter transport handler.

After creating the send port, you will need to spend some time getting communication between BizTalk and your JMS server working.  Some of the less obvious issues you will run into could be security related but those are too environment specific to go into detail here.  Additionally, you may run into issues with how the JMS server is configured.  For instance, I had created multiple ports to communicate with the JMS server, but the server was configured to only allow one connection.  This caused the JMS adapter to throw infrequent errors when trying to communicate.  It took several tests to isolate the issue and change the configuration on the JMS server to allow multiple connections.

Lastly, there are other aspects to consider before implementing the use of the JMS adapter.  In my opinion, two of the more important aspects are that of security and cost.    Depending on how mission critical this adapter is to your business, the former may weigh more heavily on the decision rather than the later.  This was the case in my scenario.

Security
From a high level, this should always be in the back of your mind as to how to handle security with any adapter being used.  In my opinion, defense in depth is always the way to go.  If your unfamiliar with this term, it essentially means the more security you have, the better.  Still, there are considerations to take, like whether messaging is internal only or external (meaning outside of your network).  What kind of infrastructure support you have to help set up security.  What type of data is being messaged could come into play as well.

Cost
The only negative about this adapter has been the cost.  Since my environment consists of migrating code through four different environments, there was a cost associated with the adapter for each environment. Since my current company leverages an integrated test environment that mimics our production environment, the cost for the adapter was even higher.  This is something to consider before going forward with the adapter.

All in all, I've been really happy with the adapter and how we were able to spin up a proof of concept fairly quickly.  Not having any prior experience with integrating BizTalk and JMS, this adapter helped bridge that gap.  It does a lot of the heavy lifting so that I didn't have to become an expert in JMS.

BizTalk 2010 and integrating with the Java Messaging Service (JMS) SonicMQ Part 2 Configuration

The second post on the JNBridge JMS Adapter for SonicMQ will cover configuration of the JMS send adapter transport handler.  If you haven't already read Part 1 , I suggest you do so before continuing.  That post covers the installation of the adapter onto a BizTalk 2010 Server.  Although installation is pretty straightforward, the post also details how to get the third party adapter from JNBridge, as this adapter doesn't come out of the box with BizTalk.

After installing the JMS Adapter, configuring the adapter falls into two parts.  The first part covers configuring the send adapter transport handler and will be explained in this post.  The second part will cover creating the send port in a BizTalk Application and will be detailed in a subsequent post. Setting up the JMS adapter for the receive transport handler and receive location should be very similar to the send handler and port and will not be covered.

Configuring the JMS Adapter Send Transport Handler

When configuring the send transport handler for JMS, there are several properties that are important and dependent upon the SonicMQ which is being integrated with.  If your unfamiliar with SonicMQ (like me), it's important to have a detailed discussion with whoever is responsible for this application.  Doing so will ensure the below properties are properly configured which in turn will save a lot of time and effort.

In order to edit the properties of the send handler, you will need to open BizTalk Administrator, and navigate to "Platform Settings", "Adapters", "JNBridge SonicMQ", and finally right click the "SendHost" and select the properties option:

The first section of properties that need to be configured are under the section "JMS Properties".  I started with the property called "Acknowledge Mode"  A pull down list is provided for this property.  There are three choices, "AUTO ACKNOWLEDGE", "CLIENT ACKNOWLEDGE", and "DUPS OK ACKNOWLEDGE".  In my case, I selected the "AUTO ACKNOWLEDGE" which also happens to be the default:

The second property to be configured is called the "Initial Context Factory".  This is an editable field and needs the name of a JNDI class.  I'm not going to explain what JNDI is, as it's outside the scope of this post.  In my scenario, a named SonicMQ domain is required, which the JNBridge Adapter supports.  In order for the JMS adapter to support this, I am required to use com.sonicsw.jndi.mfcontext.FixedMFContextFactory as the initial context factory:

The JMS Scheme is the protocol used to connect to the JMS service. In the case of SonicMQ, that is tcp:

The next property(s) that need to be configured are the Queue Factory or Topic Factory.  This is the class name of the SonicMQ factory.  In my case, I only filled out the Queue Factory property since that is what I'm connecting to.  I left the Topic Factory property blank:

The Security mode property is the last under the "JMS Properties" section of the send handler.  This property is another drop down list and allows the passing of credentials to the JMS server.  The options are none, simple, and strong.  In my case, I need to pass credentials so I selected the simple option:

The second section of properties that need to be configured are under the "JNBridge Properties" section. These properties are JN Bridge specific and I'm not going to go into details on the set up of these properties. The documentation provided by JN Bridge is pretty thorough and will guide you on the proper values required for each property.  One point I do want to comment on is the "JVM Arguments" property.  This property allows me to pass a named SonicMQ Domain and ties directly to the "Initial Context Property" configured above:

.

The third section called "Security Properties" has to do with credentials needed to connect to the JMS server.  Since I selected the "simple" in the Security Mode Property under the JMS Properties section, I will need to pass a username and password.  This should be supplied by the owner of the SonicMQ application:

The last section "Debug Properties", isn't required, but is worth mentioning.  I relied upon this heavily when first trying to connect to the JMS SonicMQ Queue.  It provides verbose logging when the JMS adapter is trying to connect to SonicMQ.  After your successful in connecting, you can just switch the "Log Errors" flag to false

That is the last section of the JMS adapter send handler transport properties to configure.  In the next post, I'll cover creating the send port in BizTalk Administrator and what you should consider before using the JNBridge JMS adapter.  

BizTalk 2010 and integrating with the Java Messaging Service (JMS) SonicMQ Part 1 Installation

I recently worked on a Proof of Concept (POC) with BizTalk 2010 and integrating with a Java Messaging Service (JMS) queue called SonicMQ.  Having never done this, I did a quick Google search to see if I could leverage any existing adapters.  Luckily, there is already a a third-party JMS adapter created by JNBridge.   They offer a 30 day trial version of the software, so I downloaded the install bits and went through the installation and configuration.  I want to use this post to go through the installation of the adapter.  I'll follow up with a second post on the configuration of the adapter.

Installing the JMS Adapter from JNBridge

The installation of the JMS adapter by JNBridge is fairly straight forward.  Since my environment is 64 bit, I downloaded the msi file for the 64 bit environment (latest version is JMS Adapter for BizTalk Server v3.0.1 x64).  Double clicking the msi file brings up the installation wizard:

I accepted the License agreement terms:

I kept the default install directory and installed the software:

After the installation completes, I went ahead and added the adapter in BizTalk Administrator.  To do this, I opened BizTalk Administrator, navigated to "Platform Settings" and expanded the group, then right clicked the "Adapters" option and selected "New" and "Adapter...":

Under the Adapter properties, from the Adapter pull down list, I selected "JNBridge JMS":

After giving the adapter a name and clicking the "OK" button, the installation of the JNBridge JMS adapter is complete.  Don't forget to update the adapter to use specific hosts.  If your like me, I have a different host for sending and receiving.  Lastly, the JMS Adapter requires an install of the Java Runtime Environment (JRE), which you can download from Oracle.

Up next, the steps needed to configure the JNBridge adapter.  In my particular instance, I am connecting to SonicMQ, but the JNBridge JMS Adapter can work with multiple vendors.  The JNBridge website has a list of those vendors and the supported JMS versions.  It's important to check that list to ensure the adapter will work with your particular implementation.