BizTalk Wiretap
During my years of supporting BizTalk Solutions I have seen a very common technique of using dedicated file-adapter send ports that subscribe to messages for inspection and for other purposes following the wiretap messaging pattern.
Granted, using the wiretap pattern provides a very flexible way of adding and removing the secondary destination components without affecting the core solution, however, in most cases I have seen, it was implemented for the wrong reasons.
In this article I will share some of the cases I have seen using this pattern, and the caveats that arise from misusing this pattern in BizTalk solutions.
1. File-based Business Activity Monitoring
I can’t tell you how many times I have seen the wiretap pattern being implemented in order to provide business-related reports and auditing purposes.
If there were exiting auditing downstream systems in the organization that rely on these files that would have been fine, but to add insult to injury, either the business users will manually search for these files in a network folder, or additional downstream custom components are built to handle transformations, manipulations, and user interface for business users.
This latter approach is very common with teams who are new to BizTalk, as they feel at home when they develop their own tools using the flat files outside the BizTalk realm.
This is an example of a misuse of the pattern for the following reasons:
Performance Impact
Additional subscriptions, port-level transformations, and pipeline components logic will introduce more load on the BizTalk engine in terms of compute, memory and IO to process these file-adapter messages. This can get much worse quickly with large message payloads, multiple logging stages, and under realistic production load.
BizTalk BAM
Another reason is that BizTalk already ships with a business activity monitoring (BAM) engine which could be leveraged to seamlessly provide business-related reports for BizTalk Applications using configurations and without writing a single line of code. BAM also comes with a portal, infrastructure database model, and data maintenance background jobs.
Importantly, BizTalk tracking process is optimized to minimize the tracking performance overhead, as it temporarily gives higher priority for processing live transactions over tracking when the engine is under high loads.
It is worth mentioning that there is also a prominent Business Activity Monitoring product called Atomic Scope by Kovai, which provides rich, contemporary features that span across BizTalk and Azure Integration Services components. You can check my post where I review Atomic Scope and BizTalk BAM.
Data Maintenance and Governance
Data maintenance and governance are usually an afterthought, files containing sensitive data will require obfuscation, older files will require regular archiving or cleanup, and folders will require proper access control and governance.
2. File-based Solution Tracing
This another case of misusing the wiretap pattern in BizTalk solutions that I have seen time and time again. Developers use it to conduct technical tracing throughout a BizTalk Solution for debugging and troubleshooting purposes.
Tracing a solution can get very granular to be able to adequately troubleshoot a solution, this usually manifests in capturing many fields of the published messages across multiple stages of the message journey. This in turn leads to having numerous trace files, or even a single trace file with huge amount of entries, all of which will actually lower the visibility of the solution tracing.
Again, developers who are new to BizTalk will be tempted into building and maintaining a user interface that allows them and the support team to find specific trace information to troubleshoot in a timely manner. In this case, you can also trust these developers will be lured into building that user interface with all the bells and whistles for searching, sorting, access control, and reading files’ trace contents.
In some other cases I find the support team manually looking among the trace files in the target folder(s) and even on multiple nodes in a clustered BizTalk Server, using no clues other than date and time range!
As with the Business Activity Monitoring approach, this comes with the same caveats in terms of the performance impact, data governance and access control, and not using the right tool for the job.
BizTalk CAT Instrumentation Framework
CAT Instrumentation Framework (CATIF) is the right tool to instrument BizTalk solutions, it was created by Microsoft’s BizTalk Customer Advisory Team and it is based on Event Tracing for Windows (ETW) infrastructure, so it provides tracing with very low performance overhead.
Based on my experience, I find CATIF performance overhead to be really negligible in my integration scenarios, the overhead is usually in the range of double-digit milliseconds.
You can check my AIMS webinar where I show a demo on BizTalk CAT Instrumentation Framework and dive into the different techniques for Tracing Solutions in BizTalk.
Ad-hoc Wiretap Troubleshooting
I like to complement CATIF with file-based tracing for ad-hoc, on-demand troubleshooting to inspect messages at the solution edges. This comes in handy when you are trying to troubleshoot end-to-end messaging scenarios.
I use a single file-adapter send port that is dedicated for debugging and that is always maintained in an un-enlisted state. The port is only enabled by a developer in my team when she wants to start debugging.
You need to be extra careful when using this technique, if this port is accidentally left enabled, it could unintentionally be subscribing to messages which could potentially be hiding critical subscription-related issues in your solution.
3. File-based Business Process Automation
It goes without saying, using file adapter for a secondary system in a publish-subscribe manner as a part of the business process automation is absolutely fine, after all, this is where BizTalk shines, by integrating disparate systems with different integration capabilities.
The target systems in these cases are usually legacy systems that ingest flat files to do some sort of processing that is part of a business process.
By legacy systems, I don’t only mean old systems, this also includes systems that are beyond the organization’s control to change and adapt to modernized integration styles.
I have seen this in cases where flat files with specific format are handed off through a local/network folder to an existing legacy system which will conduct business-related, secondary processing; even if it is used for logging, auditing, reporting purposes, as long as it is a part of the organization’s business process.
The key difference that favors the use of wiretap pattern in this case over the previous two cases, is that it is integrating with an existing system as part of a business process automation implementation.
Conclusion
The wiretap pattern is a very common practice that I have seen many times in BizTalk support projects, as it is very easy to implement in BizTalk solutions. However, it is because of that ease of use, it will likely be misused when not taking into consideration the common caveats like the performance impact, data maintenance and governance, and reinventing the wheel by not using the right tool for the job.