XML Batching/Grouping/Assembling in Azure Logic Apps using the Batch feature

We are having a requirement to batch XML messages per the interface type (what I mean by interface here is Purchase Order interface, Invoice interface, etc.) before sending them over to the destination. I decided to utilize Logic Apps’ own Batching feature to see if this can be done. I did a proof of concept for a batch count of 500 messages and tested it using Postman by firing 1000 messages consecutively. All messages got batched successfully and also the messages were grouped (2 * 500) by the interface type. For grouping the messages that you are sending to the batcher, Logic Apps has a property called Partition Name. You can also add a unique id for every message that you send to the batcher.


Here is how I did that…

I created three logic apps: two batch senders and one batch receiver (and release).

Let’s take a look at the batch receiver first.

Batch Receiver Logic App:

This logic app uses batch trigger and will keep on collecting messages until the release criteria is met. The release criteria can be based on message count (say 50 messages per batch), batch size (say 1 MB) or schedule (say every 2 hours). Learn more here..

I set the message count to 500. Upon reaching the criteria the batch would be released. I initialized a variable to create the root element of the message batch then a for each action to loop over the released items and kept on appending the content of each item. Once the loop is done, I closed the root element tag. I made use of Partition Name for my root element name to make the batching logic app generic. For example, I can send purchase orders, invoices to the same batching logic app as long as the release criteria is the same.

Batch Trigger

1_BatchTrigger

Initialize variable to append XMLs with Partition Name as the root element

2_VariableInitialization

Loop over the batch release content and append each item to the variable

3_ForEachAndAppend.PNG

Close the root element tag

4_CloseRootElement.PNG

Store the appended messages in OneDrive

5_SaveAppendedFile


Then I created two batch sender logic apps. One to send Purchase Order interface files and the other to send Invoice interface files. Please note both of them send their messages to the same batcher Logic App. But they are grouped by Partition Names automatically. How awesome!

First Batch Sender Logic App:

This logic app will send the individual messages (in this case Purchase Order) to the batching logic app with the partition name “PurchaseOrders”. I exposed this Logic App as a HTTP end point and would post the Purchase Order XML payload from Postman. Then it would call the batcher Logic App to send the messages with the batch name, partition name and the message content.

6_BatchSender1


Second Batch Sender Logic App:

This logic app will send the individual messages (in this case Invoice) to the batching logic app with the partition name “Invoices”. Everything else is the same as the first batch sender Logic App.

7_BatchSender2


All set! I made use of Postman collection runner to send 500 messages from each sender logic app. All the 1000 messages were split into two files of 500 each and stored in my PurchaseOrders and Invoices OneDrive folders as PurchaseOrders.xml and Invoices.xml respectively.

8_PostmanRunner

Never felt super cool like this in a while.

This excellent post by Microsoft MVP Toon Vanhoutte explains how it can be done utilizing Azure Service Bus. I think the current Batching feature of Logic Apps also utilize Service Bus behind the scenes.

I hope you enjoyed reading my blog. Please feel free to leave your comments/suggestions. Thank you!

Shri

Advertisements

One thought on “XML Batching/Grouping/Assembling in Azure Logic Apps using the Batch feature

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s