Ordered Processing / FIFO Sequencing with Azure Logic Apps and Service Bus Topics

I was trying to figure out how to ensure sequenced processing of the messages (the messages are processed in the same order they are received) with Azure Logic Apps. Here my requirement is to make sure the subsequent message should not be processed until the previous one’s processing is complete and it should work the same way for the lifetime of the process. Example: Multiple senders can send (related or unrelated) messages (new, updates within milliseconds or minutes or days apart) to my topic subscription. The first messages that comes in, should always go out processed before the second message gets picked up for processing.

I utilized the following properties of Azure Service Bus and Logic Apps to achieve my desired requirement.

 

Service Bus Sessions: To realize FIFO guarantee in messaging, Microsoft recommends to use Service Bus Sessions. Sessions enable joint and ordered handling of unbounded sequences of messages which is what I exactly need. Any sender can create or share a session while submitting messages into a topic or queue by setting the SessionId property to some application-defined identifier that is unique to the session. So I can create a session and keep it on forever by publishing the messages to the topic subscription with the same SessionId.

Service Bus Message Sequencing: Senders can submit messages to the topic within  milliseconds timeframe. I would need a better Sequencing technique to absolutely order the messages. Service Bus came to the rescue again. Message Sequencing and timestamping are two features that are always enabled on all Service Bus entities and surface through the SequenceNumber and EnqueuedTimeUtc properties of received or browsed messages. The sequence number can be trusted as a unique identifier since it is assigned by a central and neutral authority and not by clients. It also represents the true order of arrival, and is more precise than a time stamp as an order criterion, because time stamps may not have a high enough resolution at extreme message rates and may be subject to (however minimal) clock skew in situations where the broker ownership transitions between nodes. I don’t have to do anything for this as the sequencing is auto-enabled.

Service Bus Peek-Lock Mode: This mode puts an exclusive lock on the message/session and retains the lock for the set duration until the receiving client (Logic App) completes (dequeues) the message. This is very critical in my case as I don’t want any subsequent messages to be made available until the current message is explicitly marked complete.

Logic Apps Next Available property on the Service Bus trigger: Enabling this property makes sure every time the logic app triggers, it will get only the next available message on the session.

I think we are all set to address the requirement by leveraging Azure Logic Apps and Azure Service Bus.

Azure Service Bus Standard Tier can process message up to 256 KB in size. If your message size exceeds the limit, you can keep the message content elsewhere (on a blob storage or a file share or a FTP site and publish only the message metadata to the Service Bus Topic and have your Logic App directly read and process the content from where you have stored your file) or you can consider to go Premium that can process 1 MB.

Here is what I did:

• Created a Service Bus namespace and a Topic on the Azure Portal
• Leveraged Service Bus Explorer to create subscriptions and filters (Enabled Sessions on just one so we can differentiate how the Session-aware and Session-unaware messages are processed) (Azure Portal doesn’t have the feature to create Filters on Subscriptions yet)
• Create three Logic Apps (First to publish PO transactional messages (ordered processing required) and Supplier master messages (ordered processing not required), second to sequentially process the PO messages and the third to process the Supplier data )

Let’s take a detailed look.

Service Bus Namespace creation:

1. Log on to Azure Portal –> Create a resource –>  Search for “Service Bus” and create the namespace.

2. Open Service Bus namespace on the Service Bus Explorer by copying over the connection string from the Azure Portal. Create Subscriptions and the appropriate Filters to receive only the intended messages. And set the Lock duration to 1 minute.

Important: Enable Session on the Purchase Order subscription

The client holds an exclusive lock on all messages with that session’s SessionId that exist in the subscription, and also on all messages with that SessionId that still arrive while the session is held. The lock duration for each message on the session should be carefully set. If the client takes more time to process a message than the set duration, the client would lose the lock on the message and the message would be released for the next available subscriber or in our case, the next logic app instance. If required, lock on the session can be renewed by the client while the message is being processed. In case a message failed processing inside the application, the message should be dead-lettered so the next message will be made available.

 

 

This slideshow requires JavaScript.

3. Now the Service Bus Topic is ready. The subscription ‘subscription-purchaseorders’ will receive the purchase order messages that would be sent with the message property {Interface = ‘PurchaseOrder’} and SessionId = PO and the subscription ‘subscription-suppliers’ will receive the supplier messages that would be sent with the message property {Interface = ‘Supplier’} and no SessionId

Let’s see how to design the Logic Apps:

Logic App 1: Publish Purchase Order and Supplier messages to the Topic

I used HTTP trigger and sent the Purchase Order and Supplier messages from Postman. Once triggered, it would publish 9 messages to the topic. Based on the message content if it’s Purchase Order, SessionID will be set to ‘PO’ and in case of Suppliers, SessionID will be nothing. Message Properties need to be set while the messages are being published so the correct subscription would receive them per the filter defined.

Logic App 2: Consume Purchase Order Messages from the Topic (Ordered processing)

Once again, the requirement is to process Purchase Order in the sequence they are received. And also the previous purchase order needs to be completed (dequeued) before the next one waiting on the line is made available and picked up for processing.

Here is what I did: I made the Service Bus trigger on the second logic app to poll every 5 seconds and deliberately added a delay action to hold the processing of every instance by at least for 50 seconds (The below screen shot incorrectly shows 1 Minute. Apologies!). I did this to check if the subsequent triggers of logic app (every 5 seconds) are getting the next messages on the topic before the current message being processed is marked complete. To achieve this you have to set a property called “Session id” to “Next Available” on the Service Bus Trigger and use Peek-Lock mode while receiving the message so an exclusive lock would be applied on the message by the Logic App instance that picked it.

Service Bus Session would ensure only the top message (least sequenced) is available for the subscribers though it has 9 messages with the same Session Id.

Logic App 3: Consume Supplier Messages from the Topic

This is a straightforward Logic App. Set up the trigger to consume messages from the Supplier subscription and upon getting triggered, it will receive all the messages that are on the Topic and it will complete. Here the Peek-Lock mode puts the lock on just the message (so it won’t be available for any other subscriber for the set lock duration) and it doesn’t affect the other messages on the subscription. Though the Service Bus adds the Sequence number to the messages upon arrival, it doesn’t guarantee FIFO or ordered release unlike Sessions.

So all our Logic Apps are created. Let’s test them.

Let’s trigger a PO message from the Postman App.

Logic App 1 published 9 messages to the Service Bus Topic “sb-topic-interfacemessages”. Service Bus auto-sequenced them.

Let’s see what the second purchase order logic app did.

 

This slideshow requires JavaScript.

 

Though the logic app was running every 5 seconds, the messages were picked up only after the previous message was complete i.e., every instance took almost a minute to process and mark the message complete so the next message was made available. Also take a look at the skipped triggers which means while a message was being processed, even when the logic app attempted to receive messages from the subscription, no message was available though they were on the subscription and trigger was skipped. Hooray! 

Let’s see what the third logic app did.

 

This slideshow requires JavaScript.

Though the Logic App was running every 30 seconds, upon seeing messages on the subscription, it went ahead and received all the messages and processed them all. The lock on the message was held only on that message and it didn’t affect the other messages’ availability.

Conclusion: So Azure Logic Apps + Azure Service Bus Topic Subscriptions enabled with Sessions can guarantee you ordered processing or FIFO processing for your messages. Any messages that are published to the subscription (milliseconds/seconds/days/weeks/months apart) would always be processed in the order they are received.

One caveat is here that huge load of files could create more wait time depending on the processing time of each message. In such case, multiple sessions can be created perhaps based on sender or something as long as one session’s batch processing doesn’t affect the other.

There is already an excellent article on technet for ordered delivery but I don’t want to have a single instance of logic app looping over again and again looking for messages and my session is also not a short-lived one and my session wouldn’t be closed by the sender.

I hope this is helpful to you in case you are trying to solve FIFO processing with Azure Logic Apps.

Cheers!

Shri