I was discussing an architecture with a friend the other day which was put together well on the whole but one thing stood out and I felt compelled to write about it. Anybody that’s been in IT and finance has had to deal with high volume messaging applications. In fact most of the better people I know have based their entire career on this. This is what this post is about.
Let’s define some terms up front. A queue is a structure which follows the FIFO principle. Azure primarily uses queues to communicate between web and worker roles. It’s important to understand the limits to this when planning an architecture for an azure service. Web roles communicate with worker roles via small non volatile messages and all of the features such as fault tolerance, guaranteed delivery etc. are provided through the queue so it’s something you can rely on. Here’s what the queue is though … It’s Azure Storage with a nice queue algorithm wrapped around it. In effect the two roles are looking at the same disk location or abstraction of that location. It’s important to understand this. To me the queue should be used for inter role communication.
When we write messaging applications we treat messaging as a subsystem. We identify message endpoints and nodes. If you can visualise your application in this way at all then the queue may be wrong for you. In the case of the architecture I was looking at a queue was in place solely because the author saw two endpoints. One node enqueued and the other dequeued. While entirely relevant I wanted to further open the conversation here because I think this is a common pattern which breaks down when the context is not understood. The key is how much tight coupling you want and what else the receiving node is doing. These things should be considered in conjunction with the fact that a high volume of messages matter. If you’re dealing with thousands of messages an hour then your queue is going to be constantly full and you’ll have to continually write a lot of code pull off each message. Maybe you’ll have too many messages and your queue will fill up beyond its sustainable buffers. You could throw raw compute at it but what’s the point when you don’t have to.
Let’s take a look.
var storageAccount = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
var queueClient = storageAccount.CreateCloudQueueClient();
var queue = queueClient.GetQueueReference("happyqueue");
while (true)
{
Thread.Sleep(1000);
if (queue.Exists())
{
var msg = queue.GetMessage();
if (msg != null)
{
// do something here!
queue.DeleteMessage(msg);
}
}
}
We can see from the above some very interesting things.
- We place a sleep which is indicative of what this loop looks for – a pattern to conserve compute cycles
- Pulling a message off the queue – or even using a Peek call to check whether there is a message first
- Offloading some processing and continuing on the next polling iteration
No surprise here. This is well-known and one of fundamental tenets of this type of queue-based programming.
Now, enter tens to hundreds of messages per second and here is where the problems start (Microsoft state that the queue should be able to process 500 msgs/second however this value is not SLA driven). Let’s say that your expectation is anywhere between 50-500 messages per second, your main thread is going to be checking the queue continually and whilst this is not a problem since there is no physical queue size limit you’ll have to be clever in your algorithm to avoid 10′s if not 100′s of millions of unnecessary transaction and additional cost. At this level you’re paying for $1 per 1 million transactions.
Enter the posh queue! Well the service bus. Spend some extra time and development on this and employ a free messaging subsystem, well almost free! You no longer have tight coupling, continuous polling etc. life is much easier because you can do everything asynchronously and calculate different strategies to deal with the queue load. There are lots of ways to use the service bus: relaying, topics, subscriptions and queues – all fairly intuitive and much more open and recognisably like middleware than the queue.
There are some fantastic resources on MSDN about the service bus and a great comparison here so I won’t go into implementation details here:
The message I wanted to get across here is that if there are two endpoints and messages need to be passed between roles don’t automatically assume you should use Azure Storage Queues; it may be pricier and harder to implement, you may be volume restricted and end up paying more in compute anyway.
I’m currently working on the load test tool whereby I’ll publish part II of the series shortly so I probably won’t have time to do any real investigation but I’ll ask Rebecca Martin if she’ll look at this and post hard data if she has time to do some tests.
Happy coding!
