KnowledgeTree Blog

Clouds & XMPP: A match made in heaven?

It has been a few days since OSCON 2008 has drawn to a close, and the two topics that interested me most have made a few appearances on the all knowing web. One of the two topics is cloud computing. No surprise there as I gave a presentation about it at OSCON. The other is XMPP. Until OSCON I didn’t really give much thought to the possibilities that XMPP is offering, but more on that in a sec.

Cloud computing really has reached buzz word status. Looking at the number of presentations at OSCON dealing with clouds as a proxy of this, shows how just a year ago not a single presentation dealt with cloud computing. OSCON 2008 had about half a dozen presentations dealing with cloud computing in one way or another. The first two days of this week have brought news of HP, Intel and Yahoo’s entry into cloud computing, and then there is a rumor going around that Microsoft is also looking at entering this market. All of this activity and we don’t even yet have an accepted definition of cloud computing.

You could be forgiven if you thought that traditional colo’s are about to disappear from the face of the earth, washed away by heavy rains brought by the clouds. But I would not run out and write their obituaries just yet. As Thorsten von Eicken pointed out in his presentation “Scale into the Cloud with Open Source” the real benefits of cloud computing reveal themselves to applications that are able to scale horizontally. In theory that should be possible with every application, but in reality there usually is some development needed to achieve this. The more significant the modification required, the higher the cost is to switch. The counter argument is that switching will have to be done sooner or later as the demand for the application grows. That is true, yet the cost of switching now could be higher than it will be in a year or two.

There is not only the application to consider. Another important puzzle piece is the management of your cloud. Most conventional management and reporting tools don’t readily map onto the cloud computing paradigm perfectly. And this brings me to XMPP. The development of XMPP has largely fallen under my radar, so I was taken a bit by surprise at the versatility of this protocol.

Let’s just take a step back. One of the biggest challenges when moving to the cloud is the amount of automation that such a move involves. You have servers coming and going, and certain configuration changes need to happen based on this. Your servers also need to co-ordinate with each other and communicate with each other. I have been using Capistrano to do most of this automation. Although I have found it to be an awesome tool, it hasn’t given me everything that I require. SSH is not the perfect protocol for this type of work, and is inherently a one way communication when used in scripts. What I really would like is something that is more ‘interactive’ and real time. Enter XMPP.

People are doing some amazing stuff with XMPP, but what got my interest the most was Engine Yard’s Vertebra. Take a look at this presentation to get an idea how Engine Yard is planning to use it. Wow, some interesting stuff. But wait, there is more. One of the key problems that cloud computing is trying to solve is scalability. And this incidentally happens to be what the good folks over at Flickr are using XMPP for. Evan and Kellan’s presentation “Beyond REST? Building Data Services with XMPP PubSub” discussed how the inherent polling nature of web services, be it REST or SOAP, quickly lead to scalability problems. Have a look at slide 15 and 16 to get an idea of the problems that polling poses for Flickr.

So, XMPP and cloud computing, is this a match made in heaven? Guess time will tell, but I would recommend you keep an eye on this space, I suspect there is more to come.