A New Kind of Application Server

As you probably know, I'm a cofounder of Coversant which, at its heart, is an XMPP development platform. Most of our larger customers (thousands of simultaneous users) are ISV's that have built on the SoapBox Platform®. We allow you to easily develop XMPP applications using .NET technology.

A really long time ago, I wrote about some possibilities for using the SoapBox Platform including examples of what our customers were doing at the time. This was before there microblogging was popular, or I probably would have used that example too. :)

The last couple of weeks there seems to be quite a bit of buzz around the subject of using XMPP as an application server, and that gets me really excited! A friend/competitor Matt Tucker of Jive Software wrote in his company blog about how XMPP is the future for cloud services. A "real" online author (aka not a member of an XMPP company) even picked up Matt's article and ran with it. Yesterday, a little buzz hit Slashdot when another friend/competitor Mickael Raymond of Process One wrote about introducing the XMPP application server (when I wrote this, it seems Process One was experiencing a bit of the Slashdot effect -- hopefully by the time you read this it will be gone and you can read his article), which is an exploration of building a Twitter-like microblogging system on top of their XMPP server. Great stuff, indeed!

This is wonderful news and very validating for me personally! It seems after six years of committing to the infant technology, I wasn't crazy after all, and XMPP is a good platform for presence/messaging systems! And if you're in the market for .NET based XMPP solutions, head on over to the SoapBox Developer site. :)

XMPP Presence Priority

In XMPP (the messaging and presence protocol SoapBox uses) you can log in simultaneously on multiple systems. How your contat list is displayed and how messages are routed are based on hints the client software provides to the server. One of these is presence priority. Priority is just an integer value between -128 and 127 defined in RFC 3921 Section 2.2.2.3.

Managing priority is fairly straightforward when you control all client implementations used by a bare JID, but in a heterogeneous client environment it is more difficult as each developer chooses his/her own path and only learns by spying on what others do. I think presence priority is something that should be covered in the implementation guide so the actual values become more uniform on the network. We currently do pretty basic priority management, but I have some thoughts on a better implementation I hope others implementations might help refine.

Priority is typically used by both clients and servers to determine which resource is most likely to be the one you want to interact with. This could be for routing messages sent to bare JID’s, or deciding which presence status to display in a contact list. So, after much deep thought and implementation trial/error, here is my proposal.

Unless a user explicitly specifies she does not want to receive any messages, never set the priority below 0.

In desktop software, presence priority is based first on user input activity. It is more likely that if a user is active, she will be there to respond to a message, regardless of the presence status that is set in the client software. The actual times may vary based on implementation, but the general idea is to have four base values:

• User input idle less than 1 minute: 100
• User input idle more than 1 minute: 80
• User input idle less than 5 minutes: 60
• User input idle less than 30 minutes: 40

When these idle thresholds are met, presence should be broadcast. The current show value is then applied.

• chat: -4
• none (normal/available): -4
• dnd: -4
• away: -8
• xa: -12

Thus, if a user has been idle for more than 30 minutes and has a show of xa, her priority is 28 (40-12). If a user has been idle for more than 30 minutes and has a show of away her priority is 32 (40-8). By starting at a high number and leaving padding between each show value, we leave room for implementations to add additional rules to adjust priority. Dnd has a higher priority than away and xa because, although the user does not want to be disturbed they are more likely to be at the dnd resource than the away or xa resource. It is up to the client implementation to insure they are not disturbed per the user’s configuration. A show of dnd uses the same priority as none and chat since you are likely equally available, you just don’t want to be disturbed.

Priority and, optionally, show values are adjusted automatically based on common “away from the computer” events: system input idleness (how long since the user interacted with the system), when a session is locked, or a session screen saver is active. When these events occur, the priority rules must be processed again. For Example, my client is configured to go to a show of away after 5 minutes with a status of “AFK”. After 30 minutes it is set to a show value of xa and a status of “AFK for a long time”. When my system is locked or the screen saver comes on, my client automatically adjusts my show value to away and a status of “Not here”. Important: even if an implementation does not change status messages or show values after these “away from the computer” events it must still adjust its priority as defined above.

After the “away from the computer” rules are processed, the relative availability of your peer resources is taken into account. If any peer was more recently more available (had a more available show value than your current value – chat, none/dnd, away, then xa) and has a non-negative priority, subtract 20 from your current priority. For example: I’m logged in at home and work. At home I last announced available presence 4 hours ago. My software at home never puts me into “xa” and instead decides I am “away” after 30 minutes. Home priority is now 32 (40-8). At work I last announced available presence 3 hours ago with a “none” show value. I go to lunch at work but it is configured to reach a show value of xa after 30 minutes. Its priority is now 28 (40-12). However, since work was my last active resource, when my home resource receives the broadcast from work it adjusts itself to a presence priority of 12 (32-20). The purpose of this is to insure that only the most recently available resource above more available than away/xa is the one that takes the higher priority.

It is important to never adjust your priority relative to the priority of a peer resource without taking much care. Doing so can cause a “race” for the highest or lowest priority (depending on the relative direction your adjusting).

That's about it. I also posted this on the XMPP standards mailing list. Reaching a concensus on how to adjust presence priority sure will help user perception of XMPP software.

Validating JabberID Nodes (XMPP/SoapBox User Names)

One of the most common, and tedious, tasks that comes up while writing software is validating user input. "Don't trust the user," the mantra goes. Even with current development tools it is still suprisingly difficult (i.e. not automatic) to validate user input and provide useful feedback to people who enter bad data.

For the web (http/dhtml), ASP.Net provides the best implementation I've seen so far. Controls like the RegularExpressionValidator combined with a ValidationSummary or the more recent Validator Callout allow you to create a very intuitive and reactive interface while still providing server side input validation. Well, how does all this apply to JabberID Nodes?

A JabberID Node is essentially your user name on an XMPP domain. I am [email protected], thus "jconley" is my JabberID Node. A JabberID is represented by the ABNF form: [ node "@" ] domain [ "/" resource ] (for more details see the Addressing Scheme section of RFC 3920). Now, as with any good RFC the exact definition of what is permitted is available. It's, long, it's complicated, it's (eek) Unicode, it's called StringPrep. To be more precise, the NodePrep Profile Prohibited Output. The domain and resource also have their own rules, but maybe we'll talk about that another day.

Up until recently I haven't had to worry much about StringPrep. Chris was the unlucky soul here to bite that one off, so he is officially our in-house Unicode geek. Way down deep in our coversant.corlib assembly of the SoapBox Framework there is a namespace called "Coversant.StringPrep". This is automatically used by our JabberID class to make sure that JabberID's are always in the right format and do not contain any bad data. Well, this last week I had to build a web page that someone could enter JabberID Node information on. I wanted this to be, for the most part, handled on the client side to provide instant feedback and correction for common mistakes (like including a space), but still use the server for final validation (don't want those evil bots trying to mess with my page!).

The most elegant way I could think of was to create a regular expression to validate the input. This would let me utilize the controls already in ASP.Net for client and server side validation. As it turns out this isn't straighforward. From what I could figure out, the Regex engine included with Microsoft.Net (and today's browsers) do not support unicode code points above 16bits. Nodeprep Prohibit says There are thousands of disallowed code points above there (it goes up to 32 bits).

So, in the end, we validate as much as we can on the client with the following expression (I shamelessly stole the ASCII portions from Artur - a guy I met at the XMPP Interop Event last year): ^([\x29\x23-\x25\x28-\x2E\x30-\x39\x3B\x3D\x3F\x41-\x7E\xA0 \u1680\u202F\u205F\u3000\u2000-\u2009\u200A-\u200B\u06DD \u070F\u180E\u200C-\u200D\u2028-\u2029\u0080-\u009F \u2060-\u2063\u206A-\u206F\uFFF9-\uFFFC\uE000-\uF8FF\uFDD0-\uFDEF \uFFFE-\uFFFF\uD800-\uDFFF\uFFF9-\uFFFD\u2FF0-\u2FFB]{1,1023}) Oh, there are some artificial line breaks in that Regex. Don't include those. :) We leave the rest up to a CustomValidator on the server side and the SoapBox JabberID.ValidateUserText method.

SoapBox Platform Possibilities

Our customers do some very interesting things with our platform, including:

• RPC – The asynchronous nature of XMPP along with its addressing and our platform make it ideal for application to application messaging. One of our customers wrote a .NET Remoting transport on SoapBox, and many use it for other custom RPC needs (some are below).
• Network gaming infrastructure – sending game data, hosting chat rooms, private chat during games, etc
• In class test taking – Tests are distribute to Tablet PC’s when students log in and results are tallied.
• Geographical Chat – Whiteboarding combined with maps and group conferencing.
• Financial Market Data – Real time data from the market flows into applications used by traders.
• Social Networking – Consumer social networking site using our platform for chat, content, and advertising delivery to the desktop.
• Emergency Alerting – “The nuclear power plant is melting down. Evacuate!”
• Remote Surveillance Control – Watch your surveillance cameras at home in any web browser and control them.
• Automated Manufacturing Alerting – “Line 5 is clogged. Attention required!”.
• Plain ole’ chat built into an existing application – Give users access to the people they need in the application they are familiar with.
• Build System Alerts – Our build system sends us messages as our daily builds runs, letting us know the status.
• Web Based Live Support – Communicate with customers live, through a web site.
• Voice and Chat on Ruggedized Handhelds - Push to talk through a contact list with presence, send messages, pictures, or go into walkie talkie mode.
• And the list goes on…

Some have been in the news, some are still in "stealth mode", but unfortunately I can't mention any names of companies. We will have some case studies coming out for a few of them.

Some of these applications are built on our 2005 platform, but many are built on our upcoming 2007 release. We've been working hard with our partners to make sure this upcoming release is something special. Here's a little overview of the new SoapBox platform.

SoapBox Framework (now SoapBox Studio)

The SoapBox Framework was our first product offering. We built and productized a framework knowing we’d want to build a server and an advanced communicator client. It started out as one framework, and has grown into quite a bit more. With the 2007 product release we will be distributing our frameworks in one package called the SoapBox Studio. They are all based on the same code base, which allows us to quickly add features to our entire product line. These frameworks include:

• Desktop Edition – Build desktop and web applications in .Net for Windows, Linux, and Solaris.
• Mobile Edition – Build mobile applications that run on PocketPC, Smart Phone, Windows Mobile, or Windows CE operating systems.
• Web Service Edition – This is my favorite. J A standard SOAP web service, built using our Desktop Edition on the back end, that allows you to integrate with any language that supports web services. This includes Java, ColdFusion, Flash, C++, Perl, Php, and more (basically every language out there).
• Server Administration Edition – All of the features that are available in our Management Console are available to you through public API’s. You can quickly perform tasks such as adding users, managing contact lists, and retrieving message archives.
• Server Edition –This allows you to build plug-ins to the SoapBox Server to manage users from your own custom user store, create custom components to service client requests, filter messages, do custom logging, and manipulate the way the server works in general.

SoapBox Server

The SoapBox Server is our flagship XMPP server product. It is based on our Desktop Edition Framework with additional layers to do everything a server needs to do. You can easily customize the Server to meet your specific needs through the SoapBox Framework Server and Administration SDK’s.

SoapBox Communicator

The SoapBox Communicator is our client software built on our SoapBox Framework Desktop Edition (as well as another layer that we will be productizing soon) and serves as an example of how to best utilize the framework for client side development.

Other Key Platform Features

• Open, well normalized database – We use a strict Enterprise nTier model in the SoapBox Server. Our database is well normalized with stored procedures for all interactions. This makes it extremely easy to integrate users, contact lists, message archiving, and presence with any environment that can read or write to a database.
• Shared code base - Whether you’re writing an application for a Smartphone, or a server plug-in to handle workflow in a CRM application, your code interacting with SoapBox will look strikingly similar. This better utilizes your developer resources by focusing them on their business problem, not re-learning another API.
• Unprecedented vertical scalability - SoapBox Server will fully utilize any hardware you can throw at it.

We have been hard at work creating the best documentation and samples out there for any XMPP platform. Here are a couple of (draft) samples.

• Building Custom Packets
• Building a Custom User Manager

If you would like advanced access to the SoapBox 2007 Platform, shoot us an email over at [email protected] and we'll get you on the beta list.

More on Interop

I'm sitting on a plane on my way back to Sacramento (a whopping 1 hour flight) and I thought I'd try to give a little more information about the interop event. After two days of testing it appears as though we're actually very close to having quite a few fully interoperable XMPP implementations. As I mentioned last time, Monday was a cake walk. Unfortunately Tuesday wasn't quite so easy.

Consensus was reached on Monday to start later on Tuesday. We wanted to do 10am, but had reservations at Jive's office to sample beers of the northwest at 5 (yeah they have some good beers) and play some XBox, so we decided to start at 9 instead. The morning session consisted mostly of protocol discussions that were very productive. We decided on the general protocol flow of PEP (Personal Eventing via Pubsub). This protocol addition allow us to create some very innovative and interesting extended presence features (more on this later) in future versions of SoapBox Communicator, and they'll be compatible with features in other clients such as Google Talk, GAIM, etc.

I had some really good Thai food from a street vendor, and I'm still not sick, so that's good. In fact it was some of the best Thai food I've had in a long time, and I eat Thai about once a week.

After lunch we got down to business with more interoperability testing. The goal: mutually authenticated TLS streams between servers per RFC 3920. A certificate authority was created, X.509 Certificates were generated, servers were configured, and then... it didn't work. Unlike the overwhelming success of Monday, Tuesday brought the skeletons out of the closet. We soon realized there were numerous breaking differences between OpenSSL, Java, and .NET based implementations of TLS. When we started, absolutely none of the servers were able to talk to each other over a fully trusted TLS connection. The interesting thing was we could all talk to another instance of our own servers. Hmmm.

After a few hours of hacking and debugging we realized there was significant work that needed to be performed and we didn't have enough time to do it. We were able to get connected with a few of the servers, and vice versa, but there definitely wasn't a Happy Path for all. As a result the server to server TLS specifications in RFC3920 will be clarified, as we eventually reached consensus on what it all really means and how it should be implemented.

In the very near future the JSF will be facilitating ad-hoc interoperability testing over the internet. It will be managing domains (such as soapbox.xmpp.org, google.xmpp.org, etc) where all participating vendors and open source projects will host servers. These will be semi-private domains without open registration, but open to anyone developing XMPP applications that need to test interoperability.

All in all, this was a very successful couple of days. We probably saved a good two months worth of bickering over e-mail lists to figure out protocol issues, verified that XMPP is in fact interoperable, and set the stage for future interoperability testing. We'll also be exploring fully automated tests, which Coversant will likely contribute to the JSF, to make sure everyone continues to play nice in the future. :)

Interoperability - Yup, we got it covered

Today was the first day of the first ever official XMPP Interop Event. In fact, it was probably the first day of any open instant messaging and presence interop event, ever.

We had showings from Coversant (yeah, that's us), Sun, SixApart (LiveJournal), Google, Jive Software, Jabber Inc, Sun Microsystems, and Process-one. Some of those names might sound familiar, and others not, but in the end what we ended up with was seven completely different XMPP server code bases/implementations both open and closed source, setup on a LAN and federating with each other. We spent more time configuring DNS, IP addresses, and other networking junk than we did fixing bugs that were hampering interoperability.

As far as I know (who knows what everyone was hiding on their laptop screens) there were only a few major bugs that were found and they were on two of the very freshest of server implementations, one of which the vendor considers in "pre-alpha" release status. No, I won't tell you who, but I bet they release patches soon. :)

So what the heck did we do all day? Well, we tested interoperability, obviously! But more specifically:

• Inter-domain roster manipulation - Add and remove contacts on other servers.
• Inter-domain messaging - Can we actually hold conversations with each other?
• Inter-domain presence - Avatars, status messages, show states, etc
• pizza
• UTF8 support - Strange unicode characters in other languages in addresses, messages, and presence.
• Discussed quite a few of the enhancement protocols pending for the XMPP specification and came to concensus on some issues.
• Mingled with everyone and shared some anecdotes on implementing XMPP that pretty much only the people in that room would understand.
• beer

What does that mean to you, a loving Coversant customer -- you are a customer, right? Well, it means that you can securely talk through IM in a federated manner to your trading partners, friends, family, and arch enemies even if they aren't one of our customers. Yeah, we support that, and apparently it works. ;)

#1 on the list for tomorrow, trusted TLS based inter-server connections. So we have the joy of setting up a trusted certificate authority and distributing certs. I bet setup will take longer than the interoperability testing itself, again.