XMPP Presence Priority

In XMPP (the messaging and presence protocol SoapBox uses) you can log in simultaneously on multiple systems. How your contat list is displayed and how messages are routed are based on hints the client software provides to the server. One of these is presence priority. Priority is just an integer value between -128 and 127 defined in RFC 3921 Section 2.2.2.3.

Managing priority is fairly straightforward when you control all client implementations used by a bare JID, but in a heterogeneous client environment it is more difficult as each developer chooses his/her own path and only learns by spying on what others do. I think presence priority is something that should be covered in the implementation guide so the actual values become more uniform on the network. We currently do pretty basic priority management, but I have some thoughts on a better implementation I hope others implementations might help refine.

Priority is typically used by both clients and servers to determine which resource is most likely to be the one you want to interact with. This could be for routing messages sent to bare JID’s, or deciding which presence status to display in a contact list. So, after much deep thought and implementation trial/error, here is my proposal.

Unless a user explicitly specifies she does not want to receive any messages, never set the priority below 0.

In desktop software, presence priority is based first on user input activity. It is more likely that if a user is active, she will be there to respond to a message, regardless of the presence status that is set in the client software. The actual times may vary based on implementation, but the general idea is to have four base values:

• User input idle less than 1 minute: 100
• User input idle more than 1 minute: 80
• User input idle less than 5 minutes: 60
• User input idle less than 30 minutes: 40

When these idle thresholds are met, presence should be broadcast. The current show value is then applied.

• chat: -4
• none (normal/available): -4
• dnd: -4
• away: -8
• xa: -12

Thus, if a user has been idle for more than 30 minutes and has a show of xa, her priority is 28 (40-12). If a user has been idle for more than 30 minutes and has a show of away her priority is 32 (40-8). By starting at a high number and leaving padding between each show value, we leave room for implementations to add additional rules to adjust priority. Dnd has a higher priority than away and xa because, although the user does not want to be disturbed they are more likely to be at the dnd resource than the away or xa resource. It is up to the client implementation to insure they are not disturbed per the user’s configuration. A show of dnd uses the same priority as none and chat since you are likely equally available, you just don’t want to be disturbed.

Priority and, optionally, show values are adjusted automatically based on common “away from the computer” events: system input idleness (how long since the user interacted with the system), when a session is locked, or a session screen saver is active. When these events occur, the priority rules must be processed again. For Example, my client is configured to go to a show of away after 5 minutes with a status of “AFK”. After 30 minutes it is set to a show value of xa and a status of “AFK for a long time”. When my system is locked or the screen saver comes on, my client automatically adjusts my show value to away and a status of “Not here”. Important: even if an implementation does not change status messages or show values after these “away from the computer” events it must still adjust its priority as defined above.

After the “away from the computer” rules are processed, the relative availability of your peer resources is taken into account. If any peer was more recently more available (had a more available show value than your current value – chat, none/dnd, away, then xa) and has a non-negative priority, subtract 20 from your current priority. For example: I’m logged in at home and work. At home I last announced available presence 4 hours ago. My software at home never puts me into “xa” and instead decides I am “away” after 30 minutes. Home priority is now 32 (40-8). At work I last announced available presence 3 hours ago with a “none” show value. I go to lunch at work but it is configured to reach a show value of xa after 30 minutes. Its priority is now 28 (40-12). However, since work was my last active resource, when my home resource receives the broadcast from work it adjusts itself to a presence priority of 12 (32-20). The purpose of this is to insure that only the most recently available resource above more available than away/xa is the one that takes the higher priority.

It is important to never adjust your priority relative to the priority of a peer resource without taking much care. Doing so can cause a “race” for the highest or lowest priority (depending on the relative direction your adjusting).

That's about it. I also posted this on the XMPP standards mailing list. Reaching a concensus on how to adjust presence priority sure will help user perception of XMPP software.

Building Reactive User Interfaces in .NET: ISynchronizeInvoke on Idle Time

One of the most common things to do in a multi-threaded .NET Windows Forms application is to use the ISynchronizeInvoke interface on a Control to marshal things into the UI thread. The basic jist of things is: you can have the Windows Forms engine call your delegate on the thread that created the Control instance you're using. It does this using the window message pump. Typical uses of the ISynchronizeInvoke are: you want to read some data from a file, database, socket, or web service and you want your application to be responsive while you do it. When you're done you do a BeginInvoke/Invoke on the Control to update some UI elements. Well, sometimes using a Control's ISynchronizeInvoke leads to hung GUI's.

If you have a lot of asynchronous operations pending and they complete in bunches the affect on your UI can be the appearance of a hang or sluggishness as drawing is queued behind the delegates you registered with BeginInvoke on your control. During the login sequeunce in SoapBox Communicator there are potentially thousands of events that need to be processed on the UI thread. Yup, thousands. Your roster arrives from the server. You receive the current avialability from every online person on your roster. You got 10 messages while you were offline. Each of your contact's cached profiles (avatars, names, client capabilities, etc) are read from disk and compared with the user's current presence. This onslought of activity can be handled a number of ways. The basic principles I stumbled into after much trial and error is:

1. Never, ever, ever, ever, ever do IO on a UI thread. Even something as simple as reading a 1KB image from disk can bring your UI to a halt under the right circumstances.
2. Use timers and update common pieces of UI (like list views) in batches whose changes were caused changes by background operations. This will reduce flicker as you invalidate areas to redraw.
3. Use Application.Idle to your advantage. (see my take on this below)
4. Be careful how many window handles you create. Controls are useful, but sometimes you've just gotta draw your own.
5. Profile your code where it seems sluggish.

I mentioned above that you should use Application.Idle to your advantage. The articles I've found on it all mention coding precise things you want to do in that event, like updating a single form. I wanted to do it more generically. So, I created an implementation of ISynchronizeInvoke that uses the Application.Idle event to process qeued items. I've created a (only slightly) contrived example that uses a Pi Calculator I found and an animated Gif to demonstrate a hanging GUI.

Both buttons calculate Pi to 50 digits on the UI thread using 100 separate ISynchronizeInvoke.BeginInvoke calls. The "Idle Invoke" button does so using my ApplicationIdleSynchronizer. The "Direct Invoke" button calls BeginInvoke on the Form directly. You can click the buttons any number of times and more and more Pi calculation runs will be queued. The completed label is incremented after every run. Here's the code:

    public partial class Form1 : Form
 {
     private const int WorkItems = 100;
     private const int PiDigitsToCalc = 50;

     private int _workItemsCompleted = 0;

     private ApplicationIdleSynchronizer _idleSynchronizer = new ApplicationIdleSynchronizer();

     public Form1()
     {
         InitializeComponent();
     }

     protected override void OnClosed(EventArgs e)
     {
         base.OnClosed(e);
         _idleSynchronizer.Dispose();
     }

     private void button1_Click(object sender, EventArgs e)
     {
         System.Threading.ThreadPool.QueueUserWorkItem(QueueOnBackgroundThread, _idleSynchronizer);
     }

     private void button2_Click(object sender, EventArgs e)
     {
         System.Threading.ThreadPool.QueueUserWorkItem(QueueOnBackgroundThread, this);
     }

     private void QueueOnBackgroundThread(object state)
     {
         for (int i = 0; i < WorkItems; i  )
         {
             ((ISynchronizeInvoke)state).BeginInvoke(new ThreadStart(MyWorkItem), null);
         }
     }

     private void MyWorkItem()
     {
         PiCalculator.CalculatePi(PiDigitsToCalc);
         _workItemsCompleted++;
         label2.Text = _workItemsCompleted.ToString();
     }
 }

Click the buttons. Move the form around. Resize it. You might be suprised by the result (or not). Both very adequately use your CPU, but the Idle Invoke produces a much more reactive UI. You can download the full source code here: ApplicationIdleInvoker.zip. Note, this exact code isn't in use in production. I wrote it for this blog. YMMV. Let me know if it has any issues.

YouTube Getter

Usually everything I post here is serious business. Well, not today! Today, we're going to look at how to get the raw video stream for YouTube videos. Why, you ask? Well, curiosity, mostly. :) Some people like sports, I like problem solving. There are a number of browser plug-ins and web sites that do this already, but hey, it was still fun.

YouTube offers a bit of an API for developers to mess with. It's mostly for grabbing sets of videos and preview images and such. They also let you embed their player in your pages. That's awfully nice of them, but what if I don't like their player? What if I want a sexier player? Well, if you've got something that can play/convert Flash Video (FLV's), here's the full C# code to grab the FLV URI (pieces dissected below):

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
using System.Web;
using System.Collections.Specialized;

namespace JDFun
{
  public static class YouTubeGetter
  {
      public static Uri GetFlvUri(string viewUri)
      {
          return GetFlvUri(new Uri(viewUri));
      }

      public static Uri GetWatchUri(Uri flvUri)
      {
          NameValueCollection qry = HttpUtility.ParseQueryString(flvUri.Query);

          string videoID = qry["video_id"];
          string watchUri = string.Concat("http://www.youtube.com/watch?v=", HttpUtility.UrlEncode(videoID));

          return new Uri(watchUri);
      }

      public static Uri GetImageUri(Uri flvUri)
      {
          NameValueCollection qry = HttpUtility.ParseQueryString(flvUri.Query);

          string imageUri = qry["iurl"];

          return new Uri(imageUri);
      }

      public static Uri GetFlvUri(Uri viewUri)
      {
          // so either i've got the embed link or the watch link

          //watch link: http://www.youtube.com/watch?v=up-RX_YN7yA
          //embed link: http://www.youtube.com/v/up-RX_YN7yA

          string toQuery = null;

          NameValueCollection queryString = HttpUtility.ParseQueryString(viewUri.Query);

          string videoId = queryString["v"];

          if (null != videoId)
          {
              toQuery = string.Concat("http://www.youtube.com/v/", videoId);
          }
          else
          {
              toQuery = viewUri.ToString();
          }

          if (null == toQuery)
              throw new InvalidOperationException("Not a valid YouTube Uri.");

          Uri queryUri = new Uri(toQuery);
          //ok we have the uri to query, now go there and get redirected.
          HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(queryUri);
          req.AllowAutoRedirect = true;

          // make them think we're coming from a direct link
          req.Referer = string.Empty;

          // firefox rules!
          req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3";
          using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
          {
              string absoluteRedirectLocation = resp.ResponseUri.AbsoluteUri;

              //replace the swf with the get_video request
              if (!absoluteRedirectLocation.Contains("/p.swf?"))
                  throw new InvalidOperationException("Unrecognized Uri. YouTube changed something.");

              absoluteRedirectLocation = absoluteRedirectLocation.Replace("/p.swf?", "/get_video?");

              //return the absolute URI for this request
              return new Uri(absoluteRedirectLocation);
          }
      }
  }
}

Ok, wow, that's a lot bigger than I remember. Here's the basic idea.

1. Get the video ID from the given URI. The URI could be the "watch" URI, or it could be the URI from the embed code they give you.
2. Load the video directly based on the video ID, at the embed URI.
3. Make a HTTP Request and get redirected to another URI, (which happens to contain the Image URI, and a special token "t").
4. Know the magic URI where they keep the video streams (I cheated) and use it.

Now wasn't that fun? :)

Validating JabberID Nodes (XMPP/SoapBox User Names)

One of the most common, and tedious, tasks that comes up while writing software is validating user input. "Don't trust the user," the mantra goes. Even with current development tools it is still suprisingly difficult (i.e. not automatic) to validate user input and provide useful feedback to people who enter bad data.

For the web (http/dhtml), ASP.Net provides the best implementation I've seen so far. Controls like the RegularExpressionValidator combined with a ValidationSummary or the more recent Validator Callout allow you to create a very intuitive and reactive interface while still providing server side input validation. Well, how does all this apply to JabberID Nodes?

A JabberID Node is essentially your user name on an XMPP domain. I am [email protected], thus "jconley" is my JabberID Node. A JabberID is represented by the ABNF form: [ node "@" ] domain [ "/" resource ] (for more details see the Addressing Scheme section of RFC 3920). Now, as with any good RFC the exact definition of what is permitted is available. It's, long, it's complicated, it's (eek) Unicode, it's called StringPrep. To be more precise, the NodePrep Profile Prohibited Output. The domain and resource also have their own rules, but maybe we'll talk about that another day.

Up until recently I haven't had to worry much about StringPrep. Chris was the unlucky soul here to bite that one off, so he is officially our in-house Unicode geek. Way down deep in our coversant.corlib assembly of the SoapBox Framework there is a namespace called "Coversant.StringPrep". This is automatically used by our JabberID class to make sure that JabberID's are always in the right format and do not contain any bad data. Well, this last week I had to build a web page that someone could enter JabberID Node information on. I wanted this to be, for the most part, handled on the client side to provide instant feedback and correction for common mistakes (like including a space), but still use the server for final validation (don't want those evil bots trying to mess with my page!).

The most elegant way I could think of was to create a regular expression to validate the input. This would let me utilize the controls already in ASP.Net for client and server side validation. As it turns out this isn't straighforward. From what I could figure out, the Regex engine included with Microsoft.Net (and today's browsers) do not support unicode code points above 16bits. Nodeprep Prohibit says There are thousands of disallowed code points above there (it goes up to 32 bits).

So, in the end, we validate as much as we can on the client with the following expression (I shamelessly stole the ASCII portions from Artur - a guy I met at the XMPP Interop Event last year): ^([\x29\x23-\x25\x28-\x2E\x30-\x39\x3B\x3D\x3F\x41-\x7E\xA0 \u1680\u202F\u205F\u3000\u2000-\u2009\u200A-\u200B\u06DD \u070F\u180E\u200C-\u200D\u2028-\u2029\u0080-\u009F \u2060-\u2063\u206A-\u206F\uFFF9-\uFFFC\uE000-\uF8FF\uFDD0-\uFDEF \uFFFE-\uFFFF\uD800-\uDFFF\uFFF9-\uFFFD\u2FF0-\u2FFB]{1,1023}) Oh, there are some artificial line breaks in that Regex. Don't include those. :) We leave the rest up to a CustomValidator on the server side and the SoapBox JabberID.ValidateUserText method.

Installing Coversant Products on Vista

Due to the enhanced security in Windows Vista, not all Coversant products are able to be installed out of the box. Luckily, this is really easy to work around and rest assured, future version of our installation packages will not suffer from these issues.

The symptoms show up as an error message dialog with code 2869:

And then a series of empty dialog boxes:

This is apparently some sort of permissions issue. To solve it, the MSI needs to be run as an administrator. The easiest way to do this is to create a bat file to run the msi manually. It would be something like this:

msiexec.exe /i "c:\SoapBoxServer2007\files\SoapBox.Server.Enterprise.x64.3.0.213.69.msi"

Then you right click on the bat file and choose "Run As Administrator". Presto, a working installation in Vista.

XSLT For MSDN Product Keys

Here at Coversant we're Microsoft partners. We have MSDN subscriptions for all our developers/testers, and we share the same set of license keys. Rather than give everyone willy-nilly access to the MSDN download web site (ick, lots of bandwidth suck) we setup an internal file share for MSDN installation files, CD images, etc. We used to have all the product keys in there just saved as html from Microsoft's web site. However, that's no fun!

In the spirit of having fun and being developer friendly, Microsoft is nice enough to offer an "Export Key List to XML" button on the web page where we view all our product keys. So, I clicked the button. Out popped a very nicely/simply formatted XML document, like the following:

<?xml version="1.0" standalone="yes"?>
<Your_Product_Keys>
 <Product_Key
  Name="All products requiring a 10-digit product key"
  Key="xxx-xxx-xxxx"
  Key_Type="Retail"/>
 <Product_Key
  Name="Windows Vista Ultimate"
  Key="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX"
  Key_Type="Retail"
  Date_Key_Claimed="2006-11-20 17:36:11.787"/>
 <Product_Key
  Name="Windows Server 2003 R2 Standard Edition"
  Key="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX"
  Key_Type="Retail"
  Date_Key_Claimed="2006-11-20 17:36:08.270"/>
 <Product_Key
  Name="Office 2007 Desktop Programs"
  Key="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX"
  Key_Type="Retail"
  Date_Key_Claimed="2006-11-20 17:35:57.537"/>

  ...

</Your_Product_Keys>

I whipped up a quick XSLT you might find useful to transform this to a really ugly, really simple, x-html page (yeah without the namespace declaration).

<?xml version="1.0" encoding="ISO-8859-1"?>

<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="/">
 <html>
 <body>
   <h2>MSDN Product Keys</h2>
   <table border="1">
   <tr bgcolor="#9acd32">
     <th align="left">Product Name</th>
     <th align="left">Product Key</th>
   </tr>
   <xsl:for-each select="/Your_Product_Keys/Product_Key">
   <tr>
     <td><xsl:value-of select="@Name"/></td>
     <td><xsl:value-of select="@Key"/></td>
   </tr>
   </xsl:for-each>
   </table>
 </body>
 </html>
</xsl:template>

</xsl:stylesheet>

Then I made one quick edit to the xml file so friendly xslt aware web browsers will do the transform for me:

<?xml version="1.0" standalone="yes"?>
<?xml-stylesheet type="text/xsl" href="productkeys.xslt"?>
<Your_Product_Keys>
  ...
</Your_Product_Keys>

Presto-chango, we now have an easy to update central listing of our MSDN keys. Now if only MS had a web service that took my Live ID credentials and gave me back that XML...

Compact Framework WaitHandle.WaitOne Gotcha

I ran into a behavior in the 2.0 Compact Framework today that was most vexing. It wasn't hard to find like a subtle race condition. It wasn't an issue that only duplicated with a certain system configuration, under a full moon, on Wednesday. No, it duplicated every single time the code was ran. But, it wasn't documented anywhere I could find.

One of my favorite new features in the Compact Framework is the availability of the WaitHandle.WaitOne(int, bool) overload. That's something we use quite a bit in our test code and here and there in the actual SoapBox Framework. We used to have our own ManualResetEvent implementation for the Compact Framework that P/Invoked out to Windows CE. But Micrsoft was nice enough to add this into the 2.0 Framework. Yay! (In case you never had the joy of programming to the .NET 1.0 framework, the only WaitOne overload that was there was the indefinitely blocking one. No timeouts allowed.)

I was running our unit test suite against the Compact Framework and tests that used to pass until I ripped out our custom P/Invoking ManualResetEvent implementation were failing. Odd... Well, it turned out to be very easy to track down. WaitHandle.WaitOne(int, true) throws an ArgumentException every single time. That's right. If you pass true to that second parameter, the exception is thrown.

Don't get me wrong, I understand the implications of exiting the synchronization domain for the context or not. It turns out the code that was causing the error should have been passing "false" for the parameter anyway. But, why did the call throw an exception? Why not just ignore the argument when it's not relevant as the documentation alludes? And I quote from MSDN2: "The exitContext parameter has no effect unless the WaitOne method is called from inside a nondefault managed context."

Anyway, if you're doing any Compact Framework development, make sure to pass "false" into the exitContext parameter of your WaitHandle.WaitOne calls.