Friday, December 14, 2007

Unadulterated Expurgation!

Alright, so, occasionally I post something funny that happened recently. Yesterday I received a spam company broadcast email from an anonymous customer of mine. You know who you are! :)

Team Anonymous Company,

In each office Anonymous Company employees have inhabited, many attempts have been made to keep our refrigerators clean and sanitary. Keeping clean refrigerators only works if we all participate – yet despite the best team players on the planet, we’ve fallen short of the goal. Common comments run the gamut from, "I’ll never put 'my' lunch in there," to the more lyrical, "There’s a fungus among us!"

As a result of this circumstance, the many must suffer the failings of a few – meaning:

Every Friday afternoon at 4:00 p.m. our Sergeant of Sanitation, will remove and dispose of everything from every refrigerator – as in: any one thing, each thing, the sum of all things, collectively all, the aggregate of all, anything at all, the totality of all within, including or comprehending all things, a comprehensive sweep of any and all things, whatever it is, whatever is left, nothing will be left, the sum total of all things, the Full Monty, the whole enchilada (frozen and unfrozen), the whole shebang. In short, every Friday at 4:00 p.m. all the refrigerators will experience an unadulterated expurgation!

Many thanks in advance for your cooperation and understanding,

The Management

Good thing I don't use the fridge...

Tuesday, December 11, 2007

Introduction to fbasync: an Asynchronous Facebook Library (Part 1)

I'd like to start off by saying asynchronous programming is easy. Building asynchronous frameworks is hard, but using them is easy. Everybody doing programming in AJAX and Flash do it every day without thinking about it. There's no reason us C# .NET programmers can't do the same!

This first article about fbasync is going to show how to use it. Your 12 year old script kiddie nephew could do it, I promise. Then later on in the series we'll dive into how the library is implemented and see if we can forever scare him away from a programming career.

Those who know me well (or have read my blog for a while) know I'm a proponent of asynchronous patterns. Why? Well, there are a lot of reasons. I've mentioned why 1 2 3 4 5 6 times. To sum it up, the two primary reasons for using asynchronous patterns are to gain better vertical scalability (to do more with one piece of hardware) and provide a better user experience. When it's as easy as writing synchronous code, I say why not? Using asynchronous patterns in your code is especially important when it comes to communicating with a web service outside of your network (like Facebook). When you don't know how long your service calls are going to take, you simply can't expect to scale up an application without going asynchronous!

In a previous article entitled Simpler Isn't Always Better I spent a lot of time bashing the event based asynchronous pattern. Well, I take it back (some of it). It turns out in the context of ASP.NET and Winforms, the event based pattern is very nice indeed. While I implemented both the event based and IAsyncResult pattern in fbasync, I'd recommend most people use the event based interface. This is what we'll be looking at today.

What you've been waiting for

Ok, enough blabbering! The fbasync library is available on CodePlex. You can download the bits right now and follow along at home. The library comes with a simple example that allows you to browse the photo albums uploaded by you and your friends. In this sample we get all the albums and related cover photos of the requested user, and the list of friends for the authenticated user on the server through asynchronous requests. Let's get it setup and take a deeper look!

  1. Make sure you have Visual Studio 2008 installed. It's required.
  2. Download and extract the zip file somewhere interesting.
  3. Open the fbasync.sln file.

Now that you have the web site loaded just make sure your web server is setup to run on port 30912 and the virtual path is "/" (in your project properties). This is the way the sample application is configured in Facebook. If you have your own application (API Key/Secret) you want to test this with, edit the web.config.

Project Setup

With this setup, you'll be able to run the fbasync.web application, add it through Facebook, and experiment with it in the Facebook iframe.

Setting up your page

Getting your Facebook application setup and working is outside the scope of this article. Check out the Facebook Developer site for introduction information. Once you decide you want a Facebook canvas page with an iframe application, return here. :)

It's quite easy to get your page connected to Facebook. Simply inherit from the fbasync.webcomponents.FacebookPage. Once we hit the 1.0 release of the library, you'll also be able to drag a component onto an existing page without messing with your inheritance hierarchy. For now, here's how it looks.

Once you've got your page setup, using the library is simple. The base page takes care of making sure the request is authenticated and gets a Facebook session based on the authentication token provided to your application. Before we go any further, make sure you set the Async directive on your page so fbasync can do things asynchronously. How async pages work was explained well by Fritz Onion in an Extreme ASP.NET article last year.

Now that you have setup the page to process requests asynchronously, fbasync can best do its magic. Override the OnFacebookSessionCreated method in your page, or add a handler for the FacebookSessionCreated event. This will be raised when fbasync has acquired a session from Facebook, using either the authentication token provided in the Facebook iframe or after a redirect through the Facebook login page. The Facebook authentication process is covered in depth in the Facebook Developer Authentication Overview. The good news is, you probably don't need to know! Fbasync will handle the most common situation for you.

Now that you've got a Facebook session, it's time to do something interesting. Let's take a look at the code in our AlbumBrowse page.

There are two interesting bits in this code (they're in giant red boxes – hard to miss). First, we setup event handlers for the *Completed events we are interested in. Second, we call the *Async methods. Notice in this example we're doing three simultaneous asynchronous requests. That's important. These requests will happen simultaneously. That, is the power of asynchronous code in action. Imagine each call takes 250ms to complete. With three calls, that's nearly 750ms of time just waiting around for Facebook. Using the asynchronous pages we end up waiting only as long as it takes for the longest running call to complete!

At this point astute readers will point out that I am in fact lying. In the default configuration of .NET, you will only ever make two simultaneous calls to a given remote IP. This is built-in to the WebRequest class. Lucky for us, we can change it in the web.config like so!

<add address="*" maxconnection="100" />

With this configuration entry in place we can now make up to 100 simultaneous outgoing requests to the same IP address. Much better. Even if you're not using fbasync, you should make a similar change.

Back to the example. Making asynchronous requests is interesting, but requests don't usually do us much good without dealing with the results. The list of friends on the left side of the application is generated using a ListView control and some simple codebehind. I'm going to leave the HTML markup out of it, as its dirt simple. Have a look at the code if interested. Here's the codebehind that renders the user list.

Surprised? There are no red highlights. There's nothing very interesting here. If you haven't seen many lambda functions or LINQ before, the OrderBy and Single methods might look funny, but the code here is pretty darn simple. We remove our event handler (not doing this is prone to memory leaks so I always do it). Then we bind the ListView to the Result on the event argument. And finally we put the friends name who's albums we are viewing in the placeholder. Yup, that's it. No thread locking, no monitor, no mutex, no volatiles, no interlocks, none of that scary threading stuff at all.

Let's review what happened. Our page loaded. The fbasync library got a session from Facebook (or pulled one out of the cache/session – more on this in a future post). Our GetUsersAsync method was kicked off (among others). It finished, in a background thread, and raised the GetUsersCompleted event back in the context of our ASP.NET request. We took the results and bound them to a list view. No muss, no fuss. No threading required. My grandmother could do it – well, if she knew ASP.NET.

This is getting too long already. I'm starting to feel like I'm writing a Scott Guthrie article. Coming soon: we'll wrap up the rest of the example code, and move onto how the *Async methods are implemented. I'd love to get some more contributors on this project. So, if you're interested, give me a buzz or head on over to CodePlex and get crackin! Feedback is welcome and encouraged.

Thursday, December 6, 2007

Geeks With Kids

I was just IMing with a good geek friend of mine. This is what happens when you give paranoid geeks kids (names removed for obvious reasons).

(5:08:06 PM) Geek Friend: i'm outta here, i gotta go home,
looks like the kids are fighting
(5:08:15 PM) Geek Friend: no adult supervision
(5:08:20 PM) Me: haha
(5:08:26 PM) Me: you watching them on camera?
(5:08:27 PM) Geek Friend: i'm so glad my parents didn't have
video cameras all over my house when i was their age
(5:08:30 PM) Me: hahahha
(5:08:34 PM) Geek Friend: watching, listening
(5:08:39 PM) Me: and they know?
(5:08:42 PM) Geek Friend: going to go ground two of them
(5:08:44 PM) Geek Friend: yeah, they know
(5:08:47 PM) Me: wow
(5:08:56 PM) Me: and they still act up
(5:08:56 PM) Me: crazy
(5:09:04 PM) Geek Friend: kid0 was straight up punching kid1
just now
(5:09:08 PM) Geek Friend: elbowed him in the face
(5:09:17 PM) Geek Friend: tried to gouge his eyes with her
(5:09:17 PM) Me: nice
(5:09:19 PM) Geek Friend: crazy fight
(5:09:25 PM) Geek Friend: never seen them do anything like
that before
(5:09:30 PM) Me: well get goin!
(5:09:33 PM) Me: before one of them dies
(5:09:38 PM) Geek Friend: i called them
(5:09:42 PM) Me: ah
(5:09:43 PM) Geek Friend: they're sitting in the corners
of the room
(5:09:46 PM) Me: lol
(5:09:47 PM) Geek Friend: and can't move
(5:10:05 PM) Geek Friend: i turned on the alarm, i told
them if they move, the alarm will go off and the police
will come
(5:10:21 PM) Geek Friend: (not true though) just a really
loud noise until i shut it off
(5:10:49 PM) Geek Friend: later

Wednesday, December 5, 2007

Async Facebook Library Teaser

I've been trolling the Facebook Developer Forums recently and talking with other developers. I've also been reading a lot and trying to get a feel for what people might want out of the asynchronous Facebook library I've been (slowly) building. In particular, I've been participating in this thread about Facebook scalability issues. I thought I would share my most recent post as a bit of a teaser as what's to come. Here it is, reproduced:

tomten wrote:

...but it was way overkill for what I've done. ...

This is one myth I want to squash. Async stuff isn't hard. I promise. smile Let's take your page loading example, and make it asynchronous. I put in code comments for a "facebook is hammered" timing. Let's say every call takes you 250ms. That means that page will spend roughly one second waiting for facebook.

Page_Load {
GetSession(); // Call to Facebook 250ms
// Make calls to Facebook
GetFriends(); // Call to Facebook 250ms
UpdateProfile(); // Call to Facebook 250ms
SendNotification(); // Call to Facebook 250ms

Now, let's make this Async. It would become (roughly -- there is also a "Completed" event handler you'd setup for the items you cared about):

Page_Load {

Page_PreRenderCompelete {

This time, with the async calls in the server, we spend 0 time waiting for facebook. Just like in client side AJAX, we get a callback when the calls were complete! And, instead of a full second for our page to finish processing, it only takes 250ms, as all the facebook calls happen simultaneously (yeah I know this is a faulty assumption, but it is usually pretty close).

In addition to the shorter page lifecycle (250ms instead of 1 second) we are not tying up a processing thread (of which there is a small finite number). During that time we are waiting for facebook to finish our API calls, the engine happily processes more requests on the thread that originally kicked off processing of our page.

A shorter lifecycle is important, not only for throughput, but also for memory consumption and garbage collection efficiency. Because the objects do not live for as long, they will be garbage collected more quickly and your application will spend less time doing garbage collection.

Ok, so that's a bit of a spoiler, but, I hope it helps.

Hopefully some time this week I'll clean up the library enough that I think it's worthy to be posted publicly. I spent last night refactoring a bit and removing a lot of "who wrote this crap" code. :)

As a bit of an aside, I was excited to see the team working on the "almsot official" Microsoft .NET library for Facebook released an asynchronous version of the FacebookService. I thought maybe I wouldn't have to write one myself. However, after looking at the code, I was very disappointed by the fake asynchronous support that's in the Facebook Developer Toolkit. It made me shed a tear. I really wish they'd put a disclaimer up: "This AsyncFacebookService is not really asynchronous I/O. It will only make things worse by tying up an additional threadpool thread waiting for the delegate to the synchronous method to return.".

Friday, November 30, 2007

An Entrepreneurial Story

I don't remember how exactly I got there, but while clicking through my morning feeds I ended up at the very interesting story of an Australian .NET micro ISV startup called Shuffle Text. They outline pretty much everything we learned when we started Coversant nearly seven years ago.

Some Gems
"And while we believed that technical differentiation would be enough to sell our product, we ended up being dead wrong."
"Change, it turned out, ended up being one of our worst development enemies."
"Of all those Google Ads signups, we got 0 feedback. All feedback on the product beta came from friends or friends of friends."
"There’s a bajillion ISV’s out there all competing to sell their crappy untested memory leaking piece of junk software. No matter how much time you spend perfecting your algorithm, improving the performance, and making sure you’ve got zero tolerance for leaks, NOBODY CARES. At the end of the day, it’s getting noticed that matters, convincing people that they should take a bet on you to solve their problems."

Don't give up guys. The product's been out less than a month and people are blogging about you! Your marketing seems pretty solid to me. While I don't have a use for your software at this very second. I'll certainly keep it on my radar. Oh yeah, consider putting out a completely free version, as in beer, not a trial.

Wednesday, November 28, 2007

Linq to Sql Surprise Performance Hit

A couple days ago I built Photo Feeds. I wrote about it here. After using the app a bit I noticed something peculiar. The page took a really long time to load. From my local web server it was on the order of 2-5 seconds. That's way too slow.

Having written the code a few hours prior, I immediately knew what was wrong. I thought "Well, duh, either Facebook sucks, I'm hitting my database too often, or I'm calling the Facebook service too often." I/O is, after all, usually what causes an application this simple to be slow. I glanced over the code and thought to myself "Holy crap, who wrote this junk? It's calling the database waaaaay too much. That has to be the problem."

Having been down this path many times I decided to analyze a bit more before officially jumping to conclusions. I popped open task manager to see if I was in fact I/O bound (i.e. to see if my CPU was racing or idle) – usually a good rough indicator of what's going on. To my jaw-dropped amazement the CPU was hitting 100% for the majority of the page request time. "Could it be the JIT?" I thought. So I hit refresh 50 times and watched in amazement as the CPU spiked each time. I switched over to the Processes tab and, again to my amazement, noticed it was the web server chewing up the cycles, and NOT SQL server. Did I mention I was amazed?

Ok, so apparently I was wrong [insert shock and awe]. The web site was way too CPU hungry! Good thing I didn't put any bets on that one. I then theorized that maybe Cassini just sucked (my code couldn't be that bad). I whipped up a virtual directory in IIS and ran my tests again.

Nope, it wasn't Cassini. Alright, so now let's get scientific. I pop open ANTS and setup a profiling session on my virtual directory. I loaded the page a few times and, lo and behold, the worst offender screen pops up:

Wow! That's a long time (and a lot of hits). I was definitely right. I'm hitting the database way too often. But, I know I saw the CPU racing... What the heck?

Here's the offending code:

public static Feed GetOrCreateFeed(
FeedsDataContext dc,
string owner,
FeedSourceType sourceType,
string facebookId)
var q = from f in dc.Feeds
where f.OwnerId == owner &&
f.FeedSourceType == (byte)sourceType &&
f.FacebookId == facebookId
select f;

Feed existingItem = q.SingleOrDefault();


So I dug into the call graph a bit and found out the code causing by far the most damage was the creation of the LINQ query object for every call! The actual round trip to the database paled in comparison. Now that was, again, a huge surprise. Check out the hit counts on this call – holy cow!

1,176,879 calls to SqlVisitor.Visit. Over a million calls to anything for loading a page three times can't be good.

I started doing some research. I remembered reading about using compiled LINQ to SQL queries for optimization a few months ago. I did a little more searching and ran into this gem for anybody scared of lambda functions. Warning, scary .NET 3.5 lambda functions ahead. Here's the magic fix:

private static readonly Func<
FeedsDataContext, string, FeedSourceType,
string, IQueryable<Feed>> _compiledGet =
(FeedsDataContext dc, string owner,
FeedSourceType sourceType, string facebookId) =>
from f in dc.Feeds
where f.OwnerId == owner &&
f.FeedSourceType == (byte)sourceType &&
f.FacebookId == facebookId
select f);

public static Feed GetOrCreateFeed(
FeedsDataContext dc,
string owner,
FeedSourceType sourceType,
string facebookId)
Feed existingItem = _compiledGet(
dc, owner, sourceType, facebookId)


If that's your first time seeing lambdas in C#, I feel for you. I'd suggest trying them out and doing a lot of research. That simple change yielded the following result:

That method is still by far the worst offender in the application, but is five times faster than it was and barely registers on the CPU when the page loads. This one change netted sub second page rendering times which is good enough for me. Obviously I still have major issues, hitting the database way too often, but I'm a big believer in not doing any premature optimization. But now, at least I'm back in my comfort zone and am I/O bound.

So, what have we learned? Never, ever, ever, ever, ever, ever, ever, ever, ever blindly edit code to fix performance issues. Always run it through a profiler. No matter how much experience you have performance tuning, the worst offenders will still surprise you 99% of the time. Oh, and sometimes LINQ to SQL is slow at creating its query objects. Use compiled queries in these cases!

Tuesday, November 27, 2007

How much space are all my sql tables using?

I deal with a few dozen databases on a daily basis. Often times I wonder "What the heck is making this database 100GB?" Sure, you can click around in Management Studio and find figures on a table by table and index by index basis, but there has to be a better way!

So, I asked a colleague of mine – who happens to be the guy that did all the database work on the SoapBox products and the best database dude I know. He didn't have a good answer for me so he whipped up this tiny script [Edit: After reading my blog he told me he actually adapted the script from one he found on the net that didn't quite work] in a few minutes. It makes my head spin. It's not perfect, but it's darn close.

WITH table_space_usage
( schema_name, table_name, index_name, used, reserved, ind_rows, tbl_rows )
AS (
, o.Name
, coalesce(i.Name, 'HEAP')
, p.used_page_count * 8
, p.reserved_page_count * 8
, p.row_count
, case when i.index_id in ( 0, 1 ) then p.row_count else 0 end
FROM sys.dm_db_partition_stats p
INNER JOIN sys.objects as o
ON o.object_id = p.object_id
INNER JOIN sys.schemas as s
ON s.schema_id = o.schema_id
LEFT OUTER JOIN sys.indexes as i
on i.object_id = p.object_id and i.index_id = p.index_id
WHERE o.type_desc = 'USER_TABLE'
and o.is_ms_shipped = 0
SELECT t.schema_name
, t.table_name
, t.index_name
, sum(t.used) as used_in_kb
, sum(t.reserved) as reserved_in_kb
, case grouping(t.index_name)
when 0 then sum(t.ind_rows)
else sum(t.tbl_rows) end as rows
FROM table_space_usage as t
, t.table_name
, t.index_name
, t.schema_name
, grouping(t.table_name)
, t.table_name
, grouping(t.index_name)
, t.index_name

I hope that hurts your head as much as it does mine. Just goes to show that I'm no SQL guru, I guess. But hey, if you ever want to know where that space is going in your database, this script will tell you!

Photo Feeds Facebook Application

I've built quite a few applications for Facebook. These include SoapBox, Friend Photosaver, Photozap, and my latest creation (late last night) is called Photo Feeds.

Photo feeds, as of the written-in-12-hours v1.0, automatically creates RSS feeds for the photos of you and your friends. You can subscribe to these feeds using anything that supports RSS photos. Your news reader, your favorite photo screen saver, windows vista sidebar, google sidebar, digital photo frames, etc, etc.

In the near future I hope to make this more compelling with new features including adding deeper integration into your Facebook profile and publishing to the Facebook news feed. In addition, I'll be integrating the three photo products (Friend Photosaver, Photozap, and Photo Feeds) to provide a rich add-on suite to Facebook Photos aimed at getting your photos to your desktop outside of Facebook.

Of course, this latest application was written using .NET Framework 3.5, Visual Studio 2008, and my top secret (though soon open source) asynchronous Facebook library. If you're a .NET developer interested in writing Facebook apps let me know and I'll be sure to blog about the technical details in the future. There doesn't seem to be many of us in the Facebook world.

Monday, November 26, 2007

FileVersionInfo - My Long Lost Friend

I've been doing .NET development professionally since the first public beta release of the 1.0 framework. That's... well, a long time. It constantly amazes me how big the framework really is. There are a number of times I've written tens or hundreds of lines of code for a task that I later realized – or was smacked over the head with by a colleague with – that were in the framework. Heck, even after writing my Extension Methods post a couple months ago some friendly readers pointed out how I was writing too much code.

Today, I was updating a client's web application to show the version number at the bottom of the screen based on the actual version that was running (it used to just be hard coded). The version number on the site was supposed to show the full revision of the assembly – which is stored in the assembly file version. The process to build the web site follows Microsoft version standards so the assembly version number does not necessarily change with each revision, but the file version does. Therefore, the normal System.Reflection.Assembly.GetExecutingAssembly().GetName().Version.ToString() isn't really the version number we want to display.

But how do you get the file version? Well, with System.Diagnostics.FileVersionInfo.GetVersionInfo of course! I swear I have looked for how to do this numerous times with only win32 responses to my query. Anyway, apparently it's dirt simple. This time I ran into this thread with the answer. Check out the System.Diagnostics.FileVersionInfo class for more info. Duh!

I leave you with this to ponder: WHY, WHY, WHY is this feature not exposed through System.IO.FileInfo or some other class that makes sense (and I did check in Reflector to be sure I wasn't mad)?!

Wednesday, November 21, 2007

Command Line VS 2008 Project Upgrade Tool

I've converted a few projects to 2008 RTM now (from 2003 to 2005 to 2008 beta projects), and I wish John Robbins had published this blog earlier. For those of us that don't like clicking in things he presents a built-in command line switch on devenv to upgrade solutions. I decided to take it to the next level.

I built a small command line application that recursively scans a directory for solution files and prompts you to upgrade all of them. I tried it on a bunch of my solutions and it seemed to work pretty well.

To use the tool you just run it from the command line and specify the directory where you want to start searching. Or you can specify no arguments to search the current working directory.

    upgrader.exe c:\root\of\your\tree



That's it! Quick and easy. The output will look something like:

    Searching for solutions under c:\svn\websites
Upgrade "c:\svn\websites\mysite\my.sln"? [Y/N] n
Skipped "c:\svn\websites\mysite\my.sln"
Upgrade "c:\svn\websites\broken\site.sln"? [Y/N] y
Converting "c:\svn\websites\broken\site.sln"
Unable to convert solution "c:\svn\websites\broken\site.sln"
Check out the errors in the UpgradeLog.xml file.
Upgrade "c:\svn\websites\ok\site.sln"? [Y/N] y
Converting "c:\svn\websites\ok\site.sln"
Converted "c:\svn\websites\ok\site.sln"

Well, that's my simple app that took just a few minutes to write and saved the pain of many clicks through a wizard. Download it here.

Tuesday, November 20, 2007

More Visual Studio 2008 Beta 2 to RTM Upgrade Gotchas

Yesterday I posted on the first thing I ran into. Today I ported another project and hit a couple more snags.

  1. Linq to Sql Unicode Byte Header
  2. Linq to Sql Add/Remove Method Changes
  3. ASP.NET Listview Changed from itemContainer to itemPlaceholder
Linq to Sql Unicode Byte Header

The xml document that makes up the Linq to Sql dbml file is declared to be in unicode format. For some reason my file was not saved with the unicode byte header. Apparently this didn't matter to the beta 2 framework, but in RTM release the project failed to build and double clicking on the designer caused the error "There is no Unicode byte order mark. Cannot switch to Unicode."

When the error occurs it opens the dbml file as XML. So, I just moved my cursor to the first position. Hit enter then backspace and saved the file. Visual Studio 2008 RTM put the Unicode Byte Header in for me and everything was kosher.

Linq to Sql Add/Remove Method Changes

The Linq to Sql Table<T> and ITable has changed the signature for adding and removing entities. Instead of the Add and Remove methods there are now InsertOnSubmit and DeleteOnSubmit. This makes sense, as it was not obvious what Add and Remove would do. These new names make a lot of sense to me and were outlined in a msdn forum posting a few months ago.

ASP.NET Listview Changed from itemContainer to itemPlaceholder

It looks like a few bloggers already beat me to this one, but the behavior of the itemContainer in an ASP.NET Listview has changed and the new control ID you have to use is called itemPlaceholder. Daniel Moth has a good description on his blog here. The annoying thing about this change for me was that it showed up at runtime, and not compile time. Though the runtime message is pretty descriptive: System.InvalidOperationException: An item placeholder must be specified on ListView 'ListView1'. Specify an item placeholder by setting a control's ID property to "itemPlaceholder". The item placeholder control must also specify runat="server".

Wrap Up

Well, that's all I've run into so far. I'm sure I'll find more issues as I wrote quite a bit of (production) code under 2008 Beta 2. I'll make sure to summarize any more findings here.

Monday, November 19, 2007

Small Gotcha in .NET Framework 3.5 Beta 2 to RTM Upgrade

Visual Studio 2008 was officially released today to MSDN subscribers so I took the leap and did the upgrade. I ran into an issue with a simple fix that I thought everyone should be aware of. You'll get an error message stating: "Could not load file or assembly 'System.Data.DataSetExtensions, Version=". This is somewhat cryptic, but easy to fix.

If you have a web project that was running under .NET 3.5 beta 2 you will have an assembly reference as follows:

<add assembly="System.Data.DataSetExtensions, Version=, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>

Notice the version number is This was apparently changed at the RTM and the version number should be like:

<add assembly="System.Data.DataSetExtensions, Version=, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>

Tuesday, November 13, 2007

ASP.NET MVC Framework

I'm a huge fan of Test Driven Development and I've been looking forward to the ASP.NET MVC (Model View Controller) Framework to increase the level of testability of web projects. Scott Gu just posted a great and frighteningly in-depth article – I don't know how he has the time. If you like releasing quality/tested, code, check it out. If you're not a developer, make your developers look at it.

Friday, November 9, 2007

Dvorak's Gone Mad

Maybe I'm just young, arrogant, and naive, but I think John's getting a little senile. In his latest rant for his PC Mag column he bashes Google's plans for the Open Handset Alliance with the title "The Google Phone is Doomed". Go read that if you haven't already and work your way back here. Don't get me wrong, I'm not convinced the project will work either – more on that later, but his reasoning confuses me.

Why Dvorak Thinks it Won't Work
  1. Google isn't charismatic
  2. Smartphones suck
  3. Nobody wants the internet on their phone

Google isn't charismatic? People don't love Google? Really? Google has more mindshare than pretty much any internet company out there. Ok, so they put out a crappy social networking tool called Orkut that nobody has heard of. I'm not sure how that relates to the potential success of an open phone product.

Smartphones currently suck. I definitely won't be arguing with that. The operating system user interfaces are crap and the software packages suck. However, I don't get how that has anything to do with a vaporware smarthpone due out some time in the future. If anything, an open platform should lead to a bunch of startups trying to build the next great phone interface. If they can pull it off, this should jumpstart an entire industry of reform. Apple did some good things with the iPhone. I see no reason why the same can't happen with OHA.

I'm not sure what year he's living in, but pondering why anyone would want internet on the phone when they could use the phone is just plain dumb. How do you know the phone number of the restaurant without the internet (or finding a phone book somewhere)? And why would I want to talk to someone when I can get more information myself on Google or [insert name of search startup here]? Maybe it's just me, but this seems like someone arguing to keep their fax machine instead of getting email, or keep their email instead of getting instant messaging.

Why I think it might not work

Alright, so maybe I don't agree with everything John brings up, but I do agree with one. Google wants to roll up the entire mobile market. The carriers, the operating systems, the software, and the hardware manufacturers. I don't see one of the biggest industries in the world taking that lying down.

In addition to the huge hurdles on the business side of things the technical challenges are staggering. Phones are not simple platforms to build on. And from what I've seen Google has yet to produce a large, complex, consumer facing product. They have a great back end platform and a boatload of itty bitty – dare I say widget – sized front end applications. An operating system and SDK that runs on a phone will prove to be a huge task, indeed.

In any case, "doomed" is pretty uncalled for. Of course, I did read his article, look at his ads, and now all of you are going to head over there too. So, I guess that's a success... And hey, I'm just a punk kid anyway! :)

Sunday, October 7, 2007

Vista Rant

Next month Vista will have been out in RTM for a year (it was released on 11/30/2006 to the volume/dev world). Why is it then, that it still doesn't work?

When I originally bought my current workstation, almost two years ago, I set it up with four partitions (I had a terrabyte RAID). I installed XP Pro 32 bit for games and Windows Server 2003 x64 for development. I left two open partitions, one for Vista, and one for some flavor of Linux. For the first six months of its life I spent most of my time in XP Pro x86. It worked, but I was wasting my fancy 64 bit hardware!

The beginnings — x64, take 1

On June 6 2006 I blogged about my experience installing Vista on my workstation. At the time it went reasonably well. Heck, my system passed the hardware compatibility wizard with flying colors! However, I really should have posted some follow-ups.

A week later — x64, take 2

After I started actually using Vista, it started crashing, a lot. I'd get at least one blue screen a day. I looked at the minidumps and they were all related to one of my NVidia drivers. It would be either the NIC or the sound or the SATA or the RAID — yeah, pretty much everything that comes with my motherboard (I bought it for the pretty colors). One day I noticed a Windows update for my NVidia RAID drivers. "Cool", I thought. Maybe they sped them up or improved stability. The drivers installed; my system rebooted; it bluescreened; repeat. After much frustration I switched to XP Pro x64.

A month later — x64, screw you

XP Pro x64 was fairly stable, but I still had serious issues with the interaction of the drivers. If I was using the network heavily and then started using the sound card heavily: Kablammo! BSOD! I sort of learned how to work around this and would mute my music before doing anything intensive over the network. Needless to say this was very annoying. I switched to XP Pro x86.

December, 2006 — Vista x64 RTM install

"Ok, Vista is RTM now", I thought. "Surely NVidia has got their act together now!" They're working closely with Microsoft on this stuff, right?!

December, 2006 — Vista x86 RTM install

"Ok, x64 is the bastard child of hardware", I thought. "Surely this will work."

December, 2006 — Back to the trusty XP Pro x86 partition

Yeah, back on XP. I'll spare you the details.

August, 2007 — Vista x86 install

"Ok, it's been a million computer years now", I thought. "Surely all my crash reports and support incidents have caused some bug fixes." Not so much. I even installed it on a PATA drive, no RAID required. To add to the pain, I also purchased a D-Link DWA-130 USB Wireless N Adapter for my desktop at the same time. I can't pass on the promise of speed with no wires. "What?! No Vista drivers?! I quit!" What's with a new product (released in July 2007) not having Vista drivers? Oh, there are beta ones available now — they don't work either.

August, 2007 — XP x86 Pro for life

After over a year of struggling and giving it a chance, I've given up. Either Microsoft or these hardware/driver vendors need to get their heads out of their asses, take some initiative, and fix this crap. I'm officially a hateful bastard until someone shows me some reason not to be.

October, 2007 — The world should know

I wrote this blog. It feels good to vent. Much cheaper than a therapist.

Friday, September 28, 2007

Asyncify Your Code

Asyncify your code. Everybody's doing it. (Chicks|Dudes)'ll dig it. It'll make you cool.

Pretty much everything I build these days is asynchronous in nature. In SoapBox products we are often waiting on some sort of IO to complete. We wait for XMPP data to be sent and received, database queries to complete, log files to be written, DNS servers to respond, .NET to negotiate Tls through a SslStream, and much more. Today I'll be talking about a recent walk down Asynchronous Lane: the AsynchronousProcessGate (if you don't like reading just download the package for source code goodness).

I ran into a problem while working on a new web application for Coversant. I needed to execute an extremely CPU and IO intesive process: creating and digitally signing a self extracting compressed file -- AKA The Package Service. This had to happen in an external process, and it had to scale (this application is publicly available on our consumer facing web site). Here's a basic sequence of the design I came up with:

Do you notice the large holes in the activation times? That's because we're asynchronous! The BeginCreatePackage web service method the page calls exits as soon as the BeginExecute method exits, which is as right when the process starts. That means we're not tying up any threads in our .NET threadpools at any layer of our application during the time a task is executing. That's a Good Thing™.

At this point I'm used to writing highly asynchronous/threaded code. However, I still wouldn't call it easy. Why do it? I'd say there are three main reasons.

  1. To provide a smooth user experience. The last thing a developer wants is for his/her software to appear sluggish. There's nothing worse than opening Windows Explorer and watching your screen turn white (that application is NOT very asynchronous).
  2. To fully and most appropriately utilize the resources of the platform (Runtime/OS/Hardware). To scale vertically, you might call it.
  3. Because it makes you cool. AKA: To bill a lot more on consulting engagements.

Microsoft recommends two asynchronous design patterns for .NET developers exposing Asynchronous interfaces. These can be found on various classes throughout the framework. The Event Based pattern comes highly recommended from Microsoft and can be found all over new components they build (like the BackgroundWorker). Personally I think the event based pattern is overrated. The hassle of managing events and not knowing if the completed event will even fire typically steers me away from this one. However, it is certainly easier for those who are new to the asynchronous world. This pattern is also quite useful in many situations in Windows Forms and ASP.NET applications, leaving the responsibility of the thread switching to the asynchronous implementation (the events are supposed to be called in the thread/context that made the Async request -- determined by the AsyncOperationsManager). If you've ever used the ISynchronizeInvoke interface on a Winforms Control or manually done Async ASP.NET Pages you can really appreciate the ease of use of this new pattern...

The second recommended pattern, and usually my preference, is called the IAsyncResult pattern. IAsyncResult and I have a very serious love/hate relationship. I've spent many days with my IM status reading "Busy - Asyncifying" due to this one. But, in the end, it produces a simple interface for performing asynchronous operations and a callback when the operation is complete (or maybe timed out or canceled). Typically you'll find IAsyncResult interfaces on the more "hard core" areas of the framework exposing operations such as Sockets, File IO, and streams in general. This is the pattern I used for the Asynchronous Process Gate in the Package Service.

The Package Service has a user interface (an AJAXified asynchronous ASP.NET 2.0 page) which calls an asynchronous web service. The web service calls another asynchronous class which wraps a few asynchronous operations through the AsynchronousProcessGate and other async methods (i.e. to register a new user account) and exposes a single IAsyncResult interface to the web service.

Confused yet? Read that last paragraph again and re-look at the sequence. In order to make this whole thing scale it had to be asynchronous or we'd be buying a whole rack of servers to support even a modest load. Also because of the nature of the asynchronous operation (high cpu/disk IO) it had to be configurably queued/throttled. I went through a few possible designs on paper. But in the end I chose to push it down as far as possible. The AsynchronousProcessGate, quite simply, only allows a set number of processes to execute simultaneously, the number of CPU's reported by System.Environment.ProcessorCount by default. It does this by exposing the IAsyncResult pattern for familiar consumption. The piece of magic used internally is something we came up with after writing a lot of asynchronous code: LazyAsyncResult<T>.

LazyAsyncResult<T> provides a generic implementation of IAsyncResult. It manages your state, your caller's state, and the completion events. It also uses Joe Duffy's LazyInit stuff for better performance (initializing the WaitHandle is relatively expensive and usually not needed).

Using the asynchronous process gate is straight forward if you're used to the Begin/End IAsyncResult pattern. You create an instance of the class, and call BeginExecuteProcess with your ProcessStartInfo. When the process is complete you will get your AsyncCallback, or you can also wait on the IAsyncResult.WaitHandle that is returned from BeginExecuteProcess. You then call EndExecuteProcess and the instance of Process that was used is returned. If an exception occurred asynchronously, it will be thrown when you call EndExecuteProcess.

The Begin Code:
static void StartProcesses()
AsynchronousProcessGate g = new AsynchronousProcessGate();
while (!_shutdown)
//keep twice as many queued as we have cpu's.
//for a real, CPU or IO intensive, operation
//you shouldn't do any throttling before the gate.
//that's what the gate is for!
if (g.PendingCount < g.AllowedInstances * 2)
new ProcessStartInfo("notepad.exe"),
The End Code:
static void ProcessCompleted(IAsyncResult ar)
AsynchronousProcessGate g =

using (Process p = g.EndExecuteProcess(ar))
Console.WriteLine("Exited with code: " +
p.ExitCode + ". " +
g.PendingCount + " notepads pending.");
catch (Exception ex)
+ ex.GetType().ToString()
+ ") - " ex.Message);

Phew! After all that, the end result for SoapBox: a single self extracting digitally signed file someone can download. Oh, and a simple library you can use as an Asynchronous Process Gate! Enjoy. Look, another download link so you don't even have to scroll back up. How nice am I?

Thursday, September 27, 2007


A task that I often end up doing when coding an actual web site (i.e. not writing a sample or some such) is adding client script to a page/control in codebehind using the ClientScriptManager. Let's say you've got the following alert script you want to add to the page so you can use it in a control:

function doAlert()
Well, there are now a number of ways to get this into your page, but the quickest, in-line way is to use the ClientScriptManager. Like so:
if (!Page.ClientScript.IsClientScriptBlockRegistered(
this.GetType(), "alert"))
System.Text.StringBuilder sb = new System.Text.StringBuilder();
sb.Append("function doAlert() {");
sb.Append("alert ('welcome!'); }");
this.GetType(), "alert", sb.ToString(), true);

Ok, so that's a bit contrived. You're not going to use a StringBuilder for something that simple. But, with a more complicated script you probably would, especially if the script will be different depending on the state of the page.

I had a little Winforms utility sitting around I've been using for a while to automate that process for me, and I decided to put it up on the web. So, here you go: You feed the engine a multi-line string (like a script) and it gives you back a StringBuilder with everything properly escaped and such. The code's a bit of a mess at this second, that's why I didn't post it. Enjoy!

Wednesday, September 19, 2007

ASP.NET Centric Extensions

I've been loving the new Extension Methods in Orcas. If you haven't had the pleasure of working with them, well, you're really missing out. They are utility at its finest. Here is a list of my favorite extensions I've written thus far.


Ok, I know this sounds stupid, but for some reason I really hate typing HttpUtility.UrlEncode(myString) all over the place so I wrote this extension.

public static string UrlEncode(this string toEncode)
return HttpUtility.UrlEncode(toEncode);

For some reason my memory for .NET format strings really sucks. I blame Intellisense. I always forget how to hex encode integers. In case you're wondering, this is quite useful for giving users short strings that can be very unique and directly represent a primary key in a database. Hex is generally friendlier on the eyes than a number alone.

public static string ToHexString(this int val)
return string.Format("{0:x2}", val);


Likewise, when you have a hex encoded integer, it's quite useful to get that value back out. I have no idea why, but I forget this neat trick all the time and constantly have to look it up. No more!

public static int FromHexString(this string val)
return Convert.ToInt32(val, 16);

I posted this a few weeks ago. I love it...


This is something I have struggled with since the early days of ASP. There has never been a straightforward way to get the root of the URI of a request. This method extends HttpRequest and gives you the scheme (http/https) and host/port of the request. So, provided a request to this extension would return

public static string GetRootPath(this HttpRequest r)
string hostPort = r.Url.Authority;
string scheme = r.Url.Scheme;

string path = string.Concat(scheme, @"://", hostPort);

return path;

A common task in a web app is to send out an email for one reason or another. Maybe you want to send someone a link to a download, confirm their registration, or bug them when they haven't visited your site in a while. This method takes an application relative path stored in your App.Config appSettings and translates it into a fully qualified URL based on the current executing page. Your configuration might look like:

<add key="EventDetailsFormatString" value="~/events/{0}.aspx"/>

You could then use the following extension to turn that relative URL into a fully qualified based on the current page that is executing. It relies on the GetRootPath method above.

public static string GetConfigBasedPath(
this Page p,
string configKey,
params object[] formatArgs)
if (null == p)
throw new ArgumentNullException("p");

if (string.IsNullOrEmpty(configKey))
throw new ArgumentNullException("configKey");

string valFromConfig = ConfigurationManager.AppSettings.Get(configKey);

if (null == valFromConfig)
throw new ArgumentOutOfRangeException("configKey",
"The specified key was not found in the configuration.");

Uri rootUri = new Uri(p.Request.GetRootPath(), UriKind.Absolute);

Uri relativeUri = new Uri(
p.ResolveUrl(string.Format(valFromConfig, formatArgs)),

Uri fullUri = new Uri(rootUri, relativeUri);

return fullUri.ToString();

In a Treeview you often want to see if a node or any of its children are selected to help determine how to display the tree. This is especially useful when using the Treeview for navigation.

public static bool IsDescendantOrSelfSelected(this TreeNode node)
if (node.Selected)
return true;
else if (node.ChildNodes.Count > 0)
foreach (TreeNode n in node.ChildNodes)
if (IsDescendantOrSelfSelected(n))
return true;

return false;
else { return false;

This one is quite simple. It just makes sure that the provided node is visible in the tree by making sure all of its parent nodes are expanded. This is a useful function for stateless Treeviews or synchronizing the same Treeview across multiple pages.

public static void EnsureVisible(this TreeNode node)
if (null != node.Parent)

Well, that's it for now! I hope someone finds these as useful as I do.

Wednesday, September 5, 2007

List of Country Names

Today I had to build yet-another-list of country names for a web site drop down form. I did some Google searching for about 10 minutes and didn't find anything that was free. I dunno, maybe I'm just blind... I did, however, find a list on Wikipedia. So, I decided I'd pull it down and parse it up. Here's the code:

static void Main(string[] args)
string fullpage;
using (WebClient wc = new WebClient())
byte[] pageBytes = wc.DownloadData(
string encoding = wc.ResponseHeaders["Content-Encoding"];
fullpage = wc.Encoding.GetString(pageBytes);

MatchCollection countryMatches = Regex.Matches(fullpage,
@"<p><b><a href="". ""\s*title="". "">(?<country>. )</a></b>:</p>",

foreach (Match m in countryMatches)
Group g = m.Groups["country"];


As you can see it's dirt simple. Just a WebClient to download the page (holy crap it's big) and a simple Regex call. Presto! A list of countries I'll be dropping into a database so we can edit it later.

Thursday, August 30, 2007

Don't forget to close your Linq to Sql connections

Today I was working on some Linq to Sql code that dealt with an event management system. There are events people can attend, and attendees fill out a form to register for the event. The fields for each event are configurable by the person that creates the event. When I save a set of event fields I want to do it in a transaction so I don't end up with a partial "record". Here's what my code looked like originally:

public void SaveEventAttendeeFields(EventAttendeeField[] fields)
using (LinqToSqlDataContext dc = new LinqToSqlDataContext())
dc.DeferredLoadingEnabled = false;

try { dc.Connection.Open(); dc.Transaction = dc.Connection.BeginTransaction(); foreach (EventAttendeeField f in fields)
if (f.EventAddendeeFieldId > 0)
dc.EventAttendeeFields.Attach(f, true);
else dc.EventAttendeeFields.Add(f); } dc.SubmitChanges(); dc.Transaction.Commit(); } catch { if (null != dc.Transaction)


I assumed this was perfectly fine, until I ran it through my unit test library. My test library automatically creates a new database, tables, sprocs, and then populates it with seed test data. At the end of the test run it drops the database that was created. Well, the drop was failing because the database was in use! I thought "hmm, I wonder if my connections aren't being closed." Sure enough, if you manually open the connection in the DataContext you had better manually close it as well!

public void SaveEventAttendeeFields(EventAttendeeField[] fields)
using (LinqToSqlDataContext dc = new LinqToSqlDataContext())
dc.DeferredLoadingEnabled = false;

try { dc.Connection.Open(); dc.Transaction = dc.Connection.BeginTransaction(); foreach (EventAttendeeField f in fields)
if (f.EventAddendeeFieldId > 0)
dc.EventAttendeeFields.Attach(f, true);
else dc.EventAttendeeFields.Add(f); } dc.SubmitChanges(); dc.Transaction.Commit(); } finally { if (null != dc.Connection && dc.Connection.State != System.Data.ConnectionState.Closed)

This code didn't cause any leaks. So, moral of the story, close your connections in your DataContext if you open them.

Tuesday, August 28, 2007

Clean Up a String for a Url

This is a quickie. Yesterday I was doing some Url rewriting for a project and accepting user input for said Url. It's basically like this blog system. I can specify the friendly url of each post. So, I wrote a little extension method to clean up a string and make it URL friendly. Yes, you can just escape everything someone enters, but it's much friendlier to just make it work. Be lenient in what you accept! Here it is:

public static string ToUrlString(this string s)
if (null == s)
throw new NullReferenceException();

string tmp = Regex.Replace(s.ToLower(), @"\s", @"-");
tmp = Regex.Replace(tmp, @"[^-_\.\w]", @"");

return tmp;

This simply replaces all whitespace with a "-", and all characters that are not "-", "_", ".", or Word Characters with an empty string.

In the end, if your input is "this is a @#)(*&*@$^ crock!" you'll get "this-is-a--crock". As a guy who was, at one time, scared of regular expressions I hope this helps someone.

Wednesday, August 22, 2007

using (wcfclient) { } kaboom!

I've been doing a lot of WCF development lately (love it) and ran into a bit of a stumbling block. The proxy client generated by Visual Studio 2008 Beta 2 doesn't handle fault conditions and dispose well. I have to admit, I haven't used WCF in Visual Studio 2005. If you're impatient here's the project (Visual Studio 2008 Beta 2 project): The situation unfolds:

The Problem - Dispose Isn't Safe!

  1. Create a WCF service.
    public interface IGoodBadService
    void Good();

    void Bad();
  2. Add a service reference to your service in another project.
  3. Create a service client and use it, with a "using" block (it is IDisposable, afterall).
    using (GoodBadServiceClient c =
    new GoodBadServiceClient())
    catch (Exception ex)
  4. A call to your service, for whatever reason, throws an exception.
    public class GoodBadService
    : IGoodBadService
    public void Good()
    throw new Exception("GOOD!");

    public void Bad()
    throw new Exception("BAD!");
  5. You get a cryptic System.ServiceModel.CommunicationObjectFaultedException instead of the real exception when using calls Dispose() on your proxy client. It reads: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

The Solution - A Simple Generic Wrapper

public class ServiceProxy<TClient>
: IDisposable
where TClient : ICommunicationObject
public ServiceProxy(TClient client)
Client = client;

public TClient Client { get; private set; }

public void Dispose()
if (null != Client &&
Client.State != CommunicationState.Closed &&
Client.State != CommunicationState.Faulted)

Using the wrapper is pretty straightforward, but a tad more cryptic than just using the client directly. Some of this can be avoided with a generic factory.

So, here's how you use it:

using (ServiceProxy<GoodBadServiceClient> c
= new ServiceProxy<GoodBadServiceClient>(
new GoodBadServiceClient()))
catch (Exception ex)
Run the project and check out the results. The wrapper gives you the actual exception, whereas "using" the client directly causes the CommunicationObjectFaultedException. Here's the project (Visual Studio 2008 Beta 2 project):

Tuesday, August 21, 2007


Well, I've officially relocated my blog here, to The old blogging engine I was using really sucked. This one seems a lot better. And if I don't like something, the source code looks pretty clean so I can get my hands dirty and change it. I'm also a fan of one of the Haacks that wrote it -- well, his blog at least.

In addition to a new blog engine my new location has implicitly granted me a little freedom. Fear not, my blog will no longer be 100% about Coversant related items. I've been exploring the wide world of .NET 3.5, Visual Studio 2008, and everything in between. I've got a few articles that need some polish and I'll be posting them here.

Wednesday, June 20, 2007

Friend Photosaver for Facebook

By now everybody who knows what Facebook is, knows they announced an open platform for integrating with it. Interestingly enough Microsoft was a launch partner of theirs so there is also a shared source .NET API layer available. We decided to jump on the opportunity and published SoapBox for Facebook. Chris mentioned this in his announcement blog earlier.

But, as with anything that goes to production, adding cool features quickly turned into fixing uncool bugs, scope discussions, planning sessions, and then it became work.

I had a bit of a realization yesterday. I hadn't written code for the sheer fun of it in a really long time. Don't get me wrong, I love the stuff I do (almost) every day. We're coding cutting edge stuff here and it's a blast. But, I wanted to do something for me! So I wrote a screensaver.

Strange, huh? The guy that writes blogs on memory management, async programming, and other maddening things decided to hack together a screensaver. You'll just have to get over the shock of it all... Research was done, scope was decided, fingers flew. A few hours later, I had a screensaver! At 2am I decided on its name (probably a bad idea), Friend Photosaver for Facebook, whipped up an installer and submitted the app to Facebook. Catchy isn't it? Yeeeah. Anyway, though the name stinks, I think it's a pretty cool screensaver.

It aggregates the list of all your photos with all of the photos of your friends, and picks them at random. It then stuffs the photo in a random spot on the screen where it will be visible. Right now it only displays on the primary screen. Secondary screens are blank. Here's a little preview:

Screen Saver

If you're a Facebook user you can login and install my screensaver. The rest of you... well, this is pretty useless. But you should be on Facebook too. Oh, and (shameless plug), don't forget to use SoapBox for Facebook. :)

Friday, June 8, 2007

Simpler isn't always better - AsyncOperationsManager

AsyncOperationManager... die!

We have a new invitation system coming out soon for SoapBox Communicator and I was doing a little premature optimization, err, I mean, capacity planning. :) I set about to perform the simple task (or so I thought) of sending all of our email based invitations asynchronously. The architecture of the system is pretty simple. We have an ASP.NET 2.0 web site that you use inside SoapBox for sending invitations to your friends. The web site talks to an internal web service. The web service talks to a database. It's a pretty classic nTier design, with one exception. All layers are asynchronous.

The UI uses asynchronous ASP.NET pages and RegisterAsyncTask to make the call to the EnqueueEmailInvites web service. The web service uses asynchronous web methods and reads from/writes to our database using the asynchronous data methods in SqlClient.

Async Web Method

public IAsyncResult BeginEnqueueEmailInvites(EmailInvitation invite, AsyncCallback callback, object state)
return InvitationFactory.BeginEnqueueEmailInvites(invite, callback, state);

public string[] EndEnqueueEmailInvites(IAsyncResult ar)
return InvitationFactory.EndEnqueueEmailInvites(ar);
Factory Calling DAL

public static IAsyncResult BeginEnqueueEmailInvites(EmailInvitation invite,

AsyncCallback callback, object state)
InviteEmailAsyncState myAs = new InviteEmailAsyncState();
myAs.Invitation = invite;
InviteEmailAsyncResult myAR = new InviteEmailAsyncResult(myAs, callback, state);

int emailInviteId = invite.Mutual ? GetInvitationTypeId(Constants.InviteTypeEmailMutual) :

string csvAddresses = string.Join(",", new List<string>(invite.Addresses).ToArray());

Data.Invitations.BeginCreateReadInvitation(emailInviteId, invite.FromJid, invite.FromName, invite.Subject, invite.UserBodyPlain, string.Empty, csvAddresses, EmailInvitationCreatedCallback, myAR);

return myAR;

I write a lot of asynchronous code. A lot. I'm a bit of an asynchronous I/O zealot. In fact, I've had an article on the shelf called "Asyncify your code" for months now. l I just haven't got the sample code written for it yet. Anyway, my experience has led me to a simple conclusion: I hate the AsyncOperationManager.

Maybe it's because I write a lot of asynchronous code, or maybe just because people tend to Mistake Familiarity with Superiority, but whatever the reasoning, I can't stand it. The AsyncOperationManager is used all over the .NET 2.0 framework where you see calls ending in "Async", even though you probably don't even know it. One such instance that rubbed me the wrong way yesterday (and into this morning) is the SmtpClient. Don't get me wrong, the SmtpClient is a great improvement over the old CDO based classes, but the Async support leaves something to be desired.

First off, you can't call Send or SendAsync until the previous operation has completed. That means if you're sending out more than one email in succession you're going to need to do your own queueing. So I said to myself "Ok, this is easy, I'll rip out the loop, register an eventhandler for SendCompleted, and pass a Queue through as state to SendAsync." This led me to the next problem.

The asynchronous web method called the asynchronous data method, which led to a callback. Inside of this callback I grabbed some info from the data reader result and composed the MailMessage. I created a SmtpClient, registered for the SendCompleted event kicked off the first SendAsync, and completed the asynchronus web method. The mail sent great, but my SendCompleted event handler was never called.

Apparently, if you make an async call there is the potential that the completed event handler you registered will never get called. Yep. You can call SendAsync, and the SendCompleted event will never be raised. I admit, the circumstances this happened in were kind of abnormal, but still. That just shouldn't happen.

After a little reflectoring it was apparent this had something to do with the SynchronizationContext at the time of the call used by the AsyncOperationManager inside of the SmtpClient. I got a litte cross-eyed looking through the code in reflector so I gave up my hunt for the exact answer. My workaround: Call ThreadPool.QueueUserWorkItem and create the SmtpClient and call SendAsync inside of that callback. This gets you the generic SynchronizationContext that doesn't have anything weird associated with it, just plain ole async delegates. If someone has a better solution, please let me know.

The Hack

if (messagesToSend.Count > 0)
//this seems strange, but we queue up the creation of the SmtpClient to a threadpool thread.
//the synchronization context seems to become invalid when we complete the async web method call
//and we don't want to wait for the emails to go out to complete the web service call.

System.Threading.ThreadPool.QueueUserWorkItem(StartSendingEmailCallback, new object[] { myAr.MyAsyncState.Invitation, messagesToSend });

I have a plea to all framework designers out there. Have the decency to give me the good ole IAsyncResult based async pattern without all that extra AsyncOperationManager baggage! I don't want to have to worry about all that. I'm a big boy. I can handle my own synchronization, and I like it that way.

Tuesday, May 8, 2007

XMPP Presence Priority

In XMPP (the messaging and presence protocol SoapBox uses) you can log in simultaneously on multiple systems. How your contat list is displayed and how messages are routed are based on hints the client software provides to the server. One of these is presence priority. Priority is just an integer value between -128 and 127 defined in RFC 3921 Section

Managing priority is fairly straightforward when you control all client implementations used by a bare JID, but in a heterogeneous client environment it is more difficult as each developer chooses his/her own path and only learns by spying on what others do. I think presence priority is something that should be covered in the implementation guide so the actual values become more uniform on the network. We currently do pretty basic priority management, but I have some thoughts on a better implementation I hope others implementations might help refine.

Priority is typically used by both clients and servers to determine which resource is most likely to be the one you want to interact with. This could be for routing messages sent to bare JID’s, or deciding which presence status to display in a contact list. So, after much deep thought and implementation trial/error, here is my proposal.

Unless a user explicitly specifies she does not want to receive any messages, never set the priority below 0.

In desktop software, presence priority is based first on user input activity. It is more likely that if a user is active, she will be there to respond to a message, regardless of the presence status that is set in the client software. The actual times may vary based on implementation, but the general idea is to have four base values:

  • User input idle less than 1 minute: 100
  • User input idle more than 1 minute: 80
  • User input idle less than 5 minutes: 60
  • User input idle less than 30 minutes: 40

When these idle thresholds are met, presence should be broadcast. The current show value is then applied.

  • chat: -4
  • none (normal/available): -4
  • dnd: -4
  • away: -8
  • xa: -12

Thus, if a user has been idle for more than 30 minutes and has a show of xa, her priority is 28 (40-12). If a user has been idle for more than 30 minutes and has a show of away her priority is 32 (40-8). By starting at a high number and leaving padding between each show value, we leave room for implementations to add additional rules to adjust priority. Dnd has a higher priority than away and xa because, although the user does not want to be disturbed they are more likely to be at the dnd resource than the away or xa resource. It is up to the client implementation to insure they are not disturbed per the user’s configuration. A show of dnd uses the same priority as none and chat since you are likely equally available, you just don’t want to be disturbed.

Priority and, optionally, show values are adjusted automatically based on common “away from the computer” events: system input idleness (how long since the user interacted with the system), when a session is locked, or a session screen saver is active. When these events occur, the priority rules must be processed again. For Example, my client is configured to go to a show of away after 5 minutes with a status of “AFK”. After 30 minutes it is set to a show value of xa and a status of “AFK for a long time”. When my system is locked or the screen saver comes on, my client automatically adjusts my show value to away and a status of “Not here”. Important: even if an implementation does not change status messages or show values after these “away from the computer” events it must still adjust its priority as defined above.

After the “away from the computer” rules are processed, the relative availability of your peer resources is taken into account. If any peer was more recently more available (had a more available show value than your current value – chat, none/dnd, away, then xa) and has a non-negative priority, subtract 20 from your current priority. For example: I’m logged in at home and work. At home I last announced available presence 4 hours ago. My software at home never puts me into “xa” and instead decides I am “away” after 30 minutes. Home priority is now 32 (40-8). At work I last announced available presence 3 hours ago with a “none” show value. I go to lunch at work but it is configured to reach a show value of xa after 30 minutes. Its priority is now 28 (40-12). However, since work was my last active resource, when my home resource receives the broadcast from work it adjusts itself to a presence priority of 12 (32-20). The purpose of this is to insure that only the most recently available resource above more available than away/xa is the one that takes the higher priority.

It is important to never adjust your priority relative to the priority of a peer resource without taking much care. Doing so can cause a “race” for the highest or lowest priority (depending on the relative direction your adjusting).

That's about it. I also posted this on the XMPP standards mailing list. Reaching a concensus on how to adjust presence priority sure will help user perception of XMPP software.

Tuesday, April 10, 2007

Building Reactive User Interfaces in .NET: ISynchronizeInvoke on Idle Time

One of the most common things to do in a multi-threaded .NET Windows Forms application is to use the ISynchronizeInvoke interface on a Control to marshal things into the UI thread. The basic jist of things is: you can have the Windows Forms engine call your delegate on the thread that created the Control instance you're using. It does this using the window message pump. Typical uses of the ISynchronizeInvoke are: you want to read some data from a file, database, socket, or web service and you want your application to be responsive while you do it. When you're done you do a BeginInvoke/Invoke on the Control to update some UI elements. Well, sometimes using a Control's ISynchronizeInvoke leads to hung GUI's.

If you have a lot of asynchronous operations pending and they complete in bunches the affect on your UI can be the appearance of a hang or sluggishness as drawing is queued behind the delegates you registered with BeginInvoke on your control. During the login sequeunce in SoapBox Communicator there are potentially thousands of events that need to be processed on the UI thread. Yup, thousands. Your roster arrives from the server. You receive the current avialability from every online person on your roster. You got 10 messages while you were offline. Each of your contact's cached profiles (avatars, names, client capabilities, etc) are read from disk and compared with the user's current presence. This onslought of activity can be handled a number of ways. The basic principles I stumbled into after much trial and error is:

  1. Never, ever, ever, ever, ever do IO on a UI thread. Even something as simple as reading a 1KB image from disk can bring your UI to a halt under the right circumstances.
  2. Use timers and update common pieces of UI (like list views) in batches whose changes were caused changes by background operations. This will reduce flicker as you invalidate areas to redraw.
  3. Use Application.Idle to your advantage. (see my take on this below)
  4. Be careful how many window handles you create. Controls are useful, but sometimes you've just gotta draw your own.
  5. Profile your code where it seems sluggish.

I mentioned above that you should use Application.Idle to your advantage. The articles I've found on it all mention coding precise things you want to do in that event, like updating a single form. I wanted to do it more generically. So, I created an implementation of ISynchronizeInvoke that uses the Application.Idle event to process qeued items. I've created a (only slightly) contrived example that uses a Pi Calculator I found and an animated Gif to demonstrate a hanging GUI.

Both buttons calculate Pi to 50 digits on the UI thread using 100 separate ISynchronizeInvoke.BeginInvoke calls. The "Idle Invoke" button does so using my ApplicationIdleSynchronizer. The "Direct Invoke" button calls BeginInvoke on the Form directly. You can click the buttons any number of times and more and more Pi calculation runs will be queued. The completed label is incremented after every run. Here's the code:

    public partial class Form1 : Form
private const int WorkItems = 100;
private const int PiDigitsToCalc = 50;

private int _workItemsCompleted = 0;

private ApplicationIdleSynchronizer _idleSynchronizer = new ApplicationIdleSynchronizer();

public Form1()

protected override void OnClosed(EventArgs e)

private void button1_Click(object sender, EventArgs e)
System.Threading.ThreadPool.QueueUserWorkItem(QueueOnBackgroundThread, _idleSynchronizer);

private void button2_Click(object sender, EventArgs e)
System.Threading.ThreadPool.QueueUserWorkItem(QueueOnBackgroundThread, this);

private void QueueOnBackgroundThread(object state)
for (int i = 0; i < WorkItems; i )
((ISynchronizeInvoke)state).BeginInvoke(new ThreadStart(MyWorkItem), null);

private void MyWorkItem()
label2.Text = _workItemsCompleted.ToString();

Click the buttons. Move the form around. Resize it. You might be suprised by the result (or not). Both very adequately use your CPU, but the Idle Invoke produces a much more reactive UI. You can download the full source code here: Note, this exact code isn't in use in production. I wrote it for this blog. YMMV. Let me know if it has any issues.

Thursday, April 5, 2007

YouTube Getter

Usually everything I post here is serious business. Well, not today! Today, we're going to look at how to get the raw video stream for YouTube videos. Why, you ask? Well, curiosity, mostly. :) Some people like sports, I like problem solving. There are a number of browser plug-ins and web sites that do this already, but hey, it was still fun.

YouTube offers a bit of an API for developers to mess with. It's mostly for grabbing sets of videos and preview images and such. They also let you embed their player in your pages. That's awfully nice of them, but what if I don't like their player? What if I want a sexier player? Well, if you've got something that can play/convert Flash Video (FLV's), here's the full C# code to grab the FLV URI (pieces dissected below):

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
using System.Web;
using System.Collections.Specialized;

namespace JDFun
public static class YouTubeGetter
public static Uri GetFlvUri(string viewUri)
return GetFlvUri(new Uri(viewUri));

public static Uri GetWatchUri(Uri flvUri)
NameValueCollection qry = HttpUtility.ParseQueryString(flvUri.Query);

string videoID = qry["video_id"];
string watchUri = string.Concat("", HttpUtility.UrlEncode(videoID));

return new Uri(watchUri);

public static Uri GetImageUri(Uri flvUri)
NameValueCollection qry = HttpUtility.ParseQueryString(flvUri.Query);

string imageUri = qry["iurl"];

return new Uri(imageUri);

public static Uri GetFlvUri(Uri viewUri)
// so either i've got the embed link or the watch link

//watch link:
//embed link:

string toQuery = null;

NameValueCollection queryString = HttpUtility.ParseQueryString(viewUri.Query);

string videoId = queryString["v"];

if (null != videoId)
toQuery = string.Concat("", videoId);
toQuery = viewUri.ToString();

if (null == toQuery)
throw new InvalidOperationException("Not a valid YouTube Uri.");

Uri queryUri = new Uri(toQuery);
//ok we have the uri to query, now go there and get redirected.
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(queryUri);
req.AllowAutoRedirect = true;

// make them think we're coming from a direct link
req.Referer = string.Empty;

// firefox rules!
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv: Gecko/20070309 Firefox/";
using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
string absoluteRedirectLocation = resp.ResponseUri.AbsoluteUri;

//replace the swf with the get_video request
if (!absoluteRedirectLocation.Contains("/p.swf?"))
throw new InvalidOperationException("Unrecognized Uri. YouTube changed something.");

absoluteRedirectLocation = absoluteRedirectLocation.Replace("/p.swf?", "/get_video?");

//return the absolute URI for this request
return new Uri(absoluteRedirectLocation);

Ok, wow, that's a lot bigger than I remember. Here's the basic idea.

  1. Get the video ID from the given URI. The URI could be the "watch" URI, or it could be the URI from the embed code they give you.
  2. Load the video directly based on the video ID, at the embed URI.
  3. Make a HTTP Request and get redirected to another URI, (which happens to contain the Image URI, and a special token "t").
  4. Know the magic URI where they keep the video streams (I cheated) and use it.

Now wasn't that fun? :)

About the Author

Wow, you made it to the bottom! That means we're destined to be life long friends. Follow Me on Twitter.

I am an entrepreneur and hacker. I'm a Cofounder at RealCrowd. Most recently I was CTO at Hive7, a social gaming startup that sold to Playdom and then Disney. These are my stories.

You can find far too much information about me on linkedin: No, I'm not interested in an amazing Paradox DBA role in the Antarctic with an excellent culture!