ValidateInput Attribute Doesn't Work in ASP.NET 4.0

Today I decided to upgrade some of our new projects (top secret, shhh) to Visual Studio 2010, ASP.NET 4.0, and ASP.NET MVC 2.0. There are about a million new features that look quite useful in all of these new releases. We have some fairly complex projects so I was excepting a few speed bumps, but, not this one.

It seems with every new release Microsoft adds annoying (isn't security always?) features to protect us from ourselves. Way back when they added the idea of Request Validation. If ASP.NET thinks a user is posting something "bad" to the server (i.e. things that lead to XSS attacks), the request is denied. This is cool except when you want to, say, allow the user to input HTML or have a web service that takes XML as a parameter in a form, or use a ":" in your URL. In ASP.NET web forms you work around this feature by turning it off at the page level or globally in your web.config through the validateRequest option. In MVC you use the ValidateInput attribute on your action.

This is really really important:

In .NET 2.0-3.5 the runtime only validated requests sent to .aspx pages, but that has now changed and any request will be validated, even it is sent to a custom handler, or an MVC application.

ASP.NET MVC implements its own request validation, which is also on by default. To turn it off you simply slap a ValidateInput(false) attribute on your controller action. This is fine and dandy, except with the latest ASP.NET 4.0 changes, it no longer works and an exception is thrown. So you might see an error like:

A potentially dangerous Request.Form value was detected from the client or A potentially dangerous Request.Path value was detected from the client

The workaround is pretty easy. Just follow the instructions on the ASP.NET 4.0 Breaking Changes page. Stick this XML in your web.config to revert to the behavior as it were in ASP.NET 2.0-3.5. This will put request validation back in the hands of the ASP.NET MVC engine and your ValidateInput attribute will start working again.

Angry Players Make Sunday More Interesting

Youtopia has been growing quickly the last couple of weeks. It's fun to watch and the team is really excited about it. Of course, with the growth comes a lot of performance tuning with our code. Today we hit an issue I wasn't expecting at all. . .

We've been running Windows 2008, IIS7, and ASP.NET 3.5 in production for a while now, but haven't had to do much of any performance tuning. It just works, and is fast. Which is awesome!

But today, Youtopia was running slowly and requests were hanging so I investigated. The databases were performing normally and not having any locking issues. The network looked good. The memcached cluster was healthy. The queueing service looked great. The ASP.NET performance counters even looked good at first glance.

None of the diagnostic performance monitors I'd used in the past (such as Requests in Application Queue) showed the issue, but requests were absolutely being queued -- or otherwise not processed immediately. There were also plenty of free worker and IOCP threads. The only thing that clued me in was the Pipeline Instance Count and Requests Executing counters were exactly the same (96) on all the servers. So I started investigating from there.

It turns out that due to the way IIS7 ASP.NET integrated mode threading model functions there is a (configurable) request limit of 12 per CPU. We hit this limit in Youtopia today because we hold open requests for asynchronous Comet-like communications and there were over 288 people online simultaneously. Our three eight core web servers each had 96 (8*12) people connected to them and weren't really serving any other requests. We aren't running into any thread configuration limits as the long running requests are asynchronous and not using ASP.NET worker threads.

Here are a few great links that came out of my research.

• http://blogs.msdn.com/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx
• http://support.microsoft.com/kb/816995
• http://msdn.microsoft.com/en-us/library/ee377050(BTS.10).aspx
• http://blogs.msdn.com/webtopics/archive/2009/02/13/asp-net-hang-in-iis-7-0.aspx

With ASP.NET 3.5 SP1 it boils down to a simple configuration file change. Use something like this in the aspnet.config file (in x64 it's at C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet.config). This is the default. Adjust maxConcurrentRequestsPerCPU to suit your needs.

<system.web>
   <applicationPool maxConcurrentRequestsPerCPU="12" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000"/>
</system.web>

In addition, the application pool needs to be configured to allow more requests. By default it only allows 1000 concurrent requests. This is done under the Advanced Settings for the application pool in the IIS 7 manager. Set Queue Length to 5000 to match this system level configuration.

ASP.NET MVC sucks and so does jQuery and PHP

Apparently, saying something sucks gets you a lot of hits. I think I'll use this tactic more often. My post on 10 Reasons ASP.NET Webforms Suck has been quite the talk in our tiny little .NET blog world this week. Who knew you all had such strong opinions on the matter!

ASP.NET doesn't completely suck

Saying something sucks doesn't mean it isn't good enough, or isn't the best option. In my mind everything sucks. My cell phone sucks, my laptop sucks, my operating system sucks, my car sucks. They all need improvements. They are all far from perfect. It is this mindset that drives me to build better software. If you can't see the flaws around you how can you improve on them? ASP.NET 4.0 didn't come around for fun. It came around because 3.5 sucks and needs improvement, and so on, and so forth.

Quite a few of you wrote some great rebuttals. Some were utter nonsense, but hey, this is the internet. That's to be expected. I'd like to talk about my favorite comments:

"It's obviously not a perfect design, but, it did it's job." – Robert Sweeney

Indeed, my thoughts exactly. It did its job. The internet has moved along really fast. Webforms are lagging behind a bit. Sure, it's still perfect for RAD business type applications. But build a web game on it, or a "Web 2.0" web site, or other consumer facing web product. The level of customization you end up doing to work within the bounds of the framework's abstraction starts to become silly.

"I agree that is does suck now." ... "over time, better ways to do things are created and naturally, the old ways get laid to rest" – shaun

Yeah, this is the nature of things. The technology will be around forever, but the development world will pass it by. We like shiny new things.

"Anyway, this 'hidding how HTTP works' philosophie that ASP.NET follows in every single corner of the framework is the real problem. Django, Ruby on Rails, and PHP doesn't try to hide the fact that you are building a website/page/app and help you in the process of coding with helpers, decorators or simple functions." – Angel

YES! HTTP, HTML, CSS, Javascript. These are the technologies we work with on the web. They're simple. Learn them, love them, embrace them. It'll also make your skills a lot more transferable should you ever be looking for work.

"Newb" – rabbit

pwned!

"loooks like newly migrated from php/java." – web spider

Did you read my post? I have been using ASP.NET since it was in beta. I live and breathe this stuff and use it every day. I'm simply pointing out the flaws I see.

"I've used PHP, Drupal, Rails and even FastCGI in the bad old early days and find I'm always coming back ASP.Net. Security, data abstraction layers, controls, validation, scalability, application recycling, caching, session management and great development/debugging environments are just to hard to pass up." – Mike

Yeah, I love ASP.NET too. I don't use anything else on a serious basis. You definitely mentioned my favorite parts of it, especially the last one.

"This model has been created back in what 1999/ 2000 when MS started working on .NET 1.0 (was released in 2002). So we are talking the model/architecture is almost a decade old, way before the Web 2.0/Ajax days." – Bart Czernicki

Yes! It's old (mature as some say)! Is it at all possible there is a better way now?

Chris Vanderheyden said a lot

"Honestly, after more than 8 years of professional experience: YES, your SHOULD be out of that highschool mentality. (Look my editor can do only 2 colors i am so L33T...) "

"I am a developer, i write logic, not translations. I WANT my HTML abstracted. I don't want to write zeroes and ones down to the NIC now do i. "

Since you likely know me only from my 'it sucks' post, you're going to find this shocking. I agree with most of the justifications you made (except the ones I listed above). My biggest beef is with your comment on my #1. You obviously have no sense of humor. ;) And, why wouldn't you want to write html? It's a simple human readable markup language, not binary networking protocols. XHTML+CSS is abstraction at its best. In fact it's usually just as simple as the abstractions provided by ASP.NET controls. I mean, really, can you actually point at one of your ASP.NET apps that would run outside of the context of your modern web browser? Something other than html 4.0 or whatever you're using? You have to learn a lot about the quirks of ASP.NET to get things done well. Why not learn the quirks in html/css/js? Oh wait, you dohave to do that too. The leaky abstraction abounds.

"I only have 1 reason... Leaky abstraction over HTTP that introduces instead of removing complexity. Every other reason is a derivative or effect of this one reason" – Greg Young

Thanks Greg. You always have a nice way of distilling things down. But that wouldn't make nearly as fun of a blog post!

"I think that JD Conley is just sarcastic. Actually he loves .NET" - br_other

No, I'm not being sarcastic. Perhaps dramatic. Yes, I love .NET. But it's not without its flaws.

"sos un boludo" – Sebastian

This is cooler than the "newb" comment! Thrasing in a foreign language!

"You don't have to use <%= ClientID%> stuff at all There is a much better way. I claim ASP.NET webforms has the "best" integration with client side DOM. You think I am kidding ? Have you ever heard IScriptControl ? I guess you didn't" – onur

Indeed I have heard of IScriptControl and use it quite a bit. It's an interesting and often useful abstraction. Though I always laugh at myself since to use it I add some C# code to generate some js code to call some other js code I could have just called in the first place if I were working in the markup.

And finally, we have the people who decided to write a full rebuttal on their slice of the net. Cool. Thanks for the link backs! Hope you enjoyed my comments.

• ASP.NET WebForms DO NOT SUCK – Mohammad Azam
• Mythbusters – Busting the myths about WebControls – Thomas Hansen
• 10 Reasons ASP.NET Webforms (Still) Rock – Lee Dumond

Mike Pope also posted an interesting commentary on the matter. Us silly kids and our toys! Aren't we allowed to change our minds, backpedal, or get excited by new technology?

Happy hacking!

10 Reasons ASP.NET Webforms Suck

I've always been a .NET Fanboy. I've been on the bandwagon since its inception. I've developed quite a few shipping .NET products for the web, Windows, and Linux. I've given talks at user groups, created a consulting company, and mentored developers new to .NET. I always experiment with the latest toys and try to stay ahead of the technolgoy curve. In my most recent role at Hive7, I've been focused on web technology. We have some pretty large scale games (millions of players) built on ASP.NET webforms and ASP.NET AJAX. It's been about 8 years since I've written a full blown web app that wasn't in ASP.NET webforms. Sure, there's the occasional small PHP or static html site, but no "real" applications have been built on anything but ASP.NET. I think I've been missing out.

I'm going to preface this by saying one thing. Ever try to train someone new to ASP.NET? Especially someone with any other web programming experience. It's not easy. That to me is a sign of suck, or maybe fail.

The Reasons (in order of frustration)

1. Other web developers assume you're inferior
   

   Let's face it, if you're coding in ASP.NET you are NOT initially considered one of the cool kids. It's automatically assumed you're a corporate lackey with no programming fu. You have to prove yourself. It sucks. Yes, this is #1. Afterall, don't you want other people to think you're cool? Or am I the only one still living in high school...?



2. One form to rule us all, one form to bind us
   

   I don't think I have much to say on this, other than. Why? What was the design decision behind overloading the html form and only lettings us have one? Why? Why? Why?



3. Viewstate
   

   Ever accidentally generated a 1MB (simple) page by just using standard controls?



4. ID insanity
   

   Mapping id's in html elements to id's in code starts out innocent enough. But, throw in nested controls (a recommended design practice) and hold on for your life. Once you get used to it everything makes sense. But try showing your dhtml/javascript guy how to use codebehind to grab a ClientID and pass that to his javascript code...

   
   
5. Html abstraction
   

   I truly hate that in webforms that you don't really write html code. Browser independent rending is just a bad, horrible idea. The abstraction sounds nice on the surface, but some day it will bite you. Web developers should know how to write html code, understand the web programming model, and the cross platform implications of their code.



6. Postbacks everywhere
   

   Linkbuttons, and any of the controls with 'autopostback' should be taken out on the street and shot. Posting back to the initial page as the default to perform an action is just counter intuitive. And then, how do you consume this action? An event handler? Weird.



7. Request lifecycle
   

   Init, Load, PreRender. WTF? Try explaining that one to your javascript guy. The fact that we need a 10 step lifecycle for things to work sends off warning bells in my head.



8. Getting data to the client
   

   Ok, I've got this cool data driven web site. And now I want to do some AJAX. How do you interface your server code with your Javascript? You can pick one of 20 methods, none of which are simple, and all leave the developer scratching his head. Sometimes things magically work. Usually they don't.



9. Ugly URL's
   

   Ok, so this one is low on the list becuase the latest service pack added a routing engine. But hey, it's bugged me for the last 8 years. Customers want pretty url's. Webforms did not deliver without much hackery.



10. Codebehind
    

    I love c#. But, the concept of codebehind just seems weird. Why is there a separate file that's coupled to the html code? This nifty abstraction has been the cause of so many developer questions and Visual Studio environment issues I don't even want to go there.



11. The odd feeling that you have to beat the framework into submission to get it to do what you want
    

    Ok, this is #11, I know. But hey, something just feels wrong in webforms. Like you're trying to stick a square peg through a round hole.

As I look over this list I realized that most things I hate about ASP.NET Webforms related to the choices that were made about abstractions. I don't understand what was so scary about the web programming model that these decisions were made. In fact, now that I think about it, I'd have been happier sticking with the classic ASP programming model than using webforms. Oh well. The last 8 years will now be known as "the time in my life when I had to code on that ASP.NET webforms junk". Ok I'm done complaining for now. Posts in the forseeable future will be about happy things like ASP.NET MVC, Azure, and jQuery.

Most recently I've been working with the ASP.NET MVC framework and I have to say, wow. What a relief. It reminds me of doing web programming when I actually wrote simple Html code for my first web site. It's not really the MVC pattern per-se that attracts me. The joy comes from not being coupled to the desires of the ASP.NET framework developers whims. I can write javascript, html, and css. I can write server side code. And guess what, it's not necessarily coupled! Maybe in 8 years I'll be singing a different tune. But for now, I'm happy again.

Edit: I posted a follow up.

Debugging after a power outage

Here at Hive7 we host all our servers with a hybrid co-location/hosting provider. When you host your servers in a colo facility there are a few key things you look for. Stuff like multiple redundant internet routes, clean power, zero interruption backup power systems, adequate cooling, and decent security. While our host has all of these, after all they come standard in any decent co-location, they also provide us with a few services above and beyond a bare bones colo like good prices on rented servers of any configuration and hardware load balancers. We've had our share of small mishaps, but things have been pretty smooth sailing. That is, until, about 36 hours ago.

Sometime around 5am PDT PG&E had a major power outage. Normally when this (rarely) happens in a colo, the backup batteries and generator carry you through without even a hiccup. Well, not this time. Backup power systems faltered and servers went down. Things quickly came back up, but, there was a catch. The Air Conditioner in our data center did not! Servers rapidly began overheating. Quick to react, the colo's on site engineers hard powered off a bunch of our servers (I'm still not quite sure why we didn't get a phone call to do this on our own the safe way).

Text messages flew (Zenoss is your friend) and our chief sys admin dude rushed down to the data center to assess the damages as soon as he was alerted. On the surface there were a few major issues. Many of our larger RAID Arrays were running degraded and needed some love. Some servers had not powered back on and were stuck. He spent all day trying to right the wrongs made by our colo's (lack of) backup power.

While he was doing that, guess what I was doing? Yeah, trying to make Knighthood run. Knighthood is a pretty big web app. We have millions of users, and roughly 10,000 actively playing at any given point in time. With a few database servers running with degraded arrays, virtual machine hosts not running, and some other systems still not powered on I set forth scouring logs and troubleshooting. One by one we got the necessary systems back up and running. First it was the email service, then the email invitation service, then the Active Directory domain controllers. With all of these up and running properly, the game was still performing poorly. It was exhibiting performance traits I'd seen before. It'd go fast for a bit, then completely hang, then go fast again.

In the past, a performance pattern like this has been caused by some reader/writer lock contention where an upgrade to a write lock causes everything to stop. It's also been caused by transactions hanging in Distributed Transaction Coordinator, or SQL Blocking, or contention on a big cache item in Memcached, or some code that is infinitely recursing. So, I did my normal troubleshooting in this scenario.

I popped open perfmon to look at request rates, bandwidth, cpu, threads, memory, etc across the farm. Every single front end IIS server (7 of them for this game) was processing absolutely nothing for a 20 second period. And by nothing, I mean nothing. IIS wouldn't even serve up an image! And CPU was 0% utilized. But, once that 20 seconds was over we'd get a good 5 seconds of processing done. Everything was queued and no requests were dropped. I thought for sure the power outage had severely damaged one of our databases, DNS servers, or a big chunk of cache (we run about 60GB of memcached), or something else obvious like that. It really felt like something was timing out and then letting the flood gates open.

At this point the logs weren't showing any errors to lead me down a debugging path. Since the behavior was happening consistently I decided to grab a hang dump of the w3wp process. The dump was, well, completely surprising! Guess what was happening?

No really. Guess.

Give Up? Nothing! Yeah, nothing. During that 20 seconds all the managed and unmanaged threads were completely idle doing absolutely nothing. No locks were held. No pages were processing. I know that because of Tess's neat blog post about which threads you can ignore. It's as if IIS just decided we didn't really want to process any more requests. This had me scratching my head. I repeated the process another 4 or 5 times with the same result. It seemed the problem must be somewhere in kernel space. None of my user mode dumps found anything at all. That scared me.

I have no experience doing kernel debugging. So, well, this is where I partially threw in the towel. I called Microsoft support and opened a premier support case. It had been about 8 hours and we had a LOT of pissed off paying users. After a few hours on the phone we captured some more user mode dumps (they didn't believe me that there wasn't anything interesting there) and uploaded them. I wish saying "I am experienced doing production crash dump debugging" meant something to these guys... I'm not sure how many times I had to say "No, you see, when it hangs it uses 0% CPU!".

The Microsoft crash dump engineers went about their business and said they'd call me back when they found something (though I knew they wouldn't find anything and no amount of whining could make them skip this step). To their credit, since it was production and affecting our main line of business, they offered to do the debugging immediately rather than withing 2 business days which is the normal turnaround.

A couple hours later everything magically started working perfectly. I changed nothing. I called our sysadmin and he said he changed nothing that should have affected Knighthood. It had already been a long day and we decided to wait for the next day to find what had fixed it. Sysadmin dude sends me an IM this morning to let me know he figured out the problem. Guess what it was?

No really. Guess.

Well of course, it was IIS logging! We have all our IIS logs pointed at a NAS. I never thought this would be an issue since, well, logging happens in a background thread right? It can't possibly interfere with actual request processing. Turns out that is incorrect! The NAS was one of the last things that Sysadmin dude had brought up at the end of the day because it was only used for archiving and log files, a very low priority in a crisis scenario. Well, apparently not!

Microsoft called me back about an hour later to let me know that the hang dumps did not uncover anything and it appeared that requests were simply being delayed before they could be processed. "Well, no $@*! I told you that 12 hours ago", I thought. So, I let them know we had fixed the problem and closed the case. I'm sure we could have seen it in a kernel hang dump in the IIS kernel mode stuff, but the problem was not reproducing anymore and I didn't want to bother...

Now you know. IIS logging can clog up all your requests. So, if you're logging to a remote system over a windows share, make sure it never goes down! Or, well, don't do that and log locally and ship them out on occasion.

ASP.NET - It's not just for DataGrids after all

Last night I gave a talk at the San Francisco chapter of the Bay.NET User's group. It was a lot of fun. Thanks for the great interaction everyone! I am also thoroughly impressed that I finished nearly on time and didn't have 10 slides left! Usually I have way too much material for these things.

As promised, here are the slides from the talk.

If any other groups out there are interested in this talk, let me know!

Async Facebook Library Teaser

I've been trolling the Facebook Developer Forums recently and talking with other developers. I've also been reading a lot and trying to get a feel for what people might want out of the asynchronous Facebook library I've been (slowly) building. In particular, I've been participating in this thread about Facebook scalability issues. I thought I would share my most recent post as a bit of a teaser as what's to come. Here it is, reproduced:

tomten wrote:

...but it was way overkill for what I've done. ...

This is one myth I want to squash. Async stuff isn't hard. I promise. smile Let's take your page loading example, and make it asynchronous. I put in code comments for a "facebook is hammered" timing. Let's say every call takes you 250ms. That means that page will spend roughly one second waiting for facebook.

Page_Load {
 GetSession(); // Call to Facebook 250ms
 RenderHtml();
 Response.Flush();
 Response.Close();
 // Make calls to Facebook
 GetFriends(); // Call to Facebook 250ms
 UpdateProfile(); // Call to Facebook 250ms
 SendNotification(); // Call to Facebook 250ms
}

Now, let's make this Async. It would become (roughly -- there is also a "Completed" event handler you'd setup for the items you cared about):

Page_Load {
fbasync.GetSessionAsync();
fbasync.GetFriendsAsync();
fbasync.UpdateProfileAsync();
fbasync.SendNotificationAsync();
}

Page_PreRenderCompelete {
RenderHtml();
Response.Flush();
Response.Close();
}

This time, with the async calls in the server, we spend 0 time waiting for facebook. Just like in client side AJAX, we get a callback when the calls were complete! And, instead of a full second for our page to finish processing, it only takes 250ms, as all the facebook calls happen simultaneously (yeah I know this is a faulty assumption, but it is usually pretty close).

In addition to the shorter page lifecycle (250ms instead of 1 second) we are not tying up a processing thread (of which there is a small finite number). During that time we are waiting for facebook to finish our API calls, the asp.net engine happily processes more requests on the thread that originally kicked off processing of our page.

A shorter lifecycle is important, not only for throughput, but also for memory consumption and garbage collection efficiency. Because the objects do not live for as long, they will be garbage collected more quickly and your application will spend less time doing garbage collection.

Ok, so that's a bit of a spoiler, but, I hope it helps.

Hopefully some time this week I'll clean up the library enough that I think it's worthy to be posted publicly. I spent last night refactoring a bit and removing a lot of "who wrote this crap" code. :)

As a bit of an aside, I was excited to see the team working on the "almsot official" Microsoft .NET library for Facebook released an asynchronous version of the FacebookService. I thought maybe I wouldn't have to write one myself. However, after looking at the code, I was very disappointed by the fake asynchronous support that's in the Facebook Developer Toolkit. It made me shed a tear. I really wish they'd put a disclaimer up: "This AsyncFacebookService is not really asynchronous I/O. It will only make things worse by tying up an additional threadpool thread waiting for the delegate to the synchronous method to return.".

Linq to Sql Surprise Performance Hit

A couple days ago I built Photo Feeds. I wrote about it here. After using the app a bit I noticed something peculiar. The page took a really long time to load. From my local web server it was on the order of 2-5 seconds. That's way too slow.

Having written the code a few hours prior, I immediately knew what was wrong. I thought "Well, duh, either Facebook sucks, I'm hitting my database too often, or I'm calling the Facebook service too often." I/O is, after all, usually what causes an application this simple to be slow. I glanced over the code and thought to myself "Holy crap, who wrote this junk? It's calling the database waaaaay too much. That has to be the problem."

Having been down this path many times I decided to analyze a bit more before officially jumping to conclusions. I popped open task manager to see if I was in fact I/O bound (i.e. to see if my CPU was racing or idle) – usually a good rough indicator of what's going on. To my jaw-dropped amazement the CPU was hitting 100% for the majority of the page request time. "Could it be the JIT?" I thought. So I hit refresh 50 times and watched in amazement as the CPU spiked each time. I switched over to the Processes tab and, again to my amazement, noticed it was the web server chewing up the cycles, and NOT SQL server. Did I mention I was amazed?

Ok, so apparently I was wrong [insert shock and awe]. The web site was way too CPU hungry! Good thing I didn't put any bets on that one. I then theorized that maybe Cassini just sucked (my code couldn't be that bad). I whipped up a virtual directory in IIS and ran my tests again.

Nope, it wasn't Cassini. Alright, so now let's get scientific. I pop open ANTS and setup a profiling session on my virtual directory. I loaded the page a few times and, lo and behold, the worst offender screen pops up:

Wow! That's a long time (and a lot of hits). I was definitely right. I'm hitting the database way too often. But, I know I saw the CPU racing... What the heck?

Here's the offending code:

public static Feed GetOrCreateFeed(
FeedsDataContext dc,
string owner,
FeedSourceType sourceType,
string facebookId)
{
var q = from f in dc.Feeds
       where f.OwnerId == owner &&
       f.FeedSourceType == (byte)sourceType &&
       f.FacebookId == facebookId
       select f;

Feed existingItem = q.SingleOrDefault();

...
}

So I dug into the call graph a bit and found out the code causing by far the most damage was the creation of the LINQ query object for every call! The actual round trip to the database paled in comparison. Now that was, again, a huge surprise. Check out the hit counts on this call – holy cow!

1,176,879 calls to SqlVisitor.Visit. Over a million calls to anything for loading a page three times can't be good.

I started doing some research. I remembered reading about using compiled LINQ to SQL queries for optimization a few months ago. I did a little more searching and ran into this gem for anybody scared of lambda functions. Warning, scary .NET 3.5 lambda functions ahead. Here's the magic fix:

private static readonly Func<
FeedsDataContext, string, FeedSourceType,
string, IQueryable<Feed>> _compiledGet =
CompiledQuery.Compile(
(FeedsDataContext dc, string owner,
FeedSourceType sourceType, string facebookId) =>
   from f in dc.Feeds
   where f.OwnerId == owner &&
   f.FeedSourceType == (byte)sourceType &&
   f.FacebookId == facebookId
   select f);

public static Feed GetOrCreateFeed(
FeedsDataContext dc,
string owner,
FeedSourceType sourceType,
string facebookId)
{
Feed existingItem = _compiledGet(
dc, owner, sourceType, facebookId)
   .SingleOrDefault();

...
}

If that's your first time seeing lambdas in C#, I feel for you. I'd suggest trying them out and doing a lot of research. That simple change yielded the following result:

That method is still by far the worst offender in the application, but is five times faster than it was and barely registers on the CPU when the page loads. This one change netted sub second page rendering times which is good enough for me. Obviously I still have major issues, hitting the database way too often, but I'm a big believer in not doing any premature optimization. But now, at least I'm back in my comfort zone and am I/O bound.

So, what have we learned? Never, ever, ever, ever, ever, ever, ever, ever, ever blindly edit code to fix performance issues. Always run it through a profiler. No matter how much experience you have performance tuning, the worst offenders will still surprise you 99% of the time. Oh, and sometimes LINQ to SQL is slow at creating its query objects. Use compiled queries in these cases!

More Visual Studio 2008 Beta 2 to RTM Upgrade Gotchas

Yesterday I posted on the first thing I ran into. Today I ported another project and hit a couple more snags.

1. Linq to Sql Unicode Byte Header
2. Linq to Sql Add/Remove Method Changes
3. ASP.NET Listview Changed from itemContainer to itemPlaceholder

Linq to Sql Unicode Byte Header

The xml document that makes up the Linq to Sql dbml file is declared to be in unicode format. For some reason my file was not saved with the unicode byte header. Apparently this didn't matter to the beta 2 framework, but in RTM release the project failed to build and double clicking on the designer caused the error "There is no Unicode byte order mark. Cannot switch to Unicode."

When the error occurs it opens the dbml file as XML. So, I just moved my cursor to the first position. Hit enter then backspace and saved the file. Visual Studio 2008 RTM put the Unicode Byte Header in for me and everything was kosher.

Linq to Sql Add/Remove Method Changes

The Linq to Sql Table<T> and ITable has changed the signature for adding and removing entities. Instead of the Add and Remove methods there are now InsertOnSubmit and DeleteOnSubmit. This makes sense, as it was not obvious what Add and Remove would do. These new names make a lot of sense to me and were outlined in a msdn forum posting a few months ago.

ASP.NET Listview Changed from itemContainer to itemPlaceholder

It looks like a few bloggers already beat me to this one, but the behavior of the itemContainer in an ASP.NET Listview has changed and the new control ID you have to use is called itemPlaceholder. Daniel Moth has a good description on his blog here. The annoying thing about this change for me was that it showed up at runtime, and not compile time. Though the runtime message is pretty descriptive: System.InvalidOperationException: An item placeholder must be specified on ListView 'ListView1'. Specify an item placeholder by setting a control's ID property to "itemPlaceholder". The item placeholder control must also specify runat="server".

Wrap Up

Well, that's all I've run into so far. I'm sure I'll find more issues as I wrote quite a bit of (production) code under 2008 Beta 2. I'll make sure to summarize any more findings here.

Small Gotcha in .NET Framework 3.5 Beta 2 to RTM Upgrade

Visual Studio 2008 was officially released today to MSDN subscribers so I took the leap and did the upgrade. I ran into an issue with a simple fix that I thought everyone should be aware of. You'll get an error message stating: "Could not load file or assembly 'System.Data.DataSetExtensions, Version=2.0.0.0...". This is somewhat cryptic, but easy to fix.

If you have a web project that was running under .NET 3.5 beta 2 you will have an assembly reference as follows:

<compilation>
<assemblies>
  ...
  <add assembly="System.Data.DataSetExtensions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>
</assemblies>
</compilation>

Notice the version number is 2.0.0.0. This was apparently changed at the RTM and the version number should be 3.5.0.0 like:

<compilation>
<assemblies>
  ...
  <add assembly="System.Data.DataSetExtensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>
</assemblies>
</compilation>

ASP.NET MVC Framework

I'm a huge fan of Test Driven Development and I've been looking forward to the ASP.NET MVC (Model View Controller) Framework to increase the level of testability of web projects. Scott Gu just posted a great and frighteningly in-depth article – I don't know how he has the time. If you like releasing quality/tested, code, check it out. If you're not a developer, make your developers look at it.

Stringbuildify!

A task that I often end up doing when coding an actual web site (i.e. not writing a sample or some such) is adding client script to a page/control in codebehind using the ClientScriptManager. Let's say you've got the following alert script you want to add to the page so you can use it in a control:

function doAlert()
{
 alert('welcome!');
}

Well, there are now a number of ways to get this into your page, but the quickest, in-line way is to use the ClientScriptManager. Like so:

if (!Page.ClientScript.IsClientScriptBlockRegistered(
   this.GetType(), "alert"))
{
   System.Text.StringBuilder sb = new System.Text.StringBuilder();
   sb.Append("function doAlert() {");
   sb.Append("alert ('welcome!'); }");
   Page.ClientScript.RegisterClientScriptBlock(
       this.GetType(), "alert", sb.ToString(), true);
}

Ok, so that's a bit contrived. You're not going to use a StringBuilder for something that simple. But, with a more complicated script you probably would, especially if the script will be different depending on the state of the page.

I had a little Winforms utility sitting around I've been using for a while to automate that process for me, and I decided to put it up on the web. So, here you go: http://jdconley.com/stringbuildify. You feed the engine a multi-line string (like a script) and it gives you back a StringBuilder with everything properly escaped and such. The code's a bit of a mess at this second, that's why I didn't post it. Enjoy!

ASP.NET Centric Extensions

I've been loving the new Extension Methods in Orcas. If you haven't had the pleasure of working with them, well, you're really missing out. They are utility at its finest. Here is a list of my favorite extensions I've written thus far.

URLEncode

Ok, I know this sounds stupid, but for some reason I really hate typing HttpUtility.UrlEncode(myString) all over the place so I wrote this extension.

public static string UrlEncode(this string toEncode)
{
   return HttpUtility.UrlEncode(toEncode);
}

ToHexString

For some reason my memory for .NET format strings really sucks. I blame Intellisense. I always forget how to hex encode integers. In case you're wondering, this is quite useful for giving users short strings that can be very unique and directly represent a primary key in a database. Hex is generally friendlier on the eyes than a number alone.

public static string ToHexString(this int val)
{
   return string.Format("{0:x2}", val);
}

FromHexString

Likewise, when you have a hex encoded integer, it's quite useful to get that value back out. I have no idea why, but I forget this neat trick all the time and constantly have to look it up. No more!

public static int FromHexString(this string val)
{
   return Convert.ToInt32(val, 16);
}

ToUrlString

I posted this a few weeks ago. I love it...

GetRootPath

This is something I have struggled with since the early days of ASP. There has never been a straightforward way to get the root of the URI of a request. This method extends HttpRequest and gives you the scheme (http/https) and host/port of the request. So, provided a request to http://jdconley.com/blog/ this extension would return http://jdconley.com

public static string GetRootPath(this HttpRequest r)
{
   string hostPort = r.Url.Authority;
   string scheme = r.Url.Scheme;

   string path = string.Concat(scheme, @"://", hostPort);

   return path;
}

GetConfigBasedPath

A common task in a web app is to send out an email for one reason or another. Maybe you want to send someone a link to a download, confirm their registration, or bug them when they haven't visited your site in a while. This method takes an application relative path stored in your App.Config appSettings and translates it into a fully qualified URL based on the current executing page. Your configuration might look like:

<add key="EventDetailsFormatString" value="~/events/{0}.aspx"/>

You could then use the following extension to turn that relative URL into a fully qualified based on the current page that is executing. It relies on the GetRootPath method above.

public static string GetConfigBasedPath(
   this Page p,
   string configKey,
   params object[] formatArgs)
{
   if (null == p)
       throw new ArgumentNullException("p");

   if (string.IsNullOrEmpty(configKey))
       throw new ArgumentNullException("configKey");

   string valFromConfig = ConfigurationManager.AppSettings.Get(configKey);

   if (null == valFromConfig)
       throw new ArgumentOutOfRangeException("configKey",
           configKey,
           "The specified key was not found in the configuration.");

   Uri rootUri = new Uri(p.Request.GetRootPath(), UriKind.Absolute);

   Uri relativeUri = new Uri(
       p.ResolveUrl(string.Format(valFromConfig, formatArgs)),
       UriKind.Relative);

   Uri fullUri = new Uri(rootUri, relativeUri);

   return fullUri.ToString();
}

IsDescendentOrSelfSelected

In a Treeview you often want to see if a node or any of its children are selected to help determine how to display the tree. This is especially useful when using the Treeview for navigation.

public static bool IsDescendantOrSelfSelected(this TreeNode node)
{
   if (node.Selected)
   {
       return true;
   }
   else if (node.ChildNodes.Count > 0)
   {
       foreach (TreeNode n in node.ChildNodes)
       {
           if (IsDescendantOrSelfSelected(n))
               return true;
       }

       return false;
   }
   else     {         return false;
   }
}

EnsureVisible

This one is quite simple. It just makes sure that the provided node is visible in the tree by making sure all of its parent nodes are expanded. This is a useful function for stateless Treeviews or synchronizing the same Treeview across multiple pages.

public static void EnsureVisible(this TreeNode node)
{
   if (null != node.Parent)
   {
       node.Parent.Expand();
       EnsureVisible(node.Parent);
   }
}

Well, that's it for now! I hope someone finds these as useful as I do.

Clean Up a String for a Url

This is a quickie. Yesterday I was doing some Url rewriting for a project and accepting user input for said Url. It's basically like this blog system. I can specify the friendly url of each post. So, I wrote a little extension method to clean up a string and make it URL friendly. Yes, you can just escape everything someone enters, but it's much friendlier to just make it work. Be lenient in what you accept! Here it is:

public static string ToUrlString(this string s)
{
   if (null == s)
       throw new NullReferenceException();
  
   string tmp = Regex.Replace(s.ToLower(), @"\s", @"-");
   tmp = Regex.Replace(tmp, @"[^-_\.\w]", @"");

   return tmp;
}

This simply replaces all whitespace with a "-", and all characters that are not "-", "_", ".", or Word Characters with an empty string.

In the end, if your input is "this is a @#)(*&*@$^ crock!" you'll get "this-is-a--crock". As a guy who was, at one time, scared of regular expressions I hope this helps someone.

Simpler isn't always better - AsyncOperationsManager

AsyncOperationManager... die!

We have a new invitation system coming out soon for SoapBox Communicator and I was doing a little premature optimization, err, I mean, capacity planning. :) I set about to perform the simple task (or so I thought) of sending all of our email based invitations asynchronously. The architecture of the system is pretty simple. We have an ASP.NET 2.0 web site that you use inside SoapBox for sending invitations to your friends. The web site talks to an internal web service. The web service talks to a database. It's a pretty classic nTier design, with one exception. All layers are asynchronous.

The UI uses asynchronous ASP.NET pages and RegisterAsyncTask to make the call to the EnqueueEmailInvites web service. The web service uses asynchronous web methods and reads from/writes to our database using the asynchronous data methods in SqlClient.

Async Web Method

     [WebMethod]
     public IAsyncResult BeginEnqueueEmailInvites(EmailInvitation invite, AsyncCallback callback, object state)
     {
         return InvitationFactory.BeginEnqueueEmailInvites(invite, callback, state);
     }

     [WebMethod]
     public string[] EndEnqueueEmailInvites(IAsyncResult ar)
     {
         return InvitationFactory.EndEnqueueEmailInvites(ar);
     }

Factory Calling DAL

public static IAsyncResult BeginEnqueueEmailInvites(EmailInvitation invite, 

AsyncCallback callback, object state)
        {
            InviteEmailAsyncState myAs = new InviteEmailAsyncState();
            myAs.Invitation = invite;
            InviteEmailAsyncResult myAR = new InviteEmailAsyncResult(myAs, callback, state);

            int emailInviteId = invite.Mutual ? GetInvitationTypeId(Constants.InviteTypeEmailMutual) : 

GetInvitationTypeId(Constants.InviteTypeEmail);
            string csvAddresses = string.Join(",", new List<string>(invite.Addresses).ToArray());

            Data.Invitations.BeginCreateReadInvitation(emailInviteId, invite.FromJid, invite.FromName, invite.Subject, invite.UserBodyPlain, string.Empty, csvAddresses, EmailInvitationCreatedCallback, myAR);

            return myAR;
        }

I write a lot of asynchronous code. A lot. I'm a bit of an asynchronous I/O zealot. In fact, I've had an article on the shelf called "Asyncify your code" for months now. l I just haven't got the sample code written for it yet. Anyway, my experience has led me to a simple conclusion: I hate the AsyncOperationManager.

Maybe it's because I write a lot of asynchronous code, or maybe just because people tend to Mistake Familiarity with Superiority, but whatever the reasoning, I can't stand it. The AsyncOperationManager is used all over the .NET 2.0 framework where you see calls ending in "Async", even though you probably don't even know it. One such instance that rubbed me the wrong way yesterday (and into this morning) is the SmtpClient. Don't get me wrong, the SmtpClient is a great improvement over the old CDO based classes, but the Async support leaves something to be desired.

First off, you can't call Send or SendAsync until the previous operation has completed. That means if you're sending out more than one email in succession you're going to need to do your own queueing. So I said to myself "Ok, this is easy, I'll rip out the loop, register an eventhandler for SendCompleted, and pass a Queue through as state to SendAsync." This led me to the next problem.

The asynchronous web method called the asynchronous data method, which led to a callback. Inside of this callback I grabbed some info from the data reader result and composed the MailMessage. I created a SmtpClient, registered for the SendCompleted event kicked off the first SendAsync, and completed the asynchronus web method. The mail sent great, but my SendCompleted event handler was never called.

Apparently, if you make an async call there is the potential that the completed event handler you registered will never get called. Yep. You can call SendAsync, and the SendCompleted event will never be raised. I admit, the circumstances this happened in were kind of abnormal, but still. That just shouldn't happen.

After a little reflectoring it was apparent this had something to do with the SynchronizationContext at the time of the call used by the AsyncOperationManager inside of the SmtpClient. I got a litte cross-eyed looking through the code in reflector so I gave up my hunt for the exact answer. My workaround: Call ThreadPool.QueueUserWorkItem and create the SmtpClient and call SendAsync inside of that callback. This gets you the generic SynchronizationContext that doesn't have anything weird associated with it, just plain ole async delegates. If someone has a better solution, please let me know.

The Hack

             if (messagesToSend.Count > 0)
             {
                 //this seems strange, but we queue up the creation of the SmtpClient to a threadpool thread.
                 //the synchronization context seems to become invalid when we complete the async web method call                     
                 //and we don't want to wait for the emails to go out to complete the web service call.                     

                 System.Threading.ThreadPool.QueueUserWorkItem(StartSendingEmailCallback, new object[] { myAr.MyAsyncState.Invitation, messagesToSend });
             }

I have a plea to all framework designers out there. Have the decency to give me the good ole IAsyncResult based async pattern without all that extra AsyncOperationManager baggage! I don't want to have to worry about all that. I'm a big boy. I can handle my own synchronization, and I like it that way.

Validating JabberID Nodes (XMPP/SoapBox User Names)

One of the most common, and tedious, tasks that comes up while writing software is validating user input. "Don't trust the user," the mantra goes. Even with current development tools it is still suprisingly difficult (i.e. not automatic) to validate user input and provide useful feedback to people who enter bad data.

For the web (http/dhtml), ASP.Net provides the best implementation I've seen so far. Controls like the RegularExpressionValidator combined with a ValidationSummary or the more recent Validator Callout allow you to create a very intuitive and reactive interface while still providing server side input validation. Well, how does all this apply to JabberID Nodes?

A JabberID Node is essentially your user name on an XMPP domain. I am [email protected], thus "jconley" is my JabberID Node. A JabberID is represented by the ABNF form: [ node "@" ] domain [ "/" resource ] (for more details see the Addressing Scheme section of RFC 3920). Now, as with any good RFC the exact definition of what is permitted is available. It's, long, it's complicated, it's (eek) Unicode, it's called StringPrep. To be more precise, the NodePrep Profile Prohibited Output. The domain and resource also have their own rules, but maybe we'll talk about that another day.

Up until recently I haven't had to worry much about StringPrep. Chris was the unlucky soul here to bite that one off, so he is officially our in-house Unicode geek. Way down deep in our coversant.corlib assembly of the SoapBox Framework there is a namespace called "Coversant.StringPrep". This is automatically used by our JabberID class to make sure that JabberID's are always in the right format and do not contain any bad data. Well, this last week I had to build a web page that someone could enter JabberID Node information on. I wanted this to be, for the most part, handled on the client side to provide instant feedback and correction for common mistakes (like including a space), but still use the server for final validation (don't want those evil bots trying to mess with my page!).

The most elegant way I could think of was to create a regular expression to validate the input. This would let me utilize the controls already in ASP.Net for client and server side validation. As it turns out this isn't straighforward. From what I could figure out, the Regex engine included with Microsoft.Net (and today's browsers) do not support unicode code points above 16bits. Nodeprep Prohibit says There are thousands of disallowed code points above there (it goes up to 32 bits).

So, in the end, we validate as much as we can on the client with the following expression (I shamelessly stole the ASCII portions from Artur - a guy I met at the XMPP Interop Event last year): ^([\x29\x23-\x25\x28-\x2E\x30-\x39\x3B\x3D\x3F\x41-\x7E\xA0 \u1680\u202F\u205F\u3000\u2000-\u2009\u200A-\u200B\u06DD \u070F\u180E\u200C-\u200D\u2028-\u2029\u0080-\u009F \u2060-\u2063\u206A-\u206F\uFFF9-\uFFFC\uE000-\uF8FF\uFDD0-\uFDEF \uFFFE-\uFFFF\uD800-\uDFFF\uFFF9-\uFFFD\u2FF0-\u2FFB]{1,1023}) Oh, there are some artificial line breaks in that Regex. Don't include those. :) We leave the rest up to a CustomValidator on the server side and the SoapBox JabberID.ValidateUserText method.

.net 2.0 web service hair puller

Every couple of weeks I spend four hours doing something that should take five minutes. It just happened, and now I feel compelled to take another few minutes and explain so it doesn't happen to you. Not only did I waste my time, but the time of another one of our developers. What might waste four hours, you say?

We recently migrated our entire web site to .NET 2.0 and a new portal. Since we were building a new web site anyway we thought, "What the heck, let's re-factor the licensing subsystem. The database was hacked together over three years and we don't want no stinkin' .NET 1.1 code running on our shiny new site!" Well, this didn't turn out exactly as planned.

The SoapBox licensing web service is quite simple. There's a single method called "Activate" that takes in a unique hash of some information on the user's computer (so we can track duplicate usages) and the serial number. It returns an XML document containing all the license information, signed with our private key, that the SoapBox Server then validates with the embedded public key. That's it.

We get a call from a customer today saying they can't enter a license key into the MMC. They're getting a wonderfully descriptive "Object reference not set to an instance of an object" error. We immediately kick into debug mode. Afterall, if a potential customer can't enter in our trial key they're going to get upset and likely just give up on using it all together.

After a little debugging we found the web service wasn't returning the license xml document like it should have been. This was _very_ weird, considering our licensing page uses the same web service as our MMC and it was working just fine with the exact same parameters. Much head to wall interaction took place and we had another breakthrough, the web method wasn't receiving any data from the parameters. The serial number was null!

Now we bust out the packet sniffer. After careful examination, a new mini test application to call the service, and three Diet Cokes, we found the problem.

Apparently, during the migration and re-factoring process, we missed a character in a copy/paste operation. That's right, the trailing "/" on the namespace declaration for the web service.

This in itself is not such a suprise. However, the suprise is that .NET did not throw a "There is no matching web method you dummy" exception. No, it didn't throw an exception at all. Instead it called the correct web method, but did not pass in any of the parameters. Oiy.

It turns out our licensing page was working because the web reference was created after the new licensing service was put in place. It all makes sense now... Time for a beer.