Friday, November 30, 2007

An Entrepreneurial Story

I don't remember how exactly I got there, but while clicking through my morning feeds I ended up at the very interesting story of an Australian .NET micro ISV startup called Shuffle Text. They outline pretty much everything we learned when we started Coversant nearly seven years ago.

Some Gems
"And while we believed that technical differentiation would be enough to sell our product, we ended up being dead wrong."
"Change, it turned out, ended up being one of our worst development enemies."
"Of all those Google Ads signups, we got 0 feedback. All feedback on the product beta came from friends or friends of friends."
"There’s a bajillion ISV’s out there all competing to sell their crappy untested memory leaking piece of junk software. No matter how much time you spend perfecting your algorithm, improving the performance, and making sure you’ve got zero tolerance for leaks, NOBODY CARES. At the end of the day, it’s getting noticed that matters, convincing people that they should take a bet on you to solve their problems."

Don't give up guys. The product's been out less than a month and people are blogging about you! Your marketing seems pretty solid to me. While I don't have a use for your software at this very second. I'll certainly keep it on my radar. Oh yeah, consider putting out a completely free version, as in beer, not a trial.

Wednesday, November 28, 2007

Linq to Sql Surprise Performance Hit

A couple days ago I built Photo Feeds. I wrote about it here. After using the app a bit I noticed something peculiar. The page took a really long time to load. From my local web server it was on the order of 2-5 seconds. That's way too slow.

Having written the code a few hours prior, I immediately knew what was wrong. I thought "Well, duh, either Facebook sucks, I'm hitting my database too often, or I'm calling the Facebook service too often." I/O is, after all, usually what causes an application this simple to be slow. I glanced over the code and thought to myself "Holy crap, who wrote this junk? It's calling the database waaaaay too much. That has to be the problem."

Having been down this path many times I decided to analyze a bit more before officially jumping to conclusions. I popped open task manager to see if I was in fact I/O bound (i.e. to see if my CPU was racing or idle) – usually a good rough indicator of what's going on. To my jaw-dropped amazement the CPU was hitting 100% for the majority of the page request time. "Could it be the JIT?" I thought. So I hit refresh 50 times and watched in amazement as the CPU spiked each time. I switched over to the Processes tab and, again to my amazement, noticed it was the web server chewing up the cycles, and NOT SQL server. Did I mention I was amazed?

Ok, so apparently I was wrong [insert shock and awe]. The web site was way too CPU hungry! Good thing I didn't put any bets on that one. I then theorized that maybe Cassini just sucked (my code couldn't be that bad). I whipped up a virtual directory in IIS and ran my tests again.

Nope, it wasn't Cassini. Alright, so now let's get scientific. I pop open ANTS and setup a profiling session on my virtual directory. I loaded the page a few times and, lo and behold, the worst offender screen pops up:

Wow! That's a long time (and a lot of hits). I was definitely right. I'm hitting the database way too often. But, I know I saw the CPU racing... What the heck?

Here's the offending code:

public static Feed GetOrCreateFeed(
FeedsDataContext dc,
string owner,
FeedSourceType sourceType,
string facebookId)
var q = from f in dc.Feeds
where f.OwnerId == owner &&
f.FeedSourceType == (byte)sourceType &&
f.FacebookId == facebookId
select f;

Feed existingItem = q.SingleOrDefault();


So I dug into the call graph a bit and found out the code causing by far the most damage was the creation of the LINQ query object for every call! The actual round trip to the database paled in comparison. Now that was, again, a huge surprise. Check out the hit counts on this call – holy cow!

1,176,879 calls to SqlVisitor.Visit. Over a million calls to anything for loading a page three times can't be good.

I started doing some research. I remembered reading about using compiled LINQ to SQL queries for optimization a few months ago. I did a little more searching and ran into this gem for anybody scared of lambda functions. Warning, scary .NET 3.5 lambda functions ahead. Here's the magic fix:

private static readonly Func<
FeedsDataContext, string, FeedSourceType,
string, IQueryable<Feed>> _compiledGet =
(FeedsDataContext dc, string owner,
FeedSourceType sourceType, string facebookId) =>
from f in dc.Feeds
where f.OwnerId == owner &&
f.FeedSourceType == (byte)sourceType &&
f.FacebookId == facebookId
select f);

public static Feed GetOrCreateFeed(
FeedsDataContext dc,
string owner,
FeedSourceType sourceType,
string facebookId)
Feed existingItem = _compiledGet(
dc, owner, sourceType, facebookId)


If that's your first time seeing lambdas in C#, I feel for you. I'd suggest trying them out and doing a lot of research. That simple change yielded the following result:

That method is still by far the worst offender in the application, but is five times faster than it was and barely registers on the CPU when the page loads. This one change netted sub second page rendering times which is good enough for me. Obviously I still have major issues, hitting the database way too often, but I'm a big believer in not doing any premature optimization. But now, at least I'm back in my comfort zone and am I/O bound.

So, what have we learned? Never, ever, ever, ever, ever, ever, ever, ever, ever blindly edit code to fix performance issues. Always run it through a profiler. No matter how much experience you have performance tuning, the worst offenders will still surprise you 99% of the time. Oh, and sometimes LINQ to SQL is slow at creating its query objects. Use compiled queries in these cases!

Tuesday, November 27, 2007

How much space are all my sql tables using?

I deal with a few dozen databases on a daily basis. Often times I wonder "What the heck is making this database 100GB?" Sure, you can click around in Management Studio and find figures on a table by table and index by index basis, but there has to be a better way!

So, I asked a colleague of mine – who happens to be the guy that did all the database work on the SoapBox products and the best database dude I know. He didn't have a good answer for me so he whipped up this tiny script [Edit: After reading my blog he told me he actually adapted the script from one he found on the net that didn't quite work] in a few minutes. It makes my head spin. It's not perfect, but it's darn close.

WITH table_space_usage
( schema_name, table_name, index_name, used, reserved, ind_rows, tbl_rows )
AS (
, o.Name
, coalesce(i.Name, 'HEAP')
, p.used_page_count * 8
, p.reserved_page_count * 8
, p.row_count
, case when i.index_id in ( 0, 1 ) then p.row_count else 0 end
FROM sys.dm_db_partition_stats p
INNER JOIN sys.objects as o
ON o.object_id = p.object_id
INNER JOIN sys.schemas as s
ON s.schema_id = o.schema_id
LEFT OUTER JOIN sys.indexes as i
on i.object_id = p.object_id and i.index_id = p.index_id
WHERE o.type_desc = 'USER_TABLE'
and o.is_ms_shipped = 0
SELECT t.schema_name
, t.table_name
, t.index_name
, sum(t.used) as used_in_kb
, sum(t.reserved) as reserved_in_kb
, case grouping(t.index_name)
when 0 then sum(t.ind_rows)
else sum(t.tbl_rows) end as rows
FROM table_space_usage as t
, t.table_name
, t.index_name
, t.schema_name
, grouping(t.table_name)
, t.table_name
, grouping(t.index_name)
, t.index_name

I hope that hurts your head as much as it does mine. Just goes to show that I'm no SQL guru, I guess. But hey, if you ever want to know where that space is going in your database, this script will tell you!

Photo Feeds Facebook Application

I've built quite a few applications for Facebook. These include SoapBox, Friend Photosaver, Photozap, and my latest creation (late last night) is called Photo Feeds.

Photo feeds, as of the written-in-12-hours v1.0, automatically creates RSS feeds for the photos of you and your friends. You can subscribe to these feeds using anything that supports RSS photos. Your news reader, your favorite photo screen saver, windows vista sidebar, google sidebar, digital photo frames, etc, etc.

In the near future I hope to make this more compelling with new features including adding deeper integration into your Facebook profile and publishing to the Facebook news feed. In addition, I'll be integrating the three photo products (Friend Photosaver, Photozap, and Photo Feeds) to provide a rich add-on suite to Facebook Photos aimed at getting your photos to your desktop outside of Facebook.

Of course, this latest application was written using .NET Framework 3.5, Visual Studio 2008, and my top secret (though soon open source) asynchronous Facebook library. If you're a .NET developer interested in writing Facebook apps let me know and I'll be sure to blog about the technical details in the future. There doesn't seem to be many of us in the Facebook world.

Monday, November 26, 2007

FileVersionInfo - My Long Lost Friend

I've been doing .NET development professionally since the first public beta release of the 1.0 framework. That's... well, a long time. It constantly amazes me how big the framework really is. There are a number of times I've written tens or hundreds of lines of code for a task that I later realized – or was smacked over the head with by a colleague with – that were in the framework. Heck, even after writing my Extension Methods post a couple months ago some friendly readers pointed out how I was writing too much code.

Today, I was updating a client's web application to show the version number at the bottom of the screen based on the actual version that was running (it used to just be hard coded). The version number on the site was supposed to show the full revision of the assembly – which is stored in the assembly file version. The process to build the web site follows Microsoft version standards so the assembly version number does not necessarily change with each revision, but the file version does. Therefore, the normal System.Reflection.Assembly.GetExecutingAssembly().GetName().Version.ToString() isn't really the version number we want to display.

But how do you get the file version? Well, with System.Diagnostics.FileVersionInfo.GetVersionInfo of course! I swear I have looked for how to do this numerous times with only win32 responses to my query. Anyway, apparently it's dirt simple. This time I ran into this thread with the answer. Check out the System.Diagnostics.FileVersionInfo class for more info. Duh!

I leave you with this to ponder: WHY, WHY, WHY is this feature not exposed through System.IO.FileInfo or some other class that makes sense (and I did check in Reflector to be sure I wasn't mad)?!

Wednesday, November 21, 2007

Command Line VS 2008 Project Upgrade Tool

I've converted a few projects to 2008 RTM now (from 2003 to 2005 to 2008 beta projects), and I wish John Robbins had published this blog earlier. For those of us that don't like clicking in things he presents a built-in command line switch on devenv to upgrade solutions. I decided to take it to the next level.

I built a small command line application that recursively scans a directory for solution files and prompts you to upgrade all of them. I tried it on a bunch of my solutions and it seemed to work pretty well.

To use the tool you just run it from the command line and specify the directory where you want to start searching. Or you can specify no arguments to search the current working directory.

    upgrader.exe c:\root\of\your\tree



That's it! Quick and easy. The output will look something like:

    Searching for solutions under c:\svn\websites
Upgrade "c:\svn\websites\mysite\my.sln"? [Y/N] n
Skipped "c:\svn\websites\mysite\my.sln"
Upgrade "c:\svn\websites\broken\site.sln"? [Y/N] y
Converting "c:\svn\websites\broken\site.sln"
Unable to convert solution "c:\svn\websites\broken\site.sln"
Check out the errors in the UpgradeLog.xml file.
Upgrade "c:\svn\websites\ok\site.sln"? [Y/N] y
Converting "c:\svn\websites\ok\site.sln"
Converted "c:\svn\websites\ok\site.sln"

Well, that's my simple app that took just a few minutes to write and saved the pain of many clicks through a wizard. Download it here.

Tuesday, November 20, 2007

More Visual Studio 2008 Beta 2 to RTM Upgrade Gotchas

Yesterday I posted on the first thing I ran into. Today I ported another project and hit a couple more snags.

  1. Linq to Sql Unicode Byte Header
  2. Linq to Sql Add/Remove Method Changes
  3. ASP.NET Listview Changed from itemContainer to itemPlaceholder
Linq to Sql Unicode Byte Header

The xml document that makes up the Linq to Sql dbml file is declared to be in unicode format. For some reason my file was not saved with the unicode byte header. Apparently this didn't matter to the beta 2 framework, but in RTM release the project failed to build and double clicking on the designer caused the error "There is no Unicode byte order mark. Cannot switch to Unicode."

When the error occurs it opens the dbml file as XML. So, I just moved my cursor to the first position. Hit enter then backspace and saved the file. Visual Studio 2008 RTM put the Unicode Byte Header in for me and everything was kosher.

Linq to Sql Add/Remove Method Changes

The Linq to Sql Table<T> and ITable has changed the signature for adding and removing entities. Instead of the Add and Remove methods there are now InsertOnSubmit and DeleteOnSubmit. This makes sense, as it was not obvious what Add and Remove would do. These new names make a lot of sense to me and were outlined in a msdn forum posting a few months ago.

ASP.NET Listview Changed from itemContainer to itemPlaceholder

It looks like a few bloggers already beat me to this one, but the behavior of the itemContainer in an ASP.NET Listview has changed and the new control ID you have to use is called itemPlaceholder. Daniel Moth has a good description on his blog here. The annoying thing about this change for me was that it showed up at runtime, and not compile time. Though the runtime message is pretty descriptive: System.InvalidOperationException: An item placeholder must be specified on ListView 'ListView1'. Specify an item placeholder by setting a control's ID property to "itemPlaceholder". The item placeholder control must also specify runat="server".

Wrap Up

Well, that's all I've run into so far. I'm sure I'll find more issues as I wrote quite a bit of (production) code under 2008 Beta 2. I'll make sure to summarize any more findings here.

Monday, November 19, 2007

Small Gotcha in .NET Framework 3.5 Beta 2 to RTM Upgrade

Visual Studio 2008 was officially released today to MSDN subscribers so I took the leap and did the upgrade. I ran into an issue with a simple fix that I thought everyone should be aware of. You'll get an error message stating: "Could not load file or assembly 'System.Data.DataSetExtensions, Version=". This is somewhat cryptic, but easy to fix.

If you have a web project that was running under .NET 3.5 beta 2 you will have an assembly reference as follows:

<add assembly="System.Data.DataSetExtensions, Version=, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>

Notice the version number is This was apparently changed at the RTM and the version number should be like:

<add assembly="System.Data.DataSetExtensions, Version=, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>

Tuesday, November 13, 2007

ASP.NET MVC Framework

I'm a huge fan of Test Driven Development and I've been looking forward to the ASP.NET MVC (Model View Controller) Framework to increase the level of testability of web projects. Scott Gu just posted a great and frighteningly in-depth article – I don't know how he has the time. If you like releasing quality/tested, code, check it out. If you're not a developer, make your developers look at it.

Friday, November 9, 2007

Dvorak's Gone Mad

Maybe I'm just young, arrogant, and naive, but I think John's getting a little senile. In his latest rant for his PC Mag column he bashes Google's plans for the Open Handset Alliance with the title "The Google Phone is Doomed". Go read that if you haven't already and work your way back here. Don't get me wrong, I'm not convinced the project will work either – more on that later, but his reasoning confuses me.

Why Dvorak Thinks it Won't Work
  1. Google isn't charismatic
  2. Smartphones suck
  3. Nobody wants the internet on their phone

Google isn't charismatic? People don't love Google? Really? Google has more mindshare than pretty much any internet company out there. Ok, so they put out a crappy social networking tool called Orkut that nobody has heard of. I'm not sure how that relates to the potential success of an open phone product.

Smartphones currently suck. I definitely won't be arguing with that. The operating system user interfaces are crap and the software packages suck. However, I don't get how that has anything to do with a vaporware smarthpone due out some time in the future. If anything, an open platform should lead to a bunch of startups trying to build the next great phone interface. If they can pull it off, this should jumpstart an entire industry of reform. Apple did some good things with the iPhone. I see no reason why the same can't happen with OHA.

I'm not sure what year he's living in, but pondering why anyone would want internet on the phone when they could use the phone is just plain dumb. How do you know the phone number of the restaurant without the internet (or finding a phone book somewhere)? And why would I want to talk to someone when I can get more information myself on Google or [insert name of search startup here]? Maybe it's just me, but this seems like someone arguing to keep their fax machine instead of getting email, or keep their email instead of getting instant messaging.

Why I think it might not work

Alright, so maybe I don't agree with everything John brings up, but I do agree with one. Google wants to roll up the entire mobile market. The carriers, the operating systems, the software, and the hardware manufacturers. I don't see one of the biggest industries in the world taking that lying down.

In addition to the huge hurdles on the business side of things the technical challenges are staggering. Phones are not simple platforms to build on. And from what I've seen Google has yet to produce a large, complex, consumer facing product. They have a great back end platform and a boatload of itty bitty – dare I say widget – sized front end applications. An operating system and SDK that runs on a phone will prove to be a huge task, indeed.

In any case, "doomed" is pretty uncalled for. Of course, I did read his article, look at his ads, and now all of you are going to head over there too. So, I guess that's a success... And hey, I'm just a punk kid anyway! :)

About the Author

Wow, you made it to the bottom! That means we're destined to be life long friends. Follow Me on Twitter.

I am an entrepreneur and hacker. I'm a Cofounder at RealCrowd. Most recently I was CTO at Hive7, a social gaming startup that sold to Playdom and then Disney. These are my stories.

You can find far too much information about me on linkedin: No, I'm not interested in an amazing Paradox DBA role in the Antarctic with an excellent culture!