Sunday, November 22, 2009

Angry Players Make Sunday More Interesting

Youtopia has been growing quickly the last couple of weeks. It's fun to watch and the team is really excited about it. Of course, with the growth comes a lot of performance tuning with our code. Today we hit an issue I wasn't expecting at all. . .

We've been running Windows 2008, IIS7, and ASP.NET 3.5 in production for a while now, but haven't had to do much of any performance tuning. It just works, and is fast. Which is awesome!

But today, Youtopia was running slowly and requests were hanging so I investigated. The databases were performing normally and not having any locking issues. The network looked good. The memcached cluster was healthy. The queueing service looked great. The ASP.NET performance counters even looked good at first glance.

None of the diagnostic performance monitors I'd used in the past (such as Requests in Application Queue) showed the issue, but requests were absolutely being queued -- or otherwise not processed immediately. There were also plenty of free worker and IOCP threads. The only thing that clued me in was the Pipeline Instance Count and Requests Executing counters were exactly the same (96) on all the servers. So I started investigating from there.

It turns out that due to the way IIS7 ASP.NET integrated mode threading model functions there is a (configurable) request limit of 12 per CPU. We hit this limit in Youtopia today because we hold open requests for asynchronous Comet-like communications and there were over 288 people online simultaneously. Our three eight core web servers each had 96 (8*12) people connected to them and weren't really serving any other requests. We aren't running into any thread configuration limits as the long running requests are asynchronous and not using ASP.NET worker threads.

Here are a few great links that came out of my research.

With ASP.NET 3.5 SP1 it boils down to a simple configuration file change. Use something like this in the aspnet.config file (in x64 it's at C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet.config). This is the default. Adjust maxConcurrentRequestsPerCPU to suit your needs.

<system.web>
<applicationPool maxConcurrentRequestsPerCPU="12" maxConcurrentThreadsPerCPU="0" requestQueueLimit="5000"/>
</system.web>

In addition, the application pool needs to be configured to allow more requests. By default it only allows 1000 concurrent requests. This is done under the Advanced Settings for the application pool in the IIS 7 manager. Set Queue Length to 5000 to match this system level configuration.

Monday, November 16, 2009

Ditch Your Events (Part 1)

About four months ago Max, Hive7's Lawful Evil CEO, decided we needed to take our games to the next level and build something fun and accessible that everyone who plays "those farming games" would want to play. We all brainstormed, pitched our ideas to the company, and everyone voted by comparing every idea against every other – I wish we had a digital photo of the giant matrix on the whiteboard. There were a bunch of great ideas, but in the end... I won! Youtopia was born.

Youtopia was released to the public about three months from its inception. Hats off to the dev and art team for pulling this one together. A new technology for the developers and fully animated objects for the art team led to much blood, sweat, and tears, but we got 'er done! Of course, we're still actively developing Youtopia, and there are lots of great things planned for the future! But, back to my tech article...

It's been a long time since I've stepped out of my comfort zone and learned a new (to me) technology. Don't get me wrong, I'm always experimenting with the lastest .NET based thingie-ma-bobbers out there, but I haven't used a completely foreign development environment since C#/.NET came out over eight years ago. But for this project I needed to learn Flash/AS3, and it needed to be done yesterday. Luckily for me nobody else on our dev team knew Flash so I could still pretend like I knew what I was talking about and make lots of (un)educated architectural decisions without anyone being the wiser!

One such recent decision was to use an event driven property binding system. Youtopia's engine is based on a great open source game engine, brought to you from some of the Dyamix/GarageGames people, called the PushButton Engine (or PBE). In PBE there is a class called PropertyReference. This class facilitates a late-bound approach for one component to read the value of a property (member variable or getter/setter) on another component. It's a pretty cool pattern, but requires you to poll the target component whenever you want to know if the property changed. This works fine when you're talking about 10's or 100's of components. But in Youtopia we have thousands of entities in the scene at once. We needed this binding to be event-driven.

Of course, with my .NET background I immediately reached for the INotifyPropertyChanged pattern used in .NET's data binding infrastructure. With INotifyPropertyChanged it is the responsibility of the object owning the property to raise an event whenever a property value changes. Any listeners will then immediately know they need to poll for the new value if they want it.

This works great in .NET and is very performant. But in Flash, events are a whole other story. They are an extremely feature-rich subsystem that I don't really want to get into. In the end, all the features and memory allocations when you raise an event lead to poorer performance than we needed for Youtopia. We need every bit of CPU power on that single Flash thread and really shouldn't be wasting it raising events.

So, I shamelessly copied the .NET patterns and brought them over to AS3. Let's start at the core. In order for things to perform their best, I couldn't use the built-in Events. Though Troy did the benchmarking legwork, he didn't provide an implementation we could use to register callbacks and call multiple functions. So, I wrote a MulticastFunction that behaves a whole lot like the MulticastDelegate in .NET. Usage is really straightforward.

var func:MulticastFunction = new MulticastFunction();

//register my listener callback
func.add(
function():void
{
//this callback does amazingly cool stuff
trace("hello from the callback");
});

//calls all the callbacks that have been added, in the order they were added
func.apply();

As you can see, dealing with the MulticastFunction is a lot like the EventDispatcher, but each MulticastFunction is only designed to be used for a single event. So, to use it for events, create a public getter on your class named something reasonable and add your callbacks to it. Done!

Ok, I realize I keep talking about event dispatching speed, but haven't put my money where my mouth is. I wrote some benchmarks of my own and here is the output with a release build, in the latest standalone Flash 10 player. It does five test runs. Download the Source

running tests...
Event dispatching took 848ms
MulticastFunction took 355ms

running tests...
Event dispatching took 846ms
MulticastFunction took 351ms

running tests...
Event dispatching took 834ms
MulticastFunction took 352ms

running tests...
Event dispatching took 836ms
MulticastFunction took 351ms

running tests...
Event dispatching took 823ms
MulticastFunction took 343ms

Yup, that's right. MulticastFunction is nearly 2.5x faster, and I haven't spent much time tuning it. For example, it's using an Array under the hood and doing more work than it needs to during the apply call. Events will also become less performant over time as you have to create (and potentially clone) Event objects for every dispatch, causing a lot of garbage collection pressure. Here's the MulticastFunction, with lots of comments or you can download the source

package com.jdconley
{
/**
* A wrapper that mimics the synchronous behavior of the MulticastDelegate used in .NET for events.
* This doesn't support any of the async methods, as we don't have free threading here.
* It also doesn't support return values.
* See: http://msdn.microsoft.com/en-us/library/system.multicastdelegate.aspx
*/
public class MulticastFunction
{
private var _functions:Array = [];
private var _iterators:int = 0;

/**
* Adds a function to be called when apply is called.
* If the function is already in the list it won't be added twice.
* Returns true if the function was added.
**/
public function add(func:Function):Boolean
{
var i:int = _functions.indexOf(func);
if (i > -1)
return false;

//add new functions to the end so they are picked up live during an apply
_functions.push(func);
return true;
}

/**
* Removes a function to be called when apply is called.
* Returns true if the function was removed.
**/
public function remove(func:Function):Boolean
{
var i:int = _functions.indexOf(func);
if (i < 0)
return false;

if (_iterators == 0)
_functions.splice(i, 1);
else
_functions[i] = null;

return true;
}

/**
* Synchronously applies all functions that have been added.
* Functions can be safely added or removed during an apply and changes will take effect immediately.
* Added functions will be called, and removed functions will not.
**/
public function apply(thisArg:*=null, argArray:*=null):void
{
_iterators ;
var holes:Boolean = false;

for (var i:int = 0; i < _functions.length; i )
{
var f:Function = _functions[i];
if (f == null)
holes = true;
else
f.apply(thisArg, argArray);
}

//cleanup holes left by removing functions during this apply call.
//if any of the function apply's throw an error the state of _iterators will be off.
//but, we'll only leak array slot memory if functions are removed.
//putting a try/finally or try/catch block here significantly decreases performance.
if (--_iterators == 0 && holes)
{
for (i = _functions.length - 1; i >= 0; i--)
{
if (_functions[i] == null)
_functions.splice(i, 1);
}
}
}

/**
* Removes all functions from the list. Stops the current apply call, if there is one.
**/
public function clear():void
{
_functions = [];
}
}
}

Although capture, bubble, weak references, and priority are handy features of the Flash eventing system, they're not always necessary and will hurt your performance when you might have thousands of them firing per frame.

In Part 2 we'll put this MulticastFunction to use in a more meaningful way with the INotifyPropertyChanged implementation.

Friday, November 13, 2009

Anyone still out there?

Wow, I haven't posted in a while. In recent months I've been focused intently on a few things.

  1. Babies! My wife and I had twins in February.
  2. Learning a new technology while shipping an amazing game at Hive7.
  3. Working on a cool open source project.

I won't bore all you geeks with the baby stuff. If you can find the link to my personal blog you can go look at lots of pictures.

You should all check out Youtopia (the new game we shipped). We're really proud of this one.

So, drumroll please... *in my most awesome announcer voice* And, the new technology is... Flash! That's right, this Microsoft fanboy is now in the Flash camp. I really wish I could be working with Silverlight, but well, you can't build a game that runs on Facebook and make people install something. It just won't work. Once Silverlight has a market share more like Flash Player, then we're in business.

What do I dislike most about Flash? The development environments (yes, plural) for Flash pale in comparison to Visual Studio. Compiling is slow. Stuff crashes a lot. Heck, I even got the compiler to throw a null pointer exception on a few occasions! Debugging is a pain. The garbage collector isn't very fast. You only have one thread to work with. Hey Adobe is it still 1998?

All that being said, Flash (and more specifically Actionscript 3 and Flash Player) is actually really mature now and a decent piece of technology. It has most things a developer looks for in a language/runtime. And, well, it allows us to create a really rich and interactive experience that runs in your browser and doesn't require you to install anything. Obviously the business case here wins out over my whining.

I think I've spent enough time talking. Coming very soon, a useful post that contains lots of great technical info from the perspective of a C# junky diving head first into Flash.

About the Author

Wow, you made it to the bottom! That means we're destined to be life long friends. Follow Me on Twitter.

I am an entrepreneur and hacker. I'm a Cofounder at RealCrowd. Most recently I was CTO at Hive7, a social gaming startup that sold to Playdom and then Disney. These are my stories.

You can find far too much information about me on linkedin: http://linkedin.com/in/jdconley. No, I'm not interested in an amazing Paradox DBA role in the Antarctic with an excellent culture!