Tuesday, August 1, 2006

Using the MySql Command Line from C#/.NET

We have added two new data access providers to the upcoming SoapBox Server 2007 release. We now support PostgreSql and MySql as well as Microsoft SQL and Oracle. The code for these databases has been in our server and test libraries for a couple of months and this last week it was time to add them into our post installation configuration wizard. We strive to make our setup process as simple as possible. You'll notice many improvements over the 2005 wizard. We have better auto-configuration and fewer wizard screens.

One of our core philosphies here at Coversant is to make all of our software as easy to use as possible. Aside from testing, we spend more time doing this than any other R&D effort. Why is easy to use software so important? Well, the easier our software is to use the more you will like it and the less support we have to provide. We cut costs, you are happier. It's win/win. :)

Adding the automated setup process for these new data access layers was supposed to be very straightforward. In our configuration utility it's all abstracted so I just implement a few classes, the DB guy gives me the scripts to run and I call the appropriate tool (mysql.exe, pgsql.exe, osql.exe, etc), which we ship with our server for your convenience. However, there is one big feature missing from the mysql command line tool. There is no way to specify a file to use as input! But, it does take standard input. So, typically, if i were running the tool from the command line I would do something like: 'mysql -uroot -p < "c:\myscript.sql"'. If you're command line saavy you will know this redirects the file c:\myscript.sql as standard input into the mysql command line tool. I thought I could do this in .NET using command line arguments passed to the System.Process.ProcessStartInfo class. Well, I was wrong. It doesn't work (at least I couldn't figure out how to give it a file as stdin -- the "<" didn't work).

So, what's the answer? Simple. Set the nifty RedirectStandardInput property and read the file into the Process.InputStream. Code follows:


private void ExecuteSQLScript(string databaseName, string user, string password, string command, string filename, string server)
{
using (System.Diagnostics.Process p = new System.Diagnostics.Process())
{
#if LINUX
p.StartInfo.WorkingDirectory = System.IO.Path.GetFullPath(this.InstallOptions.InstallDirectory);
p.StartInfo.FileName = "mysql";
#else
//grab the path from our installation options
p.StartInfo.WorkingDirectory = System.IO.Path.GetFullPath(System.IO.Path.Combine(this.InstallOptions.InstallDirectory, BaseScriptDirectory));
p.StartInfo.FileName = System.IO.Path.Combine(p.StartInfo.WorkingDirectory, "mysql.exe");
#endif

//set all the startup options
p.StartInfo.CreateNoWindow = true;
p.StartInfo.WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden;
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;

//build the arguments for mysql
StringBuilder args = new StringBuilder();

if (!string.IsNullOrEmpty(password))
args.AppendFormat("-p{0} ", password);

if (!string.IsNullOrEmpty(server))
args.AppendFormat("-h{0} ", server);

if (!string.IsNullOrEmpty(databaseName))
args.AppendFormat("-D{0} ", databaseName);

if (!string.IsNullOrEmpty(user))
args.AppendFormat("-u{0} ", user);

if (!string.IsNullOrEmpty(command))
args.AppendFormat("-e\"{0}\" ", command);
else if (!string.IsNullOrEmpty(filename))
p.StartInfo.RedirectStandardInput = true;

p.StartInfo.Arguments = args.ToString();

WTrace.TraceInfo("Run Script", this.GetType(), "Executing: '{0}' with args '{1}' in working dir '{2}'", p.StartInfo.FileName, p.StartInfo.Arguments, p.StartInfo.WorkingDirectory);


//start up the process, and handle the redirected stdin and stdout -- send them to our trace lib
p.ErrorDataReceived += new System.Diagnostics.DataReceivedEventHandler(p_ErrorDataReceived);
p.OutputDataReceived += new System.Diagnostics.DataReceivedEventHandler(p_OutputDataReceived);
try
{
p.Start();
p.BeginErrorReadLine();
p.BeginOutputReadLine();

//read in the script file if one was specified and give it to stdin
if (null != filename)
{
using (FileStream f = File.OpenRead(filename))
{
using (StreamReader reader = new StreamReader(f))
{
//we could do this one line at a time to be safe memory wise, but we know our scripts are small
p.StandardInput.WriteLine(reader.ReadToEnd());
}
}

//tell mysql we want to exit, otherwise the process will hang
p.StandardInput.WriteLine("exit");
}

p.WaitForExit();
}
finally
{
p.ErrorDataReceived -= new System.Diagnostics.DataReceivedEventHandler(p_ErrorDataReceived);
p.OutputDataReceived -= new System.Diagnostics.DataReceivedEventHandler(p_OutputDataReceived);
}

if (p.ExitCode != 0)
throw new MySqlException("SQL Command failed.", p.StartInfo.Arguments, "", "");
}
}

void p_ErrorDataReceived(object sender, System.Diagnostics.DataReceivedEventArgs e)
{
if (!string.IsNullOrEmpty(e.Data))
WTrace.TraceInfo("Run Script", this.GetType(), "StdErr: {0}", e.Data);
}

void p_OutputDataReceived(object sender, System.Diagnostics.DataReceivedEventArgs e)
{
if (!string.IsNullOrEmpty(e.Data))
WTrace.TraceInfo("Run Script", this.GetType(), "StdOut: {0}", e.Data);
}

Wow, that's a lot of code. Ok, so I probably didn't have to paste that much of it, but I wanted you to get an idea of the full extent of the method, and I think being able to run mysql scripts is useful.

When all is said and done you now have two screens for installing to a MySql database: 1) enter username, password, and hostname 2) choose an existing or new database name. From there the installer figures everything else out automatically.

Wednesday, July 26, 2006

More on Interop

I'm sitting on a plane on my way back to Sacramento (a whopping 1 hour flight) and I thought I'd try to give a little more information about the interop event. After two days of testing it appears as though we're actually very close to having quite a few fully interoperable XMPP implementations. As I mentioned last time, Monday was a cake walk. Unfortunately Tuesday wasn't quite so easy.

Consensus was reached on Monday to start later on Tuesday. We wanted to do 10am, but had reservations at Jive's office to sample beers of the northwest at 5 (yeah they have some good beers) and play some XBox, so we decided to start at 9 instead. The morning session consisted mostly of protocol discussions that were very productive. We decided on the general protocol flow of PEP (Personal Eventing via Pubsub). This protocol addition allow us to create some very innovative and interesting extended presence features (more on this later) in future versions of SoapBox Communicator, and they'll be compatible with features in other clients such as Google Talk, GAIM, etc.

I had some really good Thai food from a street vendor, and I'm still not sick, so that's good. In fact it was some of the best Thai food I've had in a long time, and I eat Thai about once a week.

After lunch we got down to business with more interoperability testing. The goal: mutually authenticated TLS streams between servers per RFC 3920. A certificate authority was created, X.509 Certificates were generated, servers were configured, and then... it didn't work. Unlike the overwhelming success of Monday, Tuesday brought the skeletons out of the closet. We soon realized there were numerous breaking differences between OpenSSL, Java, and .NET based implementations of TLS. When we started, absolutely none of the servers were able to talk to each other over a fully trusted TLS connection. The interesting thing was we could all talk to another instance of our own servers. Hmmm.

After a few hours of hacking and debugging we realized there was significant work that needed to be performed and we didn't have enough time to do it. We were able to get connected with a few of the servers, and vice versa, but there definitely wasn't a Happy Path for all. As a result the server to server TLS specifications in RFC3920 will be clarified, as we eventually reached consensus on what it all really means and how it should be implemented.

In the very near future the JSF will be facilitating ad-hoc interoperability testing over the internet. It will be managing domains (such as soapbox.xmpp.org, google.xmpp.org, etc) where all participating vendors and open source projects will host servers. These will be semi-private domains without open registration, but open to anyone developing XMPP applications that need to test interoperability.

All in all, this was a very successful couple of days. We probably saved a good two months worth of bickering over e-mail lists to figure out protocol issues, verified that XMPP is in fact interoperable, and set the stage for future interoperability testing. We'll also be exploring fully automated tests, which Coversant will likely contribute to the JSF, to make sure everyone continues to play nice in the future. :)

Monday, July 24, 2006

Interoperability - Yup, we got it covered

Today was the first day of the first ever official XMPP Interop Event. In fact, it was probably the first day of any open instant messaging and presence interop event, ever.

We had showings from Coversant (yeah, that's us), Sun, SixApart (LiveJournal), Google, Jive Software, Jabber Inc, Sun Microsystems, and Process-one. Some of those names might sound familiar, and others not, but in the end what we ended up with was seven completely different XMPP server code bases/implementations both open and closed source, setup on a LAN and federating with each other. We spent more time configuring DNS, IP addresses, and other networking junk than we did fixing bugs that were hampering interoperability.

As far as I know (who knows what everyone was hiding on their laptop screens) there were only a few major bugs that were found and they were on two of the very freshest of server implementations, one of which the vendor considers in "pre-alpha" release status. No, I won't tell you who, but I bet they release patches soon. :)

So what the heck did we do all day? Well, we tested interoperability, obviously! But more specifically:


  • Inter-domain roster manipulation - Add and remove contacts on other servers.
  • Inter-domain messaging - Can we actually hold conversations with each other?
  • Inter-domain presence - Avatars, status messages, show states, etc
  • pizza
  • UTF8 support - Strange unicode characters in other languages in addresses, messages, and presence.
  • Discussed quite a few of the enhancement protocols pending for the XMPP specification and came to concensus on some issues.
  • Mingled with everyone and shared some anecdotes on implementing XMPP that pretty much only the people in that room would understand.
  • beer

What does that mean to you, a loving Coversant customer -- you are a customer, right? Well, it means that you can securely talk through IM in a federated manner to your trading partners, friends, family, and arch enemies even if they aren't one of our customers. Yeah, we support that, and apparently it works. ;)

#1 on the list for tomorrow, trusted TLS based inter-server connections. So we have the joy of setting up a trusted certificate authority and distributing certs. I bet setup will take longer than the interoperability testing itself, again.

Monday, June 26, 2006

How to Build Scalable .NET Server Applications: Memory Management

I'll get this out of the way from the start. This series of blogs will have nothing to do with ASP.NET or web services. However, if you plan on writing you own implementation of IIS in managed code this would probably be a good place to start. :) I also won't be providing very many code examples, as I'd be flogged by our intellectual property lawyers. You will not be able to copy and paste and create your own scalable server. However, I hope to provide enough insight so you can avoid a big list of gotchas we have had to figure out the hard way. This is one piece of a huge puzzle, memory management. Yes, you do have to think about that in .NET, at least if you want to build a large scale application.

For those who don't already know, SoapBox Server is a part of our SoapBox Collaboration Platform that supports the XMPP protocol as well most of the interesting JEP extensions. At the core of SoapBox Server is a highly efficient Socket server and thread machine capable of scaling into the hundreds of thousands of simultaneous users, and it's built 100% on .NET (C# now, but used to be VB).

SoapBox Server is the first multithreaded Socket based server application I've had the pleasure of working on. During the course of building the SoapBox Server into the extremely scalable and reliable system it is today I've learned a few things (as has the rest of the team, I hope). Thanks to Chris (who already had tons of experience with such things in Win32/C++), a few bloggers out there, some books, customers finding very interesting bugs, Windbg with Son of Strike, oh and Starbucks, I'd say I'm pretty well versed in the land of building scalable server applications. I'm no Jeff Richter, mind you, but I feel I have now learned enough to at least speak intelligently about it.

In that spirit I'd like to share the fruits of our tuning and debugging work, which, if history repeats itself, will continue to evolve as we begin work on our next major revision of the product. First, I'd like to repeat something I said a couple paragraphs ago, SoapBox now scales to hundreds of thousands of simultaneous connections with a single piece of server hardware. Think about that for a second. A user brings up an IM client, connects to SoapBox Server, and then holds that connection open until they Log Out. Repeat hundreds of thousands of times. This is no simple task. The .NET CLR does not provide a magic "Process.Scalable = true" property. We have invested hundreds of hours into tuning (maybe thousands) over the life of the server on classes of hardware varying from single processor laptops to 16-way Itanium2 systems with 64GB RAM. We've been through four distinct processing models as well as quite a few iterative improvements on our Socket interaction layer. Basically we have ran the server under a bunch of different profilers under many scenarios, found slow bits of code, and fixed them. But I'm not going to talk about profiling and performance tuning; perhaps another time. I'm going to talk about memory and scalable applications.

Every time your application creates a new Socket, Windows pulls memory from it's Nonpaged Kernel memory, which is simply physical memory that is reserved by the kernel and will never be paged out to disk. This block of memory has a finite limit and the kernel picks the limit based on the amount of phsyical RAM available to it. I don't know the exact algorithm, but with 4GB RAM it's usually somewhere around 150,000 TCP Socket connections, give or take. Want to see this in action? Simply create a loop that instantiates sockets. It will stop working eventually with a SocketException telling you there isn't enough buffer space. On top of this hard kernel level limitation, you also have to worry about how much memory each concurrent connection uses in your own application. In SoapBox we store a lot of information about each connection in memory in order to improve performance and decrease our IO operations. This includes things like the user's contact list, their last presence (available, away, busy, etc), authorization information, culture information, user directory information, etc. If we didn't hold this in memory we'd have to hit a file, database, or some other out of process persistent store for the information every time we needed it. Being IO bound is no fun. Believe me, we started out that way.

However, because of our extensive caching, SoapBox Server 2005 can only reliably handle about 20,000 simultaneous connections on the beefiest of 32 bit hardware (on 64 bit it's much, much, much higher -- I also have to admit we haven't stress tested the 2007 build on 32 bit hardware, it would probably be much higher now). It doesn't matter if you have 64GB RAM and 16 32bit processors, it we can still only handle 20,000 connections. Why, you ask? Well, it's because of the 2GB (well, really 3GB with a boot.ini switch) virtual memory limit per process in 32bit Windows. Without delving into managing your own memory your process is only allowed up to 3GB to play with. Typically, we use that up, or rather, .NET thinks we use it up, somewhere between 20,000 and 30,000 connections. Now why would I say ".NET thinks we use it up?" Story time!

A little over a year ago one of our customers kept running into a very bad situation. As evidenced by the Event Log, SoapBox Server was crashing (insert shock and awe here). It was an irregular occurance, but it did happen. However, we did no take this lightly. This customer was running about 2,500 simultaneous connections on a Dual Xeon with Hyperthreading and 4GB ram and the /3GB switch set. It was plenty of hardware for the job, and probably overkill. However, the service was still crashing. We set them up with the Debugging Tools For Windows and had them startup the process to wait for a crash (another blog we'll have to write some day). After a few tries we got a dump with some useful information in it. The result? We were out of memory, sort of.

In .NET when you call any socket operation and pass it a buffer, whether it be a send or receive, synchronous or asyncronous, it takes that buffer and pins it before giving it to the Winsock API's. Pinning, in a nutshell, is taking a .NET data structure and telling the .NET CLR memory manager not to move it, until it is explicitly un-pinned. The memory manager in the CLR is smart. As you allocate and deallocate memory it is constantly defragmenting it for you so the overall memory footprint is lower. There are quite a few really good/long/complicated articles on how this works so I won't bore you. However, pinning throws a wrench in this and the memory manager isn't quite smart enough to deal with it well (though it has gotten a lot better in 2.0). Basically, that buffer you want to put on the socket cannot move in memory (physically -- in terms of you virtual memory space) from the time the socket IO operation begins until it ends. If you look at the Winsock2 API's this is obvious, since the buffer is passed as a pointer. Anybody who's built this type of application in Winsock2 is probably saying "DUH!". I'd consider this a very leaky abstraction. Due to this behavior, it is quite easy to write a socket application in .NET that runs out of memory.

Back to the story! Not only were we out of memory, but the there was only about 200MB worth of data structures in the heap. For those of you like me that use calc.exe for all your basic math let me figure that out for you, 200MB > 3GB. Uhh, say what? How the heck were we out of memory? Well, we ran into the shortfall of pinning and memory fragmentation. The cause of this was a small number of small pinned buffers, in our case 2KB each, that were high enough in the heap to cause fragmentation spanning over 2.8GB. Where did the other 2.8GB go, you ask? Well, is was there, allocated by our process, but not being used by our code. In Son Of Strike (SoS -- a command line plug-in to the Windbg debugging tool I hope you never have to use) this showed up as free, empty, unused space! It was just sitting there waiting to be used, but we still ran out of memory. I think I mentioned earlier the memory manager in .NET isn't so smart when it comes to fragmented memory and pinning, well, this is what happens in the worst case.

Good thing for you, the answer to all your memory fragmentation and pinning woes is quite simple. Pre-allocate buffers for use by anything that will be causing pinning, and do it early on before there is a lot of memory thrash (when your application is rapidly allocating and deallocating a lot of memory). We created a simple class called a BufferPool that we use to pre-allocate a certain number of buffers. This pool can grow as need be, but it does so in large chunks and forces a garbage collection each time before the buffers are actually used. This considerably reduces the chances of fragmentation caused by pinned memory. If the pool starts off with 500 buffers, but then the 501st buffer is needed it will grow by a configurable value, typically another 500 buffers, and the induced garbage collection will cause these buffers to shift to the lowest possible point on the heap.

Interestingly enough when we found this bug we already knew about the pinning behavior of socket operations, but had only solved half of it. All of our BeginReceive calls were using the BufferPool because we knew the buffers would remain pinned until we received data from a client, but the BeginSend calls were not using the pool. We had not even considered the fact that sending a few KB of data might take long enough to pin memory, fragment the heap, and cause an OutOfMemoryException. But there is one case where they do, timeouts. The Windows TCP subsystem is very forgiving. If a client loses its connection and the server isn't explicitly told about it, the next piece of data you try to send to that client socket will end up being pinned while the TCP subsystem waits for the client to respond. It can take up to 5 minutes with the default configuration of Windows for the TCP subsystem to figure out the client isn't really there. During that entire time your buffer is pinned in memory. *poof* OutOfMemoryException.

Unfortunately, pre-allocating buffers does not completely fix the issue of running out of memory. There are also some other limits to the size of a .NET process's virtual memory space that are very complicated and I won't talk about, but basically you end up with anywhere from 1/2 to 2/3 usable virtual memory without running the risk of OutOfMemoryException. So, if you have 2GB virtual memory available (standard on a 32bit machine), you end up with about 1.3GB you can actually use reliably. Of course, this varies, and some applications will be able to use more, or maybe less. Your mileage may vary.

Don't fret, all of the issues I've talked about in here have been fixed since SoapBox Server 2005 SR1. And with the most common usage patterns people were not actually affected to begin with.

I hope this was at least marginally interesting to someone. :) Next up, I'll probably talk about limitations we discovered in the Windows Socket infrastructure, or maybe async IO, IOCP, and worker threadpools, or maybe how in the world we actually test at this scale. Only time will tell, unless Chris beats me to it.

Friday, June 2, 2006

Fun Installing Vista Beta 2 on AMD x64

As a self proclaimed geek and MSDN subscriber I feel as though it's my duty to explore all the new software that Microsoft comes out with. This last week I have been embarking on one such journey. Working with beta software is always a bit trying, but tack on a beta driver model and a "new" hardware platform (x64) and things get really interesting.

Vista Beta 2 was released to MSDN about a week ago. The next day I fired up my DVD burner and started messing around. About four hours later I had a working installation. Why so long? Well, because my workstation is an AMD NForce 4 x64 system and boots from the onboard SATA RAID. Apparently this is not one of Microsoft's test platforms. I had an experience similar to this guy.

I had to run the Vista install from an existing Windows installation. It simply would not work when I attempted to boot from the dvd. I never got an option to load drivers. Nvidia recently released beta Forceware drivers for Vista x64. I assumed these would have the RAID drivers I needed to install Vista, afterall they did have the appropirate txtsetup.oem file and seemed to be correct. After a few attempted installations, blue screens, automatic reboots, and hangs, I decided that my assumption was bad. Lesson learned: to install Vista x64 on an Nforce 4 RAID use the XP x64 Nforce4 RAID and SATA drivers. Yup, that's right. Well, almost.

If you're like me and want to use the latest Nvidia XP x64 drivers you'll be greeted with a black screen telling you that your drivers are corrupt after the first setup reboot. Say wha? Lucky for you, they aren't. This is a feature of Vista x64. Hit F8 at the boot screen and choose to disable driver signature verification. Of course, hitting F8 EVERY time you boot your computer is not going to be very fun. Luckily (for now) there is an application called bcdedit (just run "Bcdedit.exe –set nointegritychecks ON") that you can use to disable signature verification after you get into your desktop. Oh, don't forget to right click on the Command Prompt link in your start menu and choose "Run as Administrator" before trying to run this command, or it will tell you "Access is Denied". Yay security! I should definitely mention that Vista prompted me to allow this action (VPMTATA), at least once.

Oh yeah, I almost forgot, after you finally get to your desktop Vista will keep telling you that it has found an unknown device. This is your RAID controller; the one with XP drivers. Point the device wizard thingy to the inf file (VPMTATA) of the Forceware Vista x64 Beta drivers and this annoyance will go away. You'll also need to install drivers (VPMTATA) for the Nforce4 audio chipset.

On the plus side, Vista had drivers for my Geforce 7800GT and Nforce4 gigabit network card. It even figured out I had dual monitors (VPMTATA), picked the max resolution for both (VPMTATA), and presented a neat little dialog that let me choose the desktop layout (VPMTATA). Of course, I wanted to upgrade to the latest ones from Nvidia. This is usually straightforward.

I ran the setup exe (VPMTATA) for the Nvidia Vista Beta 2 Geforce drivers. It extracted stuff (VPMTATA), ran the second installer exe (VPMTATA), and then failed with some cryptic error messages I probably should have written down and submitted as bug reports. Subsequent attempts to run the installation package (VPMTATA) resulted in an error about running 32 bit uninstaller code on a 64 bit platform. I was very confused, didn't want to spend much time on it, and gave up.

A few days later I had an epiphany - "I should just try to update the Microsoft Geforce driver with the inf". Duh. Well, I opened up the Device Manager (VPMTATA) and clicked update drivers (VPMTATA). Viola! The drivers were upgraded. Though, I have no idea if there are any control panels with these drivers (as there are in XP) since I couldn't run the full setup. Ah well, at least I have better video acceleration.

I'll be back later for my accounts of fun with Vista. After a week of use, I think I could write a book. However, it definitely hasn't all been bad (though VPMTATA) and I will continue to use Vista as my primary OS until it does something very mean or simply won't allow me to get things done.

Friday, May 12, 2006

.net 2.0 web service hair puller

Every couple of weeks I spend four hours doing something that should take five minutes. It just happened, and now I feel compelled to take another few minutes and explain so it doesn't happen to you. Not only did I waste my time, but the time of another one of our developers. What might waste four hours, you say?

We recently migrated our entire web site to .NET 2.0 and a new portal. Since we were building a new web site anyway we thought, "What the heck, let's re-factor the licensing subsystem. The database was hacked together over three years and we don't want no stinkin' .NET 1.1 code running on our shiny new site!" Well, this didn't turn out exactly as planned.

The SoapBox licensing web service is quite simple. There's a single method called "Activate" that takes in a unique hash of some information on the user's computer (so we can track duplicate usages) and the serial number. It returns an XML document containing all the license information, signed with our private key, that the SoapBox Server then validates with the embedded public key. That's it.

We get a call from a customer today saying they can't enter a license key into the MMC. They're getting a wonderfully descriptive "Object reference not set to an instance of an object" error. We immediately kick into debug mode. Afterall, if a potential customer can't enter in our trial key they're going to get upset and likely just give up on using it all together.

After a little debugging we found the web service wasn't returning the license xml document like it should have been. This was _very_ weird, considering our licensing page uses the same web service as our MMC and it was working just fine with the exact same parameters. Much head to wall interaction took place and we had another breakthrough, the web method wasn't receiving any data from the parameters. The serial number was null!

Now we bust out the packet sniffer. After careful examination, a new mini test application to call the service, and three Diet Cokes, we found the problem.

Apparently, during the migration and re-factoring process, we missed a character in a copy/paste operation. That's right, the trailing "/" on the namespace declaration for the web service.

This in itself is not such a suprise. However, the suprise is that .NET did not throw a "There is no matching web method you dummy" exception. No, it didn't throw an exception at all. Instead it called the correct web method, but did not pass in any of the parameters. Oiy.

It turns out our licensing page was working because the web reference was created after the new licensing service was put in place. It all makes sense now... Time for a beer.

Tuesday, May 9, 2006

from the trenches of my first software startup

I was born and raised an entrepreneur (I'm pretty sure that's how you speel that). During my lifetime my dad never had a single "real job". He has always been a small business owner. From a carpet cleaning business, to a trucking company, to a coffee shop, he was always working on something he could call his own. Every time we get together we end up talking about businesses we'd like to start. If only I had the time. . . Somehow, after seeing him (and my mom) constantly working at least 80 hour weeks, stressed out, and exhausted, I still decided I wanted to start a company.

At the end of high school, and for a while after, I had a computer repair business with a really cute name and no business plan whatsoever. This lasted about six months. It turns out to be real work, and it's quite difficult to find people that are good enough to do that sort of work for very little pay. I'm really very interested in how Best Buy pulls that one off, though I assume they just have a good process and the people don't need to be all that great (no offense geeks -- I owe you guys many hours of my time that has been saved because I didn't have to help out my mother in law with her digital camera).

Ok, back on topic, sort of. A few years prior to my repair business I had fallen in love with computer programming. I wrote little programs to do all sorts of interesting things. Of course, I didn't work on anything that took more than a few weeks of my time and only built things that were interesting to me. Ah, the good old days! I went into the professional world when I turned 18 and started writing code for a living. Wow, what a change. Who knew coding could be so much fun?! "Wow, I get to program against another database and improve business processes! Woohoo!"

Needless to say, the life of a professional internal software developer was not in my cards. Even though I was making way too much money for my age as a "consultant" (that's what the recruiters like to call you, even though you're really just staff augmentation and under a different category on someone's budget), at about age 20 I started getting very restless. Luckily I met some like minded individuals (Jason and Chris) and Winfessor (Coversant's old company name) was born.

For the last few years, in between consulting engagements (we're boot strapped), we've been building the SoapBox products. Of course, we haven't only been building software, we've been building a business. Lucky for me, my role is still primarily building the technology. It continues to be a wonderful, trying, stressful, exciting, sleepless, humbling experience.

Here are some tidbits, in no particular order, that I've picked up along the way, from learning both the easy way and the hard way. It's free (and probably bad) advice from my limited experience aimed at any programmers out there that want to start a company around a software product. I'll probably write some more specifics at some point.

  • Have at least one flagship customer to start with, mainly for promotional purposes, but ideally one that will help fund the project.
  • Work full time on it. Hire others to do your consulting or other revenue generating work so you can focus, or get funding.
  • Don't work so hard. There is always more work to do, even if you work 140 hours/week.
  • Start small (ideally around 6mo development) and iterate from there.
  • Build it like you would an enterprise project - component based, easily maintainable and easily expandable.
  • Hire people to do things you aren't good at or learn to do them. Web site, graphics, brochures, accounting, sales, etc. Writing code is an itty bitty teeny weeny piece of your business.
  • Figure out how to start getting revenues, quickly, and set reasonable goals.
  • Don't look too far out into the future. Have a plan, but don't be scared to change it.
  • Any software you build needs to be extremely easy to use and install. This alone will win many sales in a head to head battle with your competitors (yes, you will have competitors).
  • FOCUS! Do one small thing better than anyone else, and then move on to the next.
  • Don't pretend to be a big company. You have a lot of advantages being the small guy and people willing to work with small companies know it.
  • Do something unique that your market needs.

That's all I've got for now. We're constantly growing and learning. I imagine my advice will evolve as the Coversant experience continues. In the end, the freedom, excitement, and financial possibilities of a startup outweigh the pains, at least for me. I also just realized I use way too many parenthesis when I write (maybe I'll work on that).

About the Author

Wow, you made it to the bottom! That means we're destined to be life long friends. Follow Me on Twitter.

I am an entrepreneur and hacker. I'm a Cofounder at RealCrowd. Most recently I was CTO at Hive7, a social gaming startup that sold to Playdom and then Disney. These are my stories.

You can find far too much information about me on linkedin: http://linkedin.com/in/jdconley. No, I'm not interested in an amazing Paradox DBA role in the Antarctic with an excellent culture!