Extra Cheese

BitBacker was a startup that I founded with two other people in March, 2006. It died for many reasons, including disagreements between the founders, changes in the target market, and changes in the economy as a whole. I never talked about it directly on this blog, but here's a summary of what happened in its three year history:

March 2006

We (Nick Barendt, Michael Branicky, and I) decide to build an online backup product. We're unimpressed with the existing solutions – we want something that can be installed and forgotten about. I'll do most of the development, Nick will do contracting with other companies to get us going, and Mike, busy enough as a tenured professor, will be in an advisory role.

We begin building a "test app" to analyze data churn rates on peoples' hard drives. This is a sort of feasability study, as well as a source of data to help us create the product. I'm still finishing undergrad, so I'm only working part-time.

June 2006 (3 months after founding)

I finish undergrad. We realize that building the test app isn't worth the time: once we get it working, we'd need to get it installed on a lot of machines to make it worth the effort. We aren't confident that we can convince random people to install an application that doesn't help them in any way. We'd rather just build a simple, first-draft backup app. The test app did have some value as a learning experience, though – it had to do a lot of the crawling and hashing that BitBacker would eventually need.

With the test app retired, we start work on BitBacker itself immediately.

August 2006 (5 months after founding)

Apple announces that Mac OS X Leopard will include Time Machine, a system for doing transparent backups to an external USB drive. This is scary, but we decide to postpone our worrying until it's released. Besides, Leopard is a year off, and we'll surely have a product out by then.

Late 2006 to early 2007 (about 9 months after founding)

I learn how to do backups well, mostly by trial and error. A lot of infrastructure shows up around the concept of a snapshot, which captures the state of the entire file system at a single point in time. BitBacker supports many snapshots, so you can step back in time and see what your folders and files looked like yesterday, or two weeks ago, or last January (much like Time Machine, but without the fancy UI).

At this point, BitBacker only has a web interface. You run a little app that has the web server in it, then point your browser at that local server. We realize that this isn't going to fly in the long term, but haven't decided what to do about it yet.

Spring 2007 (one year after founding)

We decide what we're going to do about the UI. I spend a couple weeks learning Cocoa and rebuild the whole client UI in it . At this point, we're committed to supporting only Macs, at least for the initial release. Cocoa makes me realize that, contrary to past experience, building a GUI doesn't have to be excruciating. This is a nice surprise. The early versions of BitBacker are rather silly, but that's because I'm still learning to think like a person who hasn't spent thousands of hours with this software already.

The first alpha is released. It runs live across the internet, but is only used by the founders. Everything is written in Python, including the entire Cocoa GUI.

July 2007 (one year, four months after founding)

We get a real, live office at 1677 E 40th St, Cleveland, OH. The BitBacker sign is still hanging outside the building today. We suddenly own a lot of Ikea furniture.

Until now, we've been working from our homes and various coffee shops. Having an actual, physical office raises expectations, as well as our burn rate.

Summer and Fall 2007 (about 1.5 years after founding)

The private beta begins. More users are added and I do a lot of technical waffling – having external users has made me afraid of making serious changes. The few problems that are found by users cripple my progress as I agonize over whether I've actually fixed them. Commits happen just as often as ever, but they aren't making many user-visible changes.

At this point in the project, I'm unhappy with the state of the code. I'm more disciplined than I was when we started the company in 2006, and I spend much of this period bringing the code up to my new standards.

October 2007 (1 year, 7 months after founding)

Mac OS X Leopard is released, including Time Machine, and we still don't have a released product. Because Time Machine requires a physical external drive, it achieves limited market penetration. We are relieved. However, the online backup offerings available for Mac OS X are improving. Mozy is getting quite popular, which is scary. Surely we will have a release soon!

January 2008 (1 year, 10 months after founding)

I take over as the organizer of clepy, the Cleveland Python user group, and BitBacker hosts it for the next fifteen months consecutively. Beer is now allowed at clepy meetings, which seems to improve attendance.

February 2008 (1 year, 11 months after founding)

We remove support for multiple machines being backed up with one account. This is a major turning point for me – I'm learning to throw huge chunks of a system away if they don't help the user.

March 2008 (2 years after founding)

We finally add the ability to make asynchronous S3 requests. Before this, the server had to make one S3 request at a time. When a snapshot is completed, the server potentially needs to make thousands of S3 requests quickly, so this is very important for our performance.

Python doesn't have an asynchronous HTTP library (Twisted doesn't count, at least as of March 2008). I write my own by hacking subclasses of httplib.HTTPConnection and httplib.HTTPResponse to use fake sockets (subclasses of StringIO) instead of real ones. I use asyncore to do the asynchronous communication, with the hacked httplib subclasses doing their HTTP work before and after the actual communication.

We release version 1.3.0, doubling our user base.

May 2008 (2 years, 2 months after founding)

One of our users backs up around 100 GB, roughly five times more than we'd ever tested up to this point. The client application has some performance problems, and the server takes so long that the client times out (both fixed in later versions), but the backups do complete successfully.

August 2008 (2 years, 5 months after founding)

We finally port to Leopard, which has a totally different Python-to-Objective-C bridge. The previous version of PyObjC builds apps that work in Leopard, but those apps can only be built on a development machine running Tiger. This means that I'm still using Tiger in August 2008, a year after Leopard was released. Porting to the new version of PyObjC takes about a week and is quite painful. After we finish, I'm afraid that we might've missed some corner cases. I'm really hurting for a comprehensive suite of automated full-stack tests.

September 2008 (2 years, 6 months after founding)

Until this point, I've been doing huge manual tests before release. Unit and system tests have been automated since the beginning of the project, but there have never been automated tests that actually drive the UI. I begin writing end-to-end ("acceptance") tests using appscript, which is a Python interface to the event system underlying AppleScript. These tests start the servers, build the app, launch it, and use it as the user would – clicking buttons, selecting menu items, etc. This reduces my manual test process from eight hours to one. It takes about a week and a half to implement all of the tests.

Fall 2008 (about 2.5 years after founding)

We release versions 1.4.0 and 1.4.1, which move BitBacker into the OS X menu bar. Before this, it was a normal application, showing up in the Dock and the Cmd-Tab list. At this point, I realize that building the whole client as a single application was a terrible idea due to the way that OS X "UI Element" applications work, but I can't justify taking the time to fix it. These versions also add the ability to exclude folders from backups and run at startup. We're feeling pretty comfortable about charging money for the app now.

November 2008 (2 years, 8 months after founding)

We start looking at subscription payment processing options, quickly realizing that it's much more complex than we realized. We waffle on it.

December 2008 (2 years, 9 months after founding)

A beta user's hard drive dies. He uses BitBacker to restore 17 GB of lost data. It's the first time that a large restore has been done in the wild. There is much rejoicing.

February 2009 (2 years, 11 months after founding)

The open beta begins – beta keys are no longer required for registration. Storage quotas are added (all existing users get 250 GB; new users get 2 GB). Email address confirmation is added. Legal agreements are added. A few people we don't know sign up and try the app.

March 2, 2009 (3 years after founding)

I get sick for several days and, when I get better, I find that I'm disappointed. This makes me realize that I'm not at all happy with my day-to-day life. I talk to Nick about it; we agree that we're both frustrated with the state of the company and disconcerted by the huge change in the competitive landscape since we started. We decide to stop working on BitBacker immediately. The last Clepy meeting at our office is on that night: March 2nd, 2009. We immediately begin selling our office furniture and hardware.

March 8, 2009 (3 years after founding)

We send an email to all existing beta users telling them that we're retiring the product and that the servers will go down on March 20th, 2009.

March 24, 2009 (3 years after founding)

The servers go down forever at 6:46 PM. Their uptime was 369 days. May they rest in peace... or, more likely, immediately be provisioned for some other Amazon EC2 customer. ;)

Lessons Learned

Here are the two biggest lessons I've learned from BitBacker:

1. Talk about equity early and often. Be frank about your expectations. Never say "we'll work it out later."
2. Release as early as you can, even if it doesn't do much.

This advice can be found in a thousand places, but it bears repeating. I failed to do these things even though I'd heard them so many times before. Seriously: release early, even if the thing doesn't do much. You'll always think it's not ready, but that's because you live with it every day. If it could possibly have any use to other people, release it!