Ikai Lan says

I say things!

On Hackathons, Process, Email and the Tragedy of the Commons

with 3 comments


I love hackathons. I love going to them, and I love running them. Most recently, I participated in a 48-hour hackathon in Kuala Lumpur, Malaysia. It’s one of the best parts of my job. I get to run (and sometimes participate) in both external hackathons, as well as hackathons that are internal to Google.

In early June I held an internal Hackathon at Google to teach employees how to best use the product I work on: Google App Engine. I consider the event a success: we had hundreds of RSVPs and a completely booked room. It was so successful, in fact, that I’m planning on holding at least one of these events a quarter. The breakdown was primarily newer employees, which didn’t surprise me giving the amount of hiring we’ve been doing.

A primary driver for the sheer volume of RSVPs was the fact that we advertised the event on a mailing list that went out to pretty much all of engineering. All. Of. It. An engineering company with headcount in the tens of thousands, hundreds of RSVPs was not only likely, it was pretty much a mathematical certainty. Looking back, we would probably not have received the response if we didn’t sent out such a wide blast.

As a result of what I consider to be a fairly successful event (and I don’t mean to take all the credit here, at about the same time as my event, there was another very successful internal hackathon), various teams have suggested hackathons for their product APIs. There are events on the calendar.

Therein, of course, lies our problem. The problem of noise.

What should we do? Email all of engineering for every event? Create a new list/site/page announcing new events? Let’s break down the tradeoffs for each choice:

1. email all of engineering

Pros: goes to everyone

Cons (and this is the bigger point of this post): the majority of events will be irrelevant, causing the signal-to-noise ratio on the list to significantly drop, causing people to filter out these annoucements

2. Create a new distribution channel for events

Pros: Opt in

Cons: You don’t get the distribution you’d get with #1, since only a minority of people will opt in. Also – has the same SNR problems.

Now, a hybrid solution would be to do both. High profile, important events go to all of engineering, and smaller events go to the special distribution channel. The issue here is that everyone’s event is high profile. So again, we don’t have a great solution. Not to mention: people can only attend so many hackathons and still be able to do all the stuff they’re supposed to be doing. See, that’s one of the great things about Google engineering. If you’re consistently delivering, there isn’t a manager in the company that will tell you not to attend a hackathon or internal event where you can only get better at what you do. The issue, of course, is that the more hackathons take place, you are likely taking something a resource away from another team for a non-trivial amount of time. From a hackathon organizer’s perspective, a hackathon is almost always beneficial as long as some non-zero number of participants show up: they learn about your API, provide feedback and you learn a bit about how to improve the documentation or SDK. You almost can’t afford not to throw a hackathon.

This is the classic example of the tragedy of the commons. By running an event, you consume space. You consume employee time. You generate noise on all the distribution channels. And when everyone does it, suddenly, as a whole, everyone is worse off, though you yourself may individually gain.


Another key example of the tragedy of the commons is a company’s email marketing. I worked at a consumer internet company that broke teams by product. To drive usage metrics for an individual product, the product managers would run email campaigns to the site’s millions of users. The result was that the individual product would receive for usage, and everyone would give themselves a pat on the back. What was actually happening was that it was causing users to become extremely irritated at the company (myself included) for the voluminous amounts of email being sent all the time. Sure, you could go to the site settings and disable email, but new products would automatically opt you in to receiving notifications, and you would have to log back into the site to find the settings and disable those notifications as well. Some users, like myself, have created Gmail filters to completely send all emails from this company’s domain to a “Stupid Mail” label. I can understand the individual product managers’ reasoning. You don’t want to be the one team that doesn’t deliver metrics, so you email spam. And when everyone email spams, it’s to the detriment of the company overall. An employee posted to an internal group asking if it was an example of the tragedy of the commons – I don’t know if his advice was ever heeded, but based on the complaints I see on Twitter about email, my guess is no.


I view team processes the same way, and this sometimes leads to some very heated discussions with people I work with. It’s not that I don’t believe making your 1 step process a 5 step process doesn’t make your life easier or the company better organized; it’s that everybody wants to turn their processes from one step, lightweight, free form processes into full on, form driven, strict-requirements-based, signed-in-triplicate steps for doing things. I fight heavy processes when I can because I don’t believe enough people do so. Why? The tragedy of the commons. An extra 20 minutes here, and extra 20 minutes there, and suddenly, I am spending most of my day tangled in process instead of getting things done.

There are no easy solutions to this, of course. Some process is necessary, though from the onset, it isn’t always obvious which ones. How do you know, for instance, if a process is unnecessary? A good example is a managerial approval step in a process. Let’s say I need approval to do something. How do I evaluate if managerial approval is working?

  • What is the cost of doing it wrong? What was the bad outcome?
  • What was the number of incidences in which, prior to the institution of the process, that approval would have prevented a bad outcome?
  • Is the manager rubber stamping requests?

What absolutely needs to be done are constant evaluations of process. Don’t create a process and sit on it. Make it better. What can you take away, and still have it work? Think about your last trip to the DMV. How many steps could have been eliminated?

Awareness of the bigger picture

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

Antoine de Saint-Exupery
French writer (1900 – 1944)

I suppose that’s the solution to fighting the tragedy of the commons. A constant awareness of the bigger picture and a real desire to make things better. An understanding that many things in this world are a zero sum game. I’ll issue caution, of course: you can probably only champion a few things. Championing fixing everything, and people stop listening to you, you lose focus, and you will end up fixing very little. What do we call this effect? No, I won’t bother. Hopefully you actually read this and already know.

– Ikai


Written by Ikai Lan

July 16, 2011 at 2:24 pm

3 Responses

Subscribe to comments with RSS.

  1. Great article. Good insite. Thanks for sharing.


    July 16, 2011 at 2:34 pm

  2. […] Read about the hackathon from Ikai’s blog! […]

  3. Hi Ikai – isn’t this an inevitable problem of a large organization? Most companies (IBM for instance) arrive at a matrix model: when you sit at the intersection of engineering vs. education for example you would be interested in events for both.

    Matrix models get hideous quickly: if your event is aimed at a small audience (say, your peers in engineering / education) all is well and whatever you have to say will likely be relevant to them and you will have the ability to spam them without extra approvals. Going beyond that (all of engineering, or all of education) will eventually require an increasingly complex review/approval process, exacerbating problem #2, so on and so forth. Even from the outside we can clearly see the growing bureaucracy.

    I don’t think this is an engineering problem, unfortunately, and I don’t think there are any nice clean models for how to do this right – other than splitting the company up into semi-autonomous units that run like hell but will occasionally compete (SAIC had been a prime example of this, with different business units literally in a bidding war with one another for govt. contracts!), this is the price of growth.

    Jan Zawadzki

    August 11, 2011 at 1:32 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s