0:
  [I started out with a completely extempore reference to a previous talk that
  I can't think of before or since. If anybody remembers what it was, tell me:
  ak@akkartik.com]

  Large software projects suffer many of the same problems that cities do.
  Software could be more fluid, but so far we're locked into a more physical
  way of looking at it.

  Drew said yesterday that he wants to outsmart the designers. I want to
  design artifacts that don't need outsmarting.

  You IPO'd and now you're all set, financially.
  But while you're hanging out at the beach, what happens to your baby, the
  software you sweated and bled over for all those years?

1:
  The typical way the programmers you leave behind think of work uses these
  words: plans, problems, automation, staying lean, iterating fast. Assumes
  that once you've tried a solution you don't need to revisit it.

2:
  In reality, the same issues recur.
  In tennis you often see some noob come into Wimbledon and throw double-fault
  after double-fault. Double-faults are unforced errors. Serves have been
  practiced millions of times by these professionals. So what's going on?
  Answer: pressure. You often learn a lesson and then have to relearn it under
  circumstances of greater pressure[1].
  Software is like that. Problems recur and their solutions are often
  different each time they recur.

3-8:
  I want to illustrate what I mean by circles of hell with a story.

  In the beginning you start a startup and just modify live code on a
  production server. Super convenient, but over time you realize that that's
  not a bad idea. You find out that there's some tricks to doing this that
  people have picked up over the last 30 years, of maintaining a staging
  server, deploying code without bringing the site down, rolling changes back
  gracefully when a bug is discovered. So you do all that, and you can now get
  the convenience of editing code live on production but with a measure of
  safety.

  Life is good for a time, but after a while you notice problems. There's more
  and more rollbacks, and it gets harder and harder to be sure that something
  won't break on production. You discover that it's worth automating a certain
  class of tests. Adding tests to an existing codebase is super hard, but
  let's assume that in this ideal world you manage and pay back all the
  technical debt. Life is good again, you test changes, you deploy them with a
  single command, the rollbacks cease.

  After a while you again notice issues. Stuff is getting rolled back. Wasn't
  it tested? Yes it was, but they didn't test for this one thing, and that one
  thing.. What's going on is that the space of possible tests is now starting
  to undergo combinatorial explosion, and it's growing impossible to test for
  everything. Instead what you need is to centrally record the assumptions
  various parts of your product make about the world, about the state of data,
  about each other.
  
  Already we're in virgin territory. Nobody does this well today. But let's
  assume that you somehow fix all these issues and record all the assumptions.
  Your programmers are now on one page, and rollbacks drop down again.

  After a while you again notice issues. Rollbacks are spiking again. Jumping
  down the levels you notice that your assumptions too are growing too many to
  manage. As feature requirements change your initial design is no longer
  suitable for what it currently does. It needs to somehow jump to an adjacent
  design with totally different tests. Again, impossible to do today. But
  somehow you start doing this. You allow programmers to make radical
  architecture changes without introducing regressions, to track past
  architectures, and to notice when some past assumption is no longer valid
  and allows returning to a previous architecture.

  After a while you again notice issues. All this while you've been adding
  programmers, and they're now causing edit wars over the fundamental
  architecture of the codebase, ping-ponging between states. Part of the
  problem is that they are incented to show work for their bonuses, to get
  greater responsibility, to get promoted. Their short-term incentives have
  diverged from the long-term needs of the codebase. Somehow you learn to
  detect and squash over-engineering -- an incredibly hard problem because in
  complex systems what's over-engineered today some other subsystem may
  legitimately rely on tomorrow. But let's say we pruned it all out and
  returned to an elegant, parsimonious architecture.

  Again, there's issues. It turns out that over-engineering is just one form
  of capture by employees and there's lots of other less-common tricks that
  now start to flourish in the space you just cleared out. Codebases rely on
  'building relationships' and buying the right guy donuts just like cities
  do. So what now?

9:
  These aren't just sub-problems of each other. Yes you need to solve levels
  above to attack a level below, but they don't stay solved. Solutions are in a
  state of dynamic equilibrium, and if you lose time your gains will be clawed
  back by the issues you haven't yet tackled.

  It's possible my levels are just one possible sequence. But deploying really
  does seem like a place where developers who've been playing with changes in
  individual sandboxes have to now let loose their creations into the world
  and compete, revealing all sorts of externalities they didn't consider.

10:
  This trajectory has probably triggered a few solutions in some of the
  programmers. Many of the obvious solutions turn out not to work for subtle
  reasons.

  Most engineers react to issues with rules. We've seen this change cause
  problem once, twice, thrice, let's outlaw it because others clearly aren't
  smart enough to use it well.

  The problem with rules is that they're straight lines in a phase space that
  don't perfectly isolate good states from bad. Now there's bad states that
  are permitted by your rule, and good states disallowed by it. So you add a
  second line, and a third, and eventually your lines aren't tracking good and
  bad states at all. The underlying terrain of good and bad is fractally
  distributed, and utterly hostile to straight lines.

  Most rules are irrelevant to the actual problem. Jason's talk mentioned all
  the different ways cities in the world have of enabling rich, fulfilling
  lives. Codebases are similar; if two smart people disagree violently about
  the value of a feature, the question is probably irrelevant to creating a
  good codebase. It's clear there's value on both sides. Look for better
  questions.

  One example was a question from yday: Should cities be planned or unplanned?
  You need both, because plans considered certain scenarios and have value,
  but scenarios change and we can't be overly attached to plans. Look for a
  better question.

  Another idea that's been tried is to put engineers in charge. That doesn't
  work because engineers tend to overuse rules, and because they're really
  good at making their arbitrary rules resistant to hacks. A business guy
  wouldn't notice or care that you aren't using the company-mandated language
  for a new feature. But an engineer would.

  Finally, people respond to recurring problems by saying let's setup a metric
  and track it as a team. But metrics gradually gamed as people become
  familiar with them and notice loopholes in how they're gathered, or how they
  can be traded off against other important features that are not being
  tracked.
  
  Metrics are the single biggest cause for commons getting captured by
  insiders.

11:
  So if these don't work, what might?

  Hire less. Since the circles of hell are made worse by the number of people
  involved, reducing that number makes all the problems easier to solve.

  Fewer programmers implies more space for each programmer, but that's ok.
  More space doesn't actually imply more work if you have the right design.
  It's also a lot more pleasant for programmers; their cortexes were evolved
  on the african savannah to want broad horizons and lots of choice. They
  don't want to be hemmed in by abstraction boundaries and neighboring
  programmers.

  Use metrics wisely. There's two hacks here so far:

    a) Don't split metrics between organizations. If your goal is more
    iterations without rollbacks, don't create one team to grow the number of
    iterations and another (testing) to avoid rollbacks. Now you've created
    new externalities.

    b) Focus on lots of sufficing metrics rather than trying to optimize a
    single metric. Rely on human interpretation of these metrics; if a
    manager's job is just to monitor a single number why do we need a human
    there?

  Keep things easy to change. I was talking to Harry yesterday about how
  Kiva's frontend is much cleaner even though there's a lot more cowboy coding
  there than in the backend. That seems to fit my experience as well. More
  rules paradoxically make for more cruft that's now harder to fix because of
  all the rules.

    [Yao: I've seen crappy frontends too. They may seem superficially nicer.
    Me: Yeah, perhaps it's a case of the grass being greener, since I tend to
    be a backend guy.]
  
  We do want some rules, but the key is to find the minimal set of rules that
  discourage change along carefully-chosen directions without slowing things
  down in other axes. Like an eggshell[2] that is much harder to break from
  one side than the other.

  [The GNU copyleft license is such an eggshell.]

  [Venkat: this list of solutions is uncannily like Blitzkrieg. Books: todo -
  Certain to Win.]

  [Megan: This is very poetic. Isn't this just the cycle of life that applies
  to everything? Things get captured and are supplanted. Why is this a
  problem?]

  [Mike Travers: when you make things easier to change, you'll end up with a
  lot more forks.]

[1] Wanted to insert a quote from Devil's Advocate here: "Changes everything,
pressure." Left it on the cutting floor at the last minute as too arcane --
and Alec Resnick made a Devil's Advocate reference in the next talk!

[2] Recurring Jurassic Park references throughout the conference.