Nov 26, 2012
Software libraries don't have to suck

When I said that libraries suck, I wasn't being precise.1 Libraries do lots of things well. They allow programmers to quickly prototype new ideas. They allow names to have multiple meanings based on context. They speed up incremental recompiles, they allow programs on a system to share code pages in RAM. Back in the desktop era, they were even units of commerce. All this is good.

What's not good is the expectation they all-too-frequently set with their users: go ahead, use me in production without understanding me. This expectation has ill-effects for both producers and consumers. Authors of libraries prematurely freeze their interfaces in a futile effort to spare their consumers inconvenience. Consumers of libraries have gotten trained to think that they can outsource parts of their craft to others, and that waiting for 'upstream' to fill some gap is better than hacking a solution yourself and risking a fork. Both of these are bad ideas.

To library authors

Interfaces aren't made in one big-bang moment. They evolve. You write code for one use case. Then maybe you find it works in another, and another. This organic process requires a lengthy gestation period.2 When we try to shortcut it, we end up with heavily-used interfaces that will never be fixed, even though everyone knows they are bad.

A prematurely frozen library doesn't just force people to live with it. People react to it by wrapping it in a cleaner interface. But then they prematurely freeze the new interface, and it starts accumulating warts and bolt-on features just like the old one. Now you have two interfaces. Was forking the existing interface really so much worse an alternative? How much smaller might each codebase in the world be without all the combinatorial explosion of interfaces wrapping other interfaces?

Just admit up-front that upgrades are non-trivial. This will help you maintain a sense of ownership for your interfaces, and make you more willing to gradually do away with the bad ideas.

More changes to the interface will put more pressure on your development process. Embrace that pressure. Help users engage with the development process. Focus on making it easier for users to learn about the implementation, the process of filing bugs.

Often the hardest part of filing a bug for your users is figuring out where to file it. What part of the stack is broken? No amount of black-box architecture astronomy will fix this problem for them. The only solution is to help them understand their system, at least in broad strokes. Start with your library.

Encourage users to fork you. "I'm not sure this is a good idea; why don't we create a fork as an A/B test?" is much more welcoming than "Your pull request was rejected." Publicize your forks, tell people about them, watch the conversation around them. They might change your mind.

Watch out for the warm fuzzies triggered by the word 'reuse'. A world of reuse is a world of promiscuity, with pieces of code connecting up wantonly with each other. Division of labor is a relationship not to be gotten into lightly. It requires knowing what guarantees you need, and what guarantees the counterparty provides. And you can't know what guarantees you need from a subsystem you don't understand.

There's a prisoner's dilemma here: libraries that over-promise will seem to get popular faster. But hold firm; these fashions are short-term. Build something that people will use long after Cucumber has been replaced with Zucchini.

To library users

Expect less. Know what libraries you rely on most, and take ownership for them. Take the trouble to understand how they work. Start pushing on their authors to make them easier to understand. Be more willing to hack on libraries to solve your own problems, even if it risks creating forks. If your solutions are not easily accepted upstream, don't be afraid to publish them yourselves. Just set expectations appropriately. If a library is too much trouble to understand, seek alternatives. Things you don't understand are the source of all technical debt. Try to build your own, for just the use-cases you care about. You might end up with something much simpler to maintain, something that fits better in your head.

(Thanks to Daniel Gackle; to David Barbour, Ray Dillinger and the rest of Lambda the Ultimate; and to Ross Angle, Evan R Murphy, Mark Katakowski, Zak Wilson, Alan Manuel Gloria and the rest of the Arc forum. All of them disagree with parts of this post, and it is better for it.)

notes

1. And trying to distinguish between 'abstraction' and 'service' turned out to obfuscate more than it clarified, so I'm going to avoid those words.

2. Perhaps we need a different name for immature libraries (which are now the vast majority of all libraries). That allows users to set expectations about the level of churn in the interface, and frees up library writers to correct earlier missteps. Not enough people leave time for gestating interfaces, perhaps in analogy with how not enough people leave enough time for debugging.

comments

      
  • johndurbinn, 2012-11-26: This is all completely wrong.   
        
    • Anonymous, 2012-11-27: Okay. Now, if you actually want to be helpful, tell us why.   
          
      • johndurbinn, 2012-11-27: Libraries are still needed even in the age of Google. I understand that you can find everything using the internet, but many older people rely on "old fashioned" technology like bound paper books, and buildings to house those books with indexes to locate those books. There's no reason to attack libraries for being out of date.   
            
        • Kartik Agaram, 2012-11-27: Ah, I apologize, I'm a programmer and I'm referring to software libraries, which are very different. Somebody else pointed this out as well. My girlfriend is a librarian, so I better fix this title before she sees it.. :)
      
  • Anonymous, 2012-11-27: No one's going to take the time to upgrade unless the replacement can be dropped in with little effort. It just doesn't happen in the real world.

    So you have two choices: backport all bug fixes to every version of your library where you made a breaking change or simply don't make breaking changes.   

        
    • Kartik Agaram, 2012-11-27: Lots of things didn't used to happen in the real world -- until they did.

      New ideas take time to percolate through and be acted on. And that's fortunate, because I'm not nearly confident enough about this to claim everybody should start doing this right this instant. Instead I'm experimenting with profligate forking in a little side project of mine: http://github.com/akkartik/wart.

      
  • rocketnia, 2012-11-27: Great points! Now I have some complementary additions....

    We already build software with such complex social dependencies that no one developer team can really afford to take ownership of it all. For instance, consider how expensive it has been to "seek alternatives" for entrenched consumer platforms like Windows, Flash, and von Neumann architectures. At some point, we mostly-isolated developers must find places to rest, and API documentation gives us something to believe in. The reassurance we get from this documentation may still be feeble, heavily dependent on social faith, but fortunately we continue to find objective ways to validate it.

    As you say, division of labor does require knowing what guarantees our components need and what guarantees external components can be expected to fulfill. However, sometimes we must be willing to impose requirements on systems we don't quite understand, since at least one of the systems we try to interact with is the outside world!   

        
    • Kartik Agaram, 2012-11-27: Complementary additions, are those like objections? ^_^

      I'm not trying to be purist about this -- especially since we don't understand most of even the software that's technically owned by us. I'm just asking that we think of the entire stack as under our ownership. When you find out about something broken in it, begin first by fixing it in your stack. Then worry about what to do next.

      Hey, it just occurred to me that I'm asking for an attitude of Kaizen [1] [2] towards our software stack.   

          
      • rocketnia, 2012-11-28: They're more like apologies than objections. :-p

        I find it hard to believe in a universal concept of "broken." Even if I trip on a rug, I don't want to pull it out from under someone else. ;)

        I'm afraid I have to settle into a culture—get with the program, so to speak—before I'm confident enough to derug it.   

            
        • Kartik Agaram, 2012-11-28: I see what you mean. Perhaps tuning things for yourself doesn't have to 'derug' anybody else?

          It's something so pervasive that we all take it for granted, this idea that we have to put the communal good above our own when it comes to these levels of abstraction. But individualism hath its merits.

      
  • Kartik Agaram, 2013-05-20: An opposing view: "Code you can reason about is better than code you can't. Rely on libraries written and tested by other smart people to reduce the insane quantity of stuff you have to understand. If you don't get how to test that your merge function is associative, commutative, and idempotent, maybe you shouldn't be writing your own CRDTs just yet. Implementing two-phase commit on top of your database may be a warning sign." http://aphyr.com/posts/286-call-me-maybe-final-thoughts   
  • Kartik Agaram, 2014-05-20: "Users of libraries ought to know something about what goes on inside." -- Donald Knuth, http://www.informit.com/articles/article.aspx?p=2213858&WT.mc_id=Author_Knuth_20Questions   
  • Kartik Agaram, 2015-10-04: "Reusing code without reevaluating its design decisions means we are prone to inherit all of its constraints as well. Often there can be a better, simpler way of doing things, but this requires asking why the code is the way it is. Otherwise, like the proverbial hoarder, we end up with piles and piles of stuff in the mistaken belief that since it was useful yesterday, it must still be useful today." -- http://www.tedunangst.com/flak/post/hoarding-and-reuse   
  • Kartik Agaram, 2016-03-03: "The difference is, are you abstracting away so that you truly can say “I don’t have to worry about this”? Or are you abstracting away because you’re aware of those guts, but want to focus your attention right now in this area. That is what we’re looking for." -- John Allspaw, Etsy CEO (http://thenewstack.io/etsy-cto-qa-need-software-engineers-not-developers)

Comments gratefully appreciated. Please send them to me by any method of your choice and I'll include them here.

archive
projects
writings
videos
interviews
subscribe
Mastodon
RSS (?)
twtxt (?)
Station (?)