Nov 26, 2012
Software libraries don't have to suck
When I said that libraries suck, I wasn't being
precise.1 Libraries do lots of things
well. They allow programmers to quickly prototype new ideas. They allow names
to have multiple meanings based on context. They speed up incremental
recompiles, they allow programs on a system to share code pages in RAM. Back
in the desktop era, they were even units of commerce. All this is good.
What's not good is the expectation they all-too-frequently set with their
users: go ahead, use me in production without understanding me. This
expectation has ill-effects for both producers and consumers. Authors of
libraries prematurely freeze their interfaces in a futile effort to spare
their consumers inconvenience. Consumers of libraries have gotten trained to
think that they can outsource parts of their craft to others, and that waiting
for 'upstream' to fill some gap is better than hacking a solution yourself and
risking a fork. Both of these are bad ideas.
To library authors
Interfaces aren't made in one big-bang moment. They evolve. You write code for
one use case. Then maybe you find it works in another, and another. This
organic process requires a lengthy gestation period.2
When we try to shortcut it, we end up with heavily-used interfaces that will
never be fixed, even though everyone knows they are bad.
A prematurely frozen library doesn't just force people to live with it. People
react to it by wrapping it in a cleaner interface. But then they prematurely
freeze the new interface, and it starts accumulating warts and bolt-on
features just like the old one. Now you have two
interfaces. Was forking the existing interface really so much worse an
alternative? How much smaller might each codebase in the world be without all
the combinatorial explosion of interfaces wrapping other interfaces?
Just admit up-front that upgrades are non-trivial. This will help you maintain
a sense of ownership for your interfaces, and make you more willing to
gradually do away with the bad ideas.
More changes to the interface will put more pressure on your development
process. Embrace that pressure. Help users engage with the development
process. Focus on making it easier for users to learn about the
implementation, the process of filing bugs.
Often the hardest part of filing a bug for your users is figuring out where to
file it. What part of the stack is broken? No amount of black-box architecture
astronomy will fix this problem for them. The only solution is to help them
understand their system, at least in broad strokes. Start with your library.
Encourage users to fork you. "I'm not sure this is a good idea; why don't we
create a fork as an A/B test?" is much more welcoming than "Your pull request
was rejected." Publicize your forks, tell people about them, watch the
conversation around them. They might change your mind.
Watch out for the warm fuzzies triggered by the word 'reuse'. A world of reuse
is a world of promiscuity, with pieces of code connecting up wantonly with
each other. Division of labor is a relationship not to be gotten into lightly.
It requires knowing what guarantees you need, and what guarantees the
counterparty provides. And you can't know what guarantees you need from a subsystem you don't understand.
There's a prisoner's dilemma
here: libraries that over-promise will seem to get popular faster. But hold
firm; these fashions are short-term. Build something that people will use long
after Cucumber has been replaced with Zucchini.
To library users
Expect less. Know what libraries you rely on most, and take ownership for
them. Take the trouble to understand how they work. Start pushing on their
authors to make them easier to understand. Be more willing to hack on
libraries to solve your own problems, even if it risks creating forks. If your
solutions are not easily accepted upstream, don't be afraid to publish them
yourselves. Just set expectations appropriately. If a library is too much
trouble to understand, seek alternatives. Things you don't understand are the
source of all technical debt. Try to build your own, for just the
use-cases you care about. You might end up with something much simpler to
maintain, something that fits better in your head.
1. And trying to distinguish between
'abstraction' and 'service' turned out to obfuscate more than it clarified, so
I'm going to avoid those words.
2. Perhaps we need a different name for
immature libraries (which are now the vast majority of all libraries). That
allows users to set expectations about the level of churn in the interface,
and frees up library writers to correct earlier missteps. Not enough people
leave time for gestating interfaces, perhaps in analogy with how
not enough people leave enough time for debugging.
Nov 24, 2012
Comments in code: the more we write, the less we want to highlight
That's my immediate reaction watching these programmers argue
about what color their comments should be when reading code. It seems those
who write sparse comments want them to pop out of the screen, and those who
comment more heavily like to provide a background hum of human commentary
that's useful to read in certain contexts and otherwise easy to filter out.
Now that I think about it, this matches my experience. I've experienced good
codebases commented both sparsely and heavily. The longer I spend with a
sparsely-commented codebase, the more I cling to the comments it does have.
They act as landmarks, concise reminders of invariants. However, as I grow
familiar with a heavily-commented codebase I tend to skip past the comments.
Code is non-linear and can be read in lots of ways, with lots of different
questions in mind. Inevitably, narrative comments only answer some of those
questions and are a drag the rest of the time.
I'm reminded of one of Lincoln's famous quotes,
a fore-shadowing of the CAP theorem.
Comments can be either detailed or salient, never both.
Comments are versatile. Perhaps we need two kinds of comments that can be
colored differently. Are there still other uses for them?
Nov 12, 2012
Software Libraries suck
Here's why, in a sentence: they promise to be abstractions, but they
end up becoming services. An abstraction frees you from thinking
about its internals every time you use it. A service allows you to never learn
its internals. A service is not an abstraction. It isn't 'abstracting' away
the details. Somebody else is thinking about the details so you can remain
Programmers manage abstraction boundaries, that's our stock in trade. Managing
them requires bouncing around on both sides of them. If you restrict yourself
to one side of an abstraction, you're limiting your growth as a programmer.1
You're chopping off your strength and potential, one lock of hair at a time,
and sacrificing it on the altar of convenience.
A library you're ignorant of is a risk you're exposed to, a now-quiet frontier
that may suddenly face assault from some bug when you're on a deadline and can
least afford the distraction. Better to take a day or week now, when things
are quiet, to save that hour of life-shortening stress when it really
You don't have to give up the libraries you currently rely on. You just have
to take the effort to enumerate them, locate them on your system, install the
sources if necessary, and take ownership the next time your program dies
within them, or uncovers a bug in them. Are these activities more
time-consuming than not doing them? Of course. Consider them a long-term
Just enumerating all the libraries you rely on others to provide can be
eye-opening. Tot up all the open bugs in their trackers and you have a sense
of your exposure to risks outside your control. In fact, forget the whole
system. Just start with your Gemfile
They're probably lowest-maturity and therefore highest risk.
Once you assess the amount of effort that should be going into each library
you use, you may well wonder if all those libraries are worth the effort. And
that's a useful insight as well. “Achievement unlocked: I've stopped
adding dependencies willy-nilly.”
(This birth was midwifed by conversations with Ross Angle, Dan Grover, and Manuel
1. If you don't identify as a programmer, if that
isn't your core strength, if you just program now and then because it's
expedient, then treating libraries as services may make more sense. If a major
issue pops up you'll need to find more expert help, but you knew that already.
Jun 18, 2012
How to use a profiler
All of us programmers have at some point tried to speed up a large program.
We remember "measure before optimizing" and profile it, and end up (a few
hours later) with something intimidating like this and.. what
next? If you're like me, you scratch your head at the prospect of optimizing
StringAppend, and the call-graph seems to tell you what
you already know: Your program spends most of its time in the main loop,
divided between the main subtasks.
I used to imagine the optimization process like this:
1. Run a profiler
2. Select a hot spot
But the details were hazy. Especially in step 3. Michael Abrash was clearly
doing a lot more than this. What was it?
Worse, I kept forgetting to use the profiler. I'd have a split-second idea and
blunder around for hours before remembering the wisdom of "measure before
optimizing." I was forgetting to measure because I was getting so little out
of it, because I'd never learned to do it right.
After a lot of trial and error in the last few months, I think I have a
better sense of the process. Optimization is like science. You can't start
with experiments. You have to start with a hypothesis. "My program is spending
too much time in _." Fill in the blanks, then translate the sentence for a
profile. "I expect to see more time spent in function A than B." Then run the
profile and check your results. Skip the low-level stuff, look for just A and
B in the cumulative charts. Which takes more time? Is one much more of a
bottleneck? Keep an eye out for a peer function that you hadn't considered,
something that's a sibling of A and B in the call-graph,
about the same level of granularity.
Do this enough times and you gain an intuition of what your program is doing,
and where it's spending its time.
When you do find a function at a certain level of granularity that seems to be
taking too long, it's time to focus on what it does and how it works. This is
what people mean when they say, "look for a better algorithm." Can the data
structures be better organized from the perspective of this function? Is it
being called needlessly? Can we prevent it being called too often? Can we
specialize a simpler variant for 90% of the calls?
If none of that generates any ideas, then it's time to bite the
bullet and drop down to a lower level.
But remember: optimization is about understanding your program. Begin there,
and profiling and other tools will come to hand more naturally.
Jul 14, 2011
Evolution of a rails programmer
Idiomatic rails action for registering a user if he doesn't exist:
After a year of programming in lisp, I find it most natural to write:
Is this overly concise/obfuscated? I like it because it concisely expresses
the error case as an early exit; most of the space is devoted to the
successful save, which is straight-line code without distracting branches.
It's clearer that we either pick an existing user or create a new one. Form
Aug 3, 2009
Tests are a technique to manage programmer anxiety about code. When I feel
anxious about some aspect of my code I write a test case.
Does programming language affect level of anxiety? Definitely. Does
programmer skill affect level of anxiety? Absolutely.
Writing tests becomes more important when you're part of a team. Your
choices affect not just your anxiety but that of your teammates. That's why
it's reasonable to be more dogmatic about TDD in a team.
A lot of 'getting better at TDD' is just getting better at listening to
yourself. When I started programming the little anxieties would pile up
until I'd painted myself into a corner. With experience I pay more attention
to the little anxieties.
The secondary effect: after some time doing TDD I feel less anxious just
knowing that I can write a test if I want. The benefit of the tests as an
artifact is secondary to me; what they primarily do is keep me from getting
stressed and giving up to go play poker.
May 23, 2009
Realism in simulation
"The goal of simulation is not to simulate reality as closely as possible.
With an accurate model you cannot find commonalities."
— Tom Slee
Oct 8, 2008
You’re going to have to rewrite code. All the time. You have to be ok with it. You have to be willing
. This is the secret of the great programmers, and the great stumbling block of those new to refactoring and TDD.
The bad news: Refactorings like extract object will require you to test drive a new class from scratch. All the time.
The good news: It’s much easier to rewrite when you have tests. You just haven’t noticed yet.
Aug 26, 2008
in scaling software projects is not a communication problem; it’s a dependency problem. Control quality by being hierarchical and rigorous about how you test and accept changes, not how you generate them.