Posts tagged with 'Programming'
Nov 26, 2012
Software libraries don't have to suck

When I said that libraries suck, I wasn't being precise.1 Libraries do lots of things well. They allow programmers to quickly prototype new ideas. They allow names to have multiple meanings based on context. They speed up incremental recompiles, they allow programs on a system to share code pages in RAM. Back in the desktop era, they were even units of commerce. All this is good.

What's not good is the expectation they all-too-frequently set with their users: go ahead, use me in production without understanding me. This expectation has ill-effects for both producers and consumers. Authors of libraries prematurely freeze their interfaces in a futile effort to spare their consumers inconvenience. Consumers of libraries have gotten trained to think that they can outsource parts of their craft to others, and that waiting for 'upstream' to fill some gap is better than hacking a solution yourself and risking a fork. Both of these are bad ideas.

To library authors

Interfaces aren't made in one big-bang moment. They evolve. You write code for one use case. Then maybe you find it works in another, and another. This organic process requires a lengthy gestation period.2 When we try to shortcut it, we end up with heavily-used interfaces that will never be fixed, even though everyone knows they are bad.

A prematurely frozen library doesn't just force people to live with it. People react to it by wrapping it in a cleaner interface. But then they prematurely freeze the new interface, and it starts accumulating warts and bolt-on features just like the old one. Now you have two interfaces. Was forking the existing interface really so much worse an alternative? How much smaller might each codebase in the world be without all the combinatorial explosion of interfaces wrapping other interfaces?

Just admit up-front that upgrades are non-trivial. This will help you maintain a sense of ownership for your interfaces, and make you more willing to gradually do away with the bad ideas.

More changes to the interface will put more pressure on your development process. Embrace that pressure. Help users engage with the development process. Focus on making it easier for users to learn about the implementation, the process of filing bugs.

Often the hardest part of filing a bug for your users is figuring out where to file it. What part of the stack is broken? No amount of black-box architecture astronomy will fix this problem for them. The only solution is to help them understand their system, at least in broad strokes. Start with your library.

Encourage users to fork you. "I'm not sure this is a good idea; why don't we create a fork as an A/B test?" is much more welcoming than "Your pull request was rejected." Publicize your forks, tell people about them, watch the conversation around them. They might change your mind.

Watch out for the warm fuzzies triggered by the word 'reuse'. A world of reuse is a world of promiscuity, with pieces of code connecting up wantonly with each other. Division of labor is a relationship not to be gotten into lightly. It requires knowing what guarantees you need, and what guarantees the counterparty provides. And you can't know what guarantees you need from a subsystem you don't understand.

There's a prisoner's dilemma here: libraries that over-promise will seem to get popular faster. But hold firm; these fashions are short-term. Build something that people will use long after Cucumber has been replaced with Zucchini.

To library users

Expect less. Know what libraries you rely on most, and take ownership for them. Take the trouble to understand how they work. Start pushing on their authors to make them easier to understand. Be more willing to hack on libraries to solve your own problems, even if it risks creating forks. If your solutions are not easily accepted upstream, don't be afraid to publish them yourselves. Just set expectations appropriately. If a library is too much trouble to understand, seek alternatives. Things you don't understand are the source of all technical debt. Try to build your own, for just the use-cases you care about. You might end up with something much simpler to maintain, something that fits better in your head.

(Thanks to Daniel Gackle; to David Barbour, Ray Dillinger and the rest of Lambda the Ultimate; and to Ross Angle, Evan R Murphy, Mark Katakowski, Zak Wilson, Alan Manuel Gloria and the rest of the Arc forum. All of them disagree with parts of this post, and it is better for it.)


1. And trying to distinguish between 'abstraction' and 'service' turned out to obfuscate more than it clarified, so I'm going to avoid those words.

2. Perhaps we need a different name for immature libraries (which are now the vast majority of all libraries). That allows users to set expectations about the level of churn in the interface, and frees up library writers to correct earlier missteps. Not enough people leave time for gestating interfaces, perhaps in analogy with how not enough people leave enough time for debugging.


* *
Nov 24, 2012
Comments in code: the more we write, the less we want to highlight

That's my immediate reaction watching these programmers argue about what color their comments should be when reading code. It seems those who write sparse comments want them to pop out of the screen, and those who comment more heavily like to provide a background hum of human commentary that's useful to read in certain contexts and otherwise easy to filter out.

Now that I think about it, this matches my experience. I've experienced good codebases commented both sparsely and heavily. The longer I spend with a sparsely-commented codebase, the more I cling to the comments it does have. They act as landmarks, concise reminders of invariants. However, as I grow familiar with a heavily-commented codebase I tend to skip past the comments. Code is non-linear and can be read in lots of ways, with lots of different questions in mind. Inevitably, narrative comments only answer some of those questions and are a drag the rest of the time.

I'm reminded of one of Lincoln's famous quotes, a fore-shadowing of the CAP theorem. Comments can be either detailed or salient, never both.

Comments are versatile. Perhaps we need two kinds of comments that can be colored differently. Are there still other uses for them?


* *
Nov 12, 2012
Software libraries suck

Here's why, in a sentence: they promise to be abstractions, but they end up becoming services. An abstraction frees you from thinking about its internals every time you use it. A service allows you to never learn its internals. A service is not an abstraction. It isn't 'abstracting' away the details. Somebody else is thinking about the details so you can remain ignorant.

Programmers manage abstraction boundaries, that's our stock in trade. Managing them requires bouncing around on both sides of them. If you restrict yourself to one side of an abstraction, you're limiting your growth as a programmer.1 You're chopping off your strength and potential, one lock of hair at a time, and sacrificing it on the altar of convenience.

A library you're ignorant of is a risk you're exposed to, a now-quiet frontier that may suddenly face assault from some bug when you're on a deadline and can least afford the distraction. Better to take a day or week now, when things are quiet, to save that hour of life-shortening stress when it really matters.

You don't have to give up the libraries you currently rely on. You just have to take the effort to enumerate them, locate them on your system, install the sources if necessary, and take ownership the next time your program dies within them, or uncovers a bug in them. Are these activities more time-consuming than not doing them? Of course. Consider them a long-term investment.

Just enumerating all the libraries you rely on others to provide can be eye-opening. Tot up all the open bugs in their trackers and you have a sense of your exposure to risks outside your control. In fact, forget the whole system. Just start with your Gemfile or npm_modules. They're probably lowest-maturity and therefore highest risk.

Once you assess the amount of effort that should be going into each library you use, you may well wonder if all those libraries are worth the effort. And that's a useful insight as well. “Achievement unlocked: I've stopped adding dependencies willy-nilly.”

Update: Check out the sequel. Particularly if this post left you scratching your head about what I could possibly be going on about.

(This birth was midwifed by conversations with Ross Angle, Dan Grover, and Manuel Simoni.)


1. If you don't identify as a programmer, if that isn't your core strength, if you just program now and then because it's expedient, then treating libraries as services may make more sense. If a major issue pops up you'll need to find more expert help, but you knew that already.


* *
Jun 18, 2012
How to use a profiler

All of us programmers have at some point tried to speed up a large program. We remember "measure before optimizing" and profile it, and end up (a few hours later) with something intimidating like this and.. what next? If you're like me, you scratch your head at the prospect of optimizing StringAppend, and the call-graph seems to tell you what you already know: Your program spends most of its time in the main loop, divided between the main subtasks.

I used to imagine the optimization process like this:

1. Run a profiler
2. Select a hot spot
3. ...
4. Speedup!

But the details were hazy. Especially in step 3. Michael Abrash was clearly doing a lot more than this. What was it?

Worse, I kept forgetting to use the profiler. I'd have a split-second idea and blunder around for hours before remembering the wisdom of "measure before optimizing." I was forgetting to measure because I was getting so little out of it, because I'd never learned to do it right.

After a lot of trial and error in the last few months, I think I have a better sense of the process. Optimization is like science. You can't start with experiments. You have to start with a hypothesis. "My program is spending too much time in _." Fill in the blanks, then translate the sentence for a profile. "I expect to see more time spent in function A than B." Then run the profile and check your results. Skip the low-level stuff, look for just A and B in the cumulative charts. Which takes more time? Is one much more of a bottleneck? Keep an eye out for a peer function that you hadn't considered, something that's a sibling of A and B in the call-graph, about the same level of granularity.

Do this enough times and you gain an intuition of what your program is doing, and where it's spending its time.

When you do find a function at a certain level of granularity that seems to be taking too long, it's time to focus on what it does and how it works. This is what people mean when they say, "look for a better algorithm." Can the data structures be better organized from the perspective of this function? Is it being called needlessly? Can we prevent it being called too often? Can we specialize a simpler variant for 90% of the calls?

If none of that generates any ideas, then it's time to bite the bullet and drop down to a lower level.

But remember: optimization is about understanding your program. Begin there, and profiling and other tools will come to hand more naturally.


* *
Jul 14, 2011
Evolution of a rails programmer

Idiomatic rails action for registering a user if he doesn't exist:

After a year of programming in lisp, I find it most natural to write:

Is this overly concise/obfuscated? I like it because it concisely expresses the error case as an early exit; most of the space is devoted to the successful save, which is straight-line code without distracting branches. It's clearer that we either pick an existing user or create a new one. Form follows function.


* *
Aug 3, 2009
Tests are a technique to manage programmer anxiety about code. When I feel anxious about some aspect of my code I write a test case.

A lot of 'getting better at TDD' is just getting better at listening to yourself. When I started programming the little anxieties would pile up until I'd painted myself into a corner. With experience I pay more attention to the little anxieties.

The secondary effect: after some time doing TDD I feel less anxious just knowing that I can write a test if I want. The benefit of the tests as an artifact is secondary to me; what they primarily do is keep me from getting stressed and giving up to go play poker.

Writing tests becomes more important when you're part of a team. Your choices affect not just your anxiety but that of your teammates. That's why it's reasonable to be more dogmatic about TDD in a team.


* *
May 23, 2009
Realism in simulation

"The goal of simulation is not to simulate reality as closely as possible. With an accurate model you cannot find commonalities."
Tom Slee


* *
Jan 24, 2009
How to enter the Zone at will

You know what I'm talking about. Programming can be the best of times or the worst of times. Sometimes the fingers seem clumsy on the keyboard, tool after tool acts flaky, we scream and curse, we become aware of the huge tower of complexity we rely on. At other times all thought flies except for this one thing we're working on right now. The fingers fly, the thoughts hum, and we eventually exit the trance amazed at what we have accomplished, at how cool it all is. We call this the Zone, writers and artists call it the Muse, and it has taken a while to realize that it's just a mental state only loosely influenced by external factors.

I too have chased after the Zone day after day for many years as I struggle to bring programming ideas to life. Like conversation, programming can give you anything but consistency. You never know if this session will be good or bad. Sometimes things are good from the moment you hit the keyboard. Most times, though, you struggle for a while before you find it, before you lose yourself. On most days it takes me 2 hours just to get going, to stop noticing the mechanical actions between thought and software. Since getting going takes so long I've gotten used to the idea that programming needs large quanta of time. I need my time to be divided up into chunks at least 4 hours long. The truly great sessions require 8 hour sessions, multiple of them broken up only by sleep without thought to external stimulus or time of day.

Or so I thought until now. A few days ago I noticed that I've gone months without hacking for 4 hours straight. My output is not reduced, indeed I'm accomplishing as much as I ever have. These days it seems I can make something out of even the stray hour that gets thrown my way. What changed?

What changed was that I started doing TDD. I started breaking big ideas down into lists of stories in a little text file, picking a story, writing test cases for it. That was it. Somehow, it seemed, just having stories and test cases served to focus the mind.

Did you ever find yourself thinking, "Man, for all the lengthy reasoning if I'd just heard about reason x I'd have been on board from the start?" Forget exercise-is-good-for-you, bruce-lee-beating-up-the-bad-guys, girls-like-it -- if I'd only known it would improve my posture I'd have started doing weights long ago. Forget politeness, a smooth ride, traffic safety -- if only I'd been reminded of speed limits and traffic cops and tickets, I'd have slowed right down.

And I would have been all over TDD if I'd known it would get me to the Zone at will. Forget all the lengthy rationales about project success rates, the agile manifesto, dealing with changing requirements, avoiding regressions. When I think about all those years of nursing my RSI, wondering what I was doing wrong after a lengthy disappointment, psyching myself up to the level of focus and discipline necessary for long hours at a stretch, procrastinating to avoid the grind, when I think about these things, oh man. If only I'd known.

(Inspired by this thread:


* *
Oct 8, 2008
You’re going to have to rewrite code. All the time. You have to be ok with it. You have to be willing. This is the secret of the great programmers, and the great stumbling block of those new to refactoring and TDD.

The bad news: Refactorings like extract object will require you to test drive a new class from scratch. All the time.
The good news: It’s much easier to rewrite when you have tests. You just haven’t noticed yet.

— me


* *
Aug 26, 2008
The problem in scaling software projects is not a communication problem; it’s a dependency problem. Control quality by being hierarchical and rigorous about how you test and accept changes, not how you generate them.


* *
Making the big picture easy to see, in software and in society at large.
Prose (shorter; favorites)