Posts tagged with 'Programming'
Nov 24, 2012
Comments in code: the more we write, the less we want to highlight

That's my immediate reaction watching these programmers argue about what color their comments should be when reading code. It seems those who write sparse comments want them to pop out of the screen, and those who comment more heavily like to provide a background hum of human commentary that's useful to read in certain contexts and otherwise easy to filter out.

Now that I think about it, this matches my experience. I've experienced good codebases commented both sparsely and heavily. The longer I spend with a sparsely-commented codebase, the more I cling to the comments it does have. They act as landmarks, concise reminders of invariants. However, as I grow familiar with a heavily-commented codebase I tend to skip past the comments. Code is non-linear and can be read in lots of ways, with lots of different questions in mind. Inevitably, narrative comments only answer some of those questions and are a drag the rest of the time.

I'm reminded of one of Lincoln's famous quotes, a fore-shadowing of the CAP theorem. Comments can be either detailed or salient, never both.

Comments are versatile. Perhaps we need two kinds of comments that can be colored differently. Are there still other uses for them?


* *
Nov 12, 2012
Software libraries suck

Here's why, in a sentence: they promise to be abstractions, but they end up becoming services. An abstraction frees you from thinking about its internals every time you use it. A service allows you to never learn its internals. A service is not an abstraction. It isn't 'abstracting' away the details. Somebody else is thinking about the details so you can remain ignorant.

Programmers manage abstraction boundaries, that's our stock in trade. Managing them requires bouncing around on both sides of them. If you restrict yourself to one side of an abstraction, you're limiting your growth as a programmer.1 You're chopping off your strength and potential, one lock of hair at a time, and sacrificing it on the altar of convenience.

A library you're ignorant of is a risk you're exposed to, a now-quiet frontier that may suddenly face assault from some bug when you're on a deadline and can least afford the distraction. Better to take a day or week now, when things are quiet, to save that hour of life-shortening stress when it really matters.

You don't have to give up the libraries you currently rely on. You just have to take the effort to enumerate them, locate them on your system, install the sources if necessary, and take ownership the next time your program dies within them, or uncovers a bug in them. Are these activities more time-consuming than not doing them? Of course. Consider them a long-term investment.

Just enumerating all the libraries you rely on others to provide can be eye-opening. Tot up all the open bugs in their trackers and you have a sense of your exposure to risks outside your control. In fact, forget the whole system. Just start with your Gemfile or npm_modules. They're probably lowest-maturity and therefore highest risk.

Once you assess the amount of effort that should be going into each library you use, you may well wonder if all those libraries are worth the effort. And that's a useful insight as well. “Achievement unlocked: I've stopped adding dependencies willy-nilly.”

Update: Check out the sequel.

(This birth was midwifed by conversations with Ross Angle, Dan Grover, and Manuel Simoni.)


1. If you don't identify as a programmer, if that isn't your core strength, if you just program now and then because it's expedient, then treating libraries as services may make more sense. If a major issue pops up you'll need to find more expert help, but you knew that already.


* *
Jun 18, 2012
How to use a profiler

All of us programmers have at some point tried to speed up a large program. We remember "measure before optimizing" and profile it, and end up (a few hours later) with something intimidating like this and.. what next? If you're like me, you scratch your head at the prospect of optimizing StringAppend, and the call-graph seems to tell you what you already know: Your program spends most of its time in the main loop, divided between the main subtasks.

I used to imagine the optimization process like this:

1. Run a profiler
2. Select a hot spot
3. ...
4. Speedup!

But the details were hazy. Especially in step 3. Michael Abrash was clearly doing a lot more than this. What was it?

Worse, I kept forgetting to use the profiler. I'd have a split-second idea and blunder around for hours before remembering the wisdom of "measure before optimizing." I was forgetting to measure because I was getting so little out of it, because I'd never learned to do it right.

After a lot of trial and error in the last few months, I think I have a better sense of the process. Optimization is like science. You can't start with experiments. You have to start with a hypothesis. "My program is spending too much time in _." Fill in the blanks, then translate the sentence for a profile. "I expect to see more time spent in function A than B." Then run the profile and check your results. Skip the low-level stuff, look for just A and B in the cumulative charts. Which takes more time? Is one much more of a bottleneck? Keep an eye out for a peer function that you hadn't considered, something that's a sibling of A and B in the call-graph, about the same level of granularity.

Do this enough times and you gain an intuition of what your program is doing, and where it's spending its time.

When you do find a function at a certain level of granularity that seems to be taking too long, it's time to focus on what it does and how it works. This is what people mean when they say, "look for a better algorithm." Can the data structures be better organized from the perspective of this function? Is it being called needlessly? Can we prevent it being called too often? Can we specialize a simpler variant for 90% of the calls?

If none of that generates any ideas, then it's time to bite the bullet and drop down to a lower level.

But remember: optimization is about understanding your program. Begin there, and profiling and other tools will come to hand more naturally.


* *
Jul 14, 2011
Evolution of a rails programmer

Idiomatic rails action for registering a user if he doesn't exist:

After a year of programming in lisp, I find it most natural to write:

Is this overly concise/obfuscated? I like it because it concisely expresses the error case as an early exit; most of the space is devoted to the successful save, which is straight-line code without distracting branches. It's clearer that we either pick an existing user or create a new one. Form follows function.


* *
Aug 3, 2009
Tests are a technique to manage programmer anxiety about code. When I feel anxious about some aspect of my code I write a test case.

A lot of 'getting better at TDD' is just getting better at listening to yourself. When I started programming the little anxieties would pile up until I'd painted myself into a corner. With experience I pay more attention to the little anxieties.

The secondary effect: after some time doing TDD I feel less anxious just knowing that I can write a test if I want. The benefit of the tests as an artifact is secondary to me; what they primarily do is keep me from getting stressed and giving up to go play poker.

Writing tests becomes more important when you're part of a team. Your choices affect not just your anxiety but that of your teammates. That's why it's reasonable to be more dogmatic about TDD in a team.


* *
May 23, 2009
Realism in simulation

"The goal of simulation is not to simulate reality as closely as possible. With an accurate model you cannot find commonalities."
Tom Slee


* *
Jan 24, 2009
How to enter the Zone at will

You know what I'm talking about. Programming can be the best of times or the worst of times. Sometimes the fingers seem clumsy on the keyboard, tool after tool acts flaky, we scream and curse, we become aware of the huge tower of complexity we rely on. At other times all thought flies except for this one thing we're working on right now. The fingers fly, the thoughts hum, and we eventually exit the trance amazed at what we have accomplished, at how cool it all is. We call this the Zone, writers and artists call it the Muse, and it has taken a while to realize that it's just a mental state only loosely influenced by external factors.

I too have chased after the Zone day after day for many years as I struggle to bring programming ideas to life. Like conversation, programming can give you anything but consistency. You never know if this session will be good or bad. Sometimes things are good from the moment you hit the keyboard. Most times, though, you struggle for a while before you find it, before you lose yourself. On most days it takes me 2 hours just to get going, to stop noticing the mechanical actions between thought and software. Since getting going takes so long I've gotten used to the idea that programming needs large quanta of time. I need my time to be divided up into chunks at least 4 hours long. The truly great sessions require 8 hour sessions, multiple of them broken up only by sleep without thought to external stimulus or time of day.

Or so I thought until now. A few days ago I noticed that I've gone months without hacking for 4 hours straight. My output is not reduced, indeed I'm accomplishing as much as I ever have. These days it seems I can make something out of even the stray hour that gets thrown my way. What changed?

What changed was that I started doing TDD. I started breaking big ideas down into lists of stories in a little text file, picking a story, writing test cases for it. That was it. Somehow, it seemed, just having stories and test cases served to focus the mind.

Did you ever find yourself thinking, "Man, for all the lengthy reasoning if I'd just heard about reason x I'd have been on board from the start?" Forget exercise-is-good-for-you, bruce-lee-beating-up-the-bad-guys, girls-like-it -- if I'd only known it would improve my posture I'd have started doing weights long ago. Forget politeness, a smooth ride, traffic safety -- if only I'd been reminded of speed limits and traffic cops and tickets, I'd have slowed right down.

And I would have been all over TDD if I'd known it would get me to the Zone at will. Forget all the lengthy rationales about project success rates, the agile manifesto, dealing with changing requirements, avoiding regressions. When I think about all those years of nursing my RSI, wondering what I was doing wrong after a lengthy disappointment, psyching myself up to the level of focus and discipline necessary for long hours at a stretch, procrastinating to avoid the grind, when I think about these things, oh man. If only I'd known.

(Inspired by this thread:


* *
Oct 8, 2008
You’re going to have to rewrite code. All the time. You have to be ok with it. You have to be willing. This is the secret of the great programmers, and the great stumbling block of those new to refactoring and TDD.

The bad news: Refactorings like extract object will require you to test drive a new class from scratch. All the time.
The good news: It’s much easier to rewrite when you have tests. You just haven’t noticed yet.

— me


* *
Aug 26, 2008
The problem in scaling software projects is not a communication problem; it’s a dependency problem. Control quality by being hierarchical and rigorous about how you test and accept changes, not how you generate them.


* *
Aug 5, 2008
Analytics are a 25-40% tax on your product development process. They take engineers lots of time and development effort, produce numbers that people argue about, and require additional infrastructure.

Having 1 in 4 engineers working on analytics may seem like a ton, but it helps validate assumptions, pinpoint key features and bottlenecks, model the business during decision-making. At the cost of building fewer features, learn as much as you can so that you can “run up the score” on the features that work.


* *
Making the big picture easy to see, in software and in society at large.
Code (contributions)
Prose (shorter; favorites)
favorite insights
Social Dynamics
Social Software