Our hearts do in dark forests dwell.
Each feeling, choosing, kernel core
lives in a roughly-tangled dell,
a wilderness of soul and spore.
Each forest lives, each forest grows,
unfurls through time our destinies.
Alive with buzzing thoughts, each glows,
— Andrew Hamilton, “Akrasia Forest”
I’ve always wondered — sometimes more, sometimes less — what my life might become. Nine years ago those wonderings led me to the house and grave of my favorite composer. The past few months of leisurely introspection have encouraged me again to wonder more. When the past week found me within walking distance, I went for a return visit.
I brought my changed self. Since Medtner’s resting place last observed me, I had not merely learned to play some and analyze some of his music, I’d finally composed some I was proud to call mine. On the off chance that wouldn’t suffice, I also brought flowers. Then I cued up my recorded performance of his Sonata-Reminiscenza, sat at the foot of an immense tree, and marveled at the life Medtner lived while making the music he made. Was the former a necessary precondition for the latter, or merely incidental? Are my preconditions likely similar to his? If I’m to write my music, what kind of life must I live? How can I be sure? How can I find out? The clouds parted. The birds resumed their song. I rose to my feet in appreciation.
No amount of wondering suffices to show me what my life will be. There’s only one way to find out. I hope, on my next visit, to play Medtner something wonderful and new. In the meanwhile, I know that my life is far better for his music, that it’s precious, and that it’s mine.
For a couple months now, I’ve been spoiling myself by spending the bulk of my time reducing my world’s entropy in three main ways:
1. Automate more of what I do
2. Organize more of what I know
3. Learn more of what I don’t know
I’ve begun exploring the boundaries of my attachment to Test-Driven Development. How did I get so attached in the first place? In large part, because the thinking process I’ve learned and applied by deliberately practicing TDD has, in the contexts I’ve experienced, almost always been an adaptive behavior. This is the first anecdote in a series about TDD’s value to humans.
Claiming I do my job better with TDD is equivalent to claiming I do my job worse without it — which, last I checked, was true.
Shortly after graduating with my music degree, I was hired as a software developer in a highly regulated, tightly controlled corporate environment. The tools I was hired to develop had begun life as a proof-of-concept system to manage certain kinds of identities in a heterogeneous distributed network. They put it in production. It was a roaring success. Suddenly lots of customers appeared, wanting lots more use cases, business rules, and platforms added. Suddenly the security engineers realized their proof of concept had a long life ahead of it. It would need to be managed and developed as a software product.
They may have been confident in my fitness for the job, but I sure wasn’t. While I’d been a longtime contributor to some significant open-source projects, I hadn’t been a professional software developer since 2002. And that had been my first programming job. I’d learned a whole lot very quickly, including the rudiments of TDD and other XP practices, and I was probably worth what I’d gotten paid, but I had not, by any means, mastered much of anything.
My first assignment
To their credit, my superiors had devised for me a relatively gentle introductory exercise: when an identity is no longer needed, provide a way to “disable” it. I did not need to devise a mutually authenticated client and server that spoke a network protocol in order to manipulate the server’s database of identities by issuing commands to be processed or rejected according to business rules. I didn’t even need to tweak the database schema; it already contained the needed fields. I just needed to add a “disable” command. Reading through the “create” and “update” commands, I copy-paste-edited a new “disable” command. Then I figured out how to run my copy of the server, how to point my client at it, and how to prove to myself that my copy of the database correctly marked a given identity as disabled. (When I demonstrated my progress, I found out there was considerably more to it than that. A great deal of much-needed handholding occurred.)
Then it dawned on me: I bet a bunch of the other commands need to behave differently when asked to operate on an identity that’s been disabled, now that such things exist. Taking my best guesses at those behaviors, I edited the SQL backing those other commands until they probably handled both the old and new cases. Since I didn’t understand how most of the commands were intended to be used, let alone how to verify whether they worked, we code-reviewed my very simple changes. We agreed they looked fine.
My first deployment
Guess what happened when we put my code into production? That’s right: my new “disable” command worked, but users complained that several other commands no longer did. It was easy to spot the mistakes, now that we knew where they were, and to make the obvious fixes. We scrambled to redeploy.
Before I tell you how the postmortem went, remember the context:
1. The environment was highly regulated and tightly controlled
2. The code was security-sensitive
3. I’d been uncertain of my ability to do the job before screwing up in production
“Bill,” I said, because that was Bill’s name, “if you need me to avoid screwing up like that again…”
“Uh huh,” replied Bill.
“…then I need to be doing Test-Driven Development.”
“Listen,” said Bill. “Do whatever you think you need.”
I did. And over the several years that followed, I never screwed up like that again.
Illustration by Rebekka Dohme.
You might suggest that that’s because I quickly learned the problem domain and our system’s model thereof, remembered how to write basic SQL, and made a habit of checking my assumptions and my results more carefully. You’d be right. You might suggest that I could have done all that without TDD. You’d be right again. But I’m fairly sure I did them sooner, better, and more reliably — all of which felt crucially important at the time — because of TDD.
As near as I can tell, TDD saved my job.
Given a software project whose outcome and schedule truly matter, all other things being equal, I’d choose…
- A capable team of TDD practitioners over another capable team
- To have the team practice TDD
Of course, there are always a great many other things, and they be rarely equal. Let’s scale it down a bit.
Given the responsibility to deliver working, valuable software to humans, I’d choose to drive my development with tests. I believe I fulfill my responsibility faster and better when I do. This turns out to be one of my strongest and most deeply held beliefs of any kind, right up there with the intrinsic value of human beings and an overarching respect for individual autonomy.
Yeah, that sounds a little odd to me too. Yet my professional experiences have repeatedly borne out the validity and importance of all three.
Sticking with TDD here, I’ve read and/or conjectured lots of plausible mechanisms by which it could be speeding me up, I’ve observed a bunch of them directly, and anecdotally I’m usually pleasantly surprised how smoothly things go, compared to how I imagine they might have gone instead.
Software development isn’t a controlled experiment. It’s possible I’ve been continually and subtly lying to myself under the sway of a whole bunch of cognitive biases and thus I’ve been egregiously misinterpreting my personal anecdata.
Since “I do my job better when I practice TDD” is merely a strong belief, not a result that I’ve proven, I owe it to myself and the people I work with to question it from time to time. Inspired by some recent professional experiences, now is one of those times. Where are the boundaries of my belief? If I’m trying to be effective and efficient, when wouldn’t I practice TDD?
What I mean by TDD
TDD, to me, means holding a collaborative conversation with my code. I suggest what I think I want it to do next, ask it to try, listen to what it tells me, adjust my thinking and wanting, and repeat. To listen better and adjust faster, write smaller tests (in units of code, if inside-out; in slices of functionality, if outside-in).
If that metaphor doesn’t quite work for you, try this one: TDD means pair programming with myself. Before typing anything, I communicate my intent aloud. Maybe my pair has a better idea, or a question to consider, or a drawing board we should be going back to. By working in a way that externalizes my thinking, I can be my own pair.
I wouldn’t want TDD if…
- It doesn’t need to work
- It needs to work, but it doesn’t matter when
- It’s so manifestly simple that it necessarily works
- It won’t ever need to change
- It’s so manifestly simple that it’ll be obvious how to change and necessarily still manifestly simple afterward
- It’s a throwaway experiment to learn something
- I’m being held at gunpoint by a crazed stakeholder who’ll shoot me if I write so much as a single test first (or try to have a conversation about it)
None of these cases is particularly interesting.
You might think I wouldn’t want TDD if…
- It’s legacy code
- It’s otherwise hard to write tests for
- Nobody else on the team bothers
- I’m learning a new programming language
- Unarmed stakeholders express opposition to it
You’d nearly always be wrong.
Learnings go here
I practice TDD for its many benefits. What if I could achieve those benefits another way?
As a useful oversimplification, the biggest external benefit of TDD might be that I get my work done faster and better, and the biggest internal benefit might be that I feel safer doing my work. (It seems plausible that there’s a causal relationship.) I’ve been working in a dynamic language famous for its runtime surprises, deriving a feeling of safety from my tests. Might there be a language where the compiler alone could give me that same feeling?
I can practice TDD because I’ve been targeting suitably unconstrained environments. Might there be a target whose constraints preclude TDD?
I don’t know for sure when I wouldn’t want TDD. But I’ve got some ideas, and watch out: I’m going to test them.
Illustration by Jason Crane.
My new Yak Shaving Expert t-shirt is fitting, and not only because it’s the right size. For instance, today I wanted to start backing up my email, so I found offlineimap to my taste, so I went to pkgsrc to install it and saw that we’re a couple releases behind, so I updated the package. Sometimes it turns out that in order to do what you want, it’s necessary that you first go shave a yak. Maybe I’ll get to try backing up my email tomorrow.
In software development, it’s pleasing when any desired change proves easy to make. When a change proves prohibitively difficult, we have to decide what to do: either to stop wanting it, or to want it anyway. In neither case will we be able to know the full cost of our decision. But our choice can and should be informed by our need for predictable cost of delivery in the future and our willingness to delay gratification in the present. It’s sort of like insurance. If over our product’s lifetime we need to minimize the frequency with which we have to stop everything and shave an entire yak, then we know the discipline required.
I’ve felt mildly and persistently overwhelmed since the great server crash of late 2008. First I had time to deal with the fallout, but no money; then money, but no time. Not having all my personal data organized and available has meant that for any given task, I’ve learned to assume I can’t just go and do it; instead, I’ll have to first figure out what I need, then go digging to find it all. Then, and only then, can I hope to complete the task. So the act of getting started has been harder than usual, the list of tasks has grown large, the cycle has been self-sustaining, and the effect of five years of this drip-drip-drip on my cognitive capacity has been noticeable, at least to me.
My personal yaks have finally come home to roost. (Yes, they’ve evolved the ability to roost.) I have money, time, grand ideas, and some clippers. Where to start? A few weeks ago I guessed I’d want to answer that only after refining my enormous backlog a bit. Now I think whichever item I feel like doing next is what I’ll do next. Why try to impose an ordering, when I know full well it’s yaks all the way down?
People who haven’t been programmers have no way of knowing the extent to which programming consists of making decisions. It is, let me tell you, a very large extent. Many managers expect a few large, visible decisions when we’re discussing “requirements” or “architecture” or “design”, but can’t imagine the sheer number of decisions made when we’re doing “implementation”. (These managers either haven’t encountered Agile practices or have chosen not to learn from their encounters.) Everything strange and wonderful about the art of software development, the sometimes counterintuitive stuff we do in order to do it well, stems from three premises:
- Software is made by humans for humans.
- A person developing software makes hundreds, maybe thousands of choices each day.
- For any given choice, we won’t always know its future price, but we’ll always have to pay it.
In 2009, a Fortune 100 company hired me as a security engineer to develop a custom infrastructure product. I hadn’t programmed professionally for longer than the four years I’d just spent earning a degree — not in computer science, but in music. Did I have what it took to do the job? I remember how unsure I felt. But a friend who vouched for me arranged an interview, it went well, and they made me an offer I was pleased to not refuse. A year later, I was managing the product and its team.
With a nod to Kent Beck, I’m not a great programmer, I’m just a good programmer with great judgment. My superpower is decision-making.
A developer’s choices are most often reified in code. A manager’s choices are most often reified in the product and in the team that produces it. For several years, as product manager and team lead, I did work that took full advantage of the quality of my decisions. It was an extraordinarily difficult environment in which to do the job well, and I made damn sure to do it well. But when I learned last year that my work was not widely or consistently appreciated, the struggle suddenly felt much less rewarding. It turns out that people who haven’t been software development managers have no way of knowing the extent to which managing software development teams consists of making decisions. It’s the entire first paragraph all over again, a level higher on the org chart.
I willingly accepted a reduced role with the promise that the arrangement was temporary. I knew I wouldn’t be making the big decisions anymore; this was a welcome relief from apparently thankless work. I didn’t know my informed professional opinions would not be considered when making the decisions, or that the decisions would be made poorly, or that I’d still be around long enough for that to hurt me. I certainly didn’t know, when I’d found a role elsewhere in which I’d get to deliver far more value far more visibly — a role in which I’d spend some percentage of my time coaching development teams — that my temporary arrangement, which had become untenable, wasn’t so temporary after all.
We don’t always know the price, but we always have to pay it. I was paying.
If making decisions is my superpower, then I had a decision to make. It stemmed from three premises:
- I was no longer being used to make decisions.
- I was no longer being used to support the people who were making decisions.
- I was no longer able to switch to a role where I’d make decisions or help others make them.
After checking my premises, I handed in a brief letter with my conclusion. A few weeks later, it was my last day. Yesterday.
Today’s my fourth anniversary with Bekki. As of a few months ago, we finally live in the same place. It’s a nice place. I walk to the grocery, the butcher, the pharmacy, even the dentist, and a handful of good coffeeshops. Our house is full of comforts, two of which are covered in fur of their own devising. I’m writing these words sitting in the yard because spring arrived yesterday and we have a yard.
So I’ve left my job. What next? If I could, I’d want to slow down for a while and just enjoy what I have. And I can. It’s the easiest decision in the world.