This might turn into a collection. Let’s see.
When
Joining a team.
Why
To earn trust and reciprocity.
What
Demonstrate good will and good skill.
How
Observe what’s regularly tedious, bothersome, or frustrating. Do not choose something you think they should want; this technique is to show that you’re listening, that their needs matter to you, and that you’re willing and able to help those needs get met. (Examples of what you might see or hear: make unreliable tests reliable, speed up slow tests, automate a build step.) Then go off and fix it — or if it’s too big, a tractable part of it — as promptly as possible.
By the usual criteria, this is a great gift. They want it, they’d never get it for themselves, you made it for them, and it marks a special occasion: the beginning of a fruitful collaboration.
More
- Esther Derby, Entering Groups
- GeePaw Hill, The First Coaching Days
To everyone who helped me with the search, thank you. Today marks the start of my first regular job in seven years.
In my first job as a software developer, by a string of lucky occurrences, I happened to land someplace doing Extreme Programming. I hadn’t been looking for it; as with all the other ways people make software together, I hadn’t heard of XP and had no basis for comparison. But I remember a feeling of excitement rather larger than that of getting paid to be a programmer (which, to be clear, was not inconsiderable). And I remember that in my early Google searches for the experience reports of other people doing XP, the weblog that most stood out was James Shore’s. The way he was doing it and writing about it helped me learn to do it myself.
In my first job as a software development coach, I began to suspect that I’d need to interpret the Manifesto’s “doing it and helping others do it” to mean something more specific to me: alternating between helping and doing. I worried that after too long being free of responsibility for delivery outcomes I’d stop believing my own advice. I guessed that to refresh my contextual understanding of what helps which people deliver which results under which conditions, I’d periodically need to return to working as a teammate in a team. I supposed that after another round of being accountable to teammates and customers I’d be freshly motivated to share what I’d learned. If this pendulum could keep swinging, I could perhaps sustain a consulting career.
That prediction held true. When my employment with Pillar (an XP consultancy) ended in March 2017 I was about to become a parent, with the expectation that my life was about to change in ways I could not possibly predict. That held true, too. Being an independent mostly-remote consultant proved particularly adaptive for our family life when the pandemic struck. I prioritized client opportunities according to two criteria: their compatibility with our needs at home and their impetus for my career pendulum.
I enjoyed a fantastic mix of clients — the problems they were solving, the aspects they wanted help with, the cultures they brought with them — but in the course of interviewing for this job I realized it’s been more than seven years since I personally lived the day-to-day accumulated complexity of seeing through slow changes to the cultural norms in and around a software team. I suddenly felt a hole in my heart where that feedback wasn’t. As a consultant, every now and then I’d hear (years later) of some case where I helped make a dent. But mostly I’d never know one way or the other. It’s uncomfortable to make an effort with no expectation of finding out what good it did or didn’t do. It’s a very particular kind of uncomfortable, for a devotee of Extreme Programming, to make a habit of it.
So I’m keen to take a break from “just visiting” and move in someplace. I’m eager to confound the pendulum by inhabiting a role that combines responsibility for outcomes with responsibility for techniques. I’m pleased to share that today’s my first day as a Staff Engineer XP Coach at OpenSesame, reporting to none other than Jim Shore. And I’m looking forward — as I’m doing it and helping others do it — to sharing, as always, what I learn.
A few weeks ago, Apple released new versions of Xcode and Command Line Tools. Not thinking too hard about how my pkgsrc developer environment often gets broken by OS or tool updates, I updated promptly. For one thing, I’m kinda used to it. For another, it doesn’t usually break. For a third thing, managing dependencies — anything not my code that can break my code — means responsibility for dealing with the inevitable trouble, and therefore the sooner I find it the better. (More on my approach to life with dependencies.)
A vendor-provided toolchain is a significant dependency.
So I accepted the Command Line Tools update, and it commandeered my spare time for two weeks as I hurried carefully to repair
one of the world’s biggest continuous-integration cauldrons
on
one of its most popular platforms.
When I ran my usual pkg_rolling-replace -suv to rebuild anything outdated, it did not go well at all.
This article uses “we” because the continued smooth operation of pkgsrc on macOS reflects the contributions of many developers on many occasions, including this one: I happened to be first on the scene, but several of us of were discussing the problems and potential workarounds and all of “my” commits were adjusted accordingly.
Did I mention that a few weeks ago we were aiming to stabilize for yesterday’s quarterly release? Suddenly, if we didn’t scramble to straighten things out for macOS users, we’d have to manage a complicated situation for a while. But if we created a mess on other platforms by moving rashly, that’d be even worse.
The usual feedback mechanism for proposed infrastructure changes is to compare full bulk builds before and after. There was no time for that.
Happily, the conclusion of the story is boring: as always, the pkgsrc 2024Q1 stable branch supports macOS and its developer tools, including the latest releases of each. (So does -current pkgsrc, of course, if that’s your thing.)
Curious what we had to do to keep it boring? Read on.
Stricter clang defaults
Upstream Clang 16 and GCC 14 have promoted several warnings to errors by default, and Apple’s Clang 15 has followed suit. (Gentoo has very helpfully documented this for packagers.) These changes are intentional and well-intentioned, pushing maintainers to ship more reliable code. But pkgsrc’s job is to build nearly 30,000 codebases we don’t maintain. And stricter compiler defaults break a lot of builds.
As you might hope, we can make the breakage go away in one place.
In pkgsrc, packages declare which programming languages are required for their build. The compiler framework then selects package-and-platform-appropriate compilers, places them preferentially in the package’s build environment, and — crucially — intercepts compiler invocations and rewrites them for a variety of purposes.
When we look into pkgsrc’s clang logic, we find prior art for this specific class of problem.
In September 2020, Xcode 12 (and its associated Command Line Tools) arrived even later in our quarterly schedule and promoted -Wimplicit-function-declaration to an error.
The surgical fix: on macOS only, if invoking clang reveals the new stricter default, we
pass -Wno-error=implicit-function-declaration
to demote the error back to a warning.
Apple Clang 15’s new strictures aren’t observable in the same way, so we adjust our workaround:
if clang doesn’t complain when we try demoting the new errors back to warnings, we
pass those arguments too,
via the same compiler-framework mechanism.
Missing m4 and yacc
This messy regression found only in the Command Line Tools 15.3.0.0.1.1708646388 update — not in the corresponding full Xcode 15.3 (build 15E204a) update — must have been unintended.
On macOS, some of the familiar Unix tools in /usr/bin are in fact stubs.
When invoked, they either execute into the corresponding installed program (living somewhere under /Library/Developer) or prompt the user to install the Command Line Tools.
This Command Line Tools update uninstalls m4 and yacc from /Library/Developer.
But since the OS-provided /usr/bin/m4 and /usr/bin/yacc stubs still exist, running m4 or yacc still does something:
it pops up a window prompting you to reinstall the CLT.
Unfortunately, as the latest available version doesn’t provide those tools, reinstalling is a waste of time.
As you might once again hope, we can hide the problem without personally visiting 29,000+ packages.
In pkgsrc, we also have a framework to control which non-compiler tools are invoked during builds. Packages declare which tools are required for their build. The tools framework then selects package-and-platform-appropriate incarnations of the declared tools and places them preferentially in the package’s build environment.
We just got handed a few new twists to handle in the framework, is all.
First, because this clever new CLT failure mode outfoxes our usual tool-detection mechanism, we special-case m4 and yacc detection on macOS, performing an
existence check for the stubs’ targets.
Then the selection mechanism’s usual fallback logic can provide them some other way.
This prevents the primary source of needless CLT install popups.
For non-macOS platforms, no change.
Second, because some packages might not yet be declaring all their tool dependencies, we special-case m4 and yacc handling on macOS:
when they’re not declared, we
place them in the build environment anyway,
as no-ops.
If the package build happens to invoke them, nothing happens.
This prevents the secondary source of needless CLT install popups, at the risk of breaking macOS builds for packages that are missing these tool declarations and have heretofore gotten lucky;
in such cases, the breakage will be obvious and the fix easy.
For non-macOS platforms, no change.
(At leisure, we might like to broaden this approach to help find and fix all undeclared tools on all platforms.)
Third, because the flex tool expects to invoke a GNU-compatible m4, we adjust the tools framework to
infer gm4 from a flex declaration
so that the framework controls which m4 gets found.
This more correctly expresses our intent on all platforms, and in the macOS package build environment it restores /usr/bin/flex to a working state.
Broader xcrun search
We were already relying on xcrun for a couple of things, so when our new tool-detection special cases were sometimes getting surprising results from it, that was concerning.
Turns out xcrun no longer looks solely in Apple-controlled locations, but also consults the environment’s $PATH.
By
invoking xcrun with an empty PATH and --no-cache,
we obtain controlled, predictable tool detection.
Conclusion
Under the constraints, we changed as little as possible, as safely as possible, as similarly as possible to previous proven changes, avoiding novel constructs or any whiff of unforeseen consequences. We could not have done nearly as safe or thorough a job without good abstractions already in place. Total lines of pkgsrc infrastructure code changed: less than 100. Now that 2024Q1 is out, we have room to refactor.
These 15.3 updates also include a brand new linker. So far it hasn’t given us any trouble. If that changes, wanna guess whether we have one place to take care of it?
It’s been a few weeks since my last Weekly Piano Miniature and I don’t feel bad about it. That’s a strong signal.
Why don’t I feel bad? Because Weekly Piano Miniatures lasted an entire year (and Daily Piano Miniatures for five months before that — all told, nearly 500 minutes of music). Because my 100-year-old piano finally sounds rough enough that it’s been hard to want to play. Because my energy is needed for family, health, and finding work. Because I’m not worried I’ll stop making music. Because if the last few pieces will have to keep reverberating for a while, I’m pleased that they’re three consecutive Medtners.
I hadn’t intended to pause the music project, but it’s intentional now. Recording will resume once I’m employed and have a more enjoyable instrument. Let’s all look forward to that.
Update: Found a fit!
I’m looking for my next full-time role in engineering leadership (such as Director, VP, CTO) and/or senior engineering (such as Principal or Staff).
I love people, want them to succeed, and have fairly reliably made myself helpful.
You may already know me from Agile in 3 Minutes, my Coding Tour, my PubMob, meetups I’ve run, talks I’ve given, approximately a zillion podcasts I’ve guested on, or most recently my Daily and Weekly Piano Miniatures. I’m happy to say my track record of being an effective influencer extends beyond social media.
Some core competencies
- Coaching (technical and otherwise, all levels of the org)
- Managing (risks, products, and people)
- Contributing (tests, code, relentless improvements to developer UX, you name it)
- Inviting and supporting meaningful conversations
- Making and supporting effective decisions
- Orienting myself — and helping others orient themselves — in new problem spaces
- Seeking out and attending to unmet needs
- Rooting practical daily choices in values and principles, and vice versa
- Making workplaces more humane
- Supporting people as the whole human beings they are
- Giving, always and constructively, a shit
People saying nice things about my…
A sampling of people saying nice things on the LinkedIn copy of this very post
A wonderful leader and leading practitioner. Here is a great opportunity for a software engineering organization that wants happy teams and happy customers.
— Lisa Crispin
A thoughtful and competent person. I highly recommend bringing him in to help your organization achieve better success.
— George Dinwiddie
One of the most talented people I’ve met in my Agile journey. He’d be great for your company.
— Colleen Esposito
A great technical leader. Get him and save yourselves years!
— Ewan O’Leary
I cannot say enough great things about Amitai. You should definitely hire him.
— Kristen Belcher
Excellent addition to any team. And an all around great human too.
— Doc Norton
Opportunity!! Amitai is an awesome engineering leader!
— Ruth Malan
A pleasure to work with: walks the talk and doesn’t mind getting his hands dirty. I wish we could hire him.
— Jeeva Nadarajah
An absolute wizard at helping software delivery folks get the most out of themselves and their organizations. I’ve seen first-hand his engineering expertise, and I’ve seen first-hand how great he is at helping others learn and improve. He’s one of the few engineering leaders who’s expert at true collaboration across the business-engineering chasm.
— Jim Holmes
I’m gonna be so jealous of whoever gets to hire Amitai, that’s all I’ll say.
— Dragan Stepanović
If you’re a hiring manager involved in software development, there are two questions to focus on:
- Why haven’t we hired Amitai yet?
- When is the soonest moment on my calendar I can talk with Amitai?
— Zach Bonaker
Practicalities
- For employers in the USA, I am a permanent resident and do not require a work permit
- For locations across the USA and Europe, my US/Eastern timezone ensures some synchronous working hours every day
- My résumé