No One Hands You the Right Answer

Each project at Distyl can feel like its own little startup. You have to think a bit like its CTO.

I’ve made hard technical decisions across the whole system, from how our systems wire into a customer’s stack to how an agent should work. Often, the problem itself is still a half-formed creature, stubbornly refusing to explain itself. Even in rooms with people who have decades more experience, no one has the answer for me.

This means I’ve had to become a deeper systems thinker. With each new project, I often have to throw out my assumptions and trace the wiring again. And that instinct to start from first principles makes me less willing to accept the problem as given.

Debugging the Rubric

I saw this firsthand when my team went live with an AI system designed to apply an existing rubric to student work and produce a grade for human review. When we examined the existing grading examples, we found that human graders had interpreted the rubric inconsistently. The problem started with the rubric itself: the words on the page did not draw a clear-enough line between one level of performance and the next. The AI system could apply the rubric consistently, but changing the architecture or throwing more tokens at the problem couldn’t brute-force clarity into the rubric.

Using what the system revealed, we reframed the problem with the customer and designed a new grading process. We proposed and tested an LLM first pass that could surface specific ambiguities in the rubric as targeted questions, which the human graders could then work through together in calibration sessions before the grading stage. We then adapted the system to work from the graders’ shared reading instead of trying to make sense of the rubric alone. In making their judgment part of the grading process itself, we saw accuracy improve.

Your first solution is only a hypothesis. In forward-deployed engineering, owning the outcome means knowing when to challenge the original technical scope and building with the customer to find a new path.

The Kind of Engineering I Wanted

I’ve always liked asking questions and understanding how other people see the problem. Before Distyl, some people took that as a sign that I should go into product. But I knew I wanted to build. I love writing code and solving hard technical problems, and I also want the work to be interactive. Here, those parts of the job are inseparable.

I joined Distyl when we were around 20 people. Back then, our projects had little infrastructure to build on and no established playbook to follow. Distyl is a much larger company now, yet the work still happens in small teams tackling large, open problems. I now lead one of those teams. I’m still writing code and shipping alongside them, and much of my job is making sure other engineers can move quickly and own meaningful parts of the work. Working toward a team that can operate with that much ownership has expanded how I think about building.

The technical ground has shifted in that time, too. When I joined, we leaned more on prompt engineering and deterministic workflows. As models and tooling across the field have expanded what systems can do, we’ve chosen to build more autonomous loops. The hard part is deciding which ideas are worth betting on while they’re still emerging. An approach that made sense six months ago may no longer be the right one.

As Distyl grows, I hope we preserve the willingness to cannibalize our own technology: to keep what still works, let go of what doesn’t, and always make the better technical bet. That’s the builder I want to be: willing to chase an idea while it’s still a little strange, and just as willing to outgrow it.