The interview should measure reading

The debate inside most hiring loops right now is whether to let candidates use an assistant during the technical exercise. Both camps have a case. Ban it and you are testing something the job no longer contains. Allow it and you are watching someone prompt for forty minutes and learning almost nothing.

The debate has the wrong shape, because both answers keep the build exercise. My position: drop it. Replace it with a reading exercise, where the candidate is given a change they did not write and has to decide whether it should ship, and say why. That is the job now, it is far more predictive than the thing it replaces, and it is easier to run consistently than most loops assume.

The exercise, concretely

Vague advice about “testing judgement” is how loops stay broken, so here is the actual shape.

Materials, sent in advanceA small real repository, or a convincing extract of one. Twenty minutes of reading time before the session, no exercise attached.

The artefactOne change of 150 to 400 lines against that repository, plus the ticket or specification it claims to satisfy.

The questionWould you merge this? If not, what specifically would you need first. Forty minutes, thinking out loud, interviewer answering factual questions about the system.

The turnTen minutes in, hand them one new fact about production. Watch whether the assessment updates.

The design of the artefact is where the effort goes and it is a one-off cost. Build it from a real change out of your own history, ideally one that caused an incident, with the fix removed. You already know what the right answer looked like, which solves the calibration problem that ruins most novel exercises.

The most informative single ingredient is an omission rather than an error. A missing rollback path, an error branch that was never written, a migration that only handles the forward direction. Context blindness is the characteristic failure of generated code: output that is internally coherent and incompatible with the repository around it, referencing APIs that do not exist locally or ignoring conventions everyone else follows. Candidates who read for absence find these. Candidates who read line by line for syntax do not, and the difference is stark enough to be a scoring line on its own.

What to score

Write the rubric before you run the first session, because the failure mode here is an interviewer enjoying a conversation and reporting a vibe.

Did they establish what would have to be trueBefore reading closely, did they name the conditions under which this change would be wrong? Strong candidates do this unprompted within two minutes.

Did they read for what is missingAbsent error handling, absent rollback, absent test for the case that actually breaks. This is the highest-variance signal in the exercise.

Did they check the change against the specificationNot "is this good code" but "does this do what the ticket asked, and does the ticket ask for the right thing".

Did they update on the new factCheap confidence is the thing to catch. Watch whether the mid-session fact changes the assessment or gets absorbed without effect.

Was the decision defensibleMerge and block are both acceptable answers. Only the reasoning is scored. A candidate who blocks for a reason that does not survive one question scores worse than one who merges with named residual risk.

Why allowing an assistant is a half measure

If you keep the build exercise and allow assistance, you have changed the tool and not the measurement. You are still measuring production, just production with a faster instrument, and you have made the score noisier because now it partly reflects which model the candidate happens to be fluent in.

The reading exercise dissolves the question. Let them use whatever they like. Give them a machine, a browser, an assistant, all of it. A candidate can ask a model to summarise the diff and it will not save them, because the question is not what the change does. It is whether it should exist in this system in this form, given constraints that live in the repository and in the ticket rather than in the model’s context. That is precisely the judgement you are hiring for, and it is the judgement that assistance does not currently supply.

There is a secondary benefit worth naming. Stack Overflow’s 2025 developer survey found trust in AI output falling from around 40 percent to 29 percent, and the candidates who have internalised that are exactly the ones who perform well here. You are selecting for calibrated scepticism, which no build exercise has ever managed to measure.

What this selects against

An honest hiring process names its own biases, so here are the two that matter.

It favours people who have worked in large existing codebases. Reading unfamiliar systems is a practised skill and practice comes from exposure. That disadvantages candidates from small teams and greenfield backgrounds, and it disadvantages juniors most.

Compensate deliberately rather than pretending the effect is not there. For early-career candidates, use a smaller artefact, allow more reading time, and score the process rather than the conclusion: did they build a mental model out loud, did they ask the right factual questions, did they notice they were missing something. Entry-level postings are already down around 40 percent, and a loop that quietly filters juniors on tenure is not something to add to that without deciding to.

It rewards articulacy. Thinking out loud in a second language, under evaluation, is genuinely harder for some strong engineers. The mitigation is to accept written reasoning as an alternative and to score notes as generously as speech.

Where this breaks down

Reading is not everything the job contains. Somebody still has to build the hard thing when generation fails, and those moments are usually the ones with the highest stakes. A loop that measures reading only will eventually hire a team that can evaluate anything and originate nothing. Keep one construction signal somewhere, even a small one.

Interviewer calibration is harder than in a build exercise. A candidate either passed the tests or did not. Judgement is scored by a human with opinions, and two interviewers can rate the same session very differently. This is the strongest objection to my position. The partial answer is a written rubric, two independent scorers on the same session before the loop goes live, and discarding the exercise if they disagree badly.

The artefact leaks. A single well-crafted change gets described online quickly, and the moment candidates arrive knowing where the defect is, you are measuring preparation. Budget for rotating three or four artefacts a year, which is real ongoing cost that a generic algorithm question does not carry.

Some roles genuinely still hire for production. An early-stage team that needs someone to originate a system from nothing is right to test that directly. My argument scales with the size of the existing codebase and the cost of a bad change reaching production.

And this signal may decay too. If tooling gets materially better at cross-repository reasoning, part of what this exercise measures becomes cheap in the same way construction did. I do not think that is close, and I would not bet a hiring process on more than three years without revisiting it.

The takeaway

Whether to allow an assistant is the wrong argument, because both answers preserve an exercise calibrated to a skill that is no longer scarce. Reading a change you did not write, against a specification, and deciding whether it should exist is what engineers now spend their day doing.

Build one artefact from your own incident history, write the rubric first, and let candidates use any tool they like.

If you take one thing into next week: take a real change from your last incident, strip the fix, and give it to two engineers you already rate. If they disagree wildly on whether to merge it, your loop was never measuring this.