The debrief was going sideways.
Five interviewers had spent four hours that day with the same Senior Engineer candidate. Now we were sitting around a meeting room arguing about whether to make the offer, and nobody had a single useful piece of paper between them.
Two said yes. They liked her. She was sharp, the conversation flowed, she “got it.” Two said no, with a vague unease about the depth of her answers. The fifth interviewer was quietly trying to remember which moments had been with this candidate and which had been with someone he had interviewed two days earlier.
I asked what each of them had seen, specifically, that pointed at the level we were hiring for. The room went quiet in the particular way rooms go quiet when everyone realises they have been working from impressions and hoping the impressions would line up.
I have run that meeting more times than I would like. So have you, probably.
This post is about why that meeting keeps happening, even at companies that have done the work of writing a serious career framework. It is for hiring managers, senior interviewers, and anyone who has ever sat in a debrief and realised that “she seemed strong” was the strongest argument anyone in the room could muster.
There are two layers to the problem. Most posts about interviewing fix one and ignore the other. You need both.
The wrong layer of detail
By the time a company reaches any meaningful size, it has a career framework. There are rows for each level: Junior, Mid, Senior, Staff, Principal. There are columns: Technical Skills, Scope, Communication, Impact, Ownership. Inside each cell there is a sentence.
“Works independently on complex tasks.”
“Drives improvement in team processes and tools.”
“Influences technical direction across multiple teams.”
These sentences are correct. They are also useless to the person trying to design an interview around them.
The framework is written for the wrong audience. It is written for promotion committees, performance review calibration, and HR systems. It describes outcomes you can measure if you have already worked with someone for two years. It does not describe behaviours you can observe in 45 minutes with someone you have never met.
A framework tells you what a Senior Engineer should be capable of. It is silent on how to find out whether the person sitting across from you actually is one.
This is the first half of the problem. Even good interviewers, given a vague level descriptor, will drift towards their own internal model of seniority. Their model is mostly an averaged memory of the last three Senior Engineers they liked. Two of them ask about system design because that is what the last good Senior was good at. Another asks behavioural questions because that is where the last bad Senior failed. None of them are testing the framework. They are testing their personal heuristic.
The result, predictably, is that whether someone gets hired at level depends mostly on which of your interviewers happen to be on the schedule that day.
From statement to behaviour to question
The fix for this half of the problem is a translation step that almost no company does explicitly.
I picked up a version of this from a recruiter I worked with years ago. I have since seen it reposted under various names in slide decks, but the method is the same and it is dead simple.
Three steps:
- Pick one statement from the framework at the level you are hiring for.
- Ask: what does that look like as observable behaviour, in daily work, for this specific role?
- Write a question that surfaces that behaviour.
Sounds obvious. Almost nobody actually does it.
Take the Senior Engineer line: “Drives improvement in team processes and tools.”
Step two forces you to get concrete. What does “driving improvement” actually look like in the wild for someone on a backend platform team? It might look like this. Someone notices a deployment is taking longer than it should. They do not file a ticket and forget about it. They dig into why. They write up a one-page proposal. They get the team to try it in a sprint. They measure the difference. They share what they learned.
That is observable. You can ask about it.
Step three is then almost mechanical. “Tell me about a piece of work you did that changed how your team or another team operates. How did that come about?”
Notice what the question does. It is open enough that the candidate has room to tell their actual story. It is specific enough that they cannot just answer with a hypothetical or a slogan. And it directly probes the behaviour the framework cell is trying to capture.
Compare it with the question most interviewers actually ask, which is some variant of: “Tell me about a time you showed leadership.” Sure. The candidate will produce a story. The story might be about leadership. It might be about being on a team where someone else showed leadership. It might be a project they slightly oversold. You will not know which, because the question never asked the framework question in the first place.
You can do this exercise for every cell in your framework that matters for the role. It is tedious work. It pays for itself the first time you have a calibrated debrief.
Impressions are not evidence
Solving the first half only gets you so far.
Even if every interviewer walked into the room with the right questions tied to the right framework cells, most debriefs would still be a mess. The questions are only half of it. The other half is what the interviewer brings out of the room.
I think this is where most interview discourse gets stuck. We argue about question types, about whether take-homes are fair, about whether system design is a good filter. (I have written my own version of that argument in Why I Dumped LeetCode Interviews , so I am not innocent.) We almost never argue about how interviewers record what they hear. And that is where the wheels come off.
Here is the test. Pick up your last three interview write-ups. How much of what is on the page is something the candidate actually did or said? How much of it is your interpretation of what they did or said?
In my experience, even thoughtful interviewers come back with notes that are 80% interpretation and 20% direct observation. Sentences like:
- “She seemed nervous in the system design round.”
- “He really knew his stuff on Postgres.”
- “They had a strong opinion on testing.”
None of these are evidence. They are summaries of impressions. A different interviewer in the same room might write the opposite sentence and be just as confident in it.
I came across a post by Dan North a while back called Interviewing for Evidence . It is the cleanest articulation I have seen of how to fix this side of the problem, and I have been quietly stealing from it ever since.
The core idea is exactly what it sounds like. Stop writing impressions. Start writing evidence. Things the candidate said. Things they did. Decisions they made out loud. Numbers they cited. Trade-offs they named. Questions they asked you.
“She seemed nervous” is not evidence.
“She paused for almost twenty seconds before answering, then said she was not sure how she would approach it” is evidence. The first is your interpretation. The second is something a video camera would have captured. The video camera version is what you want in your notes.
Once you start writing notes that way, two things happen. The first is that your debriefs get faster, because everyone is arguing from the same kind of material. The second, more important thing, is that you find out which of your interviewers were not actually paying attention. The ones who can only produce vague summaries usually were not capturing anything in real time. They were being polite for forty-five minutes and then writing down their general vibe afterwards.
That is a lot of decisions being made on vibes.
Four kinds of evidence
Dan’s post unpacks this further. There are four kinds of evidence you can collect in an interview, and each tells you something different.
In rough order of weight:
Experiential evidence. Things the candidate has actually done. “Tell me about a time when…” “Can you give me an example of…” “Then what happened?” This is the most valuable kind. It is also the easiest to push on, because real stories have specific details and contradictions and ambiguities. Polished stories that survive multiple follow-up questions are usually real.
Hypothetical evidence. What the candidate thinks they would do. “What would you do if…” “If you had to do that project again, what would you change?” You cannot prove a hypothetical, but you can learn how someone reasons. How do they decompose the problem? What do they ask for? What trade-offs do they name without prompting? Hypotheticals are especially useful for more junior candidates who do not yet have the experience for a strong experiential answer.
Opinion evidence. What the candidate thinks about things. “What is your view on testing?” “What do you think makes a good code review?” Opinion evidence is interesting in two layers. The first is the opinion itself, which tells you about taste and exposure. The second, often more telling, is what happens when you push back on it. Do they hold their ground with new arguments? Do they collapse and agree with whatever you said? Do they get defensive? I have learned more about candidates from the way they handled disagreement than from the opinion itself.
Credential evidence. What they are formally qualified to do. Degrees, certifications, languages, tooling. Useful, sometimes load-bearing in regulated work, but the lightest of the four for most engineering hiring.
Most of the worst interviews I have run or watched were heavy on opinion and credential, light on experiential, and almost never explicitly hypothetical. That ratio is backwards. Opinion and credential are the easiest to fake. Experiential is the hardest, especially when you keep asking “and then what did you do?” until the story either holds together or does not.
A small thing that helps me: I keep a sheet of paper divided into four quadrants, labelled with the four evidence types. As I take notes during the interview, I drop each note into the relevant quadrant. Halfway through, I can glance down and see where I am thin. If I have nothing in the experiential quadrant, my next question should not be “and what is your view on microservices?” My next question should be “give me an example of a service migration you led.”
The calibrated interview
Now stitch the two halves together, because that is where the actual fix lives.
A calibrated interview is one where each question is doing two jobs at once. It is mapped to a specific cell in the framework. And it is structured to produce one of the four kinds of evidence.
If your framework says, for Senior level, “Influences technical direction across multiple teams”, you can build a small set of questions for that cell:
- Experiential: “Tell me about a technical decision you made or shaped that affected at least one team outside your own. Walk me through how it actually played out.”
- Hypothetical: “Suppose you joined our platform team and noticed two product teams had quietly built different solutions to the same problem. What would you do?”
- Opinion: “How do you think technical direction should be set in an org with multiple engineering teams? What is the role of an individual senior engineer in that?”
Three different angles on the same framework line. Each surfacing a different kind of evidence. Each grounded in something observable.
Now compare what your debrief looks like with this setup versus without it.
Without it: “I think she was a strong Senior. She seemed to get it.”
With it: “On the framework line for cross-team influence, here is what I have. Experiential: she described a database migration involving two teams, named the political tension explicitly, told me how she handled the principal engineer who initially blocked it, and walked through what she would do differently now. Hypothetical: when I gave her the parallel-implementation scenario, she immediately asked who owned the platform charter and whether there was an existing forum for technical alignment. Opinion: she pushed back when I argued that staff engineers should make these calls unilaterally, and articulated a view about distributed decision-making with named trade-offs. I am ready to recommend hire at level on this dimension.”
Those two paragraphs lead to different organisational outcomes over a year of hiring. The first one hires whoever the loudest interviewer in the room liked. The second one hires people whose evidence matches the level you said you wanted.
Worked example: a Senior Engineer loop
Let me make this less abstract.
Suppose you are hiring a Senior Software Engineer for a backend team. Your framework, summarised, says they should be able to:
- Own a non-trivial system end to end.
- Drive improvement in team processes and tools.
- Mentor and grow more junior engineers.
- Influence technical direction within the team and occasionally beyond.
- Communicate clearly with non-engineering stakeholders.
You have a four-stage loop: hiring manager screen, technical deep-dive, system design, and a behavioural round.
Without translation, each interviewer picks up the framework, reads it, and goes “I will ask about ownership somehow.” The behavioural round becomes a wandering conversation. The technical deep-dive turns into trivia. The system design defaults to whatever pet question the interviewer has been asking for years.
With translation, the loop looks different. Each stage owns specific framework cells, and each question in that stage is mapped to an evidence type.
The hiring manager screen takes Ownership and Communication. The questions are mostly experiential and opinion. “Tell me about the last system you genuinely owned, end to end. Who else was involved? What did ‘owning’ actually mean day to day?” “How do you usually communicate technical risk to a product manager who is pushing for a deadline?”
The technical deep-dive takes Drive improvement and Mentoring. Hypothetical and experiential, mostly. “Walk me through a time you noticed something the team should change and made it happen.” “If you joined our team and saw a junior engineer repeatedly making the same mistake in their pull requests, what would you do?”
The system design takes Influence technical direction. Mostly hypothetical, with experiential pull-ins. “Design a service that does X. As you go, tell me which decisions you would actually escalate to your team or other teams, and which you would just make yourself.”
The behavioural round takes Mentoring, Communication, and Drive improvement, with deliberate cross-checks against earlier rounds. “Earlier you mentioned a refactor you led. Tell me how you brought your team along on it.” Cross-check signals from the technical deep-dive against the behavioural framing. Look for inconsistencies between hypothetical and experiential answers. Most candidates are not lying when they are inconsistent. They are revealing that the experience they described was actually shaped more by someone else than they let on.
The debrief at the end has structure. Every cell has been touched by at least two interviewers. Every interviewer has notes that mostly look like quotes and decisions, not summaries. Disagreement still happens. But disagreement happens over interpretation of evidence, not over whose vibes win.
Where this fails most often
A few things will break this if you let them.
The framework itself is terrible. If your framework is genuinely vague at every level, no amount of translation will save you. Some companies copy their levels from a cached version of an older framework, never tuned to their actual product or tech context. If you cannot pick a single sentence and turn it into a real behaviour for your engineers, the problem is the framework, not the interview. Fix that first. (I have written about a related version of this in There Is No Standard EM Role , and the same logic applies to engineers: if you have not defined the role for your context, you cannot interview for it.)
One interviewer trying to cover everything. I have seen loops where a single 60-minute round was supposed to test ownership, communication, mentoring, technical depth, and cultural fit. That is not an interview. That is an opportunity to collect impressions across five things and be confident in none of them. Pick two or three framework cells per round. Cover the rest somewhere else in the loop.
Treating “passes the bar” as the unit. The most common failure I see in evidence-based interviewing is the moment an interviewer realises they have evidence on three of the four cells they were supposed to cover and silently rounds up. The right move is the one that feels uncomfortable. Say “I do not have evidence on cell X” out loud in the debrief. Either someone else does, or you go back. Hiring at level when you have evidence on most cells but not the one that matters is how you end up with engineers who carry the title but not the responsibilities.
Senior interviewers who refuse to write evidence. The ones who have been doing this for a decade sometimes treat the move from impressions to evidence as bureaucratic. “I just know when someone is a Senior.” Maybe they do. The problem is nobody else can audit that. And if their model is wrong, you will not find out for two years, by which point a lot of damage has compounded. If you are running interviews, you owe the rest of your hiring system written evidence. Your gut is welcome to weigh in. It does not get to vote alone.
Confusing the candidate’s smoothness with their level. A candidate who interviews well is not the same thing as a candidate who works well at the level you are hiring for. Smoothness is one signal. Evidence is the test. I have hired smooth candidates who did not have the experience. I have, more than once, almost passed on excellent engineers because they were nervous and inarticulate in a stressful conversation. If your loop only rewards smoothness, you will keep getting candidates who interview better than they perform. (I wrote a separate piece on the candidate-side mirror of this in The Top 11 Technical Interview Myths , if any of you are on that side of the table right now.)
The standard
If I had to compress this into one rule, it would be: a hiring decision is only as good as the evidence you can put behind it.
Frameworks are necessary. They tell you what level you are aiming at and what that level should look like once someone is in the role. They are not sufficient. They do not tell you how to find out whether this candidate, in this conversation, is at that level.
You bridge the gap with two disciplines. You translate framework statements into observable behaviours, and then into questions. You collect evidence, by which I mean things you actually heard or saw, not generalised impressions. Then you debrief from the evidence and the framework, not from the vibe.
This sounds like more work, and it is. The first time you do it for a role, you will spend an extra hour on prep. The first calibrated debrief will take longer than your usual one. The second one will already be faster. Within a quarter, you will look back at how you used to run interviews and feel a quiet kind of horror.
I do not think hiring gets better when it gets more confident. I think it gets better when it gets more honest about the difference between what you saw and what you concluded.
Most interviews test the wrong things. The fix is not better instincts. The fix is better evidence, mapped to a framework that actually means something at your company.
That is the standard. Refuse to lower it.