Interview Scorecards: Structuring Interviews So the Decision Is Defensible

Define the criteria before you write questions

A scorecard starts from the job, not from the interview. Take the requirements you set when you wrote the job description and reduce them to four to six assessable attributes. More than six and interviewers stop scoring honestly and start pattern-matching; fewer than four and you are not really differentiating candidates.

Each attribute needs to be something an interview can actually observe. 'Passion' and 'culture fit' cannot be assessed reliably and tend to encode bias. 'Can explain a technical decision to a non-technical stakeholder' can be assessed, because you can ask for it and watch it happen.

• Derive attributes from the job description, not from the last person who held the role
• Keep every attribute observable within the interview itself
• Say which attributes are must-have and which are developable on the job
• Decide which stage assesses which attribute, so panels do not all test the same thing

Write the rating scale before the questions

A numeric scale without behavioural anchors produces meaningless averages, because one interviewer's 3 is another's 4. Define what each point on the scale looks like in behaviour, and interviewers converge without needing a meeting.

Four points works better than five: an odd-numbered scale lets an uncertain interviewer sit in the middle and avoid a judgement, which is exactly the judgement you need from them.

• 1 — No evidence, or evidence against. Would need substantial support.
• 2 — Some evidence, below the bar for this level.
• 3 — Clear evidence at the bar. Would perform this part of the role.
• 4 — Strong evidence above the bar. Would raise the team's level here.

Ask every candidate the same core questions

Consistency is what makes comparison possible and what makes a rejection defensible. If one candidate was asked about handling a difficult stakeholder and another was not, you have no basis for comparing them on that attribute — and no answer if the unsuccessful candidate asks why.

Structure does not mean rigidity. Ask the same core question of every candidate, then follow up freely on their specific answer. The core question is what you compare; the follow-ups are what you learn.

• One core question per attribute, asked of everyone at that stage
• Follow-ups improvised from the answer, not from the candidate's background
• Ask for a specific past instance rather than a hypothetical where possible
• Leave hypotheticals for genuinely novel situations the candidate could not have faced

Score independently, then discuss

The most common failure in panel hiring is anchoring: the first person to speak sets the frame, and everyone else adjusts toward it. Independent scoring before discussion removes this at zero cost.

Require every interviewer to submit their scores and written evidence before the debrief opens. Then discuss the disagreements, which is where the useful information is — two interviewers rating the same answer 2 and 4 have heard something different, and finding out what is more valuable than averaging them.

1. 1Every interviewer submits scores and evidence independently, before any discussion
2. 2Open the debrief on the attribute with the widest spread, not on the overall verdict
3. 3Ask each side what specifically they heard, rather than what they concluded
4. 4Record the final decision and the reason, whichever way it goes

Write evidence, not impressions

'Good communicator' is not evidence. 'Explained the migration rollback to me clearly without using jargon, when I said I was not technical' is. The difference matters twice: it makes the debrief useful, and it means the record can stand behind the decision months later if anyone asks.

Interviewers resist this because it takes longer. It takes about ninety seconds per attribute, and it is the difference between a hiring process and a series of opinions.

Calibrate the panel periodically

Scorecards drift. Interviewers develop private standards, and a 3 from one person stops meaning what a 3 means from another. Reviewing a handful of completed scorecards together every few months brings the panel back into alignment.

Look for interviewers who never use the extremes, interviewers who rate everyone highly, and attributes where nobody ever scores below 3 — the last usually means the attribute is not being tested at all.