The QA Manager, Part IV: Rating the Raters — and Who Never Got a Vote

📍 Industrial-Organizational Psychology 📅 July 10, 2026 · 8 min read

I designed a performance appraisal for the QA Manager role. Auditing my own design afterward — using the same criterion-theory vocabulary the course had just handed me — I found whose opinion I'd quietly written out of it.

Closing out a series, this time for real. First the job description, then a selection plan to hire against it, then an employee satisfaction survey to check the finished hire’s temperature. This last assignment asks the harder question: once someone is actually doing the job, how do you rate them — and does your rating system measure the job, or just the parts of it that are easy to watch?

Nobody looks forward to a performance review — not the person being rated, and usually not the person doing the rating either. We treat that dread as a fact of corporate life, background noise. It’s worth asking why instead: an appraisal is a measurement instrument, and most of the dread comes from instruments that were never actually validated, just inherited. Before I inherit one for a room full of real people, I wanted to build one and see where it breaks.

The scale I picked, and why

I chose a Behaviorally Anchored Rating Scale (BARS) — behavior-specific anchors instead of a bare 1-to-5 — and my own justification doubles as the first thing worth auditing. Verbatim, from the submission:

“The Behaviorally Anchored Rating Scale (BARS) was selected as the rating scale due to its accuracy. One of the goals of the company’s culture is to create a more coachable workforce, therefore this rating scale reflects that value… While it is true that BARS has not been proven to be more accurate than other formats, as was previously indicated, the scale does fit in with the overall ethos of the organization.”

Read that sentence twice. I opened by claiming the scale won on accuracy, then walked it back two lines later and admitted the actual reason was cultural fit. That’s not a contradiction I noticed while writing it — it’s one I only caught rereading it later, the same way a test suite that “passes” can still ship a defect nobody wrote a case for. I picked the scale that flattered the culture I wanted to describe, and dressed the choice up as evidence-based until my own words undercut it a paragraph later.

Who gets a vote — and the term for what’s missing

The assignment asked who would rate this role and why. My rater panel: managers up the chain, subject-matter experts, analysts working alongside the role day to day, and — reaching outside the company — vendor and program-manager contacts the role interacts with regularly.

Read that list again for who’s missing. Nowhere on it is the role’s own subordinates. This job — in the job description one assignment prior — exists specifically to mentor and build up near-shore and off-shore team members earlier in their careers. I designed an evaluation of how well someone manages and develops people, and never gave those people a formal vote in the answer.

The course had just handed me the exact vocabulary for this mistake, in the chapter on criterion theory: the gap between the ultimate criterion — everything that actually constitutes good performance — and the actual criterion, the narrower thing you actually measure, is called deficiency. A deficient measure leaves out pieces of the job that matter. My rater panel is deficient by that definition, on the exact dimension — developing people — that the role exists to perform. I gave outside vendors a seat at that table before I gave it to the team being led. For someone whose whole professional throughline is making sure underestimated voices get heard, that’s not a small miss — it’s the same blind spot showing up in the one document I had full control over.

The same chapter names the opposite failure, contamination — when your measure picks up something other than the performance you meant to capture. My external raters include vendors with an active commercial relationship to the role. A vendor who wants to keep their contract has a reason to rate generously that has nothing to do with quality of work. I built deficiency and contamination into the same instrument, on my first attempt, without the vocabulary to notice either until the reading assignment gave it to me after the fact.

How employees get compared

For ranking employees against each other, I chose a hybrid: run forced distribution first to sort out the average and low performers, then rank only within that group. I wrote that to protect strong performers from being pitted against each other over small margins — a real and defensible goal. But look at what it does structurally: the comparative scrutiny, the process explicitly built to surface who needs coaching, is aimed entirely at people already flagged as average or below. Top performers are the one group never comparatively examined at all. A fairness mechanism that only ever points at people already struggling isn’t neutral — it just moves the blind spot from “everyone” to “whoever’s on top.”

Forced distribution has a second, quieter cost that came up repeatedly in class discussion that week: when raters know they must produce a fixed share of below-average scores regardless of actual performance, the system doesn’t just sort people — it manufactures scarcity where none exists, and turns “help a teammate” into “help a competitor for the same shrinking pool of average-or-better ratings.” It’s worth naming that this isn’t a hypothetical critique of some other company’s process; it’s one I’ve watched play out in forced-ranking systems I’ve personally been rated under. The failure mode isn’t unique to my design, either, even if the mechanics differ. Deloitte’s old system wasn’t a hybrid like mine — it was a traditional annual review built on 360-degree ratings and lengthy consensus meetings where committees debated and reconciled every score. Different machinery, same underlying problem: Deloitte found the process consumed close to two million hours of leadership time a year to produce ratings nobody actually trusted, which is why they rebuilt it around frequent, unranked check-ins instead (Buckingham & Goodall, 2015). The lesson transfers even though the systems don’t — a mechanical, consensus-heavy ranking process can burn enormous time manufacturing data nobody believes, and my hybrid, for all its differences from Deloitte’s, still spends real management hours sorting people into a distribution whose accuracy nobody has actually tested. The fix I designed keeps the part of forced distribution that manufactures the problem and only patches around the edges.

What I got right

Not everything here is a defect, and a self-audit that only finds failures isn’t an honest one. The rater-error mitigation was solid: every rater completes Rater Error Training before touching a scale — covering the well-documented failure modes like halo/horn effects and distributional errors such as leniency, severity, and central tendency — plus a Frame-of-Reference session on applying the anchors consistently. That part transfers cleanly from anything I’d sign off on in a real test plan: calibrate the instrument before you trust its output, not after.

This isn’t only theoretical for me. Every real review I’ve sat through follows roughly the same shape my own assignment did: a long, careful self-account of context and judgment calls, met by a short paragraph and a number back. That asymmetry — not the rating itself — is usually what makes a review feel like paperwork instead of feedback. It’s the same deficiency problem again, just one level up: the instrument only has room to capture what the rater has time to write.

What transfers

An appraisal, like a job description, is never just a measurement — it’s a statement of whose experience of the work counts as evidence. Mine counted supervisors, peers, and external vendors, and silently excluded the very people the role exists to develop. That’s not a scoring error; it’s a scope error, invisible until someone asks who wasn’t in the room when the instrument was designed — which nobody did, including me, until the reading gave me the language to ask it myself.

There’s a broader version of this gap that the field itself hasn’t closed. Reading Miller’s (1996) work on the ethical management of human resources around the same time, I kept returning to one line of thinking: fairness in performance systems is inherently comparative — a rating that benefits one group is, structurally, a rating withheld from another — and the research base on performance appraisal has been slow to build diversity, equity, and inclusion directly into how these instruments get designed in the first place, rather than treated as a downstream fix once the numbers look uneven. My own rater panel is a small, specific instance of that larger pattern: the design choices happened before anyone asked who they’d leave out.

My throughline is negotiating for clarity, and this is where it cuts both ways: I can name exactly why BARS fit the culture, exactly why forced distribution felt fair — and still have missed, in the same document, that the people I claim to advocate for had no seat at the table measuring whether I was doing it well.

Thanks for reading. If you could add one rater to your own next performance review, who would it be — and what does it say that they don’t already have a vote?

← All articles