Catching False Passes in a QA Selection Plan

📍 Industrial-Organizational Psychology 📅 July 7, 2026 · 6 min read

I designed a hiring funnel with cutoff scores and validity coefficients to make it fair. Then I audited my own math — and found the number I'd inflated.

Yesterday I published the job description assignment for the QA Manager role—the foundational document that establishes our baseline performance requirements. This is the next structural step in that sequence: designing the selection plan. If a job description sets the acceptance criteria for a human being, the selection plan is the test suite built to verify whether a candidate actually compiles against it.

Most hiring processes are held together by faith. We trust the interview, we trust the résumé, we trust the gut feeling in the room — and we rarely ask the uncomfortable question underneath: does this step actually predict who will be good at the job? That question has an entire science behind it, and the science is often unkind to our favorite steps.

The funnel I built

I used a multiple-hurdle design ordered by cost: cheap and broad at the top, expensive and decisive at the bottom, so you only spend the costly stages on candidates who cleared the cheap ones. Three predictors, in order.

Hurdle 1 — Biodata. A biographical information blank as the wide-mouthed first filter: inexpensive, remote, and a first read on the attention to detail, negotiation, and plain textual comprehension the role demands. My written justification was blunt: past conduct predicts future behavior, because it’s the one thing a candidate has demonstrated rather than claimed.

Hurdle 2 — Assessment center. Online, three phases on consecutive days: a work-sample test (write SQL queries against a real data set — the closest thing selection science has to a crystal ball, because you stop asking “can you describe QA work?” and start watching them do it), an in-basket exercise (respond to high-pitched escalation emails), and a leaderless group discussion. The analytical exercise was scored 320 to 800 with a cutoff at 500; the stage as a whole required 60% to advance.

Hurdle 3 — Structured interview. Sixteen questions: eight behavioral scored 0–10 in one-point steps, eight technical scored 0, 5, or 10. Four interviewers — SME, principal architect, test lead, product manager — each holding ten discretionary points for “softer qualities such as communication skills, leadership, planning, organization, flexibility, and relatability.” Pass mark: 140 out of 200.

I even attached validity coefficients from the literature to each stage — biodata r = .40, assessment center r = .38, structured interview r = .71 — and computed a combined battery validity of R² = .80. Eighty percent of job performance, predicted by my funnel. On paper, rigorous. On paper.

The audit — four leaks, one of them arithmetic

Leak 1: the headline number is wrong, and it’s my kind of wrong. To get R² = .80, I added the three predictors’ r² values as if they were uncorrelated — the diagram below shows exactly where that breaks. The honest fix isn’t just a strikethrough, it’s a number: correct for the overlap and a well-built three-stage battery like this typically lands closer to R² ≈ .40–.55, not .80. Still remarkably high for behavioral prediction — just not the mythical four-fifths I’d handed myself. My funnel’s proudest statistic was an optimistic double-count, precisely the class of metric error I’m paid to catch in test reports, sitting unflagged in my own. A dashboard number that flatters the process it measures is not evidence. It’s marketing.

Leak 2: biodata rewards a résumé, and résumés carry history’s bias. “Past conduct predicts future behavior” is true — and it quietly imports every inequity that shaped who got the chance to accumulate that conduct. My submission did include a fairness section: the biodata blank stripped of anything revealing age, race, orientation, or gender; bias training for every assessor; pre-approved interview questions. I wrote precautions against the evaluators’ bias and none against the filter’s — the bias that arrives inside the candidate’s history before any evaluator sees it. A first stage that’s cheap and predictive can also be cheap and exclusionary, and I’d written the justification without the warning.

Leak 3: the unstructured backdoor inside my structured interview. Look at the arithmetic of hurdle 3 again. Forty of the 200 points — a full fifth of the decision — are discretionary, awarded by four interviewers for qualities including “relatability.” I structured sixteen questions to keep gut feel out, then handed gut feel 20% of the scoreboard through the side door. Relatability is exactly where “interviews like me” hides — and as a non-native English speaker who has sat on the scored side of that table, I know precisely which candidates lose those forty points, and it isn’t the ones who’d do the job worst. I designed a funnel I’m not sure I’d survive.

The recap above stops at three — the fourth didn’t surface until I reread my own opening paragraph.

Leak 4: my own cost ordering doesn’t survive the audit. I built this funnel on a single stated principle — cheap and broad at the top, expensive and decisive at the bottom — and never checked whether hurdle 2 actually obeys it. A three-phase assessment center run live across consecutive days, with a leaderless group discussion that needs several candidates online at once, is one of the more operationally expensive things you can schedule, and it bleeds candidates at every synchronization point. A one-hour structured interview runs one candidate at a time and clears in a single sitting. I put the heavier, more failure-prone stage in the cheap seat and called it economy of scale. The audit that caught a double-counted statistic nearly missed a mis-ordered funnel sitting one section above it.

What transfers

A selection plan and a test plan are the same document pointed at different objects. Both order checks from cheap-and-broad to expensive-and-decisive. Both are only as good as whether each check predicts the outcome you care about. And both are most dangerous when a step feels rigorous but measures the wrong thing — a smoke test that passes tells you almost nothing, and an interview stage with a discretionary fifth tells you mostly who the interviewers liked.

To its credit, the plan knew how it should be judged: I specified predictive validation, comparing every predictor score against actual job performance at eight weeks, six months, and two years. That’s the right instinct — don’t trust the funnel, test the funnel. I just should have aimed it at my own arithmetic first.

Thanks for reading. For anyone who hires: take your favorite step in your process and ask it the harder question — not “is it rigorous?” but “where does its score come from, who does it quietly screen out, and would you survive it yourself?”

← All articles