The QA Manager, Part III: The Satisfaction Survey I Built — and Every Way It Could Mislead Me

📍 Industrial-Organizational Psychology 📅 July 8, 2026 · 8 min read

I built a six-question survey to measure satisfion and audited each question for the answer it was quietly manufacturing.

Third stop in the series, not the last one after all. First I wrote the job description for a Quality Assurance Manager, then a selection plan to hire one from a pool of applicants. We now move to measuring the production environment itself: An employee satisfaction survey — one more stop remains, rating the person once they’re actually doing the job.

We treat surveys as if they simply report how people feel, like a thermometer. They don’t. A survey is an instrument, and like any instrument it can be miscalibrated, pointed at the wrong thing, or quietly bent by the person holding it. The uncomfortable version: most workplace surveys we trust have error baked in before a single person responds.

The instrument

I didn’t start from feelings; I started from the job. The competencies and KSAOs from my job description, and the high-performer behaviors anchored in the performance appraisal’s rating scale, became the survey’s backbone. From those I chose three antecedents of satisfaction — task significance, supervisor relationships, and self-esteem — and wrote two questions for each. Here is the entire instrument, verbatim:

1. How frequently do you feel motivated to come to work every day? 2. Do you feel empowered to pay extra attention to detail at work? 3. How often does your supervisor advocate for your progress at work? 4. How frequently do you get constructive criticism on the quality of your deliverables from your supervisor? 5. Do you feel supported in being your authentic self at work? 6. How committed do you feel the organization has been to your personal development over the past year?

Each answered 1 to 5, higher meaning more satisfied. Tidy: role → competencies → antecedents → items. And when the responses came back, the data even produced a legitimate finding — supervisor relationships scored lowest of the three antecedents, which is exactly the kind of result a satisfaction survey exists to surface.

That tidiness is what nearly fooled me.

The teardown — reading my own items like a code review

Then I read the instrument the way I read a build I’m about to sign off: not “does it run?” but “what would it fail to catch?”

Question 1 measures the wrong construct. “How frequently do you feel motivated…” — that’s motivation, not satisfaction. They correlate; they are not the same thing, and an item that measures the neighbor of what it claims is a construct-validity leak. It also contains a phrasing muddle: asking “how frequently” you feel something “every day” stacks a frequency on top of a stated frequency, forcing the respondent to debug the wording before they can even parse the scale. Worse, my stated design goal compounded it: I built items from the high-performer profile, to find what stops someone becoming a highly satisfied worker “based on their performance.” A miserable high performer passes my survey. That’s an instrument certifying the adjacent thing.

Question 2 is a yes/no wearing a five-point costume. “Do you feel empowered…” invites agreement, not gradation — and agreeable respondents drift toward yes regardless of truth. That’s acquiescence bias, and a positively-worded scale can manufacture the satisfaction it claims to detect. It also asks about attention to detail — a competency from my own JD — so it flatters the respondent who claims it. Social desirability, built in by me.

Questions 3 and 4 assume anonymity that may not exist. Both are about the supervisor — and in a small QA team, “which environment do you own plus how long have you been here” de-anonymizes a respondent instantly. A survey people don’t trust to be anonymous is a survey people answer strategically, and the questions most poisoned by that are precisely the supervisor questions. Which brings a harder edge to my one real finding: supervisor relationships scored lowest even under conditions that discourage saying so. The true number is plausibly worse than my instrument was able to hear.

Worse, Question 4 has a scoring bug I mapped straight past. I scored the whole scale as “higher means more satisfied” — but frequency of constructive criticism doesn’t move in one direction with satisfaction. If someone rates it a 5 (Always), are they satisfied because they’re being developed, or miserable because their work is constantly failing? If they rate it a 1 (Never), are they a flawless performer or completely neglected? The item isn’t monotonic, so the data it returns can’t be read in one direction — I built a question whose answer I can’t actually score.

Question 5 looked like the one I’d keep — until I looked closer. “Supported in being your authentic self” is the closest my survey came to measuring belonging rather than performance — the item least about being good at the job and most about being whole in it. I notice it’s also the one I wrote last-but-one, almost as an afterthought. The inclusion question usually is. But mechanically it’s just as broken as Question 2: another binary “Do you feel…” wearing a five-point costume. I gave it a pass at first because I liked what it was trying to say — which is the ultimate reviewer bias, letting a flawed implementation slide because you agree with the comment thread.

Question 6 asks for a memory audit. “Over the past year” invites recall bias — respondents answer from their most recent quarter and their current mood, not the year. And the survey’s deepest flaw answers no question at all: the satisfied-or-checked-out non-respondent is invisible. The people too busy or too disengaged to answer are exactly the signal a satisfaction survey most needs, and the instrument structurally cannot see them.

None of these failure modes are exotic. They are the everyday error budget of every engagement survey, pulse survey, and developer-experience survey I’ve ever been handed at work. I built most of them into mine in a single assignment, with good intentions, while actively trying to be rigorous.

The hotfix

If I were refactoring this instrument for a production release, I’d stop mixing binary questions with mismatched frequency scales and move to a single standardized Likert agreement scale (1 = Strongly Disagree, 5 = Strongly Agree). Rewritten:

1. I feel energized and motivated by my daily work responsibilities. — measures engagement, not attendance
2. I feel empowered to spend the time necessary to ensure high-quality outcomes. — replaces the binary “Do you” with an agreement statement
3. My supervisor actively advocates for my career progression. — kept clean
4. I receive actionable feedback that helps me improve the quality of my deliverables. — fixes the non-monotonic criticism bug
5. I feel safe expressing my authentic self in my current work environment. — fixes the binary phrasing
6. Over the past year, the organization has invested in my growth. — accepts a known system constraint. Human memory has an inherent decay rate; tracking a rolling 12-month average of organizational health will always trigger recall bias. This is intentional design debt — a vulnerability I’d log in the documentation rather than try to patch with a clever reword.

And even this cleaner build doesn’t fix the survey’s real hole. The person who never answered is still invisible, and the respondent who doesn’t trust the survey to be anonymous is still answering strategically. No amount of rewording reaches them — those are structural, not syntactic, and a refactor only patches the bugs I could see.

What transfers

A survey and a test suite share a quiet danger: both can pass cleanly and still be wrong, because a green result only proves the checks ran — not that the checks measured the right thing. A satisfaction survey that confirms what management hoped and a test suite that’s green because it never exercises the risky path are the same failure in different uniforms. The discipline is identical, and it’s the discipline of this whole trilogy: don’t ask did it pass? Ask what is this instrument structurally unable to hear?

Building the survey for my own role made that personal. I could feel myself wanting to write the items that would make my job look satisfying. Catching that impulse — in a survey, in a test plan, in a status report to executives — is the entire skill.

Thanks for reading. If you run surveys at work, here’s the exercise that taught me the most: take your instrument, go item by item, and for each one name the person whose truth it cannot record. Then ask who never answered at all.

← All articles