VulnerABLE Pan-European Survey - A Critique

📍 Survey Research Methodologies 📅 July 3, 2026 · 6 min read

A European survey set out to hear from isolated and vulnerable people. Its own method may have quietly changed what those people felt safe saying.

This started as homework: pick a real survey, find its flaws. It ended with a flaw that hides inside good intentions — and a new suspicion of anything that claims to speak for people who rarely get asked.

Almost everyone has answered a survey that felt slightly off. A question you could not answer honestly because of who might see it. A form handed to you by someone who also controls something you need. Now imagine that survey is meant to measure how isolated and vulnerable people are doing across Europe, and the results are meant to inform health policy.

Choose a real survey. Identify a core methodological problem, explain how it produces error using TSE, and propose a realistic fix.

Total Survey Error (TSE) assess a survey by utilizing a framework that treats every stage of a survey as a place where error can enter.

I chose the European Commission’s VulnerABLE pilot — a survey that is easy to admire, which is exactly why I almost let its method pass.

I should own my own blind spot first. My initial read was too kind. I saw the aims, saw who it was trying to reach, and wanted it to be good; that wanting is its own bias. It is also familiar from the other side of the table. I have been the respondent sitting across from someone official, answering in a language that was not my first, deciding on the fly how much of the truth was safe to hand over. I have managed my own answers plenty of times, and that made me slow to accuse a survey of engineering the same thing — until its method left me no other reading.

The setup

The survey attempted to reached following nine target groups of vulnerable people across 12 countries.

• Families who are in a vulnerable situation (e.g. lone parents with young children);
• Having physical, mental and learning disabilities or poor mental health;
• In-work poor;
• Older people who are in a vulnerable/isolated situation;
• People with unstable housing situations (e.g., homeless people);
• Prisoners (or ex-prisoners in vulnerable situation);
• Persons living in rural/isolated areas in a vulnerable situation;
• Long-term unemployed / inactive (not in education, training or employment); and
• Victims of domestic violence.

Six countries used a mix of two modes. France used one mode only. Five countries used the other mode only. Incentives were offered in both. No quotas were set for the target groups, and no attempt was made to measure how representative the sample was of vulnerable populations overall.

Two facts do most of the damage here, and neither is about the questions themselves.

The flaw I kept pulling on

The first is the mode split. Some people answered online and in private, through a panel provider — a computer-assisted web interview (CAWI). Others answered on paper and in person — a paper-and-pencil personal interview (PAPI), used mainly for the hardest-to-reach groups.

The second fact is who ran those in-person interviews. Local stakeholders. Social workers, charity workers, NGO staff. The same people who, from the respondent’s side of the table, may influence access to benefits, housing, and health services.

Put those together and the problem is not subtle. If the person asking whether you are satisfied with your services is also the person connected to those services, you have a strong reason to present yourself as a good client. On sensitive topics that produces under-reporting of stigmatized experiences and over-reporting of resilience, satisfaction, and positive health behaviors. Meanwhile the online respondents, answering anonymously, can admit the negative things more freely.

So the same question gets different answers depending on the mode and the presence of a gatekeeper, not because of who the respondents are. And because mode is tangled up with both country and target group, comparing countries means comparing methodologies as much as realities. That confounding is what weakens the survey’s internal validity and makes its international conclusions shaky. The survey cannot cleanly tell a real difference between countries from a difference its own design created.

The fix I proposed

The repair does not require scrapping the effort. It requires separating two jobs that got merged:

1. Let stakeholders keep the recruitment role. They are how you reach people who are otherwise invisible to a survey.
2. Take away the interviewer role. Make the survey self-administered on paper or tablet so answers to sensitive questions are more private. For low literacy or disability, bring in trained neutral interviewers after the stakeholder has made contact.
3. Add a mode indicator to the analysis (CAWI = 1, PAPI = 0) and check whether mode predicts the reported outcomes — health, barriers, satisfaction. Always read country and group differences alongside how much of each sample came from which mode.

Because this was a pilot built to scale, these are realistic changes to make before the design gets bigger, when fixing it is still cheap.

My overall verdict: a valuable, ambitious effort that genuinely reaches vulnerable groups, but not yet a sound standalone basis for European health policy while the mode and interviewer biases are unresolved.

Why this one stayed with me

The whole point of my work is negotiating for clarity, and this survey is a case study in how clarity depends on who is in the room. A well-meaning design gathered people who rarely get asked anything, handed the microphone to someone with power over them, and recorded what they said. The intention was inclusion. The effect could be a polished silence.

That is the trap you can build with the best of motives: a study generous enough to reach these groups, and structured enough to design the honesty back out of their answers. The people this survey most needed to hear are exactly the ones with the most practice at managing what they say.

Thanks for reading. One question if you run surveys of your own: do you actually know who is standing next to your respondent at the moment they answer — and what that presence is doing to the data you will go on to trust?

← All articles