Can you defend the scores your assessment produces?
Psychometric review for assessment, HR-tech, EdTech and survey businesses.
Why a high alpha isn't enough
Many assessments are built by people who know their domain well. The measurement questions arrive later, usually when a large customer asks for technical evidence. Four come up repeatedly.
An alpha of .90 shows the items hang together, not that they measure what you say. It is easy to reach by asking the same question ten ways. And a label like “at risk” or “recommended” is a claim about consequences that needs its own evidence: where the threshold was set, and how much measurement error sits around the people near it.
If items behave differently by gender, age or first language, group differences may be artefacts, and that is a legal and reputational exposure you can check with data you already have. Enterprise buyers and regulators increasingly ask for this technical evidence. A validation report is a sales asset when the deal is going well and a defence when it isn't.
What a review covers
| Question | What I examine | What you get |
|---|---|---|
| Does it measure consistently? | Internal consistency (alpha and omega), test–retest where data allow, and the standard error of measurement at the scores that matter | A reliability statement that distinguishes “consistent” from “precise enough for the decisions you make with it” |
| Does it measure one thing or several? | Exploratory and confirmatory factor analysis; model comparison | A defensible score structure, and a recommendation on whether to report a total, subscales or both |
| Do the items work? | Item response theory (discrimination, difficulty, information) and classical item statistics | Item-by-item findings: which items carry the measurement, which are redundant, which should be revised or dropped |
| Is it fair across groups? | Measurement invariance and differential item functioning by the groups that matter for your use | Evidence on whether group score differences are real or artefacts, and what to do about any flagged items |
| Does it predict or relate to what it should? | Convergent, discriminant and criterion validity, using whatever outcome or comparison data exist | A plain account of the validity evidence you have, what it supports, and what is missing |
| Are the scores and cut-offs interpretable? | Norms, score bands, classification accuracy at thresholds | Guidance on how scores should be reported and what claims your marketing can safely make |
Example: a Health Check on a 24-item workplace assessment
Read the full example report (PDF, 5 pages) (opens in a new tab)
Two ways to start
Assessment Health Check
A fixed-scope review of one assessment using your existing response data, covering the analyses above, with prioritised recommendations.
You receive: a technical review of around five to eight pages, written for a mixed audience, and a one-hour walkthrough call.
Timeline: two to three weeks.
Fee: from £2,500, fixed.
Full validation study
For a new assessment, a major revision, or an enterprise deal that needs a technical manual. Includes study design, data collection instruments and a technical report structured for procurement and expert review.
Timeline: usually six to twelve weeks, depending on whether data must be collected.
Fee: typically £8,000–£25,000, quoted as a fixed fee after scoping.
What I need from you
- A CSV of item-level responses, one row per respondent. Around 200 or more respondents gives useful results; fewer is possible with narrower conclusions.
- The scoring key: which items form which scale, any reverse-scored items, how totals and bands are calculated.
- Any demographic or group variables, and any outcome or criterion data.
- The claims you currently make about the assessment, because those are what the evidence must support.
Data arrive anonymised where possible, are held securely, used only for the agreed work and deleted on completion. I will sign your NDA or provide mine. You keep all rights to your items and data.
Questions you might have
Our sample is small.
Below about 200 respondents, some methods (particularly item response models and invariance testing) become unstable. I will say which conclusions are provisional rather than pretend otherwise.
Our items are proprietary.
I'll sign an NDA. The review needs the item text to make sense of the statistics, but nothing is retained after the engagement.
Can you get us BPS test registration?
No, I don't hold chartered status, so I can't provide that sign-off myself. I can prepare the technical evidence that a review like that would draw on.
Can you fix problems, not just find them?
Yes. Item revision, rescoring, shortened forms and re-analysis after a revision are common follow-ons.
What software do you use?
R. On request you receive the analysis scripts so that your team can re-run the checks as the assessment evolves.
Start with a conversation
Send nothing in advance. Thirty minutes is enough to work out whether a Health Check or a full study is the right starting point.
Running a programme rather than selling an assessment? See Programme Evaluation.