The Evidence Check

The Evidence Check: a programme

Route: A programme Choose a different route

JavaScript is off, so here is the whole check as a checklist you can print or fill in on paper. A key to the results is at the end.

A programme

  • 1. What’s the main thing riding on it in the next 12 months? About you, not scored

    • A renewal, tender or sale
    • A funder or board report
    • A regulator, auditor or client’s due diligence
    • Mainly our own improvement
    • Nothing in particular
  • 2. Can you say in one sentence what should be different for participants afterwards? Outcomes

    • 3 points Yes, written down, with how and when we’d expect to see it
    • 2 points Yes, written down, but not how we’d see it
    • 1 point Roughly, but it depends who you ask
    • 0 points Not really; the content is clearer than the outcome
    • – Not sure (the area counts as Not sure)
  • 3. What is the strongest evidence you currently collect? Evidence of change

    • 0 points Attendance or completion (evidence ladder rung 1)
    • 0 points Satisfaction or feedback ratings (evidence ladder rung 2)
    • 1 point A knowledge test or confidence rating (evidence ladder rung 3)
    • 2 points Skills shown in a realistic task or scenario (evidence ladder rung 4)
    • 3 points Changes in how people work, seen by others or in records (evidence ladder rung 5)
    • 3 points Changes in team or organisational results (evidence ladder rung 6)
    • Skip The programme hasn’t run yet (planning: skip questions 4 to 6 and 8, and answer the planning version of question 7)
    • – Not sure (the area counts as Not sure)
  • 4. When do you measure outcomes? Evidence of change

    Answer only if your answer to question 3 is a knowledge test or anything stronger, or Not sure.

    • 3 points Before, straight after, and again months later
    • 3 points Before and again months later
    • 2 points Before and straight after
    • 1 point Straight after only
    • – Not sure (the area counts as Not sure)
  • 5. Where do your outcome questions come from? Measure quality

    Answer only if your answer to question 3 is a knowledge test or anything stronger, or Not sure.

    • 3 points An established measure, checked for people like ours and for detecting change
    • 2 points An established measure, used as published
    • 1 point Questions we wrote ourselves
    • – Not sure (the area counts as Not sure)
  • 6. If outcomes improved, could you rule out other explanations? Attribution

    Answer only if your answer to question 3 is a knowledge test or anything stronger, or Not sure.

    • 3 points Yes: a comparison group or a trend over several groups before and after the programme
    • 2 points Partly: we compare with past groups or benchmarks
    • 1 point Not really: we only see before and after
    • 0 points We haven’t looked at outcomes that way
    • – Not sure (the area counts as Not sure)
  • 7. Do you check the programme was delivered as designed? Delivery

    Answer only if the programme has run.

    • 3 points Yes, every group, systematically
    • 2 points Sometimes, or only for some trainers or sites
    • 1 point We assume it was
    • 0 points No, and delivery varies a lot
    • – Not sure (the area counts as Not sure)
  • 7. Will you check it’s delivered as designed? Delivery

    Answer instead of the question above if the programme hasn’t run yet.

    • 3 points Yes, we’ve planned how
    • 2 points We intend to, but haven’t planned how
    • 0 points Not yet considered
    • – Not sure (the area counts as Not sure)
  • 8. What happened the last time evidence came in? Decision use

    Answer only if the programme has run.

    • 3 points It changed a decision about the programme
    • 2 points It went into a report people read
    • 1 point It was filed
    • 0 points There hasn’t been any evidence to use yet
    • – Not sure (the area counts as Not sure)

A measure

  • 1. What is it mainly used for? About you, not scored

    • Hiring or promotion
    • Development or coaching
    • Screening or identifying need
    • Measuring change over time
  • 2. What’s the main thing riding on it in the next 12 months? About you, not scored

    • A renewal, tender or sale
    • A funder or board report
    • A regulator, auditor or client’s due diligence
    • Mainly our own improvement
    • Nothing in particular
  • 3. How was the content of the measure decided? Content

    • 3 points From a clear definition, built and checked with people like those who take it
    • 2 points From a clear definition, by experts only
    • 1 point Adapted from something else without a full check
    • 0 points It grew over time
    • – Not sure (the area counts as Not sure)
  • 4. Has it been checked (piloted or validated) with people like the ones you use it with? Content

    • 3 points Yes, recently, with a good-sized sample
    • 2 points With a related group, or some years ago
    • 1 point Only where it was first developed
    • 0 points Not that I know of
    • – Not sure (the area counts as Not sure)
  • 5. Is there evidence that scores relate to what they should, such as performance, later outcomes or other measures of the same thing? Validity

    • 3 points Yes, studied with people like ours
    • 2 points In the original research only
    • 1 point We assume so
    • 0 points No
    • – Not sure (the area counts as Not sure)
  • 6. If two scores differ by a few points, could you say whether that’s real or noise? Precision

    • 3 points Yes, we know the margin of error for one person’s score
    • 2 points We know the reliability figure, but not what it means for one person
    • 1 point We assume the score is accurate
    • 0 points We’ve never considered it
    • – Not sure (the area counts as Not sure)
  • 7. Has anyone checked it works the same way across groups, such as sex, ethnicity, age or first language? Fairness

    • 3 points Yes, question by question, and differences were dealt with
    • 2 points We compare average scores between groups
    • 1 point Not formally
    • 0 points No
    • – Not sure (the area counts as Not sure)
  • 8. Do you use cut-offs or score bands? Decisions

    • Yes (answer question 8b)
    • No (skip question 8b: Decisions is Not applicable)
  • 8b. How was your cut-off or band, such as ‘high potential’ or ‘at risk’, set? Decisions

    Answer only if you answered Yes to question 8.

    • 3 points Against a real outcome, with evidence it predicts it
    • 2 points From norms or the original publisher
    • 1 point It seemed sensible
    • 0 points We don’t know where it came from
    • – Not sure (the area counts as Not sure)
  • 9. Has it been shown to pick up real change? Change

    Answer only if you use it for measuring change over time.

    • 3 points Yes, it has been tested
    • 1 point We haven’t checked
    • 0 points Results often show little or no change
    • – Not sure (the area counts as Not sure)
  • 10. If a client, auditor or regulator asked for the technical evidence tomorrow, what would you send? Evidence ready

    • 3 points A current technical report or manual
    • 2 points Some analyses, not pulled together
    • 1 point The original paper
    • 0 points Nothing ready
    • – Not sure (the area counts as Not sure)

Reading your answers

  1. Each scored question belongs to an area, named beside it. An area’s result: Not sure if you answered Not sure to any of its questions; Not applicable yet (programme) or Not applicable (measure) if you skipped all of its questions (on the programme route, Not yet reached instead if you skipped them because your answer to question 3 was attendance or satisfaction: Once you measure outcomes, you’ll also need a way to show the programme caused the change.); In place if every answer scored 3 points; Not described if every answer scored 0 or 1; otherwise Partly in place.
  2. Where to start: any area that is Not sure or Not described, then any that is Partly in place. Where two areas are level, take them in this order. Programme: Outcomes, Evidence of change, Measure quality, Delivery, Attribution, Decision use. Measure, by what it is mainly used for (question 1): Hiring or promotion: Validity, Fairness, Decisions, Content, Change, Precision, Evidence ready; Development or coaching: Content, Validity, Fairness, Decisions, Change, Precision, Evidence ready; Screening or identifying need: Decisions, Content, Validity, Fairness, Change, Precision, Evidence ready; Measuring change over time: Change, Content, Validity, Fairness, Decisions, Precision, Evidence ready.

A useful first step, by area

Programme: Outcomes
Write one sentence: who will do what differently, by when. Agree it with whoever signs off the budget.
Programme: Evidence of change
Ask participants again 6 to 8 weeks later what they have done differently, not how they rated the course.
Programme: Measure quality
Keep outcome questions word-for-word identical at every time point, and check whether they have ever been tested for detecting change.
Programme: Attribution
Collect a simple baseline before your next group starts.
Programme: Delivery
Give trainers a short after-session checklist: what was covered and what was skipped. If the programme hasn’t run yet: Before launch, agree what ‘delivered as designed’ means (the core sessions, content and contact time) and how trainers will record it.
Programme: Decision use
Before collecting anything, write down the decision the evidence will inform and who makes it.
Measure: Content
List what the measure claims to capture and map every question to one claim. Gaps and orphans show up quickly.
Measure: Validity
Pick one outcome the scores should predict or relate to, and check whether the two line up in data you already have.
Measure: Precision
Find your reliability figure and ask what margin of error it implies for one person’s score.
Measure: Fairness
Compare responses question by question across the groups you have data for, not just average scores.
Measure: Decisions
Write down where each cut-off came from. If you can’t, start there.
Measure: Change
Check the measure’s published evidence for detecting change before your next before-and-after.
Measure: Evidence ready
Gather what you have into one folder with a one-page summary.

Why it matters, by what’s riding on it

A renewal, tender or sale
This is the question a buyer comparing providers is likely to ask.
A funder or board report
This is the gap a funder or board is most likely to probe.
A regulator, auditor or client’s due diligence
This is where external scrutiny usually starts.
Mainly our own improvement
Fixing this first makes every later evaluation cheaper and more useful.
Nothing in particular
Fixing this first makes every later evaluation cheaper and more useful.

What your evidence can and can’t say (programme route)

Your answer to question 3, the strongest evidence you collect, puts you on one rung of this ladder.

  1. Rung 1, Took part. Can say: people attended. Can’t yet say: they learned or changed anything.
  2. Rung 2, Valued it. Can say: people rated it well. Can’t yet say: anything changed.
  3. Rung 3, Learned it. Can say: people knew more or felt more confident at the end. Can’t yet say: it lasted or changed how they work.
  4. Rung 4, Can do it. Can say: people can apply it in a realistic task. Can’t yet say: they do it at work.
  5. Rung 5, Do it at work. Can say: how people work has changed. Can’t yet say, without a comparison: the programme caused it. If you have a comparison (question 6): Can say, with your comparison: the programme likely contributed. If you measure straight after only (question 4): Can say: how people work now (without a before measure, not that it changed).
  6. Rung 6, Results moved. Can say: team or organisational results improved. Can’t yet say, without a comparison: it was the programme. If you have a comparison (question 6): Can say, with your comparison: the programme likely contributed. If you measure straight after only (question 4): Can say: team or organisational results now (without a before measure, not that they improved).

Your next step

A programme: Want to know what your existing evidence can support? An Evaluation Framework Review looks at what you already collect and what a stronger evaluation would need. A free 30-minute call is enough to see whether it fits.

If the programme hasn’t run yet: Starting from scratch? An Evaluation Framework Review gives you an evaluation plan you can commission against. A free 30-minute call is enough to scope it.

A measure: Want a clear verdict on your scores? An Assessment Health Check uses the response data you already have. A free 30-minute call is enough to see what it needs.

Or email your results to greg@gregorygorman.co.uk.

An initial guide based on your answers, not an independent review or a validated assessment. Nothing you entered was stored or sent.