How to Measure Training Effectiveness Beyond Completion Rates

Completion is useful. It tells you whether a defined learning event reached a person and whether they met the platform's completion rule.

It does not tell you, on its own, whether the person learned the right thing, used it at work or contributed to the outcome that justified the investment.

A practical evaluation connects five levels: reach, experience, learning, application and outcome. It also records the assumptions between them so leaders can make better decisions without overstating causation.

Begin with the decision the evaluation must support

Ask what someone will decide from the evidence:

1. continue, stop or change the programme;
2. improve content or delivery;
3. target another group;
4. increase manager support;
5. invest more or consolidate tools;
6. demonstrate that a control is operating;
7. assess whether a capability is ready to scale.

If no decision depends on a metric, question why you are collecting it.

The CIPD's learning-evaluation guidance links evaluation to organisational priorities and evidence from several sources. Plan evaluation before launch so a baseline and comparison are possible.

Build a five-level measurement chain

1. Reach and access

Did the intended population receive and access the learning?

Measures may include:

1. assignment accuracy;
2. participation and completion by due date;
3. sign-in or device failure;
4. participation by role, site or shift;
5. accessibility and language support use;
6. time to start;
7. attendance and dropout.

These measures diagnose delivery. They should not be presented as proof of performance.

2. Experience and relevance

Could people engage with the learning and did it appear useful?

Use:

1. short feedback tied to specific design questions;
2. observed friction;
3. learner questions;
4. support demand;
5. qualitative interviews;
6. facilitator and manager observations.

Satisfaction can reveal relevance and usability. A high rating does not prove learning, and a challenging but valuable activity may not maximise enjoyment.

3. Learning

What knowledge, skill, judgement or confidence changed?

Use methods that match the outcome:

1. pre- and post-assessment;
2. scenario judgement;
3. demonstration;
4. simulation;
5. work sample;
6. teach-back;
7. qualification or practical assessment.

Use equivalent rather than identical pre- and post-items where memory could distort the result. For high-consequence decisions, check assessment validity and conditions.

4. Application

Are people using the learning at work?

Use:

1. manager observation against defined criteria;
2. quality or case review;
3. workflow data;
4. customer or peer feedback;
5. confidence to act, combined with behaviour evidence;
6. follow-up scenarios;
7. repeat-error or exception patterns.

Check at a realistic interval. Application may require opportunity, tools, feedback and manager support.

5. Outcome

Did the business, service or risk measure move in the intended direction?

Examples include:

1. reduced rework or error;
2. faster readiness for independent work;
3. improved quality against a standard;
4. fewer repeat audit exceptions;
5. better customer measure;
6. stronger pipeline for a critical role;
7. reduced time to produce evidence;
8. improved retention or absence where a credible pathway exists.

Avoid claiming that training caused the change merely because the metric moved afterwards.

Write the logic before choosing metrics

Create a one-page evaluation chain:

List assumptions:

1. manager reinforces the behaviour;
2. systems and staffing allow it;
3. procedure is clear;
4. incentives do not reward the old behaviour;
5. target population is correct;
6. the business measure is sufficiently sensitive.

Evaluation should test assumptions, not just the course.

Choose a proportionate design

Pulse check

Use for low-risk, small changes. Combine access data, a short learning check and a targeted follow-up question.

Before-and-after comparison

Measure the same population before and after. Note other changes that could affect the result.

Comparison group

Where fair and operationally possible, compare similar groups receiving the intervention at different times. Avoid withholding necessary safety or compliance controls for evaluation purposes.

Cohort or time-series analysis

Review several cohorts or repeated measures. This can show whether change is sustained and whether seasonal or operational variation matters.

Qualitative case review

Use interviews, observation and work evidence to understand why a change did or did not occur. This is especially useful with small populations or complex roles.

The UK Government Project Delivery Teal Book emphasises defining outcomes and evaluating whether training addresses need. The same discipline applies beyond project delivery.

Set a baseline

Before launch, record:

1. current performance and variation;
2. population and data definitions;
3. measurement period;
4. known data-quality issues;
5. related changes underway;
6. current process cost or effort;
7. existing knowledge or competence where relevant.

If the baseline is unavailable, be honest. Start measurement now and avoid retrospective precision.

Use leading and lagging indicators together

Leading indicators show whether the conditions for change are forming. Examples: correct assignment, manager check-in, practice completed, access resolved.

Lagging indicators show later results. Examples: error rate, audit finding, service quality or readiness time.

A leading indicator gives time to act. A lagging indicator confirms whether the expected result appeared. Neither is enough alone.

Avoid common metric traps

Average completion without a denominator

Define the population, date, exemptions, leavers and status rules.

Knowledge score without a valid assessment

Easy or repeatedly memorised questions can overstate learning.

Self-confidence as competence

Confidence matters, but compare it with demonstration or work evidence where performance is important.

Activity volume as value

Courses launched, hours watched and reminders sent describe operation, not benefit.

One business metric as proof

Sales, incidents, retention and customer outcomes have many influences. Use contribution language and examine alternatives.

Only reporting positive results

Record limitations, unintended effects and groups that did not benefit. Useful evaluation improves decisions rather than defending the programme.

Build a small evaluation dashboard

For each priority programme, show:

1. intended outcome and population;
2. reach and access;
3. learning evidence;
4. application evidence;
5. outcome measure;
6. comparison or baseline;
7. confidence and limitations;
8. next decision, owner and date.

Keep operational drill-down available for managers, but give leaders a concise decision view.

A worked example: new manager conversations

Outcome

New managers set clear expectations and hold timely performance conversations.

Measures

1. Reach: correct managers assigned; access and completion.
2. Learning: scenario assessment against conversation criteria.
3. Application: manager's manager observes or reviews one conversation plan at 30 days.
4. Outcome: sample quality of documented objectives and avoidable escalation themes at 90 days.
5. Context: workload, HR process changes and manager support.

Decision

Continue if application improves; change the pathway if knowledge rises but observation does not; address process or manager capacity if opportunity is the barrier.

Use LMS data as one evidence source

An LMS can provide assignment, participation, assessment, pathway progress and selected manager actions. Join those data with relevant work evidence rather than treating the platform as the full evaluation.

Cademi's current platform pages describe live dashboards and role-based pathways. In a tailored demonstration, ask how one programme's data can be exported or combined with the organisation's agreed application and outcome measures.

Frequently asked questions

Is the Kirkpatrick model the only way to evaluate training?

No. It is a familiar structure, but use any proportionate method that links need, evidence and decision. Avoid applying a model mechanically.

When should we measure application?

Choose a point when people have had a realistic opportunity to use the learning. Some tasks can be observed immediately; others need weeks or months.

How do we prove ROI?

Use a transparent benefit and cost method, agreed baselines and cautious attribution. Not every valuable programme needs a monetised ROI.

What if completion is low?

Investigate assignment accuracy, access, protected time, relevance, manager support and communications before labelling learners disengaged.