A company builds a course for new hires: a few videos, some slides and a quiz at the end. Everyone finishes it and the survey scores are high. Three weeks later, new hires are still asking the same questions in chat. That is an illustration, not a case study, but it describes the gap this guide covers: a finished course is not the same as a changed way of working.
This guide explains why onboarding courses fail and how to measure whether yours works. It covers what failure means, what learning research says about passive design, a four-test audit, why satisfaction surveys mislead, what to measure instead and a table of warning signs. We found no study that measures how often onboarding courses fail, or which course design best predicts retention or productivity. The advice rests on learning research, a training evaluation model, newcomer research, one blog’s criteria and our reasoning.
What It Means for an Onboarding Course to Fail
We define failure as people finishing the course without changing what they know or do. A course can have full completion, friendly feedback and a tidy dashboard and still fail by that test. That definition is ours, not a standard, but it keeps attention on outcomes and not activity.
Failure usually shows up in one of three places. New hires cannot recall what the course covered a few weeks later. They remember it but cannot apply it to real work. Or they apply it but are still unclear about their role. Each needs a different fix, so measure each one separately. A practical first step is to write one sentence per module that starts “After this, a new hire can,” and to finish it with something you could watch them do. If you cannot finish the sentence, the module probably teaches information and not a skill.
Why Courses Fail: Passive Design
A large review of learning techniques gives a clue. In a 2013 review in Psychological Science in the Public Interest, John Dunlosky and colleagues rated ten techniques. Underlining, rereading and mnemonic devices were found to be “of surprisingly low utility.” Practice tests and distributed practice, meaning study spread over time, were found to be “of high utility” and “benefited students of many different ages and ability levels,” according to the Association for Psychological Science summary.
The limits matter. The summary page does not list all ten ratings, and the review concerns students, not new hires at work. Our reasoning is that many onboarding courses resemble the low-utility techniques: content is presented once, reviewed by clicking through and checked in a single sitting. If so, they have little of the practice testing and spacing that the review rated highly.
Audit Your Course Against Four Active-Learning Tests
Boterview’s guide to active learning argues that “Most learning apps are passive, but have been labeled as active learning tools.” It sets four tests for real active learning: recall rather than recognition, explained mistakes, difficulty that adjusts to performance and spaced repetition. The page is written for people choosing study apps, and it is published on the site of its author, who says he built Boterview and ranks it first. It cites no studies and describes no testing method, so the criteria are the author’s own. Treat them as a checklist, not as evidence.
Applied to onboarding, the tests become four questions:
- Recall: Does the course ask people to produce answers, such as writing the steps of a process, or only to pick from options?
- Mistakes: When someone is wrong, do they see why?
- Level: Can an experienced hire skip what they already know?
- Spacing: Do key points come back days later, not only on day one?
Two of these have support in the research above, since practice testing and distributed practice were both rated highly. We did not find support in the sources we read for the other two, so we label them as the Boterview author’s view and our reasoning. To run the audit, take one module and label every activity as watch, read or produce. Count how many ask the learner to produce an answer. If almost none do, the module is passive by this checklist, whatever the quiz at the end looks like. Our guide to the employee development plan shows how to carry learning past the first week.
Why Happy Surveys Mislead
Most course evaluation stops at reactions. The Kirkpatrick model, developed by Donald Kirkpatrick in the 1950s, has four levels: reaction, learning, behavior and results. According to Training Industry’s summary of the model, approximately 80% of training events include a level 1 evaluation. The figure is undated and secondary, so read it as a rough sign that reaction surveys are common.
Reactions are a weak guide to learning. A 2022 QIC-WD umbrella summary reviews two meta-analyses. It says that, in Sitzmann and colleagues (2008), “Reactions are only modestly related to immediate knowledge outcomes and unrelated to delayed knowledge.” It adds that in Blume and colleagues (2010), utility reactions, meaning how useful trainees found the course, had a modest connection to transfer to the job, and affective reactions, meaning how much they liked it, had none. Reactions did track motivation and self-efficacy.
There are limits. The summary reports no effect sizes, says the link to skill development is unknown and notes that no reaction measure is widely used. It suggests a survey is useful for motivation and for spotting confusing content, but not as proof that people learned. A “high satisfaction” score tells you the course was liked. Keep the survey, since it is cheap, but add one question that points toward behavior, such as what the new hire will do differently this week.
Measure What Actually Works
Use the Kirkpatrick levels as a frame, since they separate liking from learning from doing from results. These measures are our suggestions.
At the learning level, test recall two or three weeks after the course, not only at the end, and ask for written answers. At the behavior level, check a real task, such as reviewing a new hire’s first piece of work against a checklist. At the results level, track a business measure the course is meant to move, such as time to first independent task or repeat questions to support.
Newcomer research adds three more measures. A 2020 QIC-WD umbrella summary of meta-analyses, including Bauer and colleagues (2007), says newcomer adjustment is measured mainly as role clarity, self-efficacy and social acceptance. It reports that socialization strategies were moderately associated with job satisfaction, commitment and intentions to quit, and modestly with job performance and turnover. It also warns that the evidence is correlational. We suggest asking new hires about all three at 30 days.
The Kirkpatrick page notes that the hard part is showing that an outcome came from the training. A simple answer is to compare a cohort that took the revised course with an earlier one, while remembering that other things change between cohorts, such as team size, tools and the manager. Small teams may have only a few new hires per quarter, so look at the trend over several cohorts and not at one result. Our onboarding statistics page gives wider context, and the piece on measuring onboarding time with AI knowledge tools covers support teams.
A Four-Step Plan to Fix and Measure Your Course
Use this four-step plan for every onboarding course. It is our suggested process, informed by the sources above.
- Audit the course. Check it against the four active-learning tests.
- Set the behaviors. List what new hires should do differently and how you will see it.
- Test recall later. Quiz key points days or weeks after the course ends.
- Compare cohorts. Compare new hires before and after each change.
The order matters. Auditing first stops you measuring a weak course carefully, and setting behaviors before testing keeps the quiz tied to work and not trivia. Our guide to job orientation covers the first week, and the training matrix helps track who needs which module.
Warning Signs Your Onboarding Course Is Failing, and What to Do About Each
Check this table at each course review and after each new cohort. One sign means fix one thing. Three or more mean pause and review the course.
Measure the Behavior, Not the Completion
An onboarding course works when new hires remember it, use it and are clearer about their role. Audit the course for passive design, set the behaviors you want, test recall after a delay and compare cohorts. We found no study showing how often courses fail, and no proof that this plan fixes it, but each step replaces a guess with evidence. If the numbers improve after a change, you have a better reason to keep the course than a high survey score.
