The classic model of evaluation put forth by Donald Kirkpatrick (1994) comprises four levels: Reaction, learning, behavior, and results.
By looking at evaluation from these four perspectives we can address the who, what, when, why, and how of the evaluation and measurement process. All perspectives are important and no level should be by-passed.
Level One: Reaction
Let’s start with the easiest and most common level of evaluation – learner reaction.
At this fundamental level, the ETD practitioner solicits formal feedback from the learner to help determine the training’s effectiveness and how it can be improved.
Many ETD practitioners dismiss as unimportant these so-called “smile sheets.”
Some regard them as merely popularity indicators that contribute little to the evaluation process.
Kirkpatrick sees Level One as particularly important because “if they [learners] do not react favorably, they will not be motivated to learn” (Kirkpatrick 1994).
One way to measure ETD practitioner effectiveness is to develop a questionnaire for both the ETD practitioner and the learner to complete at the end of the training period (Lawson 1994).
The questionnaire compares perceptions and uncovers communication problems that may be sabotaging the training efforts.
Level One evaluation addresses the design, delivery, instruction, training site, perceived learning, and the learner’s level of satisfaction with the training experience.
When creating a reaction questionnaire, be sure to design questions to assess the ETD practitioner’s effectiveness in the following areas:
conducting the training
reinforcing the training
The questionnaire is a way for each learner to offer specific and meaningful feedback to the ETD practitioner as a means of improving the ETD practitioner’s effectiveness.
It also helps the learner to reflect on the formal training and register his or her degree of satisfaction with it.
The questionnaire can also yield valuable information to the ETD practitioner’s manager who can use the data as supporting documentation for the manager’s evaluation of the ETD practitioner’s performance.
This type of questionnaire should be used at several times throughout the training process, not merely at the end.
Level Two: Learning
Level Two evaluation deals with what is learned, the knowledge and skills the learners retain.
It addresses the learner’s demonstrated mastery of the principles, facts, techniques, and skills presented in the formal training.
Evaluation of and feedback concerning the learner’s progress should be an ongoing process, both formally and informally.
Several methods can and should be used to evaluate the learner’s skill and knowledge of the job, task, or procedures that he or she is expected to perform.
First of all, the ETD practitioner can watch the learner performing the task during the initial training period, taking notes and providing feedback. Another method is to track various predetermined metrics such as number of errors, length of time to complete a task, measurable outputs, and so forth.
Periodic testing is another option. Any formal testing should be kept simple: Short quizzes at day’s end or some type of take-home assignment.
Developing test or quiz questions is not an easy task. They can be either subjective (e.g., short answer or essay) or objective (e.g., multiple-choice or true-false).
When writing questions, consider your learner’s learning style, the time needed to grade the tests, and both the validity and reliability of each item. Above all else, remember that a question should assess the learning specified in the training and test only those skills directly related to the objective.
Rather than asking for simple recall of information, such as definitions, ask questions that require learners to apply or interpret what they’ve learned.
In most cases you will probably choose to develop multiple choice questions. If so, make sure you follow these guidelines:
Avoid “all of the above” and “none of the above” in your set of answer options.
Make sure the stem (i.e., the main part of the question) contains most of the information and defines the problem, and place missing words near the end.
Maintain grammatical consistency or parallel structure for both the stem and the answer choices.
Try to create answer choices of equal length.
Avoid ambiguity and reading difficulty by stating questions in the positive rather than in the negative.
An informal evaluation is generally based on observation of the learner’s performance during the training period. Like any other feedback, it should be delivered in private immediately following the observed behavior. The purpose isn’t only to comment on the employee’s performance, but also to encourage open, two-way communication. When informal feedback is handled haphazardly, with seemingly little thought or caring, it can have very serious consequences.