TRAINING INDUSTRY ON TRIALEvaluating Training The New World Kirkpatrick Model
In today’s financially challenged environment, expenditure (like a training department) can either be justified in clear business terms or it is at risk of being eliminated.
The latest release of the book “Kirkpatrick’s Four Levels of Training Evaluation” does an excellent job of explaining why measuring and evaluating training programs is simply not optional, but an absolute necessity. As a result, and since the inception of Skypiom, we have advocated the concept of measurability in the training environment and we have attempted to articulate our believes and philosophy in different ways, including by means of integrating relevant functionality within the KMS.
The “Four Levels” were originally published by Dr. Donald Kirkpatrick in the form of four separate articles in 1959. The articles appeared in the “Journal of the American Society of Training Directors”. Following widespread acceptance and adoption of the principles described therein, Dr. Kirkpatrick published his first book on the subject “Evaluating Training Programs” in 1993.
When he passed away in 2014, his eldest son Jim with his wife Wendy continued his work and developed the “New World Kirkpatrick Model”, resulting in the release of “Kirkpatrick’s Four Levels of Training Evaluation”. It is quite remarkable that the original model – developed 60 years ago – is more applicable than ever.
WHY EVALUATE TRAINING PROGRAMS?
Many corporate training programs are created without a full understanding of what the desired outcomes are. Such programs are then administered mostly in isolation, meaning with no measurable links to desired behaviour changes and consequently no way of knowing to what degree a program has been successful.
It is a known fact that even if a training program has been effective, i.e. knowledge has been imparted and this has been reliably demonstrated, there is absolutely no guarantee that there is a benefit to the organisation. More often than not, such benefit cannot be observed. Instead, the organisation will only benefit if the training leads to change in behaviour. Consequently, when evaluating training programs, simply measuring the effectiveness of training is not sufficient and a much more elaborate approach is required.
For a training program to truly deliver benefits to the organisation, the evaluation process must be part of the overall design and needs to be “baked in” from the early planning stages to post training procedures and monitoring. Failing to evaluate on a business level may easily result in a training department unable to demonstrate value and experience diminished funding as a consequence.
The authors of the book are quite blunt about such scenarios:
Maxine, a training specialist with a major corporation for seven years, could not believe her eyes as she read the letter that the human resources representative had just handed her:
The Training Specialist position has been eliminated as part of a necessary reduction in force to align company resources with business needs. Your last day of employment is today …
Maxine liked her job in the training department. She really felt that she had helped the business by creating good training programs and was particularly surprised to receive the notice because she had never said no to any training request sent her way.
She thought she was a great team player. How did Maxine get blindsided in this way?
Another example form the book:
Jim Kirkpatrick was invited to a technology company in Canada to do a workshop for the training group. The topic was the risk of failing to demonstrate the organisational value of training. Throughout the program, interest seemed low. When Jim probed a bit, one participant stood up and proclaimed, “Jim, we are the exception. Look around you. The organisation built this campus for us because they know that we are bringing value.”
Jim continued the workshop, but interaction was still lacklustre. At the end of the presentation, Jim offered to send more information to those who left a business card. Just one woman snuck him her card and slunk out of the room.
Six months later, the woman who provided the business card emailed Jim. “Remember those 100 people you spoke with a while ago? 75 of them are gone. Leadership had said there was no evidence that the value that training had brought warranted the cost”.
This is not an isolated incident; we could tell you more, similar stories. Do not let this happen to you and your department.
The point being made in the book is that creating and demonstrating value is an absolute necessity if you want to ensure the long-term survival of your training department. Personally, I would add another reason: Believing that you are adding value through what you do may be ok, knowing that you add value and being able to back this up with hard evidence is surely infinitely more satisfying and motivating.
THE 4 LEVELS IN BRIEF
The Kirkpatrick model appears rather simple at first; so, let’s look at what it entails. We will list the 4 levels in reverse order, since this better reflects the recommended way of dealing with the levels.
Level 4: Results
The ultimately desired results, defined as business objectives, using language understood by business- as opposed to training people.
Level 3: Behaviour
The behaviour patterns displayed by students which are deemed necessary to affect the results as defined in level 4.
Level 2: Learning
Metrics related to learning, i.e. stating the degree to which some knowledge and / or skills are present within students.
Level 1: Reaction
“Hygiene” factors and “student satisfaction”. This primarily refers to the student’s view of the training programme, i.e. how well was it presented, the state of the facility and the perceived benefit.
Research has shown that where training departments performed training programme valuation, this has mostly occurred at Levels 1 and 2. The reason is quite simply that those are relatively easy to measure, whereas Levels 3 and 4 have proven to be much harder to evaluate. The sad thing is that the most value for the organisation is created at Level 3 and again, this is backed by research: Robert O. Brinkerhoff had studied the subject matter extensively and in 2006 stated that:
A parallel group of students which was subjected to a different approach which included extended managing efforts before and after the training event, fared much better in that 85% of that group was able to apply on the job what they had learned. It is important to note that overall time spent by the training professionals was the same in both cases; it was just utilised differently.
It is important to note that overall time spent by the training professionals was the same in both cases; it was just utilised differently. What is important to understand in this context are the following points:
Many training departments spend most of their time creating training programs and administering them, when there is only a 15% benefit.
Successfully managing a training programme goes far beyond the actual training event. It starts at the planning phase and extends well beyond the training programme, to evaluate and monitor changes of behaviour on the job.
Real benefit from training only occurs if it results in changed behaviour, i.e. students are able to apply on the job what they have learned in training.
Help is required from various sources (Business Partners) to ensure that a training programme succeeds in adding value to the organisation from a business perspective.
Given the above, it could be expected that some corporate training professionals question their ability to execute on a wider scale, using their own interpretation of their current job description. There can be no doubt however, that the overriding interest of any organisation is to ensure that every department adds value to the overall business. Operating in isolation and creating little kingdoms will ultimately be detrimental to the overall business.
THE IMPORTANCE OF BUSINESS PARTNERS
Many people (myself included) will think of “external” companies like vendors, consultants, etc. when hearing the term “Business Partner”. In the Kirkpatrick context however, Business Partners are those people and / or departments within your own organisation, whom you need to work with to achieve your overall goal of creating and demonstrating value. Simply stated: Without those Business Partners a training department cannot create and demonstrate value.
So who are those Business Partners?
Well, anybody who has data you may need, can provide you with information and needed insights and who is in any way connected to your planned training programme. It all starts at the planning phase. Whoever is requesting a new training programme is the first of those Business Partners. Rather than obtaining some simple instructions as to what the training programme should include, you want to start by defining the desired results (Level 4). While it is important to define those results, it is often difficult to measure them directly. For that reason, the Kirkpatrick model caters for “Leading Indicators”. As an example, a desired result may be defined as “increased profitability” and one of the leading indicators may be “sales volume”. Whatever it is, you will need Business Partners not only for the definitions, but also for obtaining the relevant data required to evaluate your programme.
Once the desired results and leading indicators have been defined, it’s time to look at Level 3 issues. This is where most of your effort should go, because that is where the largest gains are made or lost. Defining which behaviours are required to change and to what extend is an important step. At the same time, you need to think about how to monitor and measure those behaviours. Another very important aspect is the creation of drivers and supporting measures to ensure the newly acquired knowledge or skill is indeed transferred to on-the-job behaviour.
Without drivers and supporting measures, the “transfer ratio” will be low, resulting in lost opportunity to create value. Again, you will need Business Partners for this important part of the overall project.
Your reliance on business partners lessens for Level 2 and Level 1 definitions and planning. Ensuring that measurability is defined for Levels 1 and 2 is still important, but a lot easier to achieve than for the higher levels. Planning and executing on these levels does not typically represent major challenges; it is after all what training departments have always done.
Care should be taken in not putting too much emphasis at Level 1 evaluations. While learner satisfaction and feedback are valuable criteria, the recent obsession with things like gamification and similar developments is often not in line with delivering value. Level 1 issues have a relatively small impact on overall value created, dictating that resources expended in this area need to be in line with the expected value creation and in relation to resources expended in other areas like driving Level 3 issues.
Finally, you will need Business Partners for the overall planning, setting realistic timelines, follow-up activities, ongoing programme monitoring, etc. You also need them for the ultimate determination of value created and to gather evidence of such.
THE MEASURE OF SUCCESS
So what is the ultimate measure of success of a training programme? We have already stated that it is necessary that value was created and that we were able to demonstrate this. But how exactly do we measure it? Business people tend to use “Return on Investment” or ROI as the true indicator of value added. Simply stated, ROI measures profitability and is expressed as a percentage: ROI = (Net Profit / Cost of Investment) x 100. How could this be applied to training? Well, let’s assume that a training programme costs R 1 million to implement and manage. If it can be shown that the programme was responsible in saving the company R 1.2 million, we would report an ROI of 20% (R 200,000 profit on an investment of R 1 million). It follows, that if that same programme resulted in only R 800,000 of savings, the ROI would be negative; not a good outcome.
As it happens, the Kirkpatrick methodology does not promote ROI as a final measure of success. I quote: “ROI is a defensive and reactive tactic that calls for little or no collaboration between training and the business. Rather, there is more of an effort to separate the two so that training professionals can ‘take credit’ for what they did without taking into account any efforts of business leaders, supervisors, training participants, and others who assisted with the support and accountability package after the training.”
Instead of ROI, the Kirkpatrick methodology favours another measure termed “Return on Expectations” or ROE. This measurement is seen as proactive and serves for the training department and its Business Partners to jointly define expectations. A major advantage is that ROE can include many factors which are not easily measured in financial terms, sometimes simply due to timing reasons. Making it possible to include valid business targets directly (example: measured customer satisfaction index) provides for a much easier and more definite determination of achieved results. Most important however is the fact that ROE allows all contributing parties to jointly take credit for what has been achieved.
THE BUSINESS ASPECT
Training professionals may only have had limited exposure to different business disciplines. Still, for training departments and individuals to “be appreciated”, it is vital that training professionals become acquainted with business terms and procedures. Speaking “business language”, demonstrating understanding of the overall business objectives and showing initiative in working towards stated objectives all go a long way towards gaining the respect of business leaders. While many roads may lead to Rome, you could do worse than deciding to follow the Kirkpatrick model and methodology.
Consistent report-backs on major initiatives and final evaluation reports will go a long way towards earning well deserved recognition. Reports and communication to business leaders should be free of training jargon, be concise and to the point, and include factual data.
Above all communications must be short enough to be read (business leaders are busy) using graphical depictions where applicable and be centred around business objectives.
Collecting and retaining core data for as long as possible will help when it becomes necessary to construct chains of evidence in the future, or when such data may be valuable to support potential new projects.
While it is desirable to perform such data collection on an ongoing basis, care should be taken that whatever resources are expended in performing such data collection can be justified in business terms. Fortunately, with today’s technology, the cost of such speculative data collection and retention can be at or near zero.
IMPLEMENTING THE METHODOLOGIES
I know, it sounds like a major effort to implement the proposed methodologies, but you can get lots of help. For starters, the book is of great help, not only because it explains the various concepts in great detail, but also because there are real-life examples of successful implementations. If you’re looking for specific detail, there are even example forms and surveys to borrow ideas from.
Separate chapters in the book are devoted to each of the four levels, dealing with the collection of data, guidelines, methods and tools. A separate part of the book deals with “Data Analysis and Reporting Basics”. Another section covers “Reporting Progress and Demonstrating Program Value”.
Thousands of training professionals and departments have implemented Kirkpatrick’s methodologies during the last 60 years and the model wouldn’t have become the most widely used training evaluation programme in the world if it was too difficult to implement. Sure, mistakes have been made along the way, but you will be pleased to know that the current revision of the book has a separate section which explains the most common mistakes made over the years and how to avoid them.
By now you may be wondering what role Skypiom wants to play within this context. Well, as mentioned above, since its founding, Skypiom has promoted measurability above any other aspect when talking about its KMS. In fact, Skypiom’s “Elevator Pitch” specifically mentions measurability as the major differentiating factor in the vast field of LMSs. Under the hood, our system architecture is deeply data centric, while other systems tend to put more emphasis on content delivery. Referring back to the Kirkpatrick model, our KMS has had very solid Level 2 data collection capabilities from the beginning. It has also always had capabilities to collect Level 1 and 3 data, but not in an automated manner. Recently, we have upped the ante by introducing Level 1 and Level 2 data collection capability which makes it a lot easier (and less resource intensive) to collect and analyse such data.
Besides direct system support for Levels 3 and 4 evaluation, Skypiom can assist with various service based options, including specific custom reporting, programme management and consulting services.
While we truly believe in Kirkpatrick’s model and methodologies, we are not restricted by them. Our core objective is to help our customers create business value through training. Where we see a need to deviate from or extend Kirkpatrick’s methodologies, we will do so in the interest of individual customer requirements. We want to be the “go to” vendor for all organisations who share our (and Kirkpatrick’s) philosophy. It helps to talk to somebody who speaks the same language.
Evaluating training has always been important, but has become more so in recent times. When times are tough and budget cuts are imminent, the training department is typically one of the first targets.
Ensuring that value is not only created but can be demonstrated is vital. Fortunately, there are models and methodologies to help design, implement and evaluate training programmes that deliver against predefined expectations. At the same time, the availability of system and service support will provide further peace of mind when deciding to implement what may first appear as a major task.
The Kirkpatrick book recommended has a lot of detail, right down to sample forms and various relevant and recent case studies, to help with implementing the methodologies. Finally, a proper Kirkpatrick certification is yet another option to become aligned with what is required.
The first step however would be for you to decide that you are in agreement with what is presented here and in the book. Once you have made that decision, figuring out how to get there is not that difficult anymore. While the Kirkpatrick model and methodologies will serve you well – with or without Skypiom – we would love the opportunity to be of assistance in implementing whatever course of action you have deemed appropriate for your organisation.