It’s always a good idea to use Training methods that are (1) based in evidence and proven to be effective and (2) focused on improving performance on the job.

Dr. Will Thalheimer is one of the leading research experts on evidence-based and performance-focused training methods, and he’s been kind enough to grant us an interview that we’ll present as a series of four related articles.

In this article, our interview with Dr. Thalheimer will focus on smile sheets.

You may already know the term Smile sheet, but if you don’t, you’re probably familiar with the concept. You know those surveys trainers hand out to learners who have just completed some form of training? And the learner then uses the survey to evaluate the training event, materials, and instructor? That’s a smile sheet, also known as a training evaluation sheet, reaction sheet, or a Kirkpatrick level 2 evaluation.

These learner evaluations are sometimes called Smile Sheets, however, because there’s a belief that the learners may not use them to put down truthful, objective, helpful information, but instead just write nice comments about the training and the instructor that are meant to make the trainer smile about supposedly having done a good job.

So you see the problem. If a smile sheet is nothing but a bunch of well-intended but fake or meaningless “smiles,” we’re not drawing helpful information from learners about the training material that we can use to evaluate the training and revise it if necessary so our training has a desired influence on worker job performance. And that’s why Dr. Thalheimer has done research on smile sheets, first to determine that as commonly written they’re often meaningless and second to give us tips on how to write better smile sheets that will help improve performance.

If all this is ringing a bell for you, you may be familiar with Dr. Thalheimer’s great book Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form or with the earlier article we wrote exploring his book on how to write better smile sheets.

So let’s check out the interview. Remember this is one part of a longer interview with Dr. Thalheimer, and we’ll have additional articles in which he discusses:

• Spaced practice
• The effectiveness of elearning
• Evidence-based training methods & learning myths

Where Have Smile Sheets Gone Wrong?

Convergence Training: In general, how reliable has the information been that trainers get from traditionally written smile sheets?

Dr. Thalheimer: They are 100% reliable in that they’re getting very bad information. So, the distinction is between reliability and validity, right? So it’s reliable because we’re getting the same information all the time, but it’s information that doesn’t really tell us very much. It certainly doesn’t tell us very much about the effectiveness of our learning interventions. It DOES tell us information about customer satisfaction, but just because learners are satisfied with the learning, or they are willing to make a recommendation to other people about the learning, does not mean that the learning is any good.

In fact, there’s science to back this up. There has been two meta-analyses in the last couple of decades on this looking at over 150 scientific studies, and what they found is the correlation between smile sheet ratings and learning results at 0.09–which, if you remember from your statistics, means there’s virtually no correlation at all.

You will remember from your statistics that correlations go from -1 to 1. Correlations between -.30 and .30 are considered weak correlations. Having a correlation of .09 is practically no correlation at all. It would be like correlating the household level of peanut butter use with the household level of television use.

Dr.Will Thalheimer, from his book Performance-Focused Smile Sheets

So what that means is you can get high marks on your smile sheets you could have an effective course or an ineffective course. You just really can’t tell with traditional smile sheets.

So they’re very VERY reliable in the sense that they’re always giving us the same information. In fact, that’s one of the problems with the way we use smile sheets, is that most of our ratings are very good. On a five point scale, the scores tend to be between a 3.8 and a 4.6, and that’s all of the responses we get. And if everything is so close together, then there’s no differentiation, right? So we can’t really tell what’s good and what’s bad, because it all looks about the same, and so we don’t do anything. So it creates this sort of analysis-paralysis…not even analysis, really, but just paralysis.

Convergence Training: To that point, in hopes of creating a more effective smile sheet that tells me if my training was effective and how to modify it if it was not, what are some of the common mistakes people tend to make when they’re creating questions for those smile sheets.

Dr. Thalheimer: One of the biggest mistakes is that use Likert-like scales, or numeric scales. And the problem with those is it makes it very difficult for the learner to think about their answer choices and give us precise information.

You know if you’re on a numeric scale, 1-5, 1-7, 1-10, it’s very fuzzy choosing between a 3 and a 4 or a 4 and a 5. And the same with a Likert-like scale–“strongly agree” v. “agree,” there’s just not that much differentiation there. And so it makes it harder to think about, which gives us data that’s not as precise, which makes it so that when we go to present the information, it’s not that clear what it means, and the whole reason we’re assessing in the first place is to find out what to do, or what it means. And if that whole process is messed up and fuzzy, then that’s problematic. So that’s one thing, the choices that we give learners are not very clear.

The other thing is we tend to focus on things that are not related to learning effectiveness. We can really ask three types of questions on a smile sheet. We can ask questions that get at learner satisfaction and the reputation of the course–that’s one bucket. We can have questions on there (and this isn’t done that often) that focus the learners on the goals they might have for taking the course and putting it into practice, so questions that encourage additional effort or learning on the learners part, that’s the second bucket. And the third bucket would be questions that focus on the effectiveness of the learning intervention, and that’s where my work is really focused, with my book Performance-Focused Smile Sheets. 

We know, there’s research on learning out there now, and in fact in the last 10-20 years the science of learning has really solidified. And so now we have some very clear recommendations. We know, for example, that the learner’s motivation to apply what they learn makes a BIG DEAL about whether they’re actually going to transfer the learning. Or, if there’s after-learning follow-through, or if the learning is supporting remembering. These are things that are critical, and if there’s a way to ask questions (and I’ve developed methods to ask these types of questions), then we can get better information.

Now, I always say look, smile sheets can only get us so far. They should not be the only thing we measure. And too often, that’s what organizations do, is just use a smile sheet. Smile sheets can be better than they traditionally are, but they’re not the be-all and end-all. We should do them, but we should do more than that.

How We Can Make Better Smile Sheets

Convergence Training: So if I earlier asked you what people are doing wrong with their smile sheets, and you mentioned the use of Likert-like scales and focusing on things that are not related to learning effectiveness, what then are some tips you’d give for creating better smile sheets and better smile sheet questions, in particular those that are actually measuring learning effectiveness and also giving us the kind of actionable and useful information you’re talking about?

Dr. Thalhimer: There are some factors that are SO IMPORTANT to learning effectiveness that we really should be focusing on those. Now, there are hundreds of learning factors that make a difference, but some make more difference than others.

So, in my book I offer what I call the Four Pillars of Training Effectiveness. And I say at a minimum, you want to have questions on each of these four things.

The first is learning understanding and comprehension. Did they get it? Did they learn new information? Is that new information clear? Do they feel they’ll be able to use it. So that’s learner understanding.

The next one is learner remembering. Are we doing things in the training that are likely to increase remembering? Now, the best way to measure this is to actually measure if people can remember things after three days or a week or a month or something like that. But on a smile sheet, we can ask questions that get at the factors that enable that kind of remembering. So, particularly realistic practice is something we should focus on, because it’s one of the most important things to support long-term remembering. We could in addition focus on things like the spacing effect on so on, but that’s a little more complicated to do.

So the third thing is motivation to apply what you’ve learned. I have a question I recommended in the book–a lot of my questions are evolving over time as I try them out, as we get feedback from pilot tests and things, but the wording of this one question has stayed very solid throughout.

And so the fourth bucket is after-training follow-through. Is your manager supporting you? Are you going to get coaching when you return back to the job? Are you going to be reminded of what you’ve learned? Do you have job aids and performance support?  Or are you going to do this all on your own? And we know that even if people have good intentions, they may not follow through if they don’t have this kind of support.

So those are four big areas to focus on.

Other mistakes that people make are asking biasing questions. You do that by asking a question like “What are the greatest benefits of this learning program?” or “How has this training increased your productivity?” Those are leading questions, a lot of people do that, they don’t even know they’re doing it, but they do it.

Another problem is people write questions which don’t give any negative answer choices. So, if all the answer choices are positive, if anybody bothers to answer the question, you get positive results, because all the answer choices are positive.

Another thing that people do is they focus on what you might call learner-centric questions: the learner’s feelings and things like that. And those tend to not be related to true effectiveness. In fact, there’s very good research to show that learners don’t always know their own learning. And that’s why it’s very important how we write our questions. Write them in such a way that they are the kind of things that learners can wrap their heads around, that they’re going to be good at answering. You know, sometimes we ask questions like “Predict the business results that you’ll see as a result of this course. Will you lower costs, increase revenue, increase morale, etc.?” It’s a bunch of things like that that people have no idea really, they aren’t really good at answering those questions, but we ask them anyway, and then we get results that look good but the learners don’t really have great insight into those things, so we have to be really careful to ask questions that learners can answer with some degree of accuracy.

Convergence Training: And if I recall from your book, if a bad answer option would involve a numerical answer scale or a Likert scale, a better answer choice would be specific answer options like “This training prepared me for the job task very well,” “This training raised my awareness and understanding but I’m not yet competent to perform the task on the job,” or something like that.

(Note: Read our earlier article about Dr. Thalheimer’s book and Dr. Thalheimer’s blog for some examples of question and question answer options like this.)

Dr. Thalheimer: Absolutely.

Where Smile Sheets Fit in With the Larger Training Evaluation Process

Convergence Training: Assuming people get religion and start writing good smile sheets with the examples and techniques you’re providing, how do you see the smile sheet fitting within a larger framework for the training evaluation process?

Dr. Thalheimer: That’s a great question. So, I think almost every course can use a relatively short smile sheet. But we ought to do more.

We ought to take it further and see if learners really do understand the material. Not just by asking them knowledge questions, but also by asking them to make decisions–maybe scenario-based decisions, scenario-based questions, what I used to call simulation-like questions. Or whether they can actually do tasks that are related.

So level 2, when we think about Kirkpatrick-Katzell’s four levels of evaluation, we tend to think of that as knowledge, but there should be decision making, and there should be task competence built in there as well.

In addition to that, there’s whether people are able to transfer what they learn and apply it at work. We ought to focus on that. But not just that. We should also be evaluating the supports they have to enable that. Nobody talks about that. But if we are serious about doing evaluations, we need to look at things like “Is the manager supporting the learner,” and “Do they have enough time and resources,” and “Are we providing the training too far in advance so they’ve forgotten things when they need the information?”

There are a lot of things that we could be tracking in addition to just performance results. And let me say one other thing about this. If we ask a learner a question, so let’s say we ask a learner a question about whether they’ll be apply to apply the training to their job, that is not a level three question. When you ask a learner what their perspective is, that is level 1. If you ask them “Are you going to be able to put this into practice at your work,” that’s not level 3. When you ask them “Will this increase sales,” you might think that’s level 4, but that’s not level 4, that’s still level 1, because you’re asking your learner. So that’s a big misconception that people have.

The other thing to put this all in perspective, and actually I’m in the middle of working on a new learning evaluation model, but the four levels has pushed us to do some good things, like focus on results and focus on behaviors and business results, that’s great. But at the same time, it has left a lot out, so it tends to focus us on some things that are not as useful.

Here are some early fruits from Dr. Thalheimer’s research on a new model for learning evaluation:

• What business leaders want in terms of learning evaluation 
• Don’t ask learners for level 3 and level 4 training evaluations
• Replacing the net promotor score for learning evaluations
• Kirkpatrick didn’t create the famous four-level evaluation model
• Quiz/survey: What is the purpose of workforce learning & development? 

For example, the distinction between understanding and remembering. And a good model should push us in the right direction. And so a good evaluation model ought to have that in there. So anyway, I’ve been working on one, I’ve gotten advice from many people, I’m going to be sharing this at a couple of conferences coming up to make it even better, so that’s coming down the pike.

Convergence Training: Great, we look forward to that. And I’m not shocked to hear you talk about the distinction between understanding and remembering, and later in this interview I’ll be asking you about spaced learning, which gets right to part of that. But before we move on to spaced learning, and before we leave the issue of smile sheets, is there anything about smile sheets that I should have asked but that we’ve missed?

Dr. Thalheimer: Well, there’s a lot more to be said about smile sheets, but this is probably a good introduction. But I encourage people to start here and keep working on making their own smile sheets better.

Convergence Training: Absolutely. Well, thanks again for the interview. And for the readers, keep your eyes out for three more articles based on this same interview with Dr. Thalheimer.

Conclusion: Evidence for Better Smile Sheets That Measure and Improve Job Performance

We hope you’ve found this interview with Dr. Thalheimer about smile sheets interesting, and we hope you can put some of it to use to (1) improve your own smile sheets, (2) get more meaningful and actionable information from learners after training, (3) more effectively evaluate your training, and (4) revise training when necessary so it has a better end result on job performance.

If you’re excited about doing this and are wondering how to create, deliver, and crunch the numbers on your smile sheets, there are at least three ways to do it:

1. Old-fashioned paper-based systems. Passing out paper-based surveys has worked since we ditched stone tablets and cuneiform and will continue to work. A benefit is that it’s relatively simple and it’s accessible to companies with limited or no access to computerized and/or online training technologies. On the flip-side, this comes with a lot of antiquated clerical and analytic tasks that aren’t necessary with computerized systems.

2. Online survey sites. There are plenty of online sites that you let you create surveys questions (and answer options) as we’ve been describing, that make it easy for learners to complete the survey, and even provide reporting and analytic tools. And even better, plenty of them are free or offer a free option that’s pretty powerful. For example, check out SurveyMonkey. I am admittedly not an expert on all these, so if you’re reading this article and have a system like this you’d recommend, by all means, that’s what the comments section at the bottom is for!

3. Your learning management system (LMS). Many learning management systems have some form of built-in survey making tool as well. If you’re evaluating an LMS, it might be something to ask about. If you’ve got an LMS now, check and see if you already have this capability.

See you all around next time! And one more thanks to Dr. Thalheimer.

The post Evidence-Based Training Methods and Smile Sheets (An Interview with Dr. Will Thalheimer) appeared first on Convergence Training Blog.