INDEX to reviewing.co.uk - resources for dynamic learning
HOME
 How to find your way around reviewing.co.uk
HELP

COURSE EVALUATION METHODS

Evaluative questions and tasks for participants near the end of a programme

8 bits of advice + 42 course evaluation methods + 39 links


Introduction | How to use this list | Course Evaluation Methods | Evaluation Links |
OUTDOOR
TOUR
EVALUATION
INDEX
For a free monthly supply of reviewing and evaluation methods subscribe to Roger's Active Reviewing Tips. Learn how now.
To create a favourable climate for learning, creature comforts such as good food and accommodation may help. But if you want to be in the 'Good Learning Guide' as well as in the 'Good Food Guide' you will also want feedback on some of the questions listed below.

Evaluation methods are tools for change. If they are simply used as score sheets or as pats on the back, evaluation tools can lead to complacency - rather than to change and innovation. End of course feedback can play an important part in designing an evaluation strategy - but it is by no means the only part!

When to evaluate? This page lists some of the ways in which you can generate rich insights, creative ideas, evaluative discussions and other useful responses at or near the end of a course ('summative' evaluation). Many of these ideas can also be used for 'formative' mid-course evaluation. See the article below on Facilitated Student Feedback for some detailed advice on formative evaluation.

Issues in Evaluation You will also notice plenty of messages (in the left-hand column) that will remind you of some of the wider issues that are relevant to end of course evaluation, and to evaluation as a whole.

Next

How to use this list of course evaluation methods

  1. Read through the list in the right-hand column below and pick up to 5 ideas that appeal to you (and that will engage participants).
  2. Ask at least one colleague or friend to do the same.
  3. Discuss the benefits of using the chosen ideas, and shortlist between 1-5 ideas.
  4. Write down what you expect to learn, and how you will use the information.
  5. Consider presenting this information to the learners taking part in the process - so that they are fully informed about what you are trying to achieve - and how.
  6. Re-write your chosen ideas in a suitable style and format (e.g. as a briefing sheet for a task or as a form to be completed).
  7. Once you have planned your evaluation process, do it and review it (making use of the information generated during this planning process).
  8. Let me know how you get on (I appreciate feedback too!)
  9. Take note of the warnings and guidance in the left-hand column!

    Next

EVALUATION ADVICE

Be selective! Don't bombard learners with a huge list of questions. Work out what you really want to know. Then work out the best way of finding out.

Be realistic! Form-filling is not fun (however much people may have enjoyed your course). So don't expect people to conscientiously work their way through a long, complex evaluation form (even if it is on coloured paper!). And don't read too much into the results of a rushed process. The more care people take in providing feedback, the more notice you should take of what they have to say.

Be creative! Why not create an evaluation activity that is itself engaging and enjoyable! Create evaluative processes that will fully engage learners and provide you with the feedback that you want and that you will be able to use.

Be careful! Researchers will be quick to point out the flaws in almost any process you might use. Be clear about whether you want something scientific and flawless (and expensive), or whether you want something approximate but nonetheless informative and useful. If you are after some 'fundamental insights' or 'irrefutable proof' then you will need to invest in a research study that digs deeper than the trowels and spades of 'useful' evaluation.

Be honest! If all you want is a pat on the back and some good quotes to promote what you do, then be honest and plan it all as a public relations exercise. If you want to learn from the process and improve what you do, then plan it as an evaluation exercise.

Be balanced! You may end up with a standardised evaluation process so that you can monitor results over time. But beware of applying 'production line' thinking to what is a highly variable and very human process. If you always ask the same questions, you are always looking at courses from the same perspective. Why not adopt a compromise that gives you the best of both worlds? Try combining a standardised element that allows you to make comparisons over time, with a random or changing element which allows you to get feedback from a new perspective on each occasion.

Be holistic! After a course in which people have been through a whole range of experiences, and have been communicating in many ways, and have been learning at many levels, it is not realistic to expect anyone (perhaps not even a great poet!) to express their true evaluation of a course on a piece of paper. Paper exercises can be very useful but they should be seen as part of a much wider evaluation process that includes dimensions of learning that are less easy to capture on paper.

Be human! Traditional evaluation methods tend to dehumanise the process - partly in an effort to minimise the 'Hawthorne effect' (the evaluator unwittingly influencing the process) - and partly due to a mistaken belief in what is and what isn't 'scientifically' respectable. A 'human' multi-perspective evaluation that involves a wide cross-section of significant people in the participant's life/work is likely to have far greater credibility and value for all concerned. A post-course get-together involving such people can provide an excellent forum for useful evaluation. If face-to-face meeting is impractical then consider using tele/video-conferencing or the latest internet capabilities! (How's that for 'future-proof' advice?)

COURSE EVALUATION METHODS

  1. Redesign this programme
  2. Design an extension of this programme
  3. Make an action plan or 'new year' resolution (what you would like to achieve or change as a result of attending this course?)
  4. Draw a life line with this course on it
  5. Interview each other about the course
  6. Predict how you will see this course next week, next month, next year, 10 years time.
  7. Write a self-report
  8. Devise a questionnaire
  9. Answer this questionnaire
  10. Draw a picture or diagram that represents your experiences on the course and what you think you will 'take away' from the course.
  11. Think of five very different people you will meet after this course, and what you will tell them about it.
  12. Assess your progress on each course/individual objective
  13. Assess the value of each element of the programme (using 'Plus', 'Minus', 'Interesting')
  14. Sum up each day (or other period) in five separate words
  15. How did the staff help or hinder (or generally influence) you?
  16. How did being in this group help or hinder (or generally influence) you?
  17. On this course there should have been more opportunities for ..... because.....
  18. On this course there should have been less ..... because....
  19. Things that shouldn't be changed on this course (because they worked well for me).
  20. In what ways do you think it was a well balanced programme?
  21. In what ways do you think it was not a well balanced programme?
  22. What advice would you give to people coming on this course - to get most out of it?
  23. What sort of person do you think would get most from this kind of course ... and why?
  24. What sort of person do you think would get least from this kind of course ... and why?
  25. What do you think you could have done (if anything) to get more out of this course?
  26. What do you think you have gained from this course?
  27. What might you do in the future to build on what you have gained from this course?
  28. What previous experiences of yours were most like this course ... and why?
  29. What aspects of this course stand out as being most different for you?
  30. What do you think will you remember most about this course ... and why?
  31. Were you surprised about the course in any way? If so what surprised you?
  32. Were you surprised about yourself in any way? If so what surprised you?
  33. How were the staff on this course different from other adults you come across?
  34. What qualities do you think you need to work as a facilitator or trainer on these kinds of courses?
  35. What experiences on this course do you think might have most affected your personal and social development?
  36. In what ways (if any) do you think this course will have influenced or changed you?
  37. How did this part of the project/programme influence how you approached the rest of the project/programme?
  38. Can you think of three ideas (however big or small) that would help to make this a better and more valuable course.
  39. If you could turn the clock back to the beginning of the course, what might you choose to do (or say) differently?
  40. Write an advertisement for this course.
  41. What did you think of the food? ;-)
  42. What did you think of the accommodation? ;-)

Extra ideas and comments received from readers of this page.

  • Ask participants to rate their own contribution.
  • Why limit evaluation to programme evaluation?

Do you have any ideas, advice or evaluation methods to add?

See below for evaluation methods from other sources.


LINKS: FURTHER READING ON PRACTICAL COURSE EVALUATION

Can you recommend any more good online articles about practical course evaluation?

  1. 148 tips for designing external programme evaluation [at archive.org] is a compilation of advice from hundreds of practitioners in education and evaluation derived from The Program Evaluation Standards [at archive.org] (Thousand Oaks, CA: Sage Publications, Inc., 1994). To make use of these 148 tips without getting overwhelmed, try choosing just 10 points that stand out as being important to you.
  2. 4 Tricks for Getting Better Feedback from Denise Burchell, Ideo.
  3. Another look at evaluating training programs 50 articles from Training & development and Technical training magazines that cover the essentials of evaluation and return-on-investment - compiled by Donald L. Kirkpatrick.
  4. A Tip-a-Day by and for Evaluators
  5. Better Evaluation An international collaboration to improve evaluation practice and theory by sharing information about options (methods or tools) and approaches.
  6. BOOKSHOP (Evaluation Section)
  7. Building in Research and Evaluation: Human Enquiry for Living Systems by Yoland Wadsworth
  8. Diagnising your learning evaluation needs by Kenneth Fee and Dr Alasdair Rutherford. Think in advance about the effects and impact of learning and development activities, and ensure your plan for evaluation of business impact is built in from the start. Associated free diagnosis tool from Airthrey Learning Evaluation Solutions.
  9. Evaluating Learning Booklist from Airthrey Learning Evaluation Solutions
  10. Evaluation (of teaching) by James Atherton. Includes 'How I learned to stop worrying and love evaluation' and 'Evaluating Evaluation'
  11. Evaluation: A practical guide to methods Philip Crompton (1996, 1999) for LDTI: the Learning Technology Dissemination Initiative
  12. Evaluation and Professional Development Services, Scotland, UK Lesley Greenaway describes an engaging and customised approach to evaluation that values the process in a way that enhances the quality of the outcome.
  13. Evaluation: Capturing Impact by Kenneth Fee and Dr Alasdair Rutherford (2013) summary and discussion of Dr. Rob Brinkerhoff's Success Case Method (SCM)
  14. Evaluation Cookbook: A practical guide to evaluation methods for lecturers.
  15. Evaluation of training and development programs: A review of the literature by Marguerite Foxon, Coopers & Lybrand (including a review of the Training and Development journal literature for the period 1970-1986). Also see Marguerite Foxon's A process approach to the transfer of training
  16. Facilitated Student Feedback to Improve Teaching and Learning [at archive.org] S. L. Stockham and J. F. Amann, Journal of Veterinary Medical Education, Volume 21 - Number Two, Fall 1994 
    "Summative and perhaps formative evaluation of teachers, teaching methods, or courses usually occurs at the end of courses when there is little opportunity for students to see or reap benefits from their comments. Also at this time, teachers cannot immediately implement recommended changes. Formative evaluations should occur earlier while there is time to change."
  17. Getting the most our of the evaluation investment Sarah Lewis (2013) "To design the most impactful evaluation process for your project, workshop or other intervention, there are some questions that need careful consideration ..."
  18. H form (1) (pdf file) Tips for trainers: introducing the “H­form,” a method for monitoring and evaluation Guy S; Inglis A. S. 1999. International Institute for Environment and Development, London, UK. PLA notes 34: 84­87   Also see: http://scottishparticipatoryinitiatives.co.uk/tools/h-form/#gsc.tab=0
  19. Hform (2) (pdf file) An example of how the H form process has been used [at archive.org] to evaluate a training event: pages 14-15 of the VEST project (Valuing the Extra STuff) give examples of completed H-forms
  20. Internet Resources for Higher Education Outcomes Assessment a monster list of 1,300+ links maintained by Ephraim Schechter
  21. Kirkpatrick: Another look at evaluating training programs 50 articles from Training & development and Technical training magazines that cover the essentials of evaluation and return-on-investment - compiled by Donald L. Kirkpatrick.
  22. Kirkpatrick’s Levels of Evaluation by Will and Connie Lanier presented by Debra Sieloff. These 22 slides provide a neat summary which usefully points out both the connections and disconnections between the levels.
  23. Kirkpatrick’s Levels of Evaluation by Elaine Winfrey. A brief article with diagrams. Not as clear as Debra Sieloff's presentation
  24. Learning Evaluation Glossary from Airthrey Learning Evaluation Solutions
  25. Learning Evaluation Papers from Airthrey Learning Evaluation Solutions. These 10 pdf articles include: Capturing Impact, The top ten evaluation mistakes, Learning Evaluation: Where to Start, Navigating the Learning Evaluation Maze.
  26. Methods of Evaluation by James Atherton.
  27. OERL: Online Evaluation Resource Library - plans, instruments, and reports that have been used to conduct evaluations of projects funded by the Directorate for Education and Human Resources (EHR) of the National Science Foundation (NSF). OERL also contains glossaries of evaluation terminology, criteria for best practices, and scenarios illustrating how evaluation resources can be used or adapted. OERL also provides a discussion board.
  28. Practical Assessment, Research and Evaluation (PARE) A peer-reviewed electronic journal. ISSN 1531-7714 with archives from 1988 to the present.
  29. Principles of Evaluation by James Atherton. Describes four levels of evaluation: Reaction, Learning, Behaviour, Results.
  30. Program Evaluation: Forms and Approaches [at archive.org] John M Owen, Centre for Program Evaluation, University of Melbourne, Australia. Prescribed text: Owen, J. M. with Rogers, P. Program Evaluation: Forms and Approaches (2nd Ed). Allen and Unwin, Sydney, 1999. Book Details
  31. Sample Forms is a handy site that includes Sample Evaluation Forms A useful source of ideas and questions even if you do not find an off-the-shelf ready-made solution.
  32. Some evaluation questions. Shadish, William (1998). Practical Assessment, Research & Evaluation, 6(3).
  33. Some thoughts on evaluation methods [at archive.org] including full descriptions of: 'Round Robin evaluation', 'Snowball Review' and 'What Went Well, Why, Plan' by Mark Waters, Kay Mohanna and Mike Deighan.
  34. The Art and Science of Evaluation by Hilary Briggs (2013) - a manager's perspective on the five key elements of giving an evaluation - with a focus on feedback.
  35. The Evaluation Guy Dr. Matt Champagne provides insightful glimpses into the world of evaluation in his blog, gives little away in his FAQs, but provides instant access to some of his evaluation articles. Avoid expensive mistakes or worthless evaluations by getting to know The Evaluation Guy.
  36. The nature of evaluation part i: relation to psychology. Scriven, Michael (1999). Practical Assessment, Research & Evaluation, 6(11).
  37. The nature of evaluation part ii: training. Scriven, Michael (1999). Practical Assessment, Research & Evaluation, 6(12).
  38. Thinking about how to evaluate your program? these strategies will get you started. Gajda, Rebecca & Jennifer Jewiss (2004). Practical Assessment, Research & Evaluation, 9(8).
  39. Training Evaluation - the TrainingCheck Approach Includes Kirkpatrick (ROE) and other models (Bersin, Brinkerhoff, Phillips ROI and Balanced Scorecard) within their training evaluation system.
  • Thank you to the people who have suggested the above links. If you have any suggestions or comments please send an email to roger@reviewing.co.uk



EVALUATION INDEX

 INDEX to reviewing.co.uk - resources for dynamic learning
HOME
 How to find your way around reviewing.co.uk
HELP

Copyright © Roger Greenaway, Reviewing Skills Training, who promotes ACTIVE LEARNING via
TRAINING EVENTS, CONSULTANCY, HANDBOOKS, RESEARCH, CONFERENCES, and EZINES