Program Evaluation in Adult
Evaluation Plan for the IDEA-X Program
Introduction
The evaluation plan for IDEA-X program was initially designed on the basis of utilization focused evaluation (UFE) approach in the pragmatic paradigm. However, I noticed that some stakeholders were isolated from the evaluation process and that applicants' follow-up feedback was decreasing in the past few months. Therefore, I turn to the empowerment evaluation approach, an evaluation approach that has matured over the last two decades and is used worldwide in educational programs (Fetterman, 2009).
Immersive Digital Experience Academic program (IDEA-X) is a one-month intensive program for Canadian and international undergraduate students who are interested in digital media industry. The purpose of this article is to evaluate the IDEA-X program by applying the empowerment evaluation approach. Specifically, this evaluation plan canvasses its philosophical underpinnings, institutional context, evaluation methods, design process, data collection strategies, scope and limitations. At last, some recommendations for practice and final thoughts are offered upon the current evaluation plan.
Background
Social and Institutional Context
The IDEA-X program is carried out at the Centre for Digital Media in Vancouver - a dynamic city and international digital media hub. Companies such as EA, Microsoft, Sony Pictures Imageworks and Industrial Light and Magic have studios in Vancouver, and game companies like Bandai Namco Studios Vancouver and Blackbird Interactive have studios at the Centre for Digital Media (CDM).
The CDM is jointly owned and operated by four post-secondary institutions in Canada: the University of British Columbia (UBC), Simon Fraser University (SFU), British Columbia Institute of Technology (BCIT) and Emily Carr University of Art + Design. Launched in 2007, it has become the home to four industry-relevant programs: two graduate programs (Master of Digital Media (MDM) and Pre-MDM) and two non-credit programs (one, IDEA-X, targets overseas undergraduate students, and the other, TMDM, targets local high-school students) (CDM, 2018).
The IEDA-X program benefits from the same teaching staff and facilities as the MDM program, most of whom have had several years’ working experience in some major companies in the digital technology and game industry, such as Microsoft, EA game company, and Sony Studio.
The Evaluator
An external evaluator with considerable expertise is needed to implement the evaluation plan. The empowerment evaluators serve the role as critical friends. They are able to provide professional assistance through advising and recommending; they provide a structure or set of steps to conduct an evaluation; they listen and rely on the group’s knowledge and understanding of their local situation; they serve the group or community in an attempt to help them maximize their potential and exponentially unleash their creative and productive energy for a common good (p. 201).
Philosophical and Theoretical Lens
The evaluation is guided by the pragmatic paradigm whose assumptions align closely with the idea of using the evaluation findings as a priority (Mertens & Wilson, 2012). The pragmatic paradigm is identified by some researchers as the philosophical framework that guides their choice of mixed methods (p. 91).
The Use Branch in the pragmatic paradigm illustrates how these theoretical perspectives are manifested in practice. David Fetterman (1994), who proposes the model of evaluation known as empowerment evaluation, believes that program participants who conduct their own evaluations will be more likely to use the information forthcoming from that evaluation.
Empowerment evaluation is an evaluation approach in the Use Branch that aims to increase the probability of achieving program success by
providing program stakeholders with tools for assessing the planning, implementation, and self-evaluation of their program, and
mainstreaming evaluation as part of the planning and management of the program/organization (Wandersman et al., 2005).
The Evaluation Process
The Aims of the Evaluation
By the end of the program, the evaluation plan is expected to achieve the following aims:
a. whether the participants are satisfied with the IDEA-X program,
b. whether the participants’ expectations of this program are met,
c. what improvement and modification need to be made to the curriculum, and
d. whether any participants are potentially interested in applying for the CDM’s Master of Digital Media graduate program.
The Evaluand
The evaluand is the IDEA-X program – a four-week intensive program for undergraduate students from the universities outside Canada. It aims to provide the immersive courses and team projects on digital technology and help students build connections to the cutting-edge technologies in the digital entertainment industry.
Stakeholders and Participants
Three levels of stakeholders are identified, and they are also the participants of the evaluation:
a. the administrative group at the Center for Digital Media, including those in the four co-operating institutes,
b. the international students who pay to participate in this program, and
c. the teaching staff who design and deliver the curriculum.
Method
This section consists of evaluation design, evaluation questions, and data collection strategies. Mixed methods are used in the evaluation, such as scale questions, open-ended questions, informal interviews and online surveys.
Design
In Fetterman (2001)’s evaluation, he uses a three-step approach which typically employs him as an empowerment evaluator to facilitate empowerment evaluation exercises. The empowerment evaluator helps the group to
(a) establish their mission or purpose,
(b) take stock or assess their current state of affairs by using a 1 (low) to 10 (high) rating scale, and
(c) plan for the future (specifying goals, strategies to achieve goals and credible evidence) (Fetterman & Wandersman, 2007).
The evaluator schedules a meeting among the students, teachers and the director at the beginning of the evaluation in order to get stakeholder involvement and buy in to the process. The tasks of this meeting are
(a) to build a shared vision and mission statement by exchanging students’ and teachers’ expectations for the program,
(b) to assess the current situation, and
(c) to make a specific four-week plan for the cohort. As this is a four-week intensive program, the evaluator needs to facilitate the process and be aware of the time span.
A meeting at the beginning is scheduled by the empowerment evaluator for the CDM director, teachers, and the IDEA-X cohort. It takes place on the second day of the program when the students and teachers get familiar with each other and the students get started with the courses and activities. The purpose of having the meeting on the second day instead of the first day is that the relationship nurtured by the cohort and teachers after the first day’s study will be very helpful to conduct the evaluation collaboratively.
Throughout the four-week program, the evaluation methods, goals, strategies, procedures, data, and relevant information are DIGITALIZED and shared online, either in a google shared file or a private Facebook group that only people who get the link/invitation can have access to. In this way, all the stakeholders (the CDM administrators, teachers, students) and the evaluator can monitor and keep track of the entire evaluation process through the online platform.
There are three steps need to follow at the meeting.
Step 1: What is the mission (high-level aspirational) for this evaluation group?
First, the teachers and the director present their pre-designed curriculum and teaching tools. Then, the students provide their feedback for the first day’s study and express their expectations for the courses and activities. Next, they create a mission statement together for the next four weeks’ training. For instance, the students, teachers and director of April 2018 IDEA-X program create their mission statement for the evaluation as “to collaborate in evaluating to make IDEA-X the best short-term digital media program in North America”. Finally, one teacher or a student leader is in charge of documenting and posting the outcomes on the online platform. At the meeting, the evaluator facilitates the conversation, instruct them to make a mission statement and push the meeting to the next stage.
Step 2: Take stock and assess the current situation.
The second step is devoted to brainstorming a list of the most important activities. Participants prioritize the list by “voting” for the most important activities to evaluate.
(a) Brainstorm: What are the MOST IMPORTANT things we are doing to achieve our mission?
All the students, teachers and the director are asked to write down and prioritize the things that can help achieve their mission. The following table can be employed in this step.
(b) Vote for the top five most important things we do to achieve our mission.
In this step, the empowerment evaluator acts as an advisor and a listener. He/She gives recommendations and provides a set of steps to conduct the evaluation.
Step 3: Plan for the future.
Evidence in Step 2 during the dialogue about ratings informs strategies for Step 3.
Additionally, in the last week of the IDEA-X program, the students are asked to complete an online survey to provide their feedback. The purpose of the survey is to
(1) get an overall feedback of the program from the student perspective,
(2) assess whether the (adjusted) curriculum works,
(3) appraise the effectiveness of the teaching, and
(4) improve the curriculum for the next cohort.
Evaluation Questions
Apart from the questions aforementioned in the meeting, the following shows some questions that may need during the process of evaluating.
a. Questions used in achieving the prioritized tasks:
Taking the first task in the prioritization list created in Step 3 for example, the following assessment form can be used to evaluate how well the effective teaching has achieved in terms of teaching the “agile development process”.
b. Questions used in the informal interviews/talks:
During class breaks or after school, either the teachers or student leaders ask the students how the program is going. The following questions can be used.
How do you find this class?
Do you think the way the teachers deliver the course is suitable for you?
Do you find any difficulties in understanding the content?
What do you think of the instructions of the tasks?
Do you think the activities are good for you to put theories into practice?
What role do you play in the team activities?
Do you have fun in the activities?
Do you have any suggestions for the teachers and the teaching content?
What is your ideal way of learning this specific content?
What are the three takeaways from today’s class?
c. Questions used in the online survey at the end of the program
The following questions can be considered in creating the online survey. Adjustment can be made if it is necessary.
Data Collection
Throughout the evaluation process, the critical friend - the evaluator helps the group monitor and assess the newly implemented strategies or interventions. He/She helps the group conduct online surveys, interviews and observations and inform the students about the faculty’s curriculum changes. For instance, the information gathered from the “assessment of the agile development process teaching” is given to the faculty, who are then able to make necessary changes in the corresponding curriculum and activities. If the new strategies are not viewed as timely or effective, another idea will be generated and implemented.
After completing the online questionnaires and surveys, the computer system generates the visualized charts (such as bar charts and pie charts) for the multiple choice data and provides a preliminary data analysis. The answers to the open-ended questions are organized together by the computer. For the informal interviews and talks, the teachers and student leaders who conduct them are responsible for the feedback documentation.
Interpret and Communicate the Results
All of the evaluation procedures, data, data analysis and improved curriculum plans are digitalized. Throughout the evaluation, all the stakeholders can see the data online in a google shared file or a private Facebook group which people who get the permission can have access to. For the data collected in the meeting at the beginning, they are analyzed and discussed at the scene by the teachers, the director and students together; for the data collected from the online questionnaires, the faculty make an in-depth analysis on the basis of the preliminary data analysis generated by the computer; for the data collected from the open-ended questions and informal interviews, they are also posted on the online platform anonymously; for last online assessment survey, the data are mainly interpreted and shared within the CDM faculty.
The empowerment evaluator differs from many traditional evaluators. The empowerment evaluator is critical and objective because they want the program to work better (Fetterman, 2009). Instead of being the “expert” and completely independent, separate, and detached from the people they work with, the empowerment evaluator works closely with and alongside program staff members and participants. Empowerment evaluators are like coaches. They pose difficult questions to the participants in a diplomatic fashion to facilitate the evaluation. The people they work with are in charge of the direction and execution of the evaluation.
Scope and Limitations
The scale of empowerment evaluations is continually growing. The scope of empowerment evaluation ranges from its use in a national education reform movement (Fetterman & Wandersman, 2007) to governments, foundations, businesses, and non-profits, as well as Native American reservations (Wikipedia, 2018).
The inclusion of stakeholders in the evaluation process is one of the characteristics of empowerment evaluation. Nonetheless, it can lead to some issues. One problem is that empowerment evaluation tends to generate outcome-oriented data. As the stakeholders are also the participants in the evaluation, they have the expectation of what the evaluation finding should be. Therefore, it might be hard for them to stay objective in rating the faculty and curriculum and interpreting the data.
Advice for Practice and Final Thoughts
Empowerment evaluation is a form of guided self-evaluation which requires student, faculty, and administrator participation in curricular development and refinement, facilitated by an empowerment evaluator (Fetterman, 2009).
The role of the empowerment evaluator is crucial throughout the whole evaluation process. He/She serves the role as a critical friend and helps the group monitor and appraise the newly implemented strategy or intervention. The empowerment evaluator should also be aware that necessary changes to the initial evaluation plan are needed according to the real situation.
For the purpose of enabling the stakeholders to monitor and keep track of the evaluation process, timely online posts of the evaluation procedures, data, data analysis, interview feedback and improved curriculum plans are recommended. Collaboration and cooperation among the participants -the director, teachers and students are valued.
Although some limitations aforementioned might take place inevitably, the evaluation is expected to be overall effective because it can lead to the positive and meaningful outcomes. As the director, faculty and students in the IDEA-X program evaluation are both stakeholders and participants, their interests are greatly revealed and their initiatives are significantly stimulated throughout the evaluation process. Furthermore, the collaborative and participatory engagement in the empowerment evaluation is a transformative learning process for the administrators, faculty and students. The faculty can benefit from their participation in terms of improving self-efficacy and professionalism. It can help them build evaluation capacities and may have a long-term impact on the curricular planning, sustainability of the program, and governance in the IDEA-X program.
References
CDM. (2018). Retrieved from https://thecdm.ca/our-story
Fetterman, D. M., & Wandersman, A. (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28(2), 179-198. doi:10.1177/1098214007301350
Fetterman, D. M. (1994). Empowerment evaluation. Evaluation Practice, 15(1), 1-15. doi:10.1016/0886-1633(94)90055-8
Fetterman, D. M. (2001). Foundations of empowerment evaluation Sage.
Fetterman, D. M. (2009). Empowerment evaluation at the stanford university school of medicine: Using a critical friend to improve the clerkship experience. Ensaio: Avaliação E Políticas Públicas Em Educação, 17(2), 197-204.
Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice : A comprehensive guide. New York: Guilford Publications.
Wandersman, A., Snell-Johns, J., Lentz, B., Fetterman, D., Keener, D. C. '. L., M., & et al. (2005). The principles of empowerment evaluation. in D. M. fetterman & A. wandersman (eds.). Empowerment evaluation principles in practice (pp. 27-41). New York: Guilford.
Wikipedia. (2018). Empowerment evaluation. Retrieved from https://en.wikipedia.org/wiki/Empowerment_evaluation
© Wenyi Gong 2017
Comments