Thursday 25 November
Friday 25 November
Thursday 25 November
Actionable evaluation for real-world decision makers
Presenter: Dr Jane Davidson
Institution: Real Evaluations
Have you ever been presented with evaluation findings and been none the wiser by the end of it all whether or not the whole programme or initiative was a complete waste of student and faculty time, course fees and taxpayer money? This provocative keynote presents a guided tour through the distinguishing features and natural habitats of the main species of waste-of-money evaluation and asks: Why hasn’t natural selection taken care of them already? What if evaluations actually asked evaluative questions and gave clear, direct, evaluative answers? What does it look like if they do? How does that make them infinitely more actionable for real-world decision makers? A key theme in this presentation is the need to unlearn some of our most evaluatively dysfunctional social scientist habits so we can get a firmer grip on producing – and commissioning – incisive, genuine, real evaluation that people can actually use.
Ritual or reality: do student evaluations have any effect on teacher thinking and practices?
Presenter: Sarah Stein
Institution: University of Otago
Internationally, there is widespread use of centralized systems of student evaluations of teaching in higher education. Institutions use evaluations data to monitor quality and for promotion and performance decisions. The evaluation data can also provide teachers with data to develop or refine their practice, but we are uncertain about the extent to which this occurs. Reportedly, tertiary teachers are negatively disposed towards student evaluations, but there is limited research evidence about teachers’ views of student evaluation and the way in which these perceptions influence teacher thinking and behaviours.
This paper reports on a research study that investigates teachers views of the student evaluations process and the way it impinges on all aspects of their teaching. The research was conducted with teaching staff at Otago University, Waikato University and Otago Polytechnic.
All teaching staff members at the three New Zealand institutions were invited to complete a questionnaire to elicit their perceptions about the influence of evaluations on teaching and learning processes from design to implementation, as well as their impact on teacher development. They were also invited to participate in a interview.
This paper discusses the progress of the research, including questions raised by the literature, preliminary findings from the questionnaire and the progress of interviews.
The research team hopes that the findings can be used to inform recommendations for the implementation of evaluation processes that can be most effectively used to enhance teaching and consequently student learning.
Appraisal and Evaluation: One Institution’s approach to developing a feedback loop
Presenter: Dr Trudy Harris
Institution: The University of Waikato
External institutional audit, international competition, public accountability and funding requirements have provided the catalyst and imperative for the collection of data in institutions. While knowing about how an institution works is paramount for ongoing development, the imperative often stops at the point of data collection. The follow up processes to enable development and organisational learning can be lax or lacking altogether. This paper looks at a university wide project to develop a feedback loop around the reporting of both appraisal and evaluation data. A case study will be presented centring on the Waikato Management School. The case study will outline the process taken, the data presented and the feedback obtained from all levels of teaching staff within the School. The paper will also outline the work that needs to be completed to close the feedback loop for all stakeholders concerned in the appraisal/evaluation process.
Closing the loop: A holistic approach to student feedback at Curtin
Presenters: Beatrice Tucker and Julie-Ann Pegden
Institution: Curtin University, Perth, Australia
eVALUate is Curtin’s online system for students to give feedback about their units and teachers. eVALUate comprises a unit survey and a teaching survey. Teaching survey results are confidential to teachers while unit survey results are available in various formats to provide relevant information to staff, students and the general public. The unit survey comprises eleven quantitative items and 2 qualitative items. This survey asks students for their perceptions of what helps their achievement of unit learning outcomes (items 1 to 7), their engagement and motivation (items 8 to 10) and overall satisfaction (item 11). For each quantitative item, students rate their level of agreement (strongly agree, agree, disagree strongly disagree) or select ‘unable to judge.' A Full Unit Report provides Unit Coordinators and their Head of School with a full breakdown of results and all student comments. A Unit Summary Report (USR) is published online for all students and staff and contains the percentage Agreement (Disagreement and Unable to Judge) with all quantitative items.
This paper will, firstly, provide an overview of the eVALUate system and describe the mechanisms for unit coordinators and Deans of Teaching and Learning to ‘close the feedback loop’ using: 1) the online tool and 2) unit outlines. A key responsibility of the Unit Coordinator is the maintenance and improvement of unit quality and includes the review, reflection and response to student feedback. Unit coordinators can close the feedback loop using the USR by acknowledging student feedback, assuring them that their feedback is valued and indicating how the feedback will be taken into account when the unit is next offered. Recent unit changes resulting from student feedback are published by the unit coordinator in unit outlines. The eVALUate website also contains a webpage titled ‘Student Voice: What students are telling us and what we’re doing about it’. Posted on the webpage is a general summary of the eVALUate unit survey feedback for the university overall. Deans of Teaching and Learning provide an online response to students about Faculty initiatives that are implemented and planned to improve student experiences in teaching and learning.
Students are sent an email once the webpage is updated after each semester, thanking them for their feedback and advising them how they can view Unit Summary Reports and the ’Student Voice‘ webpage. Web usage statistics show students are increasingly viewing Unit Summary Reports. For example, in July 2010 there were 2452 reports viewed by students compared to 1736 in July 2009, despite similarly high rates of Unit Summary Report publication in both years.
Secondly, this paper will describe the events that resulted in the automatic online publication of the USR for all coursework units delivered at Curtin. Curtin changed from opt-in to opt-out publication of USRs in 2009. This change to automatic publication has resulted in a 96% (n=1688) publication of USRs in Semester 1 2010 (compared with 6.7% in 2008). Another motivating influence on USR publication is Curtin’s Teaching Performance Index, which rewards Unit Coordinators for publishing USRs with a response. In 2006, 361 USRs included a unit coordinator response to students and in 2009 this number increased to 928.
As this paper will show, Curtin has adopted a comprehensive approach to closing the feedback loop, with many aspects incorporated into the eVALUate system. It is envisaged that closing the loop will become a routine part of teaching at Curtin and will contribute to ongoing improvement in response rates and survey results.
Evaluating the impact of interventions designed to improve teaching and learning
Presenter: Dr Peter Coolbear
Institution: Ako Aotearoa
As the National Centre for Excellence in Teaching and Learning, Ako Aotearoa manages a significant discretionary research and project fund. We see these funds primarily as an investment for change and have sought to position our funding processes accordingly. A critical issue for us is then to demonstrate the value of this project investment in terms of returns that benefit learners.
This raises significant multi-level evaluation challenges for us at both individual project level and at the organisational level.
This presentation will provide an update and opportunity for discussion about how we are addressing these challenges in terms of the development of a project impact framework and our own organisational outcomes hierarchy. The operation of the latter will be illustrated by the two formative evaluations we have commissioned on Ako Aotearoa to date.
Aromarau: A Wānanga way of self-evaluation
Presenters: Shelly Davies, Fui Te’evale, Thomas Tawhiri and Michelle Te Ahuru
Institution: Te Wānanga o Aotearoa
Shelly Davies (Ngātiwai, Ngāti Porou) is a subject analyst in the Te KAO Team, a curriculum realignment project at Te Wānanga o Aotearoa. Her practice has a strong focus on writing, rangahau, and ako.
Fui Te’evale (Samoan, Chinese) is a subject analyst in the Te KAO Team, a curriculum realignment project at Te Wānanga o Aotearoa. His practice has a focus on programme self review and evaluation.
Thomas Tawhiri (Te Whakatōhea, Ngāti Raukawa ki Wharepuhunga) is a subject analyst in the Te KAO Team, a curriculum realignment project at Te Wānanga o Aotearoa. His practice has a strong focus on Māori worldview and conceptual thinking, self review and evaluation systems and processes.
Michelle Te Ahuru (Ngāti Tūwharetoa, Te Rarawa) is a subject analyst in the Te KAO Team, a curriculum realignment project at Te Wānanga o Aotearoa. Her practice has a strong focus on supporting staff with their self reflective activity as well as the nurturing of healthy working relationships.
Te Wānanga o Aotearoa is constantly reflecting on our own practices as an indigenous tertiary education provider. We are keenly aware of our kaupapa, mission, and values, and are constantly reclaiming our spaces and positions in order to ensure that we are guided by āhuatanga and tikanga Māori. An ongoing challenge we face is doing this at the same time as ensuring that we meet legislative and governmental requirements placed on us as a publically funded institution—requirements which sometimes are set in stark contrast to our ways of knowing, doing and being.
With NZQA’s move to External Evaluation and Review (EER), Te Wānanga o Aotearoa saw an opportunity to develop self-evaluative practices which would primarily support the needs and aspirations of our ākonga, promote mātauranga Māori, support our tinorangatiratanga (self-determination) while at the same time . One approach being developed which is seeing success in supporting those ways and promoting tinorangatiratanga (self-determination), and is doing so in ways that are rigorous and stand up to external scrutiny, is a method of programme self-review and evaluation called “aromarau.” Aromarau is a neologism created by combining the words “aromatawai”, meaning “to self-reflect”, and “marau”, meaning programmes or curriculum. The process itself, in terms of gathering, analysing and evaluating programme-related data in order to provide recommendations for improvement, is fairly typical and can be found in other evaluative methods. What sets aromarau apart from other tools is its grounding in Kaupapa Wānanga; a set of takepū (applied principles) which guide our practice.
Ngā takepū are: kaitiakitanga (responsible trusteeship), āhurutanga (the creation of safe spaces), koha (contributions of consequence), and mauri ora (the pursuit of conscious wellbeing). When we apply these principles to our reflective practice, the responsibility to aromarau (to reflect on and improve our programmes) becomes a shared one, in which the mana to be able to effect and implement these changes rests equally on all those invested in a programme. Likewise, the drive to engage in aromarau is something which is internally driven (i.e. by the self) rather than externally imposed (either by management or by parties external to TWoA).
This presentation will discuss Ngā Takepū Kaupapa Wānanga, and how these applied principles affect the activity and outcomes of aromarau.
Friday 25 November
Closing the Loop: implications for practice
Presenter: Professor Sid Nair, Professor of Higher Education Development
Institution: University of Western Australia
The expression ‘closing the loop’ is a common expression in the higher education sector. Historically speaking in the evaluation context, the root of this phrase arises from extensive evaluation activity (in terms of surveys, observations and focus groups) with incomplete or lack of action on the results of these measures. Widespread anecdotal evidence and university quality audits indicate that student evaluations, though abundant in the system, are not systematically handled to achieve sustainable improvements in teaching and learning. Much has been reported in the Australasian region where student evaluations are used for the allocation of funding, such as for the Learning and Teaching Performance Fund (TPF), but little has been focused on achieving sustained institutional improvements in teaching and learning practices.
It is not rocket science – we all know that “closing the loop” is important in any good quality cycle. What I intend to do in this address is to probe the concept of closing the loop at the institutional level, probe the reasons why at the national level there is not a focus on improvement but on assurance, go back to the fundamentals of student evaluations to seek an understanding of the changing landscape of evaluations, look at the features of an effective loop closure and probe possible reasons why higher education institutions choose in some instances to provide good evidence of closure and in other areas does not.
Which students give feedback: An analysis of participation rates and feedback by semester weighted average
Presenters: Julie-Ann Pegden and Beatrice Tucker
Institution: Curtin University, Perth, Australia
Online unit evaluation systems are increasingly used in universities to gather anonymous student feedback. In 2005, Curtin implemented a university-wide system called eVALUate for gathering and reporting students’ perceptions of their learning experiences. eVALUate comprises a unit survey and a teaching survey. The unit survey contains eleven quantitative items and 2 qualitative items. Quantitative items ask students for their perceptions of what helped their achievement of unit learning outcomes (items 1 to 7), their engagement and motivation (items 8 to10) and overall satisfaction (item 11).
Many staff have embraced and welcomed eVALUate but some are more fearful, defensive and sceptical. In 2006, an analysis of 30,000 student comments was undertaken to determine whether students were making abusive comments in eVALUate (Oliver, Tucker and Pegden, 2007). One of the findings of this research was that students who have higher grades and are more engaged in their studies are more likely to participate in eVALUate. Whilst this research has been useful in allaying staff fears, anecdotal evidence suggests that there is still some belief amongst academic staff that eVALUate is predominantly used by academically poorer students to complain unjustly. These staff also believe survey results are negatively impacted by this perceived over-representation of lower performing students.
This study investigated differences in eVALUate participation rates by students of different semester weighted averages. The study also examined differences in survey responses by students with different semester weighted averages. Data from 154,821 surveys in four semesters (semesters 1 and 2 in 2008 and 2009) showed that students with higher semester weighted averages were more likely to give feedback. Students with a semester weighted average of 90% and higher were three times more likely to participate than students with a semester weighted average below 50%. Results also revealed that students with a high semester weighted average were more likely to agree with the survey items. This was particularly evident in survey items related to the students’ own engagement and motivation, as well as in the overall satisfaction item. Chi square analyses showed that semester weighted average was significantly impacting on response rates and survey results (p<0.05).
This investigation reinforces the previous findings from 2006 and showed that contrary to what some staff believe, students of lower semester weighted averages are less likely to participate in eVALUate and this group is under represented. In contrast, students of higher semester weighted averages are more likely to give feedback and are more likely to agree with the survey items. It is likely that higher participation by more academically accomplished and motivated students is skewing results in a positive manner when reporting aggregated university data.
The Lincon University online evaluation trial
Presenter:
Institution: Lincoln University
Traditionally Lincoln University has carried out paper based evaluations of courses and of teaching by lecturers. As information technology increasingly enables systems to be rationalised, universities are making the move to on-line evaluations. The decision to trial on-line course evaluations was based on the general trend towards blended learning with many courses having an on-line component, perceived benefits from changes to the current process at Lincoln, and a review of the literature which suggests the advantages of on-line evaluations over in-class evaluations. During Semester 1, 2010, a pilot using on-line evaluation for a number (N=85) of course evaluations was conducted at Lincoln University. This presentation discusses the results of that trial and of a survey of staff and students.
Locating evaluation ownership locally
Presenter: Linda Keesing-Styles
Institution: Unitec
Evaluation discourse at Unitec in recent years has been located in debates about universal applicability of evaluation instruments, online methodologies and response rates. These, while relevant, do not go to the heart of the evaluation issue. What appears to be missing is a culture that embeds and values evaluation as a central component of learning and teaching. Such an environment is also consistent with two other key notions. The institution's move to a Living Curriculum and Evaluative Questioning. Embedding evaluation naturally within context is key and is best integrated from the very beginning of the student experience. Evidence of teachers valuing student voice and feedback is critical to student engagement with evaluation.
In this presentation and discussion, a history with student evaluation at Unitec is briefly described and a radical new approach presented.
Evaluation in the Tertiary Sector: Value-added measurement as a quality assurance tool
Presenter: Therese Boustead,
Institution: Centre for Evaluation and Monitoring (CEM), University Canterbury, New Zealand
Value-added indicator systems have been successfully used in the New Zealand secondary sector for over a decade. These systems are focussed on professional self evaluation rather than public accountability. What can the tertiary sector learn from the success of the value-added indicator systems in the secondary sector and can this approach be used to meet tertiary quality assurance goals for professional self evaluation in higher education?
Cell! Cell! Cell!: Mobile phone text messaging as a method of enhancing participation in online student surveys
Presenters: David Collings and Luke Regan
Institution: Edith Cowan University and Murdoch University
Student evaluations in Australian universities have seen a major shift away from paper-based administration and towards an online approach over the past 10 years or so. The benefits of online surveys have been well documented, however many universities still battle with a commonly experienced disadvantage of this method – low response rates.
A range of approaches have been used in an attempt to improve response rates: For example, incentives in the form of prize draws or gifts; improved publicity via print, web-based means; closing feedback loops with students more effectively.
This presentation will consider some of the issues surrounding the use of SMS text messaging as a means of supporting online student surveys, and will look at experiences at both Edith Cowan University and Murdoch University, including examples of quasi-experimental approaches.