Program evaluations collect, analyse, and use information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. Stakeholders often want to know whether the programs are producing the intended effect. While program evaluation first focuses around this definition, important considerations often include how much the program costs per participant, how the program could be improved, whether the program is worthwhile, whether there are better alternatives, if there are unintended outcomes, and whether the program goals are appropriate and useful. Evaluators help to answer these questions.
During the Project Sunset evaluation both program and school documentation is utilized, for example how often the children attended, the descriptions of the activities and reports of student participation. Project Sunset process and impact evaluation will be performed by qualified external evaluators adhering to and practicing Ownership, Control, Access and Possession (OCAP) principles. Evaluation practices adhere to the highest ethics standards.
Site visits will be conducted each year by the evaluators where they will conduct focus groups, face to face interviews with youth, parents, teachers, school administrators, police officers and community partners as well as control group surveys. Photovoice experiences with reflection techniques will be employed to elicit responses from the participant- based on their experience in the project.
An extensive Community Partner survey will be widely administered to all community organisations engaged with Project Sunset to develop or deliver the program. It will assess the level of involvement partners have with the development and implementation of Project Sunset as well as the perceptions of community partners in the operation and impact of the program. A key objective of Project Sunset is to sustain program deliverables through a network of key community organisations, so it is essential that the evaluation determine its effectiveness so community partners can determine their ability and willingness to take on program delivery in the future.
Interviews will also be conducted to determine
- The extent to which the project has been implemented as intended
- Whether the intended outcomes have been achieved
- If there are any unintended outcomes
- Lessons learned in implementing the program (any adaptations or modifications)
Interviews would be conducted by a third party evaluation team in conjunction with Project Sunset core delivery partners including parents, community leaders or elders as well as teachers and Police Officers participating in the Project.
Focus groups with students participating in Project Sunset will be held during site visits to obtain additional information about program implementation and contextual information about student satisfaction with the program.
Finally, a non-traditional data collection method using Photovoice. Photovoice is an innovative method for showing how projects had impacted the lives of youth and/or families.
It allows for people to record and reflect on a program, highlighting its strengths and problems to promote a dialogue through discussion and photographs, to shed light on the impacts of the program. It is particularly helpful in empowering youth and allows them to give their observations in a way that values personal knowledge and experience.
The use of Photovoice promotes trust between the researcher and the youth, allowing for an honest dialogue about the project’s impact and the participants views of the experience. It creates a window from the individual experience into an understanding of community members and their needs.
As part of the evaluation of Project Sunset, program participants will be asked to take photos that demonstrate something about Project Sunset and then describe in writing, how it has impacted them. This valuable information will be collected in years 2 and 3 of the evaluation