Overview
Improving Out-of-School Time Program Quality
We knew from the outset of the Initiative that supporting high-quality out-of-school time programs was important, and we sought to fund programs with high-quality components, including strong positive adult role models, academic support and family engagement. We also knew that program quality improvement was a strong and valuable movement in the out-of-school time field.
The first group of participating program leaders encouraged us to incorporate program quality assessment and improvement into the Initiative’s learning community and evaluation. They wanted to know what they were doing well and where they could improve, to be supported in their improvement efforts, and to tell a more complete story of their work than they could with student participation or outcomes data alone.
Eugene School District 4J
After working with these leaders and other out-of-school time stakeholders to complete a thorough review of existing program quality improvement systems and tools—and their relationship to Initiative priorities and local frameworks such as the Oregon Leading Indicators (developed through the Oregon 21st Century Learning Centers in 2013)—we selected the nationally recognized, research-based Youth Program Quality Intervention and Assessment (YPQ).
Over time, program quality improvement became increasingly central to the Initiative and its learning community, and its benefits are increasingly visible to us and to participating programs.
Learn more about program quality improvement benefits in INITIATIVE & PROGRAM impacts.
Download a brief, printer-friendly synopsis of our findings on program quality.
read more about program quality in out-of-school time, how and why we selected the YPQ, and what we’ve learned through the Initiative in Program Quality: A learning brief.
About out-of-school time program quality improvement
Evidence continues to build for the importance of quality in out-of-school time, indicating that some of the variation in student outcomes can be explained by variation in program quality. A growing body of research demonstrates that participation in high-quality out-of-school time programming can lead to a wide range of positive impacts for students, including improved grades, more regular school attendance, and social and emotional learning gains like positive social behaviors (e.g., Akiva et al., 2013; Durlak et al., 2010; Kidron & Lindsay, 2014; McCombs et al., 2017; Naftzger et al., 2014; Pierce et al., 2010; Vandell, 2013).
Focusing on program quality allows organizations to change program practices that are within their control and to feel confident that these changes will improve youth outcomes. Measuring program quality is also a way to see inside the “black box”—to understand what happens during programming that might influence youth outcomes.
We cannot directly see human development; thus, it is easy to be deceived as to whether a program is having long-term beneficial effects on youth. … [M]easuring program quality features provides a shortcut around the nearly impossible task of conducting evaluation research to assess whether human development is occurring in every program (Larson et al., 2009, p. 73).
In this way, we can think of program quality as akin to program fidelity (i.e., the degree to which a program with a standardized design, curriculum or implementation is doing what it was designed to do). Given that programs vary in their content, design and context, efforts to improve programming should focus on more neutral or nonspecific quality frameworks, particularly within diverse networks like the Initiative, which include locally developed programming and culturally specific programming.
Camp Fire Central Oregon
Building Healthy Families / Wallowa Resources
High-quality out-of-school time programming is effectively defined by the frameworks and tools used to assess it, which consist of actions and features that provide programs with guideposts for building quality (Yohalem & Wilson-Ahlstrom, 2010; Larson et al., 2009).
Program quality typically focuses on the “point of service”—the interactions between staff and students that directly impact youths' experience of the program (Hirsch et al., 2010). Research shows that these interactions are better predictors of effectiveness than are structural features like student-to-staff ratios (Yohalem et al., 2009) or program attendance (Hirsch et al., 2010).
The Youth Program Quality Intervention and Assessment (YPQ) is centered in four program factors: ensuring physical and emotional safety; providing a supportive environment through positive relationships; providing opportunities for youth to learn and engage with new content, build skills and develop; and empowering youth to be leaders in their own development (Ramaswamy et al., 2013).
The nationally recognized, research-based Youth Program Quality Intervention and Assessment (YPQ) was developed by the David P. Weikart Center for Youth Program Quality and is used throughout the United States. It is grounded in research on positive youth development, supports continuous improvement and is proven to improve instructional practices and student experiences.
Given the Initiative’s focus on students in middle school, we use the Youth Program Quality Assessment (Youth PQA).
The Youth PQA is designed to empower people and organizations by helping them to envision optimal-quality youth programming, by providing a shared language for practice and decision making, and by producing scores that can be used for comparison and assessment of progress over time. The Youth PQA measures the quality of youths’ experiences and promotes the creation of environments that tap the most important resource available to any youth-serving organization: a young person’s motivation to engage critically with the world.
—Introduction, Youth PQA Form A
YPQ assessments entail both program observation and reflection on organization and program policies and practices. Assessment is followed by reflection and improvement planning, and then by implementation. This repeating or continuous assess-plan-improve cycle is supported by extensive training and, ideally, tailored implementation coaching.
We believe the YPQ is the most cost-effective and robust program quality system available. It has both a strong research base and a standardized training and coaching system (including train-the-trainer opportunities that help programs across the state in implementing the assess-plan-improve cycle). The system supports professional development and capacity-building for staff working directly with youth as well as for program leaders.
Initiative learning community
In 2014, the evaluation team worked with the Weikart Center to adapt the standard program quality assessment tools for use in Oregon. In doing so, we wanted to reflect the context and priorities of the Initiative and what we heard from out-of-school time stakeholders in Oregon more broadly, particularly regarding culturally responsive practice and support for academics.
While the Oregon Form A (observation) and Form B (reflection on organization and program policies and practices) tools are available for anyone to use, we encourage program leaders to seek training via the Institute for Youth Success or the Weikart Center to maximize their effectiveness.
Programs can use the YPQ regardless of their content focus, thus providing a shared framework for program quality and a shared language for discussing their work. This can be especially valuable for locally developed and/or culturally specific programs.
We all have the same mindset of improving quality. We are able to have a similar language now to speak to everybody. [We’re all] in the same room in the trainings and we can [learn] from each other. When we talk about something and then someone else starts to talk about the same thing, we know we’re not alone on that point.
—Program staff
[C]ulturally specific organizations are, relative to other organizations, newer. While our practices are well aligned to these practices from [YPQ], we didn’t call them what we now call them. This has allowed us to have conversations with other organizations using the same language.
—Program staff
With the support of the Initiative learning community, participating programs typically go through the assess-plan-improve cycle once per school year during each three-year grant cycle. Each time, they complete a self assessment and an external assessment, followed by targeted program improvement planning.
Self assessment consists primarily of program staff observing one another, preferably peer to peer. During each observation, staff take notes on what they see and rate various aspects of programming using a detailed rubric-based assessment tool (Form A). Ideally, they conduct multiple observations and then discuss and consolidate their ratings into a single collective set of program scores, but programs can still benefit from the process with more limited observation data.
Initiative learning community
External assessment consists of an observation and scoring by a trained outside observer who also typically asks program staff and leaders a series of questions and may review results with them to support understanding.
Periodically, participating programs also complete a questionnaire (Form B) about the organization, program policies, and structural factors that are not observable but are important to support high-quality programming. Programs can use this data as well as the self and external assessment results in their program improvement efforts.
The Initiative team does not use quality assessment results for funding decisions, but to help programs plan and improve. This helps maintain learning and improvement as priorities and encourages participating programs to engage authentically in the program quality improvement process.
read more about this in what we’RE LEARNING.
The Initiative has funded the training of a cadre of Oregon program staff and leaders as external assessors, enabling peer organizations to assess one another. This structure benefits the external assessors—who are exposed to ideas they can bring back to their own programs—as well as the programs being assessed, which gain an external perspective. External assessors complete a two-day training, which includes reliability testing, and take an annual refresher course. So far, most external assessment has happened within the Initiative, as other systems haven’t yet required it of programs, but this effort increases the feasibility of external assessment as a resource outside the Initiative as well.
Following the assessment process, program staff create plans for improvement, drawing on assessment data as well as their own knowledge of program strengths, challenges and priorities. Ideally, plans are created collaboratively by staff teams for each program and focus on making changes that are feasible within the school year. Program leader submit copies of their improvement plans via the same system that captures assessment scores; this informs the Initiative team about what programs want to work on and creates an ongoing record for programs.
Rigorous training and coaching make this heavy lift feasible for program staff and support the full assess-plan-improve cycle:
- Each fall, program staff complete a full-day of training to learn how to complete self assessments and collectively score their program.
- Mid-year, after assessment scores are gathered, programs go through a half- to full-day workshop to review and reflect on their data and begin planning for program improvement.
- Throughout the year, coaching calls offer encouragement and help with planning and troubleshooting as needed. This individualized support is complemented by annual trainings in which program leaders learn to coach staff through the YPQ process and support the organizational change that comes with program improvement.
- Year-round, but with concentrated offerings each spring, Youth Work Methods trainings support implementation of specific aspects of high-quality programming. These trainings cover topics such as active learning, building community, homework help, reframing conflict and supporting youth voice. To increase access, trainings are held throughout the state and are also available to programs outside the Initiative.
Trainings are integrated with the learning community and offered flexibly to ensure that programs get the right trainings for their staff and structure. Weikart Center staff or contractors lead some trainings, while Institute for Youth Success staff or Oregon out-of-school time program staff trained by the Weikart Center lead others. The Initiative prioritizes train-the-trainer approaches to bolster statewide expertise and capacity.