Moving Forward

As of the 2020–2021 school year, 41 programs are actively participating in the Initiative, many of which have at least one more year of eligibility. Additional funding opportunities will be announced in mid-2021. The Initiative team will continue to seek participation from diverse group-based programs. While details are still in development, program quality improvement using the YPQ will remain central. Positive adult role models, academic support and family engagement will also continue to be core to our framing of program quality.

Given that three years is the minimum needed for programs to fully implement YPQ and begin to make meaningful program improvements, the Initiative team recognizes that programs need more time to integrate and sustain the YPQ process and its benefits. In 2019, a possible fourth and fifth year of (reduced) funding and training was made available to help programs continue this work. The Initiative team expects to continue this option and is considering other ways to both expand Initiative accessibility and support deeper or more sustained program quality improvement.

OSU SMILE Program

The Initiative team will continue to prioritize learning and relationship-building with participating programs, and to focus on authentic engagement in the program quality improvement process.

The learning community will continue to be central to the Initiative’s design. While the core components of the learning community are unlikely to change, the Initiative team will continue to adjust in response to the needs and strengths of participating programs.

The Initiative will continue investing in the professional development of program leaders and staff through the learning community along with extended training and convening opportunities through the Institute for Youth Success. The team remains committed to making Youth Work Methods and other core trainings available to programs regardless of Initiative participation.

The team will continue to champion high-quality out-of-school time programs in conjunction with our broader Foundation efforts to address the opportunity gap, including through education-related advocacy.

The Institute for Youth Success will receive ongoing support as a resource for programs, facilitating the learning community and providing Youth Work Methods trainings throughout Oregon.

The Initiative team will continue coordinating and collaborating with other funders and networks to support investment in high-quality programs and program quality improvement.

While the Initiative will continue and even strengthen many of its existing components, it is also at a turning point. In 2021, we will review and refine the Initiative's goals and theory of change to better reflect what we have learned. This will help us to communicate more clearly about the Initiative and to broadly advocate for high-quality out-of-school time programs.

The team will also continue adapting to input and feedback from participating programs. Many of the ways that the Initiative will continue and deepen its work are responsive to the strengths and needs of participating programs and lessons learned through the Initiative to date. For example, our focus on program quality improvement is a direct result of programs providing evidence of its value.

Finally, the team is also considering ways to extend or deepen support for culturally specific programs as well as social and emotional learning, whether formally woven into content or more informally integrated through program elements like strong adult-youth relationships.

Adelante Mujeres—Chicas Program

In a typical school year, participating programs would have gathered once in fall for a learning community kickoff and YPQ training and completed a round of self and external assessments by the end of December. The learning community would then have had a multiday in-person convening in January, during which they would have worked to make sense of their YPQ data and started the improvement planning process for the year. The January convening would have also provided opportunities to network and learn with and from one another.

But 2020 was not a typical year. And so far, neither is 2021. When the COVID-19 pandemic began to impact Oregon schools in March 2020, it also affected out-of-school time programs. Some shifted their work to respond to the emergency, providing basic supports to their students and families well outside their normal operations (including delivering food and technology supplies). Others became vital pathways for schools to connect with students and families.

Camp Fire Central Oregon

Many programs pressed on with programming as well. Some shifted their models and delivery to meet and support students in new locations—outdoors at parks, in student driveways, and virtually. Some were approved as emergency child care providers and shifted programming to comply with those requirements. Others continued working on quality improvement, even noting that staff had more opportunities to focus on planning while they waited to see what would happen with school and out-of-school time programs in spring.

The Initiative team responded to the pandemic by releasing programs from improvement planning and reporting requirements, and by adjusting learning community activities to ensure they would be relevant while also making participation optional rather than mandatory. While some programs have continued to work on program quality assessment and improvement, the Initiative has not required internal or external assessments, or data on program improvement, since the pandemic’s onset.

Beginning in spring 2020, the Initiative supported the Institute for Youth Success in offering a series of monthly virtual conversations and learning opportunities to help out-of-school time and other youth development organizations adapt and plan during the pandemic.

These opportunities continue and are open to anyone, regardless of participation in the Initiative. They are planned on an ongoing basis, depending on needs identified by participating organizations. Sessions have featured experts who share information about various aspects of their work. For example, the Native Wellness Institute session focused on trauma-informed practice. Where appropriate, Initiative participants have access to more in-depth opportunities in addition to the shorter sessions that are open to any interested organization.

Initiative learning community

 

As the deadline for grant renewal approached in late summer 2020, the Initiative team communicated with as many participating programs as possible to learn how they were doing and about their expectations for the upcoming school year. Most requested grant renewal and signaled a desire for continued support. An entirely virtual, restructured learning community launched in fall 2020 with continued monthly sessions led by the Institute for Youth Success.

As we look forward to schools reopening fully in 2021, we also expect to see out-of-school time programs shift to in-person programming. As we find out more about what that will look like, the Initiative team will continue to adapt, seeking ways to reincorporate accountability for PQA and learning community participation while also responding to evolving program needs. 

The Initiative evaluation’s efforts to track participating students from 2014–2017 conclude with this report. We have learned a lot about these students and the programs that shared data with us, and about the process of tracking students’ paths through high school. We are incredibly grateful to the program staff, families and students who made this work possible, but our findings do not necessitate continuation of student outcome analysis.

However, the evaluation team will continue to help the Initiative team adapt and improve the Initiative and its learning community. This will entail soliciting input and feedback from participating programs as well as supporting refinement of the Initiative’s goals and theory of change based on what we’ve learned. The evaluation team will also continue to support evaluation capacity-building that complements the YPQ process and trainings for participating programs.

DOWNLOAD A BRIEF, PRINTER-FRIENDLY PDF ON REFRAMING OUT-OF-SCHOOL TIME IMPACTS.

Arts in Education of the Gorge

Out-of-school time programs can bolster opportunity for the students most likely to experience the opportunity gap: students of color, students from under-resourced rural communities, and students from low-income families. Funders and programs can prioritize engaging these students with help from intermediaries and evaluators.

Middle school is an important developmental phase; student experiences and relationships during this time can be transformative. But this age range is often overlooked by providers, funders and others in favor of early childhood and other important transition points. Funders and programs can prioritize or proactively include middle school youth in their plans, and intermediaries and evaluators can help them do so.

Out-of-school time staff are perhaps the most important program resource. They set the conditions and build the relationships that support youth success. Programs need adequate funding to support staff and provide them with professional development opportunities. Offering a living wage or otherwise incentivizing retention can mitigate turnover and build program quality. This may require creative scheduling or division of responsibilities for staff.

Funders can provide sufficient funding to encourage programs to pay a living wage and to otherwise incentivize staff to stay with programs across multiple school years. Funders and intermediaries can increase opportunities for staff development through training and professional advancement. Evaluators can attend to these organizational factors when assessing initiatives or programs.

Improving program quality is an important way to address the opportunity gap experienced by students of color, students from under-resourced rural communities, and students from low-income families. Authentic and long-term engagement in program quality improvement bolsters staff capacity and can mitigate the effects of staff turnover.

This requires capacity and commitment from program leaders and staff, as well as training, coaching and coordination, which are often provided by intermediaries and sometimes reinforced by evaluation capacity-building. Program leaders can ensure staff are included and supported in program quality improvement efforts and can model learning-focused improvement (as opposed to focusing solely on accountability).

Funders can invest in robust, long-term implementation of program quality improvement processes (three years or more) and ensure program leaders and staff are adequately trained and supported to engage in that process. Evaluations can complement or support program quality improvement rather than getting in the way. Limiting or taking great care in using program quality data will encourage authentic engagement in the improvement process.

OSU Extension Service Garden Club

Program leaders and staff benefit from being part of a network or community of practitioners. Robust and fun learning and community-building opportunities of both the “sage on stage” and peer-to-peer variety—as well as meaningful, supported networking activities—can build community and support learning across organizations, programs and places.

Program leaders can support staff in attending trainings and networking with peers, and can also model or encourage authentic engagement. Intermediaries and funders can work to increase the availability of professional development (training and networking) opportunities, recognizing that this may require funding and other support both for programs and for the people or organizations that facilitate learning (often intermediaries). Evaluators can incorporate capacity-building through participatory analysis and program-specific coaching to strengthen data collection and use.

Initiative learning community

Out-of-school time programs can position students for success and otherwise benefit communities in valuable ways. Their role in supporting students is often underappreciated within the broader educational ecosystem. Programs can invest in advocacy and work with their peers to elevate others’ understanding of the role of their programs in supporting students.

Intermediaries and funders can support programs by assisting with or funding efforts to communicate about their work, including about the quality of their programming and its impacts on students and families. Intermediaries and funders can strengthen communication about and advocacy for the role and value of out-of-school time, particularly in interactions with the education system and with policymakers. Evaluators can make findings accessible and support programs, intermediaries and funders in disseminating results in ways that further improve everyone’s work.

Out-of-school time programs are just one piece of the student experience and cannot close the opportunity gap alone. Even as students are engaged and motivated toward academic futures—and supported by peers and adults to grow their confidence, skills and abilities—larger systemic barriers remain in place, including within the educational system. Systemic change in education is needed to address structural racism and resource barriers in order to eliminate these gaps.

Camp Fire Columbia

In the meantime, the connection between out-of-school time and in-school learning can be strengthened. Locally, out-of-school time programs are well positioned to build relationships with their schools but may need support or reinforcement in making those connections. Intermediaries and funders can provide training and connections and can act as champions for coordination and collaboration between out-of-school time and in-school learning. Evaluators can include these contextual or programmatic factors in their assessment of program or initiative impacts. Social and emotional learning may provide a valuable bridge between out-of-school time programs and schools. As schools increasingly prioritize social and emotional learning, programs can provide expertise and support both locally and at a regional or statewide level.

Out-of-school time programs can play a powerful role in supporting social and emotional learning for students and elevating its importance within the educational system. Social and emotional learning is likely to be a critical bridge between schools and out-of-school time programs in the coming years—one that would benefit from investment, potentially via partnership or coordination between public education dollars and private philanthropic dollars.

We are at a pivotal moment in the history of the out-of-school time (OST) field. Multiple sectors previously siloed, have begun to collaborate under the common notion that 21st-century skills and social and emotional competencies are critical to success in school, in work, and in life. For OST, we have an unprecedented opportunity to lead—to be experts, build on our best practice, collaborate and provide systemic support, and have a voice in this movement. Together, with the fields of education, workforce development, and mental health, we can provide the platform for youth to create their positive futures (Devaney & Moroney, 2018, p. 245).

Programs can reflect on how they already support social and emotional learning, whether formally or informally, and consider building on their strengths to meet student needs. Program design and goals should determine which components of social and emotional learning to prioritize and how best to strengthen them. Programs can also attend to social and emotional learning for program staff, who will need to develop self-awareness and their own competencies as they are called on to model social and emotional learning. Last, researchers and evaluators can continue working to learn about how high-quality programs integrate and support social and emotional learning and about the various social and emotional learning competencies themselves and how best to measure them.

[Out-of-school time] organizations must clarify their approaches to supporting [social and emotional learning] and create cultures and environments throughout their networks that consistently encourage staff and volunteers to model social and emotional skills for youth. High-quality [social and emotional learning] programs and practices need to be studied and documents… and schools and [out-of-school time] providers must intentionally partner to support young people’s success (Goldberg et al., p. 240).

OSU SMILE Program

Eugene School District 4J

A growing body of research demonstrates the value of out-of-school time programs. As a result, researchers are increasingly focused on how these programs can be most effective—i.e., what high-quality programming looks like and how to improve it—as well as on student progress measures that are closer to programs’ locus of control, including social and emotional skills. The out-of-school time field and its researchers are also better contextualizing programs as one part of the larger system that supports students and their families.

In recognition of the value of out-of-school time programs and the complexity in which they operate, programs, intermediaries, funders and evaluators should shift focus and expectations away from standardized educational outcome measures and toward meaningful progress measures that connect more directly to what happens during programming.

DOWNLOAD A BRIEF, PRINTER-FRIENDLY PDF ON REFRAMING OUT-OF-SCHOOL TIME IMPACTS.

We also need to better attend to systemic factors that drive student success, including whether they have access to high-quality learning opportunities and to the resources they need to navigate the educational system. Because these factors are structural, they need not be tracked at the individual student level.

Measuring these conditions for success also shifts attention, and the burden for ensuring improvement, to adults. For example, we should ask whether students have adequate support to engage in school—and whether barriers and inequities that prevent engagement are being removed—rather than asking whether student test scores are rising.

This will contextualize what programs are doing and can do within the educational system. Coupled with program quality measurement and more relevant, realistic progress measures within programs themselves, evaluation can become a much more useful tool. Evaluators can help programs understand and communicate their value in ways relevant to the scope and focus of their programming, support program improvement, and assist programs in advocating for high-quality learning experiences in the educational system.

Rather than (or perhaps in addition to) funding specific projects or new or expanded programming, funders can support ongoing high-quality programs, which often have few funding opportunities. Funders can also support locally developed programs, which are often more responsive to student and community needs but usually lack the support that comes with being part of a national or regional organization.

The diversity of Initiative-supported programs has strengthened the learning community and added nuance to our understanding of what effective programming can look like. The YPQ defines high-quality programming in a way that is both content- and context-neutral, because a wide range of program types, approaches and contexts can be effective.

Thus, instead of focusing narrowly on specific types of programs, funders can support programs with a strong academic focus, those that are culturally specific, or those that incorporate social and emotional learning.

Hacienda CDC

Longer-term funding can stabilize and strengthen programs, but funding that lasts more than a year or two without extensive renewal or shifting eligibility requirements is rare. Three years seems to be the minimum time needed for programs to gain enough experience and expertise to begin embedding program quality improvement internally; efforts to support quality improvement should guarantee funding or require a low burden for renewal over that time period.

When funders approach initiatives with a learning mindset, it sets up the conditions to build relationships with participating programs and other stakeholders. This benefits all groups as engagement deepens, information is shared more readily and adaptations can be made when needed. This is especially important when funders are promoting or requiring programs to engage in continuous improvement, as it models and validates the value of learning and eases pressures that might otherwise become barriers to learning, such as fear of failure or loss of funding.

Setting clear expectations for engagement in learning communities or program quality improvement processes is important, as is holding programs accountable. In addition to lowering the stakes of program quality assessment or outcomes measurement—by not tying the results to continued funding, or drawing conclusions about the value of programs based on limited data sets—this allows programs to engage authentically in the process, generate accurate and useful data, acknowledge and address challenges or mistakes, and apply what they learn.

Camp Fire Columbia

To make substantial progress toward improved educational outcomes for students, funders must recognize and address the systemic barriers and structures that generate or reinforce the opportunity gap and resulting educational disparities. This includes considering the accessibility and quality of out-of-school time programs.

Systems change—including entirely different funding structures—is necessary to ensure equitable access to high-quality out-of-school time programs. Without systemic change and/or substantially expanded government investment, out-of-school time programs will continue to rely on philanthropy to maintain and improve programming. More importantly, students will continue to face barriers to accessing the support these programs provide.

Out-of-school time programs also need to be thoroughly integrated into efforts to dismantle systemic structures that currently oppress students of color, students from under-resourced rural communities, and students from low-income families. Fortunately, many programs are well positioned to support efforts to improve systems and school culture. The education system can turn to out-of-school time programs for their expertise in student and community strengths, their knowledge about the barriers students and families experience, and their expertise with the frameworks and tools needed to shift education, including youth-adult partnership, student voice, social and emotional learning, and restorative justice.

DOWNLOAD A BRIEF, PRINTER-FRIENDLY PDF ON REFRAMING OUT-OF-SCHOOL TIME IMPACTS.