Centro Cultural of Washington County
Program & Staff Impacts
Bend Science Station instructor Katrina Brandis and Development and Marketing Director Lisa Bermudez talk about the value of the Initiative learning community for their staff and program.
Initiative participation is strengthening out-of-school time programs
Our evaluation indicates that programs and staff benefit from Initiative participation in interconnected ways. Importantly, these impacts ultimately benefit students as well.
The learning community is strengthening & supporting program staff through networking & professional development.
Arts in Education of the Gorge
Program leaders and staff report that the learning community is an important source of professional development opportunities and connections that are hard to find elsewhere. Program staff see these opportunities as energizing, especially the intensive, multiday annual convenings and the program quality improvement process. They also appreciate the learning community’s safe and supportive environment, where they are free to ask questions and share challenges without fearing judgment or loss of funding.
[Sometimes], people are doing this work solo within their organization. They are the only ones that understand their program and they are the only ones getting training on [program quality improvement], so this network is their only support and connection to the work. … [S]o many participants truly appreciate the connections of this network because of the thought partners and peer coaches and conversations, with a shared language, to help them brainstorm through challenges and solutions that they can now take back to their organizations/supervisors/teams/programs.
—OST stakeholder
Staff value the program quality improvement process despite the heavy lift it requires, often naming it as one the Initiative’s most valuable aspects. Program leaders appreciate its guidance on observing programming and coaching staff on program improvements, the rich conversations they have with colleagues as they work through the process, and the new data they gather about program quality that helps them communicate about their work.
The program quality improvement process also validates the importance of program staff, helps them feel heard and supported, encourages them to work toward goals and improve their own skills. The process also helps staff feel they are being heard, supported and given the tools needed to make improvements.
Not a lot of our staff came to us as youth development professionals. They came to us with great skills and abilities, but not everyone had a ton of experience with youth. We have built youth development professionals out of a group of really great people. And I think [the YPQ] has been instrumental in that.
—Program staff
The Initiative is boosting out-of-school time program quality.
Program leaders note that the program quality improvement process, as supported by the learning community, is an opportunity to take a holistic look at program quality while focusing on making manageable, meaningful changes. Program improvements made as a result of the process and learning community include:
- Developing more inclusive, intentional transitions from one activity to another.
- Implementing or improving staff training on specific aspects of program quality (e.g., ways to create a culturally welcoming environment).
- Creating youth advisory committees and other youth leadership structures.
- Incorporating reflection strategies into program sessions.
- Developing service learning and improved academic supports.
RiverStars Performing Arts
This work is also helping program leaders develop a learning mindset for their teams; they are trying new things and continually looking for ways to iterate and improve. Programs are encouraged to celebrate what they do well, acknowledge their strengths and build on capacities they already possess. The program quality improvement process also gives program leaders an opportunity to model the strengths-based development staff should use with youth.
When I see staff practicing [YPQ] with fidelity—when they have had time to learn and try—I see more joy in their role/work, and I see more joy from the kids.
—Program staff
Program leaders report that when staff feel appreciated and are not being asked to change things that are out of their control (like whether students attend school), they are more likely to engage in difficult conversations about how to improve program quality, and to subsequently make improvements.
At every staff meeting we open up with the victories we had the previous week and then we talk about the opportunities for improvement. When we talk about the opportunities for improvement, we are focused on what we need to improve, why we need that improvement, and how we would measure success. This program instilled a culture of quality that we simply did not have before.
—Program leader
The conversations we’re having with program development because of this tool have made us want to take this and spread it throughout our organization.
—Program staff
All of this supports programs in creating the conditions that most effectively support students. As a result, program staff report myriad ways in which their programs benefit themselves and students, including improved staff-youth relationships, greater youth program ownership and engagement, and stronger relationships between students as peers.
Staff reflections on the value of program quality improvement also illustrate why the Initiative team does not use program quality assessment scores to make funding decisions or monitor for quality score changes to determine whether improvement is happening. Instead, we position program leaders and staff as the primary users of quality data and support their use of the data to inform improvement planning. This has encouraged authentic engagement in the process, resulting in more meaningful program improvement. We have found that this focus on the process, rather than the scores, is critical.
That said, the evaluation team has explored aggregate assessment scores and improvement plans to look for broad patterns and trends. In doing so, we learned what participating programs were focused on. For example, it was helpful for learning community planning to know when youth voice and reflection were an area of potential growth and of interest to participating programs. However, these patterns and trends aren’t consistent enough for us to draw any specific conclusions about program quality.
We also don’t expect to see substantial score increases within the three-year period during which most programs participate in the process. This is because we know that over the first couple of years, programs are just starting to understand the process and can often become harsher critics of their own work. Seeing assessment score changes is also unlikely when different locations, components and staff are observed each year; it is rare that a program makes adequate observations of similar enough programming to draw conclusions from year-to-year changes in scores. However, programs do have access to their historical scores through an online database because reviewing previous years’ scores can generate conversations about how practice varies and help programs plan for improvement.
The Initiative is positioning out-of-school time programs and organizations for greater stability and capacity.
Program leaders and staff report that as a result of their participation in the Initiative—especially their engagement with program quality improvement through the YPQ—they have stronger staff and greater organizational capacity, and are better positioned to seek other funding and improve other aspects of their work.
[Our] direct service staff capacity increases with grant funding, to add more opportunity takes more staff. … [I]ncreased program quality has led to more interest which then leads to more need for staff. Funding has really helped add more, and more quality, staff.
—Program leader
Latino Network
Program leaders describe three or more years of funding as stabilizing and note many benefits of repeated engagement in the program quality improvement process. They describe organizational culture changes as well as specific ways that they have improved programming. The Initiative is also validating the importance of quality, social and emotional learning, and other program priorities, which helps them when approaching other funders for support.
I see our staff who have been through the trainings, and it’s changed the way to they talk to each other, it’s changed the way that meetings happen, the way that supervisors conduct meetings. I think it’s been really subtle, but it has had an institutional impact on our office. How it runs even outside of the service side, in terms of the internal communication in person and emails.
—Program leader
[YPQ is] finally giving us something tangible, measurable, a way to accurately depict what's going on in the [program]. We have struggled with that for a while. I’m excited about displaying the impact we’re having with kids, how intentional we are about how we show up for kids in programs. This helps others understand what we are doing.
—Program leader
It taught us to not be afraid to try something new and how to adjust quickly if things didn’t work out the way we thought they would. During the COVID crisis with so much that changed daily, my staff had the confidence to lay out a plan of action but to have a plan B and a plan C, because we knew things were going to change. We learned to focus not on all the things we couldn’t do, but to focus on the things we could do. When things started to change, we immediately went to the whiteboard to use the tools to figure out why and how we needed to change to fit the situation. That is such an important skill. But to have the confidence to embrace change and know that it was OK to change was invaluable. We couldn’t have survived the COVID crisis without that confidence and skill.
—Program leader
Finally, some program leaders cite Initiative participation as beneficial for their community relationships, including with schools and school districts.
[Our] school hasn't always been such a great partner. But having them come over and say we did such a good job [implementing programming for a new group of students]. … I was really proud of our staff, using the processes they’ve learned, looking at what we were trying to accomplish. Two years ago, we wouldn’t have been able to pull this off.
—Program leader
There is a perception that [our school] is rough, but this program has been changing that. The school is becoming a more positive place, even parents are connecting more and being more involved in a positive way.
—Program leader