Urban Alliance is proud to be a continuous learning organization, dedicated to constantly refining and fine-tuning our processes to deliver programming of the highest-quality possible to our students. Central to that mission is our small but mighty evaluation and monitoring team: Lauren Rice – Senior Director of Evaluation and Learning, and Midas Hampton – Research and Program Analyst.
Small But Mighty
Urban Alliance is in a unique position in the nonprofit world, with two in-house evaluators and a streamlined, robust internal performance measurement system. Investment in internal evaluation is sometimes wrongly regarded as a luxury at other direct-service nonprofits; we see it as a necessity to ensure a culture of continuous improvement.
Over the years, we have developed internal monitoring processes to track key performance indicators such as dosage, quality, and deliverable metrics to assess program uptake and fidelity across regions. Using a mixed-method approach and proprietary tools including surveys, observational rubrics, interviews, administrative databases, and staff-reported data collection, we monitor key outcomes in real time.
This strong foundation led Urban Alliance to join a small fraction of nonprofits who have completed an independent randomized controlled trial (RCT) – regarded as the gold standard of program evaluation. The first study, completed by Urban Institute in 2017, found Urban Alliance joining an even smaller group: nonprofits who completed an RCT – and saw measurable results. That study found that completing our signature High School Internship Program had a statistically significant impact on young men attending college, mid-GPA students enrolling in college, and students’ development and retention of key soft skills. But perhaps more importantly, it taught us so much about how we run our program, as well as areas that would benefit from increased attention, resulting in innovations in areas such as mentor training, student recruitment, and more intentional integration of social and emotional learning into our curriculum.
The value of such a detailed, independent study inspired us to engage in a second RCT, essentially a replication study to see if our results held across what was then all four of our regions and to dig deeper into additional areas for improvement. The first report from that second RCT was released this week – with initial results expected in the next year.
Future of UA Evaluation
In the meantime, UA is working to continually improve our internal evaluation process.
- Youth Voice
The most important of our core values is ‘students first,’ and that extends to evaluation. Student-reported data and student interviews have always been a key part of our data collection process, but in recent years we have moved toward a more student-centered, equitable, asset-based system in which students not only provide key data but help to provide insight into the evaluation process itself.
Starting this year, Urban Alliance is launching a Youth Evaluation Advisory Council to center student voices in the evaluation process from beginning to end. The council, comprised of current students and alumni from each of Urban Alliance’s regions, will meet regularly for focus groups to provide feedback on every aspect of the process from survey design, to language, delivery of data collection methods, and more. Not only will this council be an additional professional development opportunity for our students – who will be paid for their time thanks to the support of the Annie E. Casey Foundation – it will also help us to view our internal evaluation system through a new lens and better collect the data we need to keep modifying our program.
2. Democratization of Data
Though our evaluation team is small, evaluation at UA is an organization-wide process. From our on-the-ground program staff who help to ensure timely data collection across our regions, to a cross-functional evaluation working group providing guidance, to our national team providing input on new avenues for exploration, evaluation is central to our day-to-day work. Part of the reason for that is our commitment to data democratization – ensuring that every staff member has access to our data and every staff member has a voice in the process.
UA staff regularly review our results and review and test our data collection and management tools. Information is not siloed, but shared widely so that we can utilize the combined talents and insights of our team to identify strengths and weaknesses, brainstorm solutions and innovations, and collaboratively tackle new projects. Every decision we make is driven by our data and 25 years of youth workforce development expertise.
Data-Driven Decision-Making During a Pandemic
When the COVID-19 abruptly closed down schools and businesses across the country, Urban Alliance’s strong internal evaluation system was a critical part of our ability to quickly pivot to virtual programming. We immediately activated our data collection tools to get a status update – how are our students feeling, what immediate support do they need, which students lack access to technology, how are our job partners responding, what are our school partners doing, etc.
Committed to keeping our students earning and learning during the pandemic, we used this initial data dump to help redesign our program for the virtual world – and then continue to build it as we went along, building the plane while flying it. Relying on data from students, staff, and partners throughout, we were able to make real-time course corrections to our newly virtual program as the year progressed.
Student data also enabled us to proactively plan for the current program year as well. For example, we learned that the lengthy separation from school led students to crave a sense of community, and used that to redesign our Fall 2020 pre-employment training to intentionally create space for community-building within our new student cohorts. We also used student data to identify those students most at risk of disconnection from school or work post-graduation and target alumni services to help them navigate the difficult transition from high school to adulthood after they completed our program.
Continuous Growth
Like the broader organization, our evaluation process is constantly learning how to improve, to grow, and to do better by our students. It’s not enough to know that our program works – our evaluation team is working every day to make it work even better for the next student.