Many public school districts have countless programs and initiatives and struggle to identify which ones are having the greatest impact for students. Here’s how Academic Return on Investment (A-ROI) can help.
Public education leaders are passionate problem solvers. We observe the needs of students, teachers, and other members of our teams and rush to find a solution to those problems. We use our own experiences and existing research to the best of our ability to implement initiatives as quickly as possible because we feel the urgency of doing everything we can to educate students.
However, the result of decades of implementing such solutions, compounded by changes in leadership, an evolving workforce, and shifting student populations and needs, results in what I used to refer to as the “ugly Christmas tree” when I was in district leadership. Too many decorations that, while individually well-intended, don’t work well together and weigh down the very thing they were intended to support.
At DMGroup, we observe this in districts all over the country - too many initiatives throughout the year that overwhelm, confuse, and distract school leaders and educators from meeting the needs of the students in their classrooms.
Recently, we worked with a district to develop an inventory of all central office initiatives and identified more than 260 efforts… a proliferation that left principals feeling overwhelmed and frustrated—sentiments that were clear as we conducted surveys and focus groups. The district leadership, led by a new superintendent, was excited to use the information as a baseline to implement the principles of Academic Return on Investment (A-ROI) across the district in order to better focus their central office time and energy on the efforts that would result in the right outcomes—at the right level of investment.
Academic Return on Investment (A-ROI):
What works, for whom, and at what cost?
A-ROI helps districts answer the question: What works, for whom, at what cost? While this may sound like a simple question, it requires a deeper look at the three individual components.
What Works?
In most cases, this metric will be defined using specific student outcomes - achievement, growth, attendance, survey feedback, mastery of clear learning objectives, etc. Doing this well requires going beyond simply defining the right metric and setting a goal. Leaders and initiative owners need to consider the following:
- What is the objective or problem that we are trying to solve with this effort?
- Based on that, what is the right metric and what is the specific definition of success that will allow us to see if the result of the effort is meeting the objective we set?
- How will we collect reliable data in a cost-effective way?
The key is specificity in the objective, the metric, and the definition of success to ensure that an objective and rigorous discussion will be possible at the right time.
For Whom?
Tied to setting the objective for an initiative is having clarity about which students the initiative is intended to support. While we in public education have gotten better at focusing on specific student populations and subgroups, it usually remains at the level of key demographic indicators such as IEPs, English Language Learners, free and reduced lunch, or race.
The reality is that there are many other ways to identify student segments that are much more important to understanding whether or not an initiative is meeting the needs of a more nuanced and high need set of students. For example, could you target an attendance program to students with the highest mobility rate? Or an early childhood literacy intervention to students furthest behind in fluency? Or a newcomer English learner program to refugees from a specific country?
Of course, while these considerations may allow for a higher impact effort, you need to balance with the data systems you have and make sure that you can efficiently collect data for the segment you define.
At What Cost?
This final consideration often requires the most significant shift in how districts think about the A-ROI of an initiative. It’s easy to quantify the dollars that come out of the budget to fund an initiative—hiring consultants, purchasing curriculum materials, even the cost of substitutes to release classroom teachers for training. These cash costs are readily known and commonly tracked. However, in most districts, at least 80% of the overall budget goes to people, so our most valuable resource is time.
To fully understand the cost to implement an initiative, we need to calculate the cost of the time required by teachers, school leaders, and central office staff and leadership. Even though we are all incredibly committed to working the long hours required to lead and manage districts, at the end of the day, an hour you spend on one thing is one less hour than you have to spend on something else. This is why we calculate the full and complete cost including the personnel costs as we consider the A-ROI of an effort.
Using A-ROI Analysis to Take Action Based on Evidence of Effectiveness
By thinking about the questions above at this level of depth, districts can improve budgetary decision-making and ensure that they are getting more from limited resources. At DMGroup, we’ve worked with districts across the country to apply A-ROI analysis and insight in three high-impact ways: embedding A-ROI into the budget process, using an A-ROI lens to select new initiatives, and conducting an A-ROI program evaluation.
Embedding A-ROI into the Budget Process
Ideally, A-ROI is at the core of all resource allocation decision-making processes, helping us get out of the “Keep” or “Eliminate” paradigm that we often find ourselves in where we keep programs as they are or fully cut them all together, usually due to the departure of the champion or blanket cuts given budget shortfalls.
By having a better understanding of what is working, for which students, and at what cost, district leadership and budget owners are able to make more strategic and data-driven decisions. A district using A-ROI methodology might make an informed choice to expand highly cost-effective programs, target programs that are only successful for some student segments, reduce expensive but impactful programs, and fix programs that could be strengthened were a structural or systemic problem to be addressed.
Of course, fully embedding A-ROI throughout the budget and resource allocation process takes time. Luckily, there are smaller and more rapid ways to begin to incorporate A-ROI into your district.
New Initiative Approval and Piloting
One of the fastest ways to use A-ROI to inform decision-making is to incorporate the principles into the review and approval of new ideas. By requiring all new budget requests above some set level to include a specific proposal that includes objective and success measures, target student segments, and fully loaded costs, you can both improve your decision-making and begin to shift the culture around and capacity for embedding A-ROI in your district processes. Having this level of discipline at the beginning of an effort can also set up more lean and effective piloting processes, as you can plan on looking at the effectiveness along the way to determine if new ideas should continue to be implemented and expanded.
“Quicker” Program Evaluation
While nothing is as good as comprehensive research—and many large-scale initiatives require a thorough research plan—the cost and duration required to do it well can be prohibitive. A-ROI can be applied to a specific initiative to provide faster information than a full research effort. For example, we recently worked with a district that wanted to find a reading assessment that was a strong indication of expected performance on the state assessment—at the right price and for all subgroups. With that objective, the metric became the correlation between two different assessment programs and the state assessment results overall and for specific subgroups. With this clear focus, one program was found to have a much stronger correlation than the other and at only a quarter of the full cost. While the more expensive program had more features that seemed valuable, they were not necessary to meet the specific need and objective for this program in the broader instructional strategy, so the district was able to save money and get better results.
Better Decisions, Better Budgets, and Better Student Outcomes
By taking a deep look at A-ROI and applying it to improve our ability to make resource allocation decisions in our districts, leaders can better harness our passion for problem-solving to focus and improve the support we provide to our educators and get better results for our students.
Want to learn more about Academic Return on Investment (A-ROI)? Check out these resources: