Taming the “Wild West”: Region 1 Comprehensive Center Develops Tool to Help Educators Assess Literacy Interventions

After the pandemic, many students returned to the classroom behind on reading, as shown by data in the National Assessment for Educational Progress (NAEP). Those students are now adolescents and approaching a critical turning point in their education: They are expected to read with greater independence.

Massachusetts 2022 NAEP data show eighth-grade reading scores declining. Only 40% of eighth graders demonstrate reading proficiency. Furthermore, reading score disparities exist between White students and historically marginalized students, particularly Black and Hispanic students. For multilingual learners, the difference in outcomes is especially pronounced.

For students who are still struggling to catch up in Grades 4–12, this shift can affect every aspect of their education. However, educators do not know how to gauge which interventions may be effective and targeted to a particular student’s needs.

“One of the big things we learned from educators was that the process of choosing intervention materials for older students was really a wild west,” said Katherine Tarca, director of literacy and the humanities at the Massachusetts Department of Elementary and Secondary Education (DESE). “People may end up just Googling ‘interventions for middle school’ because information is so hard to find.”

Donna Goldstein, English language arts and literacy content support lead at DESE, added that they had already made great inroads in literacy development for younger grades but saw opportunities for improvement when it came to older students.

“We didn’t have much for adolescents,” she said. “Educators were asking us to provide something to help them.”

To fill that need, DESE partnered with the Region 1 Comprehensive Center (R1CC) to develop a tool that would help districts identify and implement high-quality interventions in their classrooms—while allowing them the flexibility to choose the right curricula for themselves.

Project Overview

Together, R1CC and DESE developed the Adolescent Literacy Intervention Selection Tool (A-LIST). This tool helps schools and districts select literacy interventions that align with the Institute of Education Sciences (IES) practice guide Providing Reading Interventions for Students in Grades 4–9 (2023) and with culturally and linguistically sustaining practices.

“We would never have been able to do this without that partnership,” Tarca said. “Our R1CC partners were very knowledgeable about this already, and they were able to connect us with other experts.”

She added that IES released the practice guide at an opportune moment because it gave the experts at R1CC a solid foundation upon which to build the A-LIST rubric.

[Educators and education leaders] are really going to need the supports, the targeted thinking, and the evidence-based tools that this will guide people to.

– Donna Goldstein

Technical Assistance Approach

When developing A-LIST, R1CC first looked for tools and guidelines that already existed. R1CC reviewed education clearinghouses, including the National Center on Intensive Intervention’s Academic Intervention Tools Chart, the What Works Clearinghouse, and Evidence for ESSA (Every Student Succeeds Act).

“All of these resources are available, but can be unwieldy to use on their own,” Tarca said. “Educators cannot always sit down and read 500 pages of research. We wanted a tool that would help them sort through their options and winnow them down.”

Ultimately, the R1CC team developed two rubrics—one that measures literacy interventions against evidence-based recommendations outlined in the IES practice guide and another that measures the extent to which programs support enactment of culturally and linguistically sustaining practices. The team collected input on the rubrics from literacy specialists from DESE, the Region 9 Comprehensive Center, and the National Center on Improving Literacy.

After the rubrics were complete, DESE gathered two panels of district literacy specialists to assess a selection of literacy interventions using their rubrics. R1CC brought the panels together and facilitated training and scoring discussions. They examined eight different interventions and created a chart that outlines the panels’ findings.

This exercise allowed the team to accurately score interventions on whether they were aligned to the practice guide recommendations and were culturally and linguistically sustaining. Tarca said the last criterion really sets the A-LIST tool apart.

“Everyone is talking about culturally and linguistically sustaining practices,” she said. “But we have made it concrete by adding it to the scoring process.”

Now, the DESE and R1CC team is expanding its list of rated interventions by convening two additional scoring panels—one that includes interest holders from across the region and one consisting of Massachusetts-based literacy specialists.

Expanding Access Across States

With the scoring rubrics in place, the R1CC team held a workshop in March 2024 with literacy experts from each of the four Region 1 states (Maine, Massachusetts, New Hampshire, and Vermont). They trained participants on how to use the scoring rubrics to evaluate the alignment of an adolescent reading intervention to the evidence-based recommendations and to culturally and linguistically sustaining practices. Participants said the workshop helped them better engage in conversations with leadership and their peers.

“This process effectively lessened educator bias and subjectivity by providing the guide as a point of reference for scoring and requiring evidence to satisfy the rubric,” said one participant in the post-workshop survey. “Bringing the conversation back to the language of the rubric made the discussion less personal and more objective.”

The workshop also helped expand the impact of the A-LIST by fostering collaboration and connection between literacy experts across the region. Feedback from workshop participants was overwhelmingly positive.

  • 100% strongly agreed with the statement that “participating in this workshop has increased my capacity to understand and apply the evidence-based practice recommendations of the IES Practice Guide for adolescent literacy intervention.”
  • 94% agreed or strongly agreed with the statement that “participating in this workshop has increased my capacity to facilitate productive conversations with educators around the use of evidence-based literacy practices in adolescent intervention programs.”
  • 100% said they were likely or very likely to recommend this training to a colleague. 

Outcomes

We would never have been able to do this without that partnership [with R1CC].

– Katherine Tarca

A-LIST is a first-of-its-kind tool that is designed to provide information on both alignment to evidence-based practices and support for culturally and linguistically sustaining practices all in one place. It supports educators within Massachusetts and beyond as they better understand literacy interventions before investing time and money.

The tool helps eliminate some of the guesswork that comes from choosing interventions and also supports districts in implementing evidence-based strategies. Educators are empowered to select high-quality instructional materials that meet their specific context and needs.

“We’re really pushing the conversation forward with this tool,” Tarca said. “Maybe publishers will take notice, and it will help them improve their products. Then it becomes a positive upward cycle.”

She added that, although A-LIST was created for and tailored to Massachusetts schools, the rubric is universal. “I think what we made can be used nationwide because it’s filling such a big need,” Tarca said.

Goldstein said that A-LIST is especially valuable as students continue to recover from the pandemic. “They are really going to need the supports, the targeted thinking, and the evidence-based tools that this will guide people to,” she said.