blog posts and news stories

We Won a SEED Grant in 2022 with Georgia State University

Empirical Education began serving as a program evaluator of the teacher residency program, Collaboration and Reflection to Enhance Atlanta Teacher Effectiveness (CREATE), in 2015 under a subcontract with Atlanta Neighborhood Charter Schools (ANCS) as part of their Investing in Innovation (i3) Development grant. In 2018, we extended this work with CREATE and Georgia State University through the Supporting Effective Educator Development (SEED) Grant Program, through the U.S. Department of Education. In 2020, we were awarded additional SEED grants to further extend our work with CREATE.

Last month, in October 2022, we were notified that this important work will receive continued funding through SEED. CREATE has proposed the following goals with this continued funding.

• Goal 1: Recruit, support, retain compassionate, skilled, anti-racist educators via residency
• Goal 2: Design and enact transformative learning opportunities for experienced educators, teacher educators, and local stakeholders
• Goal 3: Sustain effective and financially viable models for educator recruitment, support, and retention
• Goal 4: Ensure all research efforts are designed to benefit partner organizations

Empirical remains deeply committed to designing and executing a rigorous and independent evaluation that will inform partner organizations, local stakeholders, and a national audience of the potential impact and replicability of a multifaceted program that centers equity and wellness for educators and students. With this new grant, we are also committed to integrating more mixed method approaches to better align our evaluation with CREATE’s antiracist mission, and to contribute to recent conversations about what it means to conduct educational effectiveness work with an equity and social justice orientation.

Using a quasi-experimental design and mixed-methods process evaluation, we aim to understand the impact of CREATE on teachers’ equitable and effective classroom practices, student achievement, and teacher retention. We will also explore key mediating impacts, such as teacher well-being and self-compassion, and conduct a cost-effectiveness and cost-benefit analysis. Importantly, we want to explore the cost-benefit CREATE offers to local stakeholders, centering this work in the Atlanta community. This funding allows us to extend our evaluation through CREATE’s 10th cohort of residents, and to continue exploring the impact of CREATE on Cooperating Teachers and experienced educators in Atlanta Public Schools.

We Won Two SEED Grants in 2020

Empirical Education began conducting the evaluation of Collaboration and Reflection to Enhance Atlanta Teacher Effectiveness (CREATE) in 2015 under a subcontract with Atlanta Neighborhood Charter Schools (ANCS) as part of their Investing in Innovation (i3) Development grant. Then, in 2018, we extended this work with CREATE and Georgia State University through the Supporting Effective Educator Development (SEED) Grant Program. And now, in 2020, we were just notified, that BOTH proposals we submitted to the SEED competition to further extend our work with CREATE were awarded grants!

One of the SEED grants is an extension to the one we received in 2018 that will allow us to continue the project for two additional years (through years 4 and 5).

The other SEED award will fund new work with CREATE and  Georgia State University by adding additional cohorts of CREATE residents and conducting a quasi-experiment measuring the effectiveness of CREATE for Cooperating Teachers (that is, the mentor teachers in whose classrooms residents are placed).  The study will examine impacts on teacher effectiveness, teacher retention, and student achievement, as well as other mediating outcomes. 

Updating Evidence According to ESSA

The U.S. Department of Education (ED) sets validity standards for evidence of what works in schools through The Every Student Succeeds Act (ESSA), which provides usefully defined tiers of evidence.

When we helped develop the research guidelines for the Software & Information Industry Association, we took a close look at ESSA and how it is often interpreted. Now, as research is evolving with cloud-based online tools that automatically report usage data, it is important to review the standards and to clarify both ESSA’s useful advances and how the four tiers fail to address some critical scientific concepts. These concepts are needed for states, districts, and schools to make the best use of research findings.

We will cover the following in subsequent postings on this page.

1. Evidence According to ESSA: Since the founding of the Institute of Education Sciences and NCLB in 2002, the philosophy of evaluation has been focused on the perfection of one good study. We’ll discuss the cost and technical issues this kind of study raises and how it sometimes reinforces educational inequity.
2. Who is Served by Measuring Average Impact: The perfect design focused on the average impact of a program across all populations of students, teachers, and schools has value. Yet, school decision makers need to also know about the performance differences between specific groups such as students who are poor or middle class, teachers with one kind of preparation or another, or schools with AP courses vs. those without. Mark Schneider, IES’s director defines the IES mission as “IES is in the business of identifying what works for whom under what conditions.” This framing is a move toward a broader focus with more relevant results.
3. Differential Impact is Unbiased. According to the ESSA standards, studies must statistically control for selection bias and other sources of bias. But biases that impact the average for the population in the study don’t impact the size of the differential effect between subgroups. The interaction between the program and the population characteristic is unaffected. And that’s what the educators need to know about.
4. Putting many small studies together. Instead of the One Good Study approach we see the need for multiple studies each collecting data on differential impacts for subgroups. As Andrew Coulson of Mind Research Institute put it, we have to move from the One Good Study approach and on to valuing multiple studies with enough variety to be able to account for commonalities among districts. We add that meta-analysis of interaction effects are entirely feasible.

Our team works closely with the ESSA definitions and has addressed many of these issues. Our design for an RCE for the Dynamic Learning Project shows how Tiers 2 and 3 can be combined to answer questions involving intermediate results (or mediators). If you are interested in more information on the ESSA levels of evidence, the video on this page is a recorded webinar that provides clarification.

Webinar: Uncovering ARRA’s Research Requirements

Researchers at Empirical Education provided a detailed overview of the various research themes and requirements of the ARRA stimulus initiatives with state department of education officials during their December 9 webinar entitled, “Meet Stimulus Funds’ Research Requirements with Confidence.“ The webinar gave specific examples of how states may start planning their applications and building research partnerships, as well as an overview of the ED’s current thinking about building local research capacity. The initiatives that were discussed included Race to the Top, Enhancing Education Through Technology, Investing in Innovation, Title I School Improvement Grants, and State Longitudinal Data Systems.

A follow-up webinar was broadcasted on January 20, 2010; it outlined a specific example of a program evaluation design that districts can use with existing data. Stay tuned for future webinar topics on more alternative experimental research designs.

Uncovering ARRA’s Research Requirements

Researchers at Empirical Education provided a detailed overview of the various research themes and requirements of the ARRA stimulus initiatives with state department of education officials during their December 9th webinar entitled, “Meet Stimulus Funds’ Research Requirements with Confidence.” The webinar gave specific examples of how states may start planning their applications, building research partnerships, as well as an overview of the ED’s current thinking about building local research capacity. The initiatives that were discussed included Race to the Top, Enhancing Education Through Technology, Investing in Innovation, Title I School Improvement Grants, and State Longitudinal Data Systems.