blog posts and news stories

Empirical Education Partners with Carnegie Learning on New Student Performance Guarantee

Schools looking to improve student Algebra and Geometry achievement have signed up for a guarantee from Carnegie Learning® that states that students using the company’s Cognitive Tutor programs will pass their math courses. Empirical Education is tasked to monitor student performance in participating schools. Starting this school year, Carnegie Learning guarantees that students who take three complete and consecutive years of Carnegie Learning’s math courses will pass their math class in the third year. The guarantee applies to middle and high school students taking the Carnegie Learning Bridge to Algebra, Algebra, Algebra II, and Geometry courses.

In the coming weeks/months, Empirical will collect roster data, course grades, and assessment scores from schools as well as usage data from Carnegie Learning’s math teaching software. These data will be combined to generate biannual reports that will provide schools with evidence they can use to effectively improve implementation of the courses and raise student achievement.

Carnegie Learning’s guarantee is part of their School Improvement Grant support efforts. “Partnering with Empirical Education will allow us to get mid- and end-of-year research reports into the hands of our school partners,” says Steve Ritter, Co-Founder and Chief Scientist at Carnegie Learning. “It’s part of our continuous improvement cycle; we’re excited to see the progress districts committed to the turnaround and transformation process can make with these new, powerful tools.”

Empirical Education Focuses on Local Characteristics at the 14th Annual CREATE Conference

Empirical Education staff presented at the National Evaluation Institute’s (NEI) 14th annual CREATE conference in Wilmington, North Carolina. Both presentations focused on the local characteristics of the evaluations. Dr. Denis Newman, president of Empirical Education, and Jenna Zacamy, research manager, presented a randomized experiment which evaluated the impact of a pre-algebra curriculum (Carnegie Learning’s Cognitive Tutor Bridge to Algebra) being introduced in a pilot program in the Maui School District. The district adopted the program based in part on previous research showing substantial positive results in Oklahoma (Morgan & Ritter 2002). Given the unique locale and ethnic makeup in Maui, a local evaluation was warranted. District educators were concerned in particular with their less experienced teachers and with ethnic groups considered at risk. Unlike in prior research, we found no overall impact although for the algebraic operations subscale, low scoring students benefited from being in the Cognitive Tutor classes indicating that the new program could help to reduce the achievement gaps of concern. We also found for the overall math scale that uncertified teachers were more successful with their Cognitive Tutor classes than their conventional classes. Dr. Newman also presented work co-authored with Marco Muñoz and Andrew Jaciw on a quasi-experimental comparison, conducted by Empirical Education and Jefferson County (KY) schools, of an activity-based middle-school science program (Premier Science) to more traditional textbook programs. All the data were supplied by the district including a rating of quality of implementation. The primary pretest and outcome measures were tests of science and reading achievement. While there was no discernible difference overall, poor readers gained more from the non-textbook approach, helping to diminish an achievement gap of concern to the district.

Reports Released on the Effect of Carnegie Learning’s Cognitive Tutor

The Maui School District has released results from a study of the effect of Carnegie Learning’s Cognitive Tutor (CT) on long-term course selections and grade performance. Building upon two previous randomized experiments on the impact of CT on student achievement in Algebra I and Pre–algebra, the study followed the same groups of students in the year following their exposure to CT. The research did not find evidence of an impact of CT on either course selection or course grade performance for students in the following school year. The study also found no evidence that variation among ethnicities in both the difficulty of course taken and course grade received depended on exposure to CT.

A concurrent study was conducted on the successes and challenges of program implementation with the teachers involved in the previous CT studies. The study took into account teachers’ levels of use and length of exposure to CT; the descriptive data comprised surveys, classroom observations, and interviews. The major challenges to implementation included a lack of access to resources, limited support for technology, and other technological difficulties. After 3 years of implementation, teachers reported that these initial barriers had been resolved; however teachers have yet to establish a fully collaborative classroom environment, as described in the Carnegie Learning implementation model.

Maui School District is the company’s first MeasureResults subscriber. A similar research initiative is being conducted at the community college level with The Maui Educational Consortium. The report for this study will be announced later this year.

Maui Community College Hires Empirical Education for an Evaluation of NSF-Funded Project

In Hawaii, Ho’okahua means “to lay a foundation”. Focusing on Hawaiian students over multiple years, the Ho’okahua Project aims to increase the number of Maui Community College (MCC) students entering, persisting, and succeeding in college level science, mathematics, and other STEM (Science, Technology, Engineering and Math) degree programs. Several strategies have already been implemented, including a bridge program with the high schools from which the MCC student community is largely drawn.

The Maui Educational Consortium provides leadership for this work and has been instrumental in a number of other initiatives for increasing the capacity to achieve their goals. For example, the implementation of Cognitive Tutor for Algebra 1 was the subject of a related Empirical Education randomized experiment. Another important capacity fostered by the Educational Consortium, working with the University of Hawai’i Office of the State Director for Career and Technical Education, is an initiative called HI-PASS, which aggregates student data across high school and community college. Initially in its evaluation, Empirical Education will be using information on math courses developed through the HI-PASS project to follow the success of students from the earlier study.

Blue Valley Schools and Empirical Education Explain Local Program Evaluations

Dr. Bo Yan, Program Evaluator for the Blue Valley Schools in Kansas, and Dr. Denis Newman, president of Empirical Education, co-presented at the Northwest Evaluation Association’s annual Members Seminar in Portland OR. The topic was how school districts can use their own testing such as that administered by the NWEA member districts to conduct their own local program evaluations. Dr. Yan, who is expecting to conduct seven such evaluations in his district this coming year, used an evaluation of READ 180 as an illustration of a comparison group design using primarily statistical controls. Dr. Newman presented the randomized control work Empirical Education has done with the Maui school system to evaluate math software and curriculum from Carnegie Learning (Cognitive Tutor: Year 1 and Year 2). Both emphasized the importance of local evaluations to estimate the impact of the programs for the specific populations and resources available to the district. They also made clear the need for a comparison group from the local district since the improvement a district can expect is anchored in its own students’ current level of achievement. While the presentation focused mainly on the use of NWEA’s Measures of Academic Progress (MAP) in the quantitative estimation of the program’s impact, the presenters also emphasized the importance of gathering information on implementation and of the conversations that must go on to integrate evaluation findings into the district’s decision-making. NWEA will be providing this presentation, along with the PowerPoint slides, as a podcast.

Maui Schools Sign Subscription Agreement for Empirical Education Research Services

Empirical Education will be providing research services to the Maui School District through an innovative subscription arrangement for MeasureResults™, an Internet-based research and consulting offering. The initial application of this service will be investigations of the longer term impact of the Cognitive Tutor program that was implemented under a Math and Science Partnership program grant.

The company’s MeasureResults service is a response to the ever increasing demands on school systems to validate their program and spending decisions based on the analysis of solid data. Most districts do not have the staff and facilities to set up data and run complex statistical analyses. In Maui, the service will take advantage of the sophisticated data warehousing capabilities being put in place statewide. MeasureResults is designed to simplify the technical and logistical steps of conducting experiments by building powerful and verified analytical techniques into an uncomplicated framework. The offering includes consultative services on research design and web-based interfaces to gather data, automate analysis, and generate reports. MeasureResults is bundled with technical support that includes review of all analyses and reports by trained statisticians.

Experiment Findings about Cognitive Tutor Not What Was Expected

Empirical Education released the results of its year-long randomized experiment on the comparative effectiveness of the Cognitive Tutor program, conducted in cooperation with the Maui School District in Hawai’i. Funded through a grant from the U.S. Department of Education to Empirical, the research evaluated the local implementation of Carnegie Learning’s Algebra I product. Company president Denis Newman explained the results, saying, “We were surprised by the results and consider this study something we can all learn from. Nobody expected a negative impact on the experienced teachers but this may show that it takes some time to get back up to their high level of performance when learning a new way of teaching. Following these teachers’ progress into a second year would help to verify this explanation. This effect was not found in previous research, which again points to the importance of conducting many experiments on any intervention before coming to a conclusion about its effectiveness.” A second experiment in the same school system, focused on the new Bridge to Algebra product, is nearing completion.