blog posts and news stories

IES Releases New Empirical Education Report on Educator Effectiveness

Just released by IES, our report examines the statistical properties of Arizona’s new multiple-measure teacher evaluation model. The study used data from the pilot in 2012-13 to explore the relationships among the system’s component measures (teacher observations, student academic progress, and stakeholder surveys). It also investigated how well the model differentiated between higher and lower performing teachers. Findings suggest that the model’s observation measure may be improved through further evaluator training and calibration, and that a single aggregated composite score may not adequately represent independent aspects of teacher performance.

The study was carried out in partnership with the Arizona Department of Education as part of our work with the Regional Education Laboratory (REL) West’s Educator Effectiveness Alliance, which includes Arizona, Utah, and Nevada Department of Education officials, as well as teacher union representatives, district administrators, and policymakers. While the analysis is specific to Arizona’s model, the study findings and methodology may be of interest to other state education agencies that are developing of implementing new multiple-measure evaluation systems. We have continued this work with additional analyses for alliance members and plan to provide additional capacity building during 2015.

Oklahoma Implements Empirical’s Observation Engine for Certification of Classroom Observers

Tulsa Public Schools, the Cooperative Council for Oklahoma Administration, and Empirical Education Inc. just announced the launch of Observation Engine to implement the Teacher and Leader Effectiveness program in the state of Oklahoma. Tulsa Public Schools has purchased Empirical Education’s Observation Engine, an online certification and calibration tool for measuring the reliability of administrators assigned to conduct classroom observations. Tulsa Public Schools developed the Tulsa Model for Observation and Evaluation, a framework for ensuring teaching effectiveness performance, as well as best practices for creating an environment for successful learning and student achievement. Nearly 500 school districts in the state are piloting the Tulsa Model evaluation system this year.

In order to support the dissemination of the Tulsa Model, the Cooperative Council for Oklahoma Administration (CCOSA) is training and administering calibration tests throughout the state to assess and certify the individuals who evaluate the state’s teachers. The Tulsa Model is embedded in Observation Engine to deliver an efficient online system for state-wide use by Oklahoma certified classroom observers. Observation Engine is allowing CCOSA to test approximately 2,000 observers over a span of two weeks.

Observation Engine was developed as part of The Bill and Melinda Gates Foundation’s Measures of Effective Teaching project in which Empirical Education has participated as a research partner conducting R&D on validity and reliability of observational measures. The web-based software was built by Empirical Education, which hosts and supports it for school systems nationwide.

For more details on these events, see the press announcement and our case study.

Join Empirical Education at ALAS, AEA, and NSDC

This year, the Association of Latino Administrators & Superintendents (ALAS) will be holding its 8th annual summit on Hispanic Education in San Francisco. Participants will have the opportunity to attend speaker sessions, roundtable discussions, and network with fellow attendees. Denis Newman, CEO of Empirical Education, together with John Sipe, Senior Vice President and National Sales Manager at Houghton Mifflin Harcourt and Jeannetta Mitchell, eight-grade teacher at Presidio Middle school and a participant in the pilot study, will take part in a 30-minute discussion reviewing the study design and experiences gathered around a one-year study of Algebra on the iPad. The session takes place on October 13th at the Salon 8 of the Marriott Marquis in San Francisco from 10:30am to 12:00pm.

Also this year, the American Evaluation Association (AEA) will be hosting its 25th annual conference from November 2–5 in Anaheim, CA. Approximately 2,500 evaluation practitioners, academics, and students from around the globe are expected to gather at the conference. This year’s theme revolves around the challenges of values and valuing in evaluation.

We are excited to be part of AEA again this year and would like to invite you to join us at two presentations. First, Denis Newman will be hosting the roundtable session on Returning to the Causal Explanatory Tradition: Lessons for Increasing the External Validity of Results from Randomized Trials. We examine how the causal explanatory tradition—originating in the writing of Lee Cronbach—can inform the planning, conduct and analysis of randomized trials to increase external validity of findings. Find us in the Balboa A/B room on Friday, November 4th from 10:45am to 11:30am.

Second, Valeriy Lazarev and Denis Newman will present a paper entitled, “From Program Effect to Cost Savings: Valuing the Benefits of Educational Innovation Using Vertically Scaled Test Scores And Instructional Expenditure Data.”

Be sure to stop by on Saturday, November 5th from 9:50am to 11:20am in room Avila A.

Furthermore, Jenna Zacamy, Senior Research Manager at Empirical Education, will be presenting on two topics at the National Staff Development Council (NSDC) annual conference taking place in Anaheim, CA from December 3rd to 7th. Join her on Monday, December 5th at 2:30pm to 4:30pm when she will talk about the impact on student achievement for grades 4 through 8 of the Alabama Math, Science, and Technology Initiative, together with Pamela Finney and Jean Scott from SERVE Center at UNCG.

On Tuesday, December 6th at 10:00am to 12:00pm Jenna will discuss prior and current research on the effectiveness of a large-scale high school literacy reform together with Cathleen Kral from WestEd and William Loyd from Washtenaw Intermediate School District.

New Education Pilot Brings Apple’s iPad Into the Classroom

Above: Empirical Education President Denis Newman converses with Secretary Bonnie Reiss and author, Dr. Edward Burger

They’re not contest winners, but today, dozens of lucky 8th grade Algebra 1 students enthusiastically received new iPad devices, as part of a pilot of the new technology.

California Secretary of Education Bonnie Reiss joined local officials, publishers, and researchers at Washington Middle School in Long Beach for the kick-off. Built around this pilot is a scientific study designed to test the effectiveness of a new iPad-delivered Algebra textbook. Over the course of the new school year, Empirical Education researchers will compare the effect of the interactive iPad-delivered textbook to that of its conventional paper counterpart.

The new Algebra I iPad Application is published by Houghton Mifflin Harcourt and features interactive lessons, videos, quizzes, problem solving, and more. While students have to flip pages in a traditional textbook to reveal answers and explanations, students using the iPad version will be able to view interactive explanations and study guides instantly by tapping on the screen. Researchers will be able to study data collected from usage logs to enhance their understanding of usage patterns.

Empirical Education is charged with conducting the study, which will incorporate the performance of over twelve hundred students from four school districts throughout California, including Long Beach, San Francisco, Riverside, and Fresno. Researchers will combine measures of math achievement and program implementation to estimate the new program’s advantage while accounting for the effects of teacher differences and other influences on implementation and student achievement. Each participating teacher has one randomly selected class using the iPads while the other classes continue with the text version of the same material.

Though the researchers haven’t come up with a way of dealing with jealousy from students who will not receive an iPad, they did come up with a fair way to choose the groups who would use the new high tech program. Classes who received iPads were determined by a random number generator.

Empirical Education Focuses on Local Characteristics at the 14th Annual CREATE Conference

Empirical Education staff presented at the National Evaluation Institute’s (NEI) 14th annual CREATE conference in Wilmington, North Carolina. Both presentations focused on the local characteristics of the evaluations. Dr. Denis Newman, president of Empirical Education, and Jenna Zacamy, research manager, presented a randomized experiment which evaluated the impact of a pre-algebra curriculum (Carnegie Learning’s Cognitive Tutor Bridge to Algebra) being introduced in a pilot program in the Maui School District. The district adopted the program based in part on previous research showing substantial positive results in Oklahoma (Morgan & Ritter 2002). Given the unique locale and ethnic makeup in Maui, a local evaluation was warranted. District educators were concerned in particular with their less experienced teachers and with ethnic groups considered at risk. Unlike in prior research, we found no overall impact although for the algebraic operations subscale, low scoring students benefited from being in the Cognitive Tutor classes indicating that the new program could help to reduce the achievement gaps of concern. We also found for the overall math scale that uncertified teachers were more successful with their Cognitive Tutor classes than their conventional classes. Dr. Newman also presented work co-authored with Marco Muñoz and Andrew Jaciw on a quasi-experimental comparison, conducted by Empirical Education and Jefferson County (KY) schools, of an activity-based middle-school science program (Premier Science) to more traditional textbook programs. All the data were supplied by the district including a rating of quality of implementation. The primary pretest and outcome measures were tests of science and reading achievement. While there was no discernible difference overall, poor readers gained more from the non-textbook approach, helping to diminish an achievement gap of concern to the district.

Empirical Education Pilots Workshop at REL-NEI Regional Meeting

Providence, RI was the site of REL Northeast and Islands’ 2007 Regional Meeting on Teacher Quality. Empirical Education staff headed out to the east coast to pilot the “Becoming Good Consumers of Research” workshop to an audience of about 30 education researchers, school administrators, university professors, and other education professionals interested in using research to inform school decisions. Gloria Miller, the company’s director of evaluation design, facilitated group discussions that touched on identifying different types of research and potential sources of bias. Gloria also provided participants with a Critical Reader’s toolkit designed to help readers evaluate the trustworthiness and relevance of various pieces of research. “Consumers of Research” is the first in a planned series of workshops focused on increasing the understanding and use of research in schools. The next round of workshops, scheduled for March 2008, is currently being developed.

Learning Point and Empirical Partner for Research on Formative Assessment

Learning Point Associates, which holds the contract for the Midwest Regional Education Lab, has contracted with Empirical Education for the operations of a large randomized experiment expected to include more than 100 elementary school teachers when in full swing in the fall of 2008. A pilot experiment, beginning this fall, involves a small number of schools in Illinois. The experiment will test the effectiveness of Northwest Evaluation Association’s formative assessment and professional development to be used in fourth- and fifth-grade classrooms. Working with the principal investigators Matt Dawson of LPA and David Cordray of Vanderbilt, Empirical will be responsible for recruiting schools, acquiring and warehousing student data, and conducting observations and surveys.