We worked with McGraw Hill in 2021 to evaluate the effect of ALEKS, an adaptive program for math and science, in California and Arizona. The School Impact report below is from the Arizona study and meets tier 3 of The Every Student Succeeds Act (ESSA) evidence standards: Promising Evidence. Read the truncated version of the report below or download the full PDF version of the report (with technical details) using the blue button below.
This study of ALEKS implementation in AZ schools during the 2018-2019 school year finds evidence of a positive effect of ALEKS on AzMERIT End of Course Algebra I and Algebra II assessments. This report identifies the school-level effects of active ALEKS usage on achievement compared to similar AZ schools not using ALEKS.
Active use of ALEKS results in improved student outcomes on AzMERIT End-of-Course (EOC) assessments in Algebra I and Algebra II compared to a matched sample of non-users. The effect of ALEKS on schools is, on average, equivalent to a 5.1 percentile point reduction in the percentage of students scoring at the lowest proficiency level (Level 1 - Minimally Proficient) and a 4.2 percentile point increase in the proportion of students performing at the highest proficiency level (Level 4 - Highly Proficient) on the Algebra I EOC assessment. We have high confidence in this result (p < .05). The result was positive but smaller for Algebra II and smaller but negative for Geometry. Figure 1 shows these results for Algebra I, Geometry, and Algebra II. These results should be taken with caution since they come from a quasi-experimental study with non-equivalent groups and use data at the course level.
Schools that were active users of ALEKS performed better on the EOC Algebra I and Algebra II assessments than schools that were not ALEKS users.
We also tested the impact of ALEKS on schools that differed in the percentage of the following subgroups of students: male/female, primary racial/ethnic groups, economically disadvantaged, English language learners (ELL), and students with disabilities. We found an additional reduction in the percentage of students scoring at the lowest proficiency level on the Algebra I EOC assessment for students classified as eligible to receive free/reduced-price lunch (FRPL), Hispanic students, English learners, and students with disabilities. The effects ranged from 4.4% to 12.3%, compared to the average effect for all students (Figure 2).
ALEKS had a positive impact on schools with higher proportions of historically disadvantaged student populations.
During the 2018–2019 school year, more than 100 schools in AZ used ALEKS. For the main analysis of this study, we limited the pool of ALEKS schools to those where at least 50% of students who were tested in a particular course took at least one ALEKS assessment during the year. This limited the sample of ALEKS schools to 31 schools. Alternate approaches to selecting the treatment group, which increase the sample size but may include less active users of the ALEKS program, are shown in the Technical Details.
Figure 3 shows the distribution of demographic variables in ALEKS schools in AZ. Each bar represents the range of a variable (10th through 90th percentile) with a white line representing the median.
Schools that used ALEKS were more likely to have higher proportions of historically disadvantaged student populations.
In this comparison study, schools that were active users of ALEKS performed better on Algebra I and Algebra II EOC math assessments than schools that were not ALEKS users. These results were even stronger across disadvantaged student subgroups in the impact on the percentage of students performing at the lowest proficiency level.
The positive average impact on the Algebra I EOC assessment for schools with active ALEKS users compared to non-users, in conjunction with the positive findings for historically disadvantaged student populations, allows for a conclusion that the study provides evidence of promise. This leads to the recommendation of additional studies in other states and in districts using student-level data. It is important to note that analysis at the aggregate school level does not allow inferences about impact on individual students. It shows that courses/grades in schools that adopted ALEKS and used it actively had better outcomes, but it does not imply that requiring an individual student to take more ALEKS assessments or retain more topics will lead to a higher test score. In order to better understand ALEKS’s impact on students, an assessment of student-level achievement and usage data is required.
To read the technical details portion of this report, please view the full report.
Reference this report: Evidentally, Inc. (2021). ALEKS Impact on AZ Schools (School Impact report). Empirical Education Inc. https://www.empiricaleducation.com/mcgraw-hill-education
