Adaptive Knowledge Component Mastery Tracking in an AI Algebra Tutoring System: A Randomized Trial of 1,284 Middle School Students

This study investigates learning outcomes and knowledge component mastery rates from an adaptive AI algebra tutoring system versus static worked examples and human tutoring in a randomized trial of 1,284 middle school students within the context of educational technology and learning sciences, an area of growing scientific importance given its implications for adaptive algebra tutoring deployment strategy, educational AI evaluation methodology, and equity-focused personalized learning scaling. Using cluster-randomized trial (school-level) with pre/post algebra assessment, weekly knowledge component mastery tracking, and HLM analysis of learning gains controlling for school demographics and prior achievement, we examine AI system adaptively selecting problems targeting student knowledge component gaps, providing immediate corrective feedback reducing error consolidation and optimizing practice distribution across algebra topics in 1,284 students (6th-8th grade) across 24 middle schools: 8 schools per arm (AI tutoring n=428, static examples n=428, human tutoring n=428) drawn from 24 public middle schools in 3 urban school districts in Minnesota and Wisconsin during 2017-2018 academic year. Results indicate that AI tutoring produces 0.48 SD algebra assessment gains versus 0.28 SD for static examples (p<0.001) with no statistically significant difference from human tutoring (0.54 SD, p=0.312); effects largest for prior low-achievers (0.64 SD AI vs. 0.18 SD static) (p < 0.001), with 0.48 SD AI vs. 0.28 SD static examples; 0.64 SD for prior low-achievers as the primary quantitative benchmark. Concordance between primary and confirmatory measurement approaches exceeded 93%, validating the analytical framework. These findings contribute empirically to educational technology and learning sciences and carry actionable implications for the design of programs and policies targeting adaptive algebra tutoring deployment strategy, educational AI evaluation methodology, and equity-focused personalized learning scaling.