Legislative leaders continued their final week of witness testimony Tuesday by calling Dr. Abel Koury to address the relationship between spending and academic growth. Koury has a Ph.D. in developmental psychology and works as a senior research scientist at Far Harbor LLC, a statistical consulting firm in Austin, Texas.
In the case, petitioners have presented extensive evidence that spending impacts student outcomes. Legislative respondents asked Koury to analyze the correlation between levels of spending on education by school districts and district scores on the Average Growth Index (AGI)—a standardized data measure that is part of the Pennsylvania Value-Added Assessment System (PVAAS) for assessing growth on the state’s standardized tests.
PVAAS data from the Pennsylvania Department of Education provides a statistical analysis to measure a teacher, school, or district’s influence on the academic progress rates of students. Unlike traditional test scores measuring student achievement, PVAAS uses those scores to compare each student against a prediction of their own expected growth based on their prior performance.
Koury analyzed the relationship between funding levels and AGI academic growth scores for English language arts, math, science, and writing assessments at several grade levels between the 2013-14 and 2017-18 school years. He found that there was no meaningful relationship between these measures of school spending and the AGI measure of academic growth, a claim that was challenged on cross examination.
Petitioner attorney Dan Cantor pointed out that the PVAAS model is intended to control for – i.e., remove the effects of – all factors outside of and not under the control of the school. He said one of those outside factors is whether a school district is chronically underfunded, a factor that potentially impacts the student’s test scores in the baseline year that they are first being measured. He suggested that the effects of underfunding would never show up in an analysis such as Koury’s because PVAAS is designed to filter out the effects of underfunding.
Koury did not immediately respond, noting that this was “a question that I hadn’t previously thought about.” While not dismissing his own analysis, he ultimately agreed with Cantor on the point that with PVAAS, “minimally, they’re trying to control for different experiences that students are having before entering the classroom,” which could include going to school in an underfunded district.
Koury also acknowledged during cross-examination that the research literature on value-added growth measures like PVAAS is “mixed,” with some researchers criticizing their validity and others supporting it.
Petitioners’ attorney Cantor showed Koury several examples of instability and inconsistency in AGI scores during cross-examination. AGI scores are color-coded—red is the category for lowest growth scores (including negative growth), and dark blue is the category for the highest growth scores that significantly exceed predicted student performance. William Penn School District’s composite AGI growth scores for English Language Arts were in the blue category in 2016-17, changed to red in 2017-18, and back to blue in 2018-19.
Cantor also showed Koury the results of an analysis from Penn State professor Matt Kelly’s rebuttal to Koury’s expert report, which found that across subject areas, the majority of schools in both the blue and the red categories for AGI in one year were not in the same category the next year. In the four-year time-period studied by Koury, 41% of districts changed their AGI color code every year.
Koury acknowledged that he had not examined variation in AGI scores over time in his analysis and agreed that there could be “wild swings” in districts’ AGI classifications from year to year that he would not have been aware of.
AGI is calculated by dividing PVAAS growth scores by a standard error for each district, Koury said. Koury agreed that the standard error is a statistical measure that indicates the amount of evidence that supports a district’s growth measure.
Cantor showed that districts with identical scores on PVAAS growth measures can have widely different AGI scores, raising questions about the value of AGI
Koury said he is aware that some students have more needs than others. “I don’t think that can be disputed,” he said. But he was not aware until today that Pennsylvania provides data on school spending weighted to account for the needs of students in poverty, students with disabilities, and others, and he was not asked by legislative respondents to conduct an analysis of weighted school spending.
Koury was not convinced that his results would have looked substantially different if he had used per-student spending that is weighted to account for need, but he acknowledged that “I do not know, I can’t say for sure how it would have changed things.”
Koury acknowledged that his analysis could not determine whether school funding was or was not the cause of increases or decreases in student growth. Establishing a causal relationship between funding and positive outcomes for students would be difficult, he said, without a randomized control trial
He said he was familiar with work from economist Rucker Johnson, an expert for petitioners in this case, and his colleagues, who in a longitudinal study that examined 28 states found a strong positive relationship between school funding increases, educational attainment, and future earnings. Though he did not agree that they had established causality, “the way that they have done it is super-rigorous,” Koury said. “I’ve read some of Rucker Johnson’s work, and I think he presents a strong argument.”
Legislative respondents expect to call their final two witnesses on Wednesday: Brian Cote, the director of curriculum for 21st Century Cyber Charter; and Eric Hanushek, an economist with the Hoover Institution. Hanushek’s testimony may continue into Thursday.