Stanford Summer Math Camp Defense Doesn’t Add Up, Either
I thank Jack Dieckmann for reading my critique of the proposed California State Math Framework (“California’s New Math Framework Doesn’t Add Up”) and for writing a response (“Stanford Summer Math Camp Researchers Defend Study”). In the article, I point to scores of studies cited by What Works Clearinghouse Practice Guides as examples of high-quality research that the framework ignores. I also mention and two studies of Youcubed-designed math summer camps as examples of flawed, non-causal research that the proposed California State Math Framework embraces.
I focused on outcomes measured by how students performed on four tasks created by the Mathematical Assessment Research Service. Based on MARS data, Youcubed claims that students gained 2.8 years of math learning by attending its first 18-day summer camp in 2015. Dieckmann defends MARS as being “well-respected” and having a “rich legacy,” but he offers no psychometric data to support assessing students with the same four MARS tasks pre- and post-camp and converting gains into years of learning. Test-retest using the same instrument within such a short period of time is rarely good practice. And lacking a comparison or control group prevents the authors from making credible causal inferences from the scores.
Is there evidence that MARS tasks should not be used to measure the camps’ learning gains? Yes, quite a bit. The MARS website includes the following warning: “Note: please bear in mind that these materials are still in draft and unpolished form.” Later that point is reiterated, “Note: please bear in mind that these prototype materials need some further trialing before inclusion in a high-stakes test.” I searched the list of assessments covered in the latest edition of the Buros Center’s Mental Measurements Yearbook, regarded as the encyclopedia of cognitive tests, and could find no entry for MARS. Finally, Evidence for ESSA and What Works Clearinghouse are the two main repositories for high quality program evaluations and studies of education interventions. I searched both sites and found no studies using MARS.
