Washington Post article

I wrote a letter to the Washington Post in response to their misleading article on progress in education in the DC area (link to the article).  It doesn’t appear that they are going to publish it.  Perhaps next I’ll try writing to the authors.  Anyway, here is my letter (I thought it was pretty good!):

This letter is in response to the October 2, 2008 column “Needy Students Closing Test Gap Under ‘No Child.’

The story begins with the exciting claim that “students from poor families in the Washington area have made major gains…and are starting to catch up with those from middle-class and affluent backgrounds.” As a statistician, however, I could not help but look carefully at whether the data supported the conclusion. They do not.

There are several problems with the presentation and with the analysis. For sake of brevity, I will focus on the major problem. Performance was measured by the percentage of students who passed the exams. In 2003, the majority of students from middle-upper income families passed the exams (in some counties the percentage was close to 100). That was not the case for students of low-income families. So, there was not much room for upper income students to improve (since the outcome was pass/fail). There was a lot more opportunity for lower income students to improve, as their pass rates in 2003 were much lower. In statistics, this is known as a ceiling effect.

From 2003 to 2007 pass rates improved. Also, the ‘gap’ in pass rates between economic groups have decreased. Does that imply that students from poor families are “starting to catch up?” Absolutely not. There are several alternative explanations that could explain the trends. An obvious one is that the tests have gotten easier. In that case, pass rates would increase, and the gap would decrease (due to the above mentioned ceiling effect). Another possible explanation is that all students have improved at the same rate. In that case, the ‘pass rate gap’ would decrease (due to the ceiling effect), but students from poor families would not be catching up with students from more affluent families.

I appreciate that The Washington Post covered such an important topic. In the future, however, I hope that more care will be taken with the data analysis and with the interpretation of the results.

About these ads