Or so proclaimed Maria Glod and Daniel de Vise of the Washington Post (link):
Since enactment of the No Child Left Behind law, students from poor families in the Washington area have made major gains on reading and math tests and are starting to catch up with those from middle-class and affluent backgrounds, a Washington Post analysis shows.
The achievement gap between economic groups, long a major frustration for educators, has narrowed in the region’s suburban schools since President Bush signed the law in 2002, according to Maryland and Virginia test data.
Isn’t that wonderful? Students from poor families are catching up to students from more affluent backgrounds. Or are they? Let’s take a look at their data.
The astute viewer might notice that they are comparing ‘economically disadvantaged’ students with ‘all students’. The latter category includes students who are in the former category. If interest is in comparing ‘economically disadvantaged’ students with middle or upper income students, then your graph should reflect that. By including low income students in both curves, you’re necessarily reducing the gap between them. Standard practice is to use mutually exclusive groups when stratifying.
There is a larger problem, however. Notice that they are not graphing test scores; rather, they are graphing the percentage of students who ‘passed’ the exam. The majority of upper income students passed the exam in 2003 (it was close to 100% in some counties). So, there wasn’t much room for upper income students to improve (since the outcome is pass/fail). There was a lot more opportunity for lower income students to improve, as their pass rate in 2003 was much lower. In statistics, this is known as a ceiling effect. The Post analysts didn’t seem to notice this problem.
Bob Somerby discussed both of these problems on Saturday here. Today, he posted an email from a very wise statistician (me!) here.
From the graphs, it’s clear that pass rates are going up. Well, it is safe to say that there is less of a gap in pass rates between lower and upper income students.
Does that mean achievement gaps have narrowed? Are students from poor families in the Washington area really “starting to catch up with those from middle-class and affluent backgrounds”?
Let’s consider some possibilities:
1. The test has gotten easier. If the test has gotten easier, then we would likely see graphs like the ones above, even if students don’t know more now than students did in 2003. The typical upper income student could pass either exam, while the lower income students might have failed the 2003 exam, but passed the 2007 exam. Did the Post even consider this (likely) possibility? As I mentioned in my email to Somerby, “if you make the test easy enough (where everyone can pass) there will be no achievement gap at all!” (based on Post logic)
2. Both upper and lower income students are testing better now than they were in 2003. Because the graphs are pass/fail, it would look like lower income students have narrowed the gap. In fact, it’s entirely possible that upper income students have improved even more during that time, widening the gap, but a pass/fail test cannot capture such an effect.
3. Lower income students really are catching up with upper income students. It’s possible, but there is not more evidence for this than for options 1 or 2.
Is it even desirable to narrow achievement gaps?
That depends. Again, quoting myself: ”There are ways to close achievement gaps that are not necessarily good. For example, schools could decide to use all of their resources on kids who are not meeting minimal standards, and ignore kids who are. Certainly achievement gaps could be narrowed, but at the expense of kids who started out ahead.”
There is no question that some kids enter school way behind other children. The school’s job is to teach those kids as much as possible. If they maximize learning from all children, will these gaps narrow? I have no idea.
Somerby mentioned an interesting theory:
When I was a teacher, we were taught a naughty theory (I think it was conventional wisdom at the time): Good teaching increases achievement gaps. If you miraculously create a situation where everyone grows at his or her maximum potential, everyone will have advanced by the end of the year. But the smarter kids will be farther ahead of the less gifted kids than they were at the start of the year.
Now, some of the most talented kids will be in the lower SES group, and therefore might make some gains on less talented kids who got a head start (due to higher SES). But, on average, will the optimal teaching situation narrow the gaps? I don’t think we know the answer to that.
