Until today I hadn’t heard of the story of Emily Rosa. Quoting wikipedia: “At age nine Rosa conceived and executed a scientific study of therapeutic touch which was published in the Journal of the American Medical Association in 1998.” She won the James Randi “Skeptic of the Year” award in 1998. She received a great deal of media attention after publication of the article.
My first thoughts were: “That’s really cool. I love it that she did an experiment and got it published!” Then I read the article.
I have never read an article more littered with errors. Almost every statement in it related to statistics is wrong.
The basic experiment was described in the abstract:
therapists were “tested under blinded conditions to determine whether they could correctly identify which of their hands was closest to the investigator’s hand. Placement of the investigator’s hand was determined by flipping a coin. […] In 1996, 15 practitioners were tested at their homes or offices on different days for a period of several months. In 1997, 13 practitioners, including 7 from the first series [my emphasis], were tested in a single day.”
Despite the repeated measures design (some practitioners were tested 10 times, others were tested 20 times), the data appear to have been analyzed as if they were independent trials (although it’s hard to tell what they actually did, because the description of the methods is so terrible).
They did a one-sided test and failed to reject the null. However, the therapists did worse than would be expected with random guessing (44% success rate with p-value of about 0.04 (I can’t give a precise p-value because it’s impossible to reconstruct their data)). It could be just chance that they performed worse than you’d expect if they were just guessing, but it does suggest that something might be going on (was the experimenter tipping them off in the wrong direction in some way? were they really sensing a difference in what they feel but attributed it to the wrong thing?). None of that was discussed.
It would take me too long to list all of the errors and potential sources of bias. But trust me, it’s that bad.
I get that a child is one of the authors and it’s cute and we could provide them some slack. But the JAMA editors could have helped make the paper better.
Most importantly, I am extremely disappointed in my fellow skeptics. They let a sensational story blind them. It’s a story that has all of the elements needed to go viral. A 9 year old took on those pseudoscientists and won! Yay us! So then the skeptic community turned off their skeptical brains and just endorsed the whole story. Shame.
Look into the Rocky Mountain Skeptics critique who did not fall in line with skeptic community at large.