A painstaking yearslong effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction.
The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.
...
More than 60 of the studies did not hold up. The project began in 2011, when a University of Virginia psychologist decided to find out whether suspect science was a widespread problem. He and his team recruited more than 250 researchers, identified the 100 studies published in 2008, and rigorously redid the experiments in close collaboration with the original authors.
Only 36% of the studies held up to scrutiny. If you add in the so-called research in other fields, even that kind of accuracy is probably too high:
Dr. John Ioannidis of Stanford, who predicted in the early 2000s that about half of the findings across medicine were inflated or wrong, said that the 36 percent replication rate for psychology might even be high once the results for other disciplines, like economics, animal research, cell biology and other areas of biomedicine, were also tested.“Many of the biases found in psychology are pervasive,” Dr. Ioannidis said.
It would be interesting to see a painstaking yearslong effort to reproduce 100 studies of education research to see what the replication rate is.
You hear people on all sides of education reform issues throw out the words "Research shows..."
Our culture has fetishized "research" and "science," as if to say if something has been found by "research" in a "scientific study," it must be so.
This is not so, as one commenter at the NY Times notes:
I applaud the researchers ability to recognize the limitations of their research and their standard of peer review. There is an old saying in medical research: "half of what is published is worthless; the challenge is figuring out which half." The measures adopted help figure that out.
The bigger danger, however, lies not with the scientists but by the talking heads that like to cite (often incorrectly) these studies. President Obama famously said he would "restore science to its rightful place", implying that science was the highest level of understanding. These findings prove what every honest researcher knows- that published studies are often incorrect, that established but untested norms of behavior are often correct, and claiming science as an authority is limiting. Placing blind trust in science is just as foolish as placing blind trust in faith.
The take home message is simple- if a single scientific study claims something different than experience suggests, it needs to be re-tested before it is claimed to be true. Remember that the next time someone proposes a sweeping social change based on shoddy and poorly explained science.
Gee, re-testing scientific research before proposing sweeping social change based on shoddy and poorly explained science - that would never happen in education, would it?
Nahh - the accuracy and validity of the studies on VAM, the Common Core State (sic) Standards, the small schools initiative, teacher evaluations based upon test scores, students who have teachers who raise test scores make more money over their lifetimes, et al. were all tested and re-tested and have been found to be "objective science," as Governor Cuomo is fond of saying.
Right?
Sure.
And "untested norms of behavior" that have stood the test of time are always suspect until Raj Chetty or some other "researcher" gets his/her hands on them for some of their rigorous, scientific research to test them.
Right?
No, not right.
As the commenter at the Times writes, blind trust in so-called science is no better than blind trust in faith.
When you get back to school in a week or so, you can bet you'll hear the words "Research shows..." at some PD meeting sooner rather than later.
It is high time to challenge the research fetishizers when they throw out their "Research shows..." jive to prove that the "research" they're citing is part of the 36% that can be replicated and not the 64% that can't.
