Monday, October 21, 2013

Big Problems in Big Science

The Economist is taking a serious look at the problems with science, hitting pretty much the same flaws as everyone else:
Too many of the findings that fill the academic ether are the result of shoddy experiments or poor analysis (see article). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 “landmark” studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties.
The problems can all be traced back to the need to publish articles in science journals.
In the 1950s, when modern academic research took shape after its successes in the second world war, it was still a rarefied pastime. The entire club of scientists numbered a few hundred thousand. As their ranks have swelled, to 6m-7m active researchers on the latest reckoning, scientists have lost their taste for self-policing and quality control. The obligation to “publish or perish” has come to rule over academic life. Competition for jobs is cut-throat. Every year six freshly minted PhDs vie for every academic post. Nowadays verification (the replication of other people’s results) does little to advance a researcher’s career. And without verification, dubious findings live on to mislead. Careerism also encourages exaggeration and the cherry-picking of results. In order to safeguard their exclusivity, the leading journals impose high rejection rates: in excess of 90% of submitted manuscripts. The most striking findings have the greatest chance of making it onto the page. Little wonder that one in three researchers knows of a colleague who has pepped up a paper by, say, excluding inconvenient data from results “based on a gut feeling”. 
Or, as psychologist Brian Nosek put it, "There is no cost to getting things wrong. The cost is not getting them published."

The way out of this mess seems obvious to me: put an end to the science journals and replace them with much more open web sites. Instead of sexing up their work to get publication credit, scientists should post it online -- all of it, including their screw-ups. If that means universities need new ways to judge professors, well, tough. Scientific journals were a great way to share information in the nineteenth century, but now they are technologically obsolete and create more problems for science than they solve.

5 comments:

Shadow said...

Perhaps scientific journals are a 19th century artifact, but where does peer review fit into your scheme? Do you think it will occur "organically?"

Shadow said...

Actually , a 20th century artifact.

John said...

One of the big issues raised in the Economist piece is that peer review is highly ineffective. People have done experiments sending papers with glaring errors to peer reviewers, who pass them. In my experience, the editor pretty much decides how the peer review will go when he chooses the reviewers; if he likes the piece, he chooses reviewers he knows are positive about that kind of work.

PLOS uses a sort of weak peer review, asking the reviewers only to pass on the general suitability of the approach and the methods. They seem to publish lots of good stuff.

Shadow said...

But the article doesn't so much recommend getting rid of peer review as moving it to post publication.

Right now anyone and everyone can publish their findings online, including shoddy research and methods. What keeps this in check is the peer review process.

Journals filter out, or attempt to filter out, the unworthy. If they aren't doing a good job of it, that needs to be fixed. There has to be some way of enforcing standards. Not having peer review, I think, strikes at the integrity of science.

Doing post reviews on research published in journals is manageable. Doing post reviews on research posted on the Internet is not.

John said...

I regard the journal publication system as completely broken. It puts huge pressure on people to cover up their mistakes and dead ends and distort their findings to make them publishable, and it does not prevent the publication of utter garbage. Those studies cited by the Economist show that less than half of peer reviewed research can be replicated. Plus, it puts research paid for by the taxpayers behind pay walls where they people who paid for it can't see it.

At I minimum, I think government-funded researchers should be required to post notices of all their failed experiments somewhere, which might keep other people from repeating their mistakes. And I think there should be serious consequences (i.e., termination of grants) for people who publish results that can't be replicated.

In the fields I know something about, a very large proportion of research is government funded. I think people who accept that funding have an obligation to the taxpayers to put all of their research where people can see it. I have to do this; I have to write detailed reports on all of my studies whether I find anything or not. Sadly, they are mainly kept in government repositories, but at least the data exists where other professionals can see it. Some agencies are moving to publish redacted versions of all their archaeological reports online. Why shouldn't biochemists have to do this?