by dbingham on 3/24/24, 6:06 PM with 160 comments
by rhelz on 3/24/24, 7:19 PM
Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it.
The solution? Its obvious. In a free market, how do you signal if you want more of something to be produced? Bueller? Bueller?
Yeah, that's right, you gotta pay for it. This cost should just be estimated and factored into the original grant proposals--if its not worth $5k or $10k to fund a round of peer review, and perhaps also funds to run confirming experiments--well, then its probably not research worth doing in the first place.
So yeah, write up the grants to include the actual full cost of doing and publishing the research. It would be a great way for starving grad students to earn some coin, and the experience gained in running confirming experiments would be invaluable to help them get that R.A. position or postdoc.
by adastra22 on 3/24/24, 9:20 PM
Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published papers by sending it off as a letter to the editor (sometimes via a sponsor if you were new to the journal), and they either accepted or rejected it as-is.
The idea of having your intellectual competitors review your work and potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective.
The rise of Arxiv is a recognition by researchers that we don’t need or want that system, and we should do away with it entirely in this era of digital print where page counts don’t matter. So please stop trying to force it back on us!
by llm_trw on 3/24/24, 7:01 PM
I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.
by zer00eyz on 3/24/24, 7:05 PM
Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar.
Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from.
The current system is glad-handing, intellectual protectionism and mastrubation.
Academia has only itself to blame for this, and they are apparently unwilling to fix it.
by timr on 3/24/24, 6:59 PM
The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. The system will self-correct over time.
IMO, the main actual problem with peer review is that non-practitioners put too much faith in it. Nobody in science actually takes a paper on faith because it's been published, and you shouldn't either. Peer review is little more than a lightweight safeguard against complete nonsense being published. It barely works for that. Just because you found a paper doesn't mean you should believe it. You have to understand it.
A secondary actual problem is that it's impossible to reproduce a lot of papers, or they're methodologically broken from the start (e.g. RCTs that are not pre-registered, or observational studies without control groups). These are problems we could actually solve. For example, just requiring that any paper publish the raw study data would help to self-control the system. There are high-profile researchers out there, right now, who do little more than statistically mine the same secret data set -- these people are likely publishing crap, but we have no way to prove it, because the data is secret.
by cachemoi on 3/24/24, 10:05 PM
Let me know if you have a project you'd like to get on there! Here's what it looks like, a paper on directed evolution [1]
[0] https://versioned.science/ [1] https://github.com/versioned-science/DNA_polymerase_directed...
by daft_pink on 3/24/24, 7:03 PM
by somethingsome on 3/24/24, 11:18 PM
So that often cited paper get a lot of different (small) public reviews that can be curated from time to time, obscure papers get at least one review justifying why it's relevant to cite them in the new work.
Some could argue that this is too much work added to the writing process.. But.. At the same time.. Shouldn't we read the papers we cite? Why not automatically write a small review of it? It has not to be huge, only the justification on why we (can) use it in our work.
by ccppurcell on 3/24/24, 9:31 PM
Clearly something is rotten about the way peer review is implemented in the empirical sciences. But think of all those high profile retractions you read about these days. Usually that comes about by a sort of post hoc peer review, not by anything resembling market forces.
by 1letterunixname on 3/24/24, 11:10 PM
by frozenport on 3/24/24, 7:43 PM
Then I can anonymously critique the paper without fear of the authors rejecting my career making Nature paper.
by milancurcic on 3/24/24, 9:32 PM
by _delirium on 3/24/24, 9:02 PM
by erictleung on 3/25/24, 3:37 PM
by lizzystrick685 on 3/26/24, 11:44 PM