by happy-go-lucky on 10/27/23, 4:16 PM with 339 comments
by thelittlenag on 10/27/23, 5:29 PM
The hiring process is setup basically to filter for folks who they think are the most likely to publish lots of papers, collaborate to push existing lines of inquiry, write lots of hopefully approved grants, and grow a lab into what is effectively a "successful small business". Quality is an after thought taken care of by what passes for peer review.
The incentives for everyone involved is just a complete and total mess. I'm reading tea leaves here, but my guess as to why she was never hired is that she was deemed "unable to get grants". Had she been, then she would have found herself hired immediately somewhere because universities are incentivized to play a numbers game and get as many folks in writing grants as possible.
by kps on 10/27/23, 5:21 PM
https://www.theguardian.com/science/2013/dec/06/peter-higgs-...
by shrubble on 10/27/23, 5:33 PM
1) Ken Iverson who invented the APL programming language and went on to win the Turing Award in 1979, had already published 'that one little book' that was considered insufficient for tenure, but which formed the basis for the award.
2) tubes remained the main focus of MIT faculty for quite some years after the transistor was invented. It was Robert Noyce and the people he worked with at Grinnell College who knew more about transistors than MIT : https://web.stanford.edu/class/e145/2007_fall/materials/noyc...
by mlsu on 10/27/23, 7:18 PM
It leaves me wondering: why do we not create any new universities? Why doesn't a Carnegie of our age create a new university? Brin University? Zuck University? This seems like a no brainer.
I think it might seem difficult to attract new talent to an "unestablished" university. But what if you make a simple promise: we will never, ever get in your way, the way that universities do today. We will never pressure you to publish subpar results. We will never nit-pick your purchase of a laptop. Have vision! Pursue things that are promising to you! We trust you, smart person, and we will give you autonomy to do what you think is promising. Based on what is discussed here, it seems like that would be extraordinarily compelling to the most optimistic, least cynical, and probably at least a handful of the most brilliant researchers out there. If the winning move is not to play the game, don't play.
I don't know. It just seems like there is a narrow-mindedness at play. A sense that "why try to fix this -- we'll never beat UPenn. Maybe not, but isn't it worth a try, based on how dysfunctional academia is? All it takes is the will.
by zaptheimpaler on 10/27/23, 9:13 PM
This isn't just one story, there are countless other researchers and even life-saving drugs that are not developed purely because of this mindset. For a brief moment in time during the COVID pandemic we saw that it is possible to have a better system but it's been forgotten just as quickly.
by contemporary343 on 10/27/23, 8:34 PM
The really distasteful thing here is Penn as an institution. They have reaped the benefits of her work in terms of mRNA patent royalties (a very large number I believe), and of course reputationally. Yet, they treated her truly terribly and have never - and it seems like will never - acknowledge it. For example, Sean Grady, mentioned here as the one that essentially cleared out her lab in 2013 without telling her is the chair of neurosurgery at Penn Medicine. Will he apologize? I doubt it.
by m_a_g on 10/27/23, 5:08 PM
by rdiddly on 10/27/23, 6:07 PM
by queuebert on 10/27/23, 5:16 PM
I have no idea how to fix this, but competition needs to be reduced, probably by more guaranteed funding for positions, not just projects, as grants are. This latest military aid package is 2x the entire NIH budget, so surely there is more money for science out there.
by seanr88 on 10/27/23, 5:32 PM
by gustavus on 10/27/23, 5:12 PM
by bell-cot on 10/27/23, 5:12 PM
by hodgesrm on 10/27/23, 5:08 PM
A brilliant woman scientist researching an uncool topic hits the trifecta of resistance to her work. It's wonderful to see her persistence vindicated but it sounds like time for a revolution in how university research is managed. The closing quote of the article is very disappointing.
by senkora on 10/27/23, 7:27 PM
While unpleasant, this is a conversation that is sometimes necessary to have as someone in a position of power communicating to a subordinate.
> In 2013, Karikó said she returned to her lab after spending time away to find all of her belongings having been packed, moved, and misplaced at Grady's direction.
But this is just petty and cruel.
by dbmikus on 10/27/23, 7:20 PM
by ceejayoz on 10/27/23, 6:52 PM
https://www.glamour.com/story/katalin-kariko-biontech-women-...
> In 2013—after enduring multiple professional setbacks, one denied grant after another, and a demotion at the institution to which she’d been devoted for decades—Katalin Karikó, Ph.D., walked out of her lab at the University of Pennsylvania’s School of Medicine for the last time.
> That morning at the lab, Karikó’s old boss had come to see her off. She did not tell him what a terrible mistake he was making in letting her leave. She didn’t gloat about her future at BioNTech, a pharmaceuticals firm that millions now associate with lifesaving vaccines but was then a relative upstart in the field. Instead the woman who had bounced from department to department, with no tenure prospects and never earning over $60,000 a year, said with total confidence: “In the future, this lab will be a museum. Don’t touch it.”
by mightyham on 10/27/23, 5:46 PM
by getpost on 10/27/23, 11:55 PM
by lacker on 10/28/23, 3:04 AM
by thimkerbell on 10/28/23, 2:59 AM
by RecycledEle on 10/27/23, 5:23 PM
If only there were a way to document disagreements publicly so they could be reviewed at a later date.
I had hopes that Internet discussion forums would be that, but the nukers destroyed that along with most training materials for LLMs.
by DoreenMichele on 10/28/23, 12:44 AM
As a woman who wonders how we solve this, I feel like the piece doesn't cast light on actionable means to make real progress. It implicitly seems to assume a thing the world calls sexism but offers no real solutions.
Presumably, Grady was a big factor here. Why was he such a problem? There's no real explanation for his behavior and we are unlikely to ever learn anything meaningful about what drove his decisions.
It airs dirty laundry, which is perhaps an essential first step in being able to analyze the problem space: First, we have to overcome the taboo against speaking of such things.
But then the initial attempts to speak of it tend to fall within some Overton Window of acceptable assumptions and this seems to do little or nothing to give women a viable path forward.
by johnp271 on 10/27/23, 7:24 PM
Arguably the story of how this researcher was treated and what she still managed to accomplish can serve as inspiration and motivation to persevere to future generations of folks with unconventional ideas or ideas that are disparaged by the 'experts'. Yes, it can also serve as motivation to research institutions to take risks and go out on limbs every now and then as there can be some wheat hidden within the chaff.
by zamalek on 10/27/23, 5:55 PM
by myth_drannon on 10/27/23, 5:10 PM
by dakial1 on 10/27/23, 8:18 PM
There is no place where you don't need good communication and selling skills. That's a fact of life and it seem impossible to remove this from any of these institutions.
Kariko seems to be that very hardworking intelligent person that really needs an eloquent and self-marketer sidekick to thrive. She is a Steve Wozniak in need for a Steve Jobs.
by anigbrowl on 10/28/23, 5:46 AM
I'm glad to see this article calling out the people who threw stumbling blocks in her way. They deserve any career damage they get.
by ilaksh on 10/27/23, 5:09 PM
by contemporary343 on 10/27/23, 8:32 PM
by godelski on 10/27/23, 6:19 PM
> One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.
> Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.
Or from Bengio
> In the rush preceding a conference deadline, many papers are produced, but there is not enough time to check things properly and the race to put out more papers (especially as first or equal-first author) is humanly crushing. On the other hand, I am convinced that some of the most important advances have come through a slower process, with the time to think deeply, to step back, and to verify things carefully. Pressure has a negative effect on the quality of the science we generate. I would like us to think about Slow Science (check their manifesto!).
> Students sometimes come to me two months before a deadline asking if I have ideas of something which could be achieved in two months.
I'm sure you can find one from LeCun too (drop it if you have it) and we have the 3 godfathers of ML. But as someone finishing my PhD, I'm utterly convinced that the whole process is psychotic and anti-scientific. I have written many rants on HN about this so what's another? Here's how I see it, and what I've been coining as Goodhart's Hell because the idea is more abstract that ML publishing or even academic publishing. There's just a huge fucking irony that this happens in ML.
It is Goodhart's Hell because everything in our world has become about easy to use metrics and bending over backwards to meet those metrics. There is not just a lack of concern about if the metric aligns with our intended goals, but an active readiness to brush off any concerns. We as a modern world just fucking embraced metric hacking as the actual goal. In ML we see this, as Hinton mentions, with benchmarkism with just trying to get top scores. But you need several (fwiw, I've held a top spot for over a year now on a popular generative dataset but the work remains unpublished because I don't have enough compute to tune other datasets. Reviewers just ask for more but not justify the ask by how another dataset says more). This is an insane world, especially as we've been degrading our statistical principles. The last 5+ years no one uses a validation set for classification but rather tunes their fucking hyperparameters on test set results. Generative models frequently measure metrics against the train set and don't have a test set! A true, honest to god, hold out set essentially doesn't exist (we might call it "zero shot", which is inaccurate, or "OOD"...). ML work has simply become a matter of compute. Like Higgs said, you need to publish fast, but these days top companies are asking for 5+ papers at a top conference for a newly minted PhD. I'm sorry, good work takes time. All this on top of several consistency experiments that demonstrate that reviewers are simply reject first ask questions later. Which why shouldn't they be? No one checks a reject and doing so increases the odds your work gets in since it's a zero sum game.
And in honesty, I don't see how conferences and journals are anything but fraud. Not in the sense that works in there are untrue (though a lot are and a lot more are junk. Regardless of field), but in the economic operation. The government and universities (double dipping on that gov money) pay for these to exist. Universities pay researchers to produce work. Researchers send to venues (journals/conferences). Researchers review other works submitted to the venue for no pay (so Uni pays). 80% of work gets rejected, and goes through the process again. And after all that, the only meaningful thing accomplished is that the university has a signal that the work that their researchers did is "good." Because the venue gets copyright ownership over the paper, which the university must now pay for to access (the "official" version, "preprints" are free). I'm sorry, but citation count is a bad metric but far more meaningful than venue publication and it's fucking free. Why don't we just fucking publish to OpenReview? The point of publishing is to communicate our work, nothing more nothing less. OR gives you hosting like arxiv but also comments and threads (and links to github). Do we need anything else? I mean no review can actually determine if a paper is valid or good work. But we forget that the world isn't binary, it's tertiary: True, False, Indeterminate (thanks Godel, Turing, and Young). In reviewing we do not have access to the "True" side, just as we don't have access to that in science in general. We do not know where the "True" direction points, but we know how to move away from the "False" and "Indeterminate" directions. That's why there's that famous substack named that way or Isaac Asimov's famous Relativity of Wrong paper[2]. We're not a religion here...
There is at least a few ways I know how to fight back. 1) Actually fucking review a work and do your god damn job. Your job isn't to be a filter, it is to earnestly read the work and to work with the authors to make it the best work it can be. Remember you're on the same side. 2) Simply don't review if you can't do #1. You're almost never required to and academic service isn't worth much, so why do it? 3) Flip the system on its head. Instead of concentrating on reasons to reject a paper (fucking easy shit right there), focus on reasons to accept a paper. Simply ask yourself "is there something __someone__ in the community would find useful here?" If yes, accept. Novelty doesn't exist in a world where we have 20k+ papers a year and produce works every few months. It's okay to move fast, but it's less novel and impactful, it's just closer to open science. Stop concentrating on benchmarks since if it's useful someone is going to tune the shit out of it anyways, benchmarks don't mean shit. These days benchmarks are better at showing overfitting than good results anyways (yes, your test loss can continue to decrease while you overfit).
[0] https://www.wired.com/story/googles-ai-guru-computers-think-...
[1] https://yoshuabengio.org/2020/02/26/time-to-rethink-the-publ...
[2] https://hermiene.net/essays-trans/relativity_of_wrong.html
by j7ake on 10/27/23, 11:58 PM
There is absolutely no incentive for UPenn to keep a professor who isn’t bringing in money.
by warbaker on 10/27/23, 11:22 PM
Grant agencies cannot always get it right. The more funding they have, the more they can give to researchers that they aren't 100% sure about.
by aborsy on 10/27/23, 6:20 PM
Does EU produce better science, I wonder?
by ubermonkey on 10/27/23, 9:23 PM
by renewiltord on 10/27/23, 5:24 PM
by drno123 on 10/27/23, 5:11 PM
by very_good_man on 10/27/23, 7:44 PM
by GabeIsko on 10/28/23, 12:09 AM
by DrNosferatu on 10/27/23, 9:38 PM
I get the feeling Katalin Karikó got a lot of that flak because she made some narcissists look bad (directly or indirectly, by comparison).
- Your views on this?
by EVa5I7bHFq9mnYK on 10/27/23, 7:09 PM
by robd003 on 10/27/23, 5:34 PM
by s1artibartfast on 10/27/23, 5:56 PM
If a researcher makes a great discovery, but can’t get funding to do anything with it, You don’t keep them around not making progress.
They got pushed out, found funding, and finally furthered the technology.
It is unclear if more scientific progress would have been made if they were kept at penn without funding.
by d--b on 10/27/23, 6:12 PM
University politics are terrible, but in this very case, whatever happened, it turned out pretty good for both her and the University.
by RecycledEle on 10/27/23, 5:20 PM
It would tell us who to listen to and who yo shun.
I had hopes that Internet forums would be that record, but the nukers destroyed that.