from Hacker News

Ask HN: Is the industry saturated with data scientists already?

by legobridge on 12/22/22, 10:38 PM with 36 comments

• by ffssffss on 12/22/22, 11:59 PM

  I think it's less a case of saturation and more a case of companies realizing that most data scientists don't actually deliver value commensurate to the salaries they were asking. Most companies simply don't have enough data, or don't have hygienic enough data, or don't have the engineering heft to build a reliable data pipeline, so the data scientists often find themselves set up to fail. I've seen that happen a few times.

  But, it's not like the fundamentals of the field are wrong. Predictive modeling is still really useful. It's just larger firms are the only ones capable of realizing that value.

• by mudrockbestgirl on 12/23/22, 6:06 AM

  The problem with the Data Scientist market is that Data Scientist is a fuzzy definition. It's pretty clear what skillset makes a good software engineer, but nearly anyone can call themselves a Data Scientist, even if they just clicking around in Excel. This has resulted in these bootcamps and certifications that "make you a Data Scientist in 14 days" - now everyone with minimal qualifications can apply for Data Scientist roles. That's why the market appears so crowded, and it is. People thought it's an easy and quick way to a high paying job, resulting in a flood of low-quality applicants for these positions.

  There is still a lot of room for people who have strong engineering AND data engineering/ML/math/statistics skills. But then don't call yourself Data Scientist because that puts you into the same low-barrier camp as all the others. From my own experience it's a clear resume red flag: Almost anyone that market themselves as primarily a "Data Scientist" has little technical skills.

• by MontyCarloHall on 12/22/22, 11:48 PM

  To be blunt, the market is saturated with people who call themselves “data scientists” but are actually just reasonably skilled software engineers with at best a college sophomore level understanding of math/stats.

  On the other hand, the market is nowhere near saturation for people with both advanced software engineering and math/stats skills (i.e. PhD-level).

• by 1attice on 12/23/22, 8:08 PM

  Call this the 'Electrician Cycle'.

  A job is initially incredibly sexy, risky, and newfangled, requiring knowledge that is not widespread. Like being an electrician in 1922.

  There are no standards, and because everyone is therefore effectively trading on reputation, salaries are high, for the same reason that Heinz ketchup costs more than Kroger: the brand carries the value.

  The job eventually becomes normalized. As part of normalization, the delta in quality between the highest and lowest earners becomes much smaller. If the industry becomes regulated, this gap narrows further. Consequently, salaries fall at the high end of the profession.

  Eventually, being an electrician in 2022 is roughly as sexy as being a plumber in 2022, and both are approximately as sexy as being a plumber in 1922.

  We've already seen this cycle consume web development and what used to be called system administration -- two positions which were HoT Sh_T in 1995, but are increasingly generic office jobs in 2022.

  This cycle will eventually consume every technical field, a kind of sociological eutrophication, but the good news is, it starts fresh with each new gyre.

  The bad news is, it happens faster with each gyre, because of the 'complexity ratchet'. You'd think the ever-increasing complexity of technical fields would slow down the cycle! But no -- the human capacity for knowing and valuing is fixed; so the complexity ratchet just means that the social-value cache gets flushed more often.

  Data scientists are just plumbers from 2052

• by sebg on 12/22/22, 10:50 PM

  > Is the industry saturated with data scientists already? No :)

  > Did all the talk of data science being the "sexiest" job cause the market to become saturated No :)

  > is it still a viable career path? Yes

  Source:

  a) I Co-run the Data Science Weekly newsletter.

  b) I was a mod of https://www.reddit.com/r/datascience/ from 15k to 30k members and people were asking that about 5 to 6 years ago. The sub now has ~829k members and that question still comes up.

  > The number of opportunities also seem to be dwindling The reason for this is that initial it was "data science", then it was "data science and machine learning researcher", then it was "data science and data engineerings and machine learning researcher", then it was "ai, data scientists, machine learning researcher, machine learning engineering, data engineer, nlp", etc. So the jobs have multiplied but so have the position titles as well. So while you could just search for data scientist positions before you now have to get a bit more specific.

• by leplen on 12/23/22, 1:59 AM

  Data science as a title has bifurcated several times. A lot of the engineering heavy DS roles are now data engineer or machine learning engineer. Probably those are the roles you should be looking for. Many of the DS jobs now are essentially analyst jobs, where SQL and stakeholder management is 75% of the role and writing code is just a bonus.

  Data science isn't going away. Leadership is always going to need numbers explained to them, but DS roles have never been as numerous or as well paid level for level as software engineering.

• by Chinjut on 12/23/22, 12:39 AM

  Perhaps what is holding you back is phrasing it as "a Master's degree in AI/DS"!

• by ProjectArcturis on 12/23/22, 12:30 AM

  Anecdotally, as a Principal Data Scientist, I'm getting a lot less recruiter spam than I did a year ago. Though I think that's probably something that's affecting everything in tech as the industry as a whole takes their foot off the gas.

  At the entry level, I'm sure there's more competition for fewer spots.

  I think the new generative AI models are absolute game-changers and will only get better. If I were starting out, I'd focus there.

• by dinkumthinkum on 12/23/22, 12:27 AM

  It seems like you we’re just chasing by trends. Trends don’t matter that much for an individual. If you focus on being an expert in something that is valuable at all then you should valuable. Trying to mom-max your career based on Gartner or something like that isn’t generally optimal for individuals.

• by DantesKite on 12/24/22, 12:32 AM

  I'm not familiar with the current job market, but data scientists being paid worse than developers is surprising to me considering the StackOverflow 2021 survey (in the US):

  Median Salary | Job Title

  ------------|----------------

  $177,500 | Senior Executive (C-Suite, VP, etc.)

  $165,000 | Engineering manager

  $150,000 | Engineer, site reliability

  $135,000 | DevOps specialist

  $133,000 | Developer, back-end

  $130,000 | Product manager

  $129,250 | Engineer, data

  $128,000 | Developer, game or graphics

  $127,500 | Marketing or sales professional

  $125,000 | Data scientist or machine learning specialist

  $120,000 | Developer, desktop or enterprise applications

  $120,000 | Developer, embedded applications or devices

  $120,000 | Developer, full-stack

  $120,000 | Developer, mobile

  $120,000 | Scientist

  https://insights.stackoverflow.com/survey/2021#work-salary

• by faangiq on 12/23/22, 4:41 AM

  Data science is a fake field and a fake career. Used Excel once? Congrats you’re a data scientist.

• by greyhound_7 on 12/23/22, 5:52 AM

  I don't know, and it's really impossible to tell since things are changing quickly. People work as "data scientists", but there are headwinds and tailwinds. the main headwind, is that companies are cutting budgets due to the recession and dropping analysis groups that are part of cost centers. It's also easy to get the basic skills (coding/stats) done in your undergrad or masters w/o having research experience. The tailwinds, are that ML capabilities are improving day by day, so the potential to use that to make money are increasing. There's also a huge digital transformation happening, and companies have more data than ever before and potential to leverage that into savings, additional revenue, or new services.

  When I started on my data science path, about 10 years ago, and there was no training pipeline, so when I dropped out of a PhD a few years later it wasn't that hard to get a data science job with the intersection of skills: math/stats/coding/research. Today that role is probably filled by someone graduating from an undergraduate or grad program, but I know the same company is still hiring for improvements on the research project I helped start.

  Good data science, for me, is when you "apply predictive models to end user problems and ship solutions in products", but when I looked around for other jobs I realized that so few companies are able to act cross functionally to exploit the value of ML in products and services. Sure, finance does it, ads does it too, but it seems like the jobs I had access to were some ill-thought out skunkworks that a VP or exec thought was a good idea, or doing work tucked away in some business unit. There are like 10 individual problems there for YC to solve, but the more fundamental issue is that as long as we are still in the hype phase of data science, there will be incentive for business leaders to spend money on it in wasteful ways (at least for your career).

  If you want to do data science or ML, it'd encourage you to find tech first companies that are actually using ML to solve real world problems for people, and avoid working on projects that haven't shipped. Also, stay under engineering orgs. In business units, you'll have a boss that doesn't understand what you do, and you'll be promoted out of tech.

  Ultimately, I left data science and am now on an infrastructure team at a database company, which is just a better fit for values. If you can get into big tech or any tech first company, the data science is mostly figured out, but in my experience lots of companies aren't offering constructive experience. Good luck.

• by mejutoco on 12/23/22, 11:11 AM

  I believe, unless the role is understood at the top and given some slack to do properly, data scientists are setup to fail.

  The expectations are generally wildly unrealistic and the work may touch on many departments. It is a minefield politically, unless it is a very clear priority for the company.

  If the value is understood at the top data science can provide immense value.

  When the media mentions x job being sexy or a shortage of x workers there is usually an agenda. I would not take those assertions at face value.

• by 0x008 on 12/24/22, 12:54 PM

  It is more a case of companies hiring drivers and then realizing they need to pave the roads first in order for them to get to a destination in a meaningful amount of time. However, paving “data roads” seems to be quite a tough problem, hence the success of companies that promise to deliver that (such as Palantir or Snowflake)

• by mdcds on 12/23/22, 1:21 AM

  a side, but a sincere question: what do Data Scientists do and what can I expect one to produce as productive output?

  I've worked as an SDE on data engineering projects myself (Spark / Hadoop stuff) and have friends who are ML researches and develop things like better recommendation results. Never met a data scientist.