from Hacker News

Ask HN: Why do people use Puppeteer for webscraping instead of SaaS services?

by proszkinasenne2 on 1/19/21, 8:39 PM with 4 comments

Hi HN

As in the title, I am wondering what are the reasons anyone would use Puppeteer/Selenium/other-browser-emu for web scraping if there are already tens if not hundreds of SaaS services offering "scraping-as-a-service". Except for JS execution.

A handful of examples: Scrapehero, Webrobots, Apify, Scrapingbee, Scrapinghub, Promptcloud

Except for the ones that require setup fee, or have ridiculous pricing models. Why would anyone want to setup Puppeteer/Selenium/other scraping bots instead of using one of the "scraping-as-a-service" platforms?

  • by phendrenad2 on 1/19/21, 8:44 PM

    Probably because people who are doing web scraping aren't professional scrapers, they're just programmers who need some data quickly. And since they're already familiar with Selenium, they think that's the state of the art. I've never seen an ad for a scraping service, so I also didn't know that they existed.

  • by billconan on 1/19/21, 8:52 PM

    my main concern is pricing. many websites use anti-scraping technologies. scraping the html doesn't work anymore. need to load everything and execute js. for example, I have seen some can detect headless / puppeteer mode too. I ended up creating my own scraping infra using vanilla chrome...

    current saas platforms charge by request count. If I need to load everything, the cost will be too high.