from Hacker News

What Are the Latest Scraping APIs or Services/Websites?

by jdcampolargo on 12/4/24, 11:12 PM with 7 comments

I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.

Anyone have recommendations for up-to-date scraping APIs or platforms that work well with a variety of websites?

  • by mike97 on 12/5/24, 12:27 AM

    I am actually working on a scraping API for serps, currently it only supports Google, may add more search engines if it gets any traction, it isn't ready yet, but hopefully it will be in a couple of days, if you want to give it a go let me know and I can give you a subscription in exchange for some feedback.

  • by davidsojevic on 12/5/24, 12:38 PM

    If you're looking to scrape any of the major search engines, eBay, Yelp, and a few others, then SerpApi has a massive number of up-to-date APIs to choose from:

    https://serpapi.com/search-api

    Disclaimer: I work there!

  • by nreece on 12/5/24, 9:27 AM

    * Shameless plug * Our feed builder combines a web scraper, AI-powered content auto-detection and data transformer to generate custom JSON and RSS feeds for any public webpage.

    https://newsloth.com

  • by bootstrpppin on 12/5/24, 4:39 PM

    We use Bright Data - so far we've found it to be pretty great, and very very cheap.

    Would highly recommend

  • by gbertb on 12/4/24, 11:16 PM