from Hacker News

Ask HN: Can we make side projects by web scraping?

by tpkahlon on 1/15/21, 2:03 AM with 3 comments

  • by pnwhyc on 1/16/21, 10:06 PM

    Yes. So long as you do it morally. Your project must extract novel insights from the scraped content, not steal it. Some of my most successful projects have been a result of web scraping.

    However, there are caveats. 1. If the scraped website receives frequent updates to its html structure, you will need to update your scraper frequently. This can turn your codeitonceandforgetit side project into a monthly maintenance headache. 2. You are entirely dependent on that site. If they shut their doors, or start blocking bots, you have to find a workaround or shut down your project completely

    Ultimately, I would recommend it as a means for quickly attracting users, or as automation for part of a larger project. Ensure that the value is > the maintenance.

  • by tpkahlon on 1/15/21, 2:05 AM

    Recently, I created a project (https://cs1000.vercel.app) which is refined and ad free version of an article. Is it fine to build such side projects via web scraping? Do we need approval from site/entity involved?