from Hacker News

Ask HN: What are the most challenging pages to scrape?

by gbajson on 3/22/18, 9:09 AM with 2 comments

• by randomerr on 3/22/18, 9:23 AM

  Try PWA based one. Since they load in segments and cache a lot you'll have fun:

  https://pwa.rocks/ - Look at the business and news webpages

  Also look for the website that use AMP. They're even more fragmented then PWA pages. Below is article about AirBNB using AMP and iFrames:

  https://medium.com/swlh/how-airbnb-is-putting-amp-at-the-cor...

  https://www.ampproject.org/

• by gbajson on 3/22/18, 9:15 AM

  I am looking for pages which are difficult to scrape, to see the applied techniques, and to learn how to bypass them.