from Hacker News

Deep dive into finding RSS feeds

by domysee on 12/6/24, 7:17 PM with 62 comments

• by HumblyTossed on 12/6/24, 8:00 PM

  Back when I was young, websites had this icon you could click that would take you straight to their RSS feed. You young whipper snappers have gone an fucked that up. Actually, I think it was Google's fault. When they killed their RSS reader people pronounced RSS dead so people just stopped publishing RSS feeds or just didn't link to them.

  * Yes, I know the article talks about the RSS icon, i'm just soapboxing.

• by aucisson_masque on 12/7/24, 8:40 AM

  I thank WordPress for most of my RSS feed.

  I follow mostly RSS on non technology website, for instance road cycling. people that wouldn't care or know about RSS because they are not very techy, yet because they are normies that use WordPress for all their website it puts a page with RSS feed automatically. You got to find it with developer tool by searching RSS but 99% of the time if it's WordPress it got RSS.

  Thank you WordPress you bloated piece of shit :)

• by superkuh on 12/6/24, 7:27 PM

  I generally try: /rss, /feed, /index.xml, /rss.xml, /feed.xml, etc. And at various root or /directory/* locations. https://blog.jim-nielsen.com/2021/feed-urls/ is a good article with statistics on naming.

  I've been adding to my feeds.opml since reddit started dying in ~2015 and now I'm up to around ~1700 feeds and mostly independent from aggregators; though I still collect new feeds from HN/IRC/etc. Mostly I just always make a point to look for them whenever I read something cool on the web.

• by LorenDB on 12/6/24, 10:58 PM

  My modus operandi for finding a non-obvious RSS feed is to check the Wayback Machine's list of saved URLs and search for "RSS", "feed", or "XML". That normally will find the feed as long as it exists.

• by sodality2 on 12/6/24, 8:13 PM

  Tried out the feed finder on my blog again and I have another bug to report - it seems the URLs on the page can cause a crash within the web app! my blog (at matthew.science) uses Zola SSG, and it seems the URLs are formatted with a preceding //: '<a href="//matthew.science/posts/riscv/">Basics of the RISC-V ISA</a>'

  This causes the following error: TypeError: URL constructor: //matthew.science/posts/riscv/ is not a valid URL.

• by fallinditch on 12/7/24, 4:41 AM

  This looks very useful. It would work well with Hoarder (would be cool if they were integrated ;)

  Note: Hoarder can automatically hoard RSS feeds as part of its 'bookmark everything' functionality. Hoarder uses AI to tag all the content (URLs, feeds, images, notes) so you can then do full text searches on your personal archive of your bookmarks etc.

  https://hoarder.app/

• by openrisk on 12/7/24, 9:16 AM

  Its interesting to contemplate an RSS-first browser that would have this functionality built-in. Think for example of promoting to full browser status a desktop RSS reader like Akregator [1] (which already embeds a webview).

  The browser as we now know it is mostly a static application that has long lost its user-centric mission. Websites might push some stuff but the user must do thinks manually. Its primary function is to provide a search window to external search. People even stopped using bookmarks and search for everything.

  This hypothetical RSS-Browser could become the main organizational tool for the users web experience, integrating the use of bookmarks.

  In fact even more "feeds" could be integrated like email and activitypub or atproto posts. It boils down to the fact that each person has a number of profiles/roles and within each they have a taxonomy of interests and we need a tool that integrates static and dynamic sources of information.

  [1] https://apps.kde.org/akregator/

• by camel-cdr on 12/6/24, 8:56 PM

  This is useful, I set up RSS on my website yesterday.

  Turns out the feed finder couldn't find the feeds even though I've linked to them using clickable RSS icons.

  I didn't know about the autodiscovery feature so I'll add that now.

• by begriffs on 12/7/24, 1:11 AM

  I created a lightweight shell script to check many url combinations on a site for feeds.

  https://github.com/begriffs/findrss

  The combinations came from what I observed in the big list of blogs I follow. The script works pretty well for most sites.

• by csswizardry on 12/6/24, 11:07 PM

  I went canvassing for RSS feeds only yesterday! Some good stuff in here: https://bsky.app/profile/csswizardry.com/post/3lckq4qo6zs22

• by 1123581321 on 12/6/24, 7:53 PM

  It’d be neat for readers to seamlessly integrate with a scraper, either self-hosted or commercial, if no feed is found. I believe Inoreader allows scraping a few sites depending on the plan level; most reader services don’t.

• by artembugara on 12/6/24, 11:33 PM

  I open-sourced pyGoogleNews and wrote a quick blog about how you can reverse engineer google news RSS to turn it into an RSS feed of any website that is supported by Google News

  https://news.ycombinator.com/item?id=42343182

  https://github.com/kotartemiy/pygooglenews

• by panozzaj on 12/7/24, 2:14 AM

  I use a Chrome extension (https://chromewebstore.google.com/detail/get-rss-feed-url/kf...) and it seems to pick out the RSS URLs fairly consistently

• by renegat0x0 on 12/7/24, 8:37 PM

  I fought this problem, since I wrote my own RSS reader in python. Might not be perfect.

  The problem with the approach presented here is speed. Most of the web pages, especially smaller are really slow.

  Crawling most of the web pages is pain, especially if you use selenium and small SBC.

  Therefore either the page presents a clean nice RSS link, or get lost.

  Most of the good, modern pages give you nice RSS. Even GitHub gives you RSS for commits.

  For other pages I try openRSS.

  For YouTube I use yt-dlp to obtain channel id, to establish RSS.

  Algorithm is crude, but gets the job done.

  https://github.com/rumca-js/Django-link-archive/blob/main/rs...

• by ks2048 on 12/7/24, 12:03 AM

  It would be nice if someone ran this on commoncrawl and published a list of all the RSS feeds. (probably someone has?)

  Or I suppose you could just find all "Content-type: application/rss+xml" in CC.

  I know in the past, when I was looking for large lists of RSS feeds, I didn't really find what I was looking for.

• by PeterStuer on 12/7/24, 7:55 AM

  You can add '/display-feed.rss' to the list of common suffixes for many .eu sites

• by ewired on 12/8/24, 3:18 AM

  Pasting a URL in NewsBlur also uses several of these techniques to find the feed(s), and it is open source, so the feed-finding code could be ripped out of NewsBlur as an alternative to this.

• by kelvinjps10 on 12/6/24, 11:50 PM

  I would like to be able put multiple websites, I had to build a script based on "Guessing the feed URL" approach to get the rss feeed of a bunch of websites that I had bookmarked

• by ulrischa on 12/6/24, 7:36 PM

  Would be nice if it is implmented in freshrss

• by benrapscallion on 12/7/24, 1:05 AM

  Does it correctly ignore the “Comments on:” feeds that are sometimes mistakenly chosen over the main feed?

• by zenlot on 12/6/24, 9:50 PM

  Came here through RSS link from miniflux, running on nvidia jetson.

• by saaaaaam on 12/6/24, 10:59 PM

  Ghost also publishes at /feed