from Hacker News

Getting all your data out of Google Reader

by kellegous on 6/28/13, 2:35 PM with 26 comments

  • by sp332 on 6/28/13, 2:42 PM

    ArchiveTeam is extracting all the data from Google Reader and uploading it to the Internet Archive. Help out by submitting your OPML file: https://news.ycombinator.com/item?id=5958119

  • by nod on 6/28/13, 5:32 PM

    Thanks mihaip!

    Worked successfully in Windows CMD for me, without using the \bin shell script:

      cd C:\mihaip-readerisdead
  set PYTHON_HOME=C:\mihaip-readerisdead
  C:\path-to-py27 reader_archive\reader_archive.py --output-directory C:\mystuff
    Locked up at 251K out of 253K items for me, though. Restarting... success! Looks like it might have locked up trying to start the "Fetching comments" section on my first try.

  • by ccera on 6/29/13, 2:31 AM

    Warning to other impatient users:

    I didn't read the instructions too well, so the half hour I spent carefully deleting gigantic/uninteresting feeds out of my subscriptions.xml file was all for naught. Because I didn't know I needed to specify the opml_file on the command line, the script just logged into my Reader account (i.e., it walked me through the browser-based authorization process) and downloaded my subscriptions from there -- including all the gigantic/uninteresting subscriptions that I did NOT care to download.

    So now I've gone and downloaded 2,592,159 items, consuming 13 GB of space.

    I'm NOT complaining -- I actually think it's AWESOME that this is possible -- but if you don't want to download millions of items, be sure to read the instructions and use the opml_file directive.

  • by Udo on 6/28/13, 3:33 PM

    This is excellent, thank you for making this! I'm using it right now to make an offline archive of my Reader stuff.

    My only gripe would be the tool's inability to continue after a partial run, but since I won't be using this more than once that's probably OK.

    All web services should have a handy CLI extraction tool, preferably one that can be run from a CRON call. On that note, I'm very happy with gm_vault, as well.

    Edit: getting a lot of XML parse errors, by the way.

  • by DecoPerson on 6/28/13, 7:13 PM

    Thank you for this! Now I can procrastinate on my own reader app for much longer :)

    Should we be concerned with errors like this?

        [W 130629 03:11:54 api:254] Requested item id tag:google.com,2005:reader/item/afe90dad8acde78b (-5771066408489326709), but it was not found in the result
    I'm getting ~1-2 per "Fetch N/M item bodies" line.

  • by pixsmith on 7/2/13, 6:27 AM

    This is an impressive bit of work. I have had, though, an interesting thing happen, in that it's apparently trying to pull every single item from explore and from suggested items in, to the extent that I get a message saying I have 13 million items, and still going strong -- it pulled about 5 or 6 gig of data down .

    Is there some way to avoid all the years of explore and suggested items with reader archive? I tried limiting the maximum number of items to 10.000 but it was still running and growing after 12 hours. Interesting though, what it was able to accomplish in that time.

  • by skilesare on 6/28/13, 3:33 PM

    If this does what I think it does(And it seems to be doing it now on my machine), then this is truly, truly awesome.

    Thank you. mihaip, if you are ever in Houston I will buy you a beer/ and or a steak dinner.

  • by dmtelf on 6/30/13, 8:35 AM

    I'm getting "ImportError: No module named site"

    echo %pythonpath% gives c:\readerisdead

    I copied 'base' from the readerisdead zipfile to c:\python27\lib & also copied the base folder into the same folder as reader_archive.py

    C:\readerisdead\reader_archive\reader_archive.py --output-directory C:\googlereader gives "ImportError: No module named site"

    What am I doing wrong? How can I get this to work?

  • by drivebyacct2 on 6/28/13, 6:27 PM

    I guess archived RSS data for me isn't terribly important since most people seem to hide the rest of their content behind a "More" link to get those precious ad views.