from Hacker News

Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

by phodo on 5/30/24, 8:24 PM with 90 comments

As the title says, I have many PDFs - mostly scans via Scansnap - but also non-scans. These are sensitive in nature, e.g. bills, documents, etc. I would like a local-first AI solution that allows me to say things like: "show me all tax documents for August 2023" or "show my home title". Ideally it is Mac software that can access iCloud too, since that where I store it all. I would prefer to not do any tagging. I would like to optimize on recall over precision, so False Positives in the search results are ok. What are modern approaches to do this, without hacking one up on my own?

  • by bastien2 on 5/30/24, 11:39 PM

    You don't. You use a full-text indexer and normal search tools. A chatbot is only going to decrease the integrity of query results.

  • by pierre on 5/30/24, 10:34 PM

    RAG cli from llamaindex, allow you to do it 100% locally when used with ollama or llamacpp instead of OpenAI.

    https://docs.llamaindex.ai/en/stable/getting_started/starter...

  • by m0shen on 5/30/24, 9:54 PM

    Paperless supports OCR + full text indexing: https://docs.paperless-ngx.com/

    As far as AI goes, not sure.

  • by Ey7NFZ3P0nzAe on 5/31/24, 5:05 PM

    I am a medical students with thousands and thousands of PDF and was unsatisfied with RAG tools so I made my own. It can consume basically any type of content (pdf, epub, youtube playlist, anki database, mp3, you name it) and does a multi step RAG by first using embedding then filtering using a smaller LLM then answering using by feeding each remaining document to the strong LLM then combine those answers.

    It supports virtually all LLMs and embeddings, including local LLMs and local embedding It scales surprisingly well and I have tons of improvements to come, when I have some free time or procrastinate. Don't hesitate to ask for features!

    Here's the link: https://github.com/thiswillbeyourgithub/DocToolsLLM/

  • by constantinum on 5/31/24, 3:13 AM

    The primary challenge is not just about harnessing AI for search; it's about preparing complex documents of various formats, structures, designs, scans, multi-layout tables, and even poorly captured images for LLM consumption. This is a crucial issue.

    There is a 20 min read on why parsing PDFs is hell: https://unstract.com/blog/pdf-hell-and-practical-rag-applica...

    To parse PDFs for RAG applications, you'll need tools like LLMwhisperer[1] or unstructured.io[2].

    Now back to your problem:

    This solution might be an overkill for your requirement, but you can try the following:

    To set things up quickly, try Unstract[3], an open-source document processing tool. You can set this up and bring your own LLM models; it also supports local models. It has a GUI to write prompts to get insights from your documents.[4]

    [1] https://unstract.com/llmwhisperer/ [2] https://unstructured.io/ [3] https://github.com/Zipstack/unstract [4] https://github.com/Zipstack/unstract/blob/main/docs/assets/p...

  • by elrostelperien on 5/30/24, 9:59 PM

    For macOS, there's this: https://pdfsearch.app/

    Without AI, but searching the PDF content, I use Recoll (https://www.recoll.org/) or ripgrep-all (https://github.com/phiresky/ripgrep-all)

  • by hm-nah on 5/31/24, 2:23 PM

    You have the find a good OCR tool that you can run locally on your hardware. RAG depends on your doc processing pipeline.

    It’s not local, but the Azure Document Intelligence OCR service has a number of prebuilt models. The “prebuilt-read” model is $1.50/1k pages. Once you OCR your docs, you’ll have a JSON of all the text AND you get breakdowns by page/word/paragraph/tables/figures/alllll with bouding-boxes.

    Forget the Lang/Llama/Chain-theory. You can do it all in vanilla Python.

  • by Kikawala on 5/30/24, 10:19 PM

  • by pixelmonkey on 5/31/24, 2:45 AM

    rga, aka ripgrep-all, is my go-to for this. I suppose grep is a form of AI -- or, at least, an advanced intelligence that's wiser than it looks. ;)

    https://github.com/phiresky/ripgrep-all

  • by SoftTalker on 5/31/24, 2:23 AM

    If you haven’t given some serious thought to getting rid of most of the documents then consider it. There is very little need to keep most routine documents for more than a few years. If you think you need your electric bill for March 2006 at your fingertips, why?

  • by Kikobeats on 6/7/24, 1:49 PM

    You can use Microlink to turn PDF into HTML, and combine it with other service for processing the text data.

    Here an example turning a arxiv paper into real text:

    https://api.microlink.io/?data.html.selector=html&embed=html...

    It looks like PDF, but it you open devtools you can see it's actually a very precise HTML representation.

  • by theolivenbaum on 5/31/24, 11:50 AM

    If you're looking for something local, we develop an app for macOS and Windows that let's you search and talk to local files and data from cloud apps: https://curiosity.ai For the AI features, you can use OpenAI or local models (the app uses llama.cpp in the background, it ships with llama3 and a few other models, and we're soon going to let you use any .gguf model)

  • by brailsafe on 5/31/24, 1:17 AM

    Like many others have suggested, local indexing is what I use for this, although some more natural interface may be better for structured search and querying.

    What I haven't seen suggested though, is the built-in spotlight. Press CMD+Space, type some unique words that might appear in the document, and spotlight will search it. This also works surprisingly well for non-OCRd images of text, anything inside a zip file, an email, etc..

  • by yousnail on 5/30/24, 10:47 PM

    PrivateGPT is a great starting point for using a local model and RAG. Text-generation-ui, oogabooga, using superbooga V2 is very nice and more customizable.

    I’ve used both for sensitive internal SOPs, and both work quite well. Private gpt excels at ingesting many separate documents, the other excels at customization. Both are totally offline, and can use mostly whatever models you want.

  • by ssahoo on 6/8/24, 2:43 PM

    This could be a humor or real hack.

    Get a copilot PC with recall enabled and quickly scan through the documents by opening in Adobe acrobat reader. Voillla! You will have an sqlite dB that has your index. Few days later, Adobe could have your data in their llm.

  • by gibsonf1 on 5/30/24, 9:36 PM

  • by pawelduda on 5/31/24, 11:07 AM

    Try https://github.com/phiresky/ripgrep-all before going down the rabbit hole of AI and advanced indexers. Quick to set up and undo if that's not what you want, but I'm pretty sure you'll be surprised how far can this get you

  • by ilaksh on 5/31/24, 3:22 AM

    If you want to run locally you can look into this https://github.com/PaddlePaddle/PaddleOCR

    https://andrejusb.blogspot.com/2024/03/optimizing-receipt-pr...

    But I suggest that you just skip that and use gpt-4o. They aren't actually going to steal your data.

    Sort through it to find anything with a credit card number or anything ahead time.

    Or you could look into InternVL..

    Or a combination of PaddleOCR first and then use a strong LLM via API, like gpt-4o or llama3 70b via together.ai

    If you truly must do it locally, then if you have two 3090s or 4090s it might work out. Otherwise it the LLMs may not be smart enough to give good results.

    Leaving out the details of your hardware makes it impossible to give good advice about running locally. Other than, it's not really necessary.

  • by bendsawyer on 5/30/24, 10:05 PM

    I looked into this for sensitive material recently. In the end I got a purpose-built local system built and am having it remotely maintained. Cost: around 5k a year. I used http://www.skunkwerx.ai, who are US based.

    The result is a huge step up from 'full text search' solutions, for my use case. I can have conversations with decades of documents, and it's incredibly helpful. The support scheme keeps my original documents unconnected from the machine, which I own, while updates are done over a remote link. It's great, and I feel safe.

    Things change so fast in this space that there did not seem to be a cheap, stable, local alternative. I honestly doubt one is coming. This is not a on-size-fits-all problem.

  • by skapa_flow on 5/31/24, 8:29 AM

    Google Drive. It doesn't fullfill the "local" criteria, but it works for us (small engineering firm). We synchronize our local file server with GD nighly and use it only for searching. Google is just good when it comes to search.

  • by phodo on 6/3/24, 6:00 AM

    Thank you all for the comments. Got a lot of good input and ways to think thru the tried and true tools (enjoying ripgrep-all + fzf) plus the standard ai/rag-style tools. I do think there is room for a bridge or an integrated way to pipe in similarity / embedding into the ripgreps of the world. Maybe something close to fzf’s piping model. Will explore if I have some time.

  • by westcort on 5/31/24, 1:33 AM

    Use Recoll on Linux or File Locator Lite on Windows to do RegEx searches. Design the RegEx searches with GPT or llama running locally (or write them yourself).

  • by hulitu on 5/31/24, 5:12 AM

    > Ask HN: I have many PDFs – what is the best local way to leverage AI for search?

    Adobe Reader can search all PDFs in a directory. They hide this function though.

  • by kkfx on 5/31/24, 7:07 AM

    Honestly?

    ocrmypdf + ripgrep-all, recoll (GUI+XLI xapian wrapper) if you prefer an indexed version, for mere full-text search, currently nothing gives better results. The semantic search it's still not there, Paperless-ngx, tagspaces and so on demand way too much time for adding just a single document to be useful at a certain scale.

    My own personal version is org-mode, I keep all my stuff org-attached, so instead of searching the pdfs I search my notes linking them, a kind of metadata-rich, taggable, quick, full-text search however even if org-ql is there I almost not use it, just org-roam-node-find and counsel-rg on notes. Once done this allow for quick enough manual and variously automated archiving, do it on a large home directory it's a very long and tedious manual work. For me it's worth done since I keep adding documents and using them, but it took more than an year to be "almost done enough" and it's still unfinished after 4 years.

  • by treetalker on 5/31/24, 2:33 AM

    On MacOS, use HoudahSpot. It’s awesome. Not AI, but as others have said, you likely want plain text search, not “AI” or a chatbot, for something like this.

    If you’re having trouble thinking of search terms to plug into HoudahSpot (or grep etc.) then I suppose you could ask a chatbot to assist your brainstorming, and then plug those terms into HoudahSpot/grep/etc.

  • by epirogov on 5/31/24, 10:58 AM

    Cheap but full featured solution for batch AI processing of PDF documents on your local is an Aspose.PDF ChatGPT plugin

    https://products.aspose.org/pdf/net/chat-gpt/

  • by dudus on 5/30/24, 9:33 PM

    I tried Google's NotebookLM for this use case and was very pleased with the experience.

    If you trust Google that is.

  • by jesterson on 5/31/24, 7:55 AM

    The best tool I found for myself for similar goal was Devonthink. Using it for many years since and quite happy with it.

    There is no AI or any other modern fad, but fulltext search (including OCR for image files inside PDFs) works great

  • by 1123581321 on 5/30/24, 10:02 PM

    Devonthink would do this with a tiny model to translate your natural length search prompts into its syntax and your folder/tag tree.

    If you're okay with some false positives, Devonthink would work as is, actually.

  • by edgyquant on 5/30/24, 10:57 PM

    Using python to dump the PDF to text then use llama3 (8B) to parse

  • by jeffreyq on 5/31/24, 2:28 PM

    Tangentially related, but you can try https://macro.com/ for reading your PDFs.

  • by hypefi on 6/1/24, 8:19 AM

    check out my app "Chofane" this is something that does it, local batch OCR scan for PDFs and PNG files, I am just launching it, you can export results to json and csv, do some text based search on results https://chofane-landing.pages.dev/

  • by Tylast on 5/31/24, 9:34 AM

  • by sciencesama on 5/30/24, 11:58 PM

    You can tabulate the info 90% of your info will be from single source. There are online tools that sort costco and walmart bills !!

  • by gandalfthepink on 5/31/24, 12:39 AM

    I use Curiosity AI. Good interface.

  • by vrighter on 6/1/24, 5:14 AM

    you use a tool intended for accurately searihing. Which is not ai based.

  • by finack on 5/30/24, 9:49 PM

    OCR and pattern matching on text are computationally cheap and incredibly easy to do. For example, tax documents often bear the name of your government's tax authority, which presumably you are familiar with and can search for. They also tend to have years on them.

  • by adyashakti on 5/30/24, 8:47 PM

    getcody.ai