from Hacker News

Meilisearch 1.0 – Open-source search engine built in Rust

by tpayet on 2/8/23, 1:31 PM with 180 comments

  • by schappim on 2/8/23, 6:38 PM

    We’ve used Meilisearch in production and it is the closest thing to self hosted Algolia you can get, which in itself is pretty amazing.

    Unfortunately the performance of indexing (constantly changing records) wasn’t great and Meilisearch would fall behind on indexing records for hours.

    Meilisearch has been amazingly great for projects where records don’t change all that much (eg docs, or even a customer database), but if you have for example a fast paced ecommerce system with 50k records constantly changing (eg product inventory), it falls over pretty quick. We had to transition over to Elastic for this aspect of our app.

    The other issue we faced is their Rails gems falling out of step with the server, and when fixes came out, the Rails gem was incompatible for a while.

    I really really hope 1.0 increases performance to the point where it becomes production ready, because the initial out of the box performance (before getting bogged down with indexing) was pretty amazing. Better than Elastic and on par with Algolia.

    I recommend keeping Meilisearch on your radar. It is going to be great.

    I wish the best for the Meili team and hope they succeed!

  • by manigandham on 2/8/23, 1:58 PM

    Congrats to the team, it's been interesting to watch the development of Meilisearch (and it's close competitor Typesense). Algolia has really paved the way here but it's nice to see the open-source options with more configurations and better default UX.

    There's also many search libraries if you want to embed search more deeply into your app. I have a list of modern search systems and libraries here: https://manigandham.com/post/search-systems-libraries

  • by kacy on 2/8/23, 1:57 PM

    We’ve been using a Meilisearch for the last six months or so and have been delighted with its performance and usability. It uses a fraction of the resources as Elasticsearch, and the language support is extensive and very active.

    That being said, our cluster is much smaller than other ones I’ve worked with in the past, so I can’t comment on its reliability at massive scale. I’ve also been very impressed with how active contributors are on GitHub and in their Discord. Everyone seems like good people, and it’s a project I’m excited to keep using.

  • by sandstrom on 2/8/23, 5:38 PM

    Great news!

    Been following along for a while and it's a great project. ElasticSearch needs some competition.

    For us, there are two things missing for us before we could make the switch:

    1. Multi-index search; Standard use-case is searching across e.g. users and companies. Common in many SaaS-applications, where you want a single search field with type-ahead for e.g. contacts/organisations/tasks/events.

    2. Decay functions; Basically to gradually phase out results for things based on age, distance or something similar. ElasticSearch has pretty good support for these. https://www.elastic.co/guide/en/elasticsearch/reference/curr...

  • by chimen on 2/8/23, 2:03 PM

    Is Rust that important that you have to place "built in Rust" in the title? Is this like a cult following that we only bet on traffic and interest coming from other evangelists where Rust is the only feature that matter?

    4 months ago: " Meilisearch, open-source alternative to Algolia in Rust lands a $15M Series A"

    It's not the first time I see, there are at least 2-3 daily submissions reaching the FP in this manner so I'm curious: "built in Rust" = marketing these days?

  • by tmikaeld on 2/8/23, 2:21 PM

    My team tried to use Meilisearch for large datasets, unfortunately, it's impossible to plan the RAM usage. If you have very little searches, it consumed very little, but if you have a lot of search traffic, it may consume more than we could provision beforehand. This made it too unpredictable and too expensive, so we went with Manticore instead. I don't know if this has been addressed in 1.0, hopefully it has.

  • by amateurdev0_07 on 2/9/23, 12:51 AM

    Thank you for making MeiliSearch. I use it for a personal project that gets a few hits a day, mostly from me and my friends.

    https://pulpflakes.com/fmisearch/

    It's a search over an index of fiction in the English language, first published in periodicals. Searchable by author, artist, magazine name and specific issue. Biggest index has about 200K documents, doc sizes are tiny.

    Integrated with my WordPress site by handwritten PHP. Which was fun.

    Performance is great. I didn't run into too many issues, and those I did i could resolve. What i remember:

    1. The rules for text searches are too strict by default and if the order of words is different, will result in no matches. A, B will not return a result if B A is in the database.

    2. Creating an index, uploading documents and changing settings required quite a bit of work. A week's worth of coding, almost. Would have loved to have a reasonably robust shell script that could take a JSON file with metadata on index and do the grunt work.

    3. I have multiple types of documents, would have liked search to cover all of them so I don't have to change search type manually each time.

    4. The default number of documents and max uploaded file size is too low. 200K and 200 MB or something. But it fails even on smaller file size.

    The above sound like complaints. They're problems I ran into and others might. I love how productive Meilisearch made me. Thank you.

  • by networked on 2/8/23, 10:13 PM

    The most specific criticism I have read of Meilisearch is https://news.ycombinator.com/item?id=32940683. It has four points: (1) words beyond 65535 are silently ignored (this is documented in https://docs.meilisearch.com/learn/advanced/known_limitation... ); (2) the position of a matching word in a document non-optionally affects ranking; (3) to get the match information you must retrieve the entire attribute; (4) the meaning of PUT and POST is switched relative to RFC 7231.

    Are points (2) through (4) true? Has any of the points been an issue for you in practice?

  • by mmachatschek on 2/8/23, 4:56 PM

    This is awesome news! We've been using meilisearch in production for a few months now and we're more than happy with its reliability. Their work of the last few months really paid off, as the search speed and especially the indexing speed has increased a lot thanks to their efforts.

    I'm excited to see all the things they'll build in the future.

  • by nop_slide on 2/8/23, 1:50 PM

    This looks really cool and I might try the self hosted option out on my small website as an upgrade from Postgres’ full text search.

    I was hoping the cloud version would be more appealing, granted there seems to be a generous free tier but the next option is $1200 a month?!

  • by leeoniya on 2/8/23, 2:20 PM

  • by rsstack on 2/8/23, 4:40 PM

    Is there a way to run it in WASM, to get something like Lunr[1]? We prefer to do our (small-index, <2MB) search client-side for a bunch of reasons, currently using Lunr.js, but it's a bit annoying and the typeahead search is something I improvised and not really official.

    [1] https://lunrjs.com/

  • by kristiandupont on 2/8/23, 2:32 PM

    I am using the core (called "Milli") in a local indexer that I run on my repositories and Obsidian files. It works like a charm and I am very happy with it. Obviously that's a use case with very little traffic but just indexing my repositories folder is quite a bit of work and it does it surprisingly fast.

    The only real thing I am missing is a typeahead feature.

  • by scop on 2/8/23, 8:25 PM

    Congrats! Question for the team as I see a possible discrepancy on the website.

    The "Comparisons" page says there is no limit for number of indices (https://docs.meilisearch.com/learn/what_is_meilisearch/compa...)

    However, the "Limitations" page says there is a limit of ~180 indices (https://docs.meilisearch.com/learn/what_is_meilisearch/compa...)

    Can you clarify what, if any, are the limitations of # indices?

  • by zX41ZdbW on 2/8/23, 7:32 PM

    You can query Meilisearch directly from ClickHouse with the integrated table function: https://github.com/ClickHouse/ClickHouse/pull/33332

    This feature was a student project, and I'm not sure if it will find its usage. If you are using Meilisearch with ClickHouse, or if you think this feature is worth something, please let me know.

  • by survirtual on 2/9/23, 5:37 PM

    This looks like an effective piece for a project I have. It would be significantly more effective if it was published on crates.io and could be instantiated within Rust, and was able to operate in memory (or have a filesystem passed to it, so that can be simulated)

    I found this issue which tracks crates.io publication: https://github.com/meilisearch/meilisearch/issues/3367

    Would be nice to see that made a priority. Having a powerful search engine that can be embedded in a larger application and made portable (like being able to deploy to WASM) would be extremely novel and valuable. Given Rust is already in use, I think it may not necessarily demand too much effort. When search becomes a focus for what I’m working on, perhaps I will make that happen if not already done yet.

    Thanks for making this available to people.

  • by jvans on 2/8/23, 8:14 PM

    This looks very cool, nice work. Any plans to support ANN vector searches in the near future?

  • by heybrendan on 2/8/23, 10:04 PM

    How would one begin to use this when data is stored in MySQL, MariaDB, or PostgreSQL?

  • by xarope on 2/10/23, 1:40 AM

    I tried RTFM'ing, but can Meilisearch handle restricted documents (P&C) and integrated with LDAP/AD to pull security groups?

    P.S. great to see your documentation search is powered by your own product (!)

  • by wiradikusuma on 2/8/23, 6:37 PM

    I see comparison against other search engines, but how does it compare to RDBMS full text search e.g. Postgre's? I know it's not apple-to-apple, but most people start with RDMBS.

  • by drcongo on 2/8/23, 2:12 PM

    Congrats team. Meilisearch is an absolute joy to work with.

  • by dawnerd on 2/8/23, 3:41 PM

    Very early adopter of meilisearch and it’s pretty great. But bumpy as the team found their footing but overall very impressed with it.

  • by hnaccountme on 2/9/23, 11:16 AM

    Anyone else having deja vu of when Java did this sort of 'X' build with Java?

  • by fyzix on 2/11/23, 3:23 PM

    They need to structure their pricing page better. A quick glance had me thinking that $1200 was the minimum for production use. But the free tier is actually pay as you go.

  • by msvan on 2/8/23, 2:05 PM

    How does Meilisearch compare to ElasticSearch from an operational point of view? I've experienced ElasticSearch to be quite painful to maintain, requiring lots of manual tweaking to balance shards and careful design of indices.

  • by cies on 2/8/23, 3:19 PM

    I think multi-lingual stemming is the point where I see this as a real ES competitor. Still they've come a long way, and burning too much RAM on ES is not the way fwd either.

  • by ckok on 2/8/23, 5:39 PM

    Does it have any kind of master/slave or replication abilities? Couldn't find anything in the docs.

  • by slig on 2/9/23, 12:41 AM

    Is there a way to somehow find related documents to a specific document?

  • by snowpid on 2/8/23, 2:37 PM

    Can you rewrite it in Rust?

  • by muhammadusman on 2/8/23, 10:07 PM

    how does this compare to Typesense? I'd like to see which one uses fewer resources for similar performance

  • by garbagecoder on 2/8/23, 4:10 PM

    What language you write a program in is not a feature, definitely not a headline one.