from Hacker News

eBay’s Fast Billion-Scale Vector Similarity Engine

by jabo on 5/5/23, 8:56 PM with 24 comments

  • by Scoundreller on 5/5/23, 9:33 PM

    Nice to see eBay invest further on its search side.

    This paper is too dense for me to read through at this moment, but eBay must be a real disappointment for its shareholders:

    Despite its head start at being a platform for online retail open to anyone, eBay stock has only doubled in 10 years, or in other words, added $12b in equity value (currently $24b). Imagine that: a successful tech company with a real product in a market that’s going stratospheric (online shopping) has lagged the S&P 500.

    Meanwhile Amazon has octupled, which works out to adding $864b in market value.

    In other words, Amazon has added 36 eBays worth of equity to itself in the last 10 years.

    Ok, I haven’t looked at corp debt to calculate these numbers based on enterprise value, but that could make the numbers worse: eBay is asset-light with little physical infra compared to Amazon.

    eBay’s fees have gone through the roof, so I suspect its volume has been decreasing.

  • by almog on 5/5/23, 9:46 PM

    One of the best things about eBay compared to other market places like Amazon and Aliexpress, is that their search operators include not just AND (space) but also OR (comma separated values inside parentheses) and NOT (minus) but more importantly - eBay's search engine respects the supported operators and denoting which items are search results and which are suggested/similar.

    This cannot be said about Amazon, Aliexpress, both which intentionally (I guess) mix similar items with results that match the search query.

    I hope that eBay keep those two things distinct.

  • by binarymax on 5/5/23, 11:20 PM

    Good to see another engine leverage ScANN outside of Google.

    HNSW uses lots of RAM and it’s interesting how all the major engines settled on that algo.

    I’m interested in how they apply filtering, since with codebook based similarity such as PQ and ScANN it’s not trivial.

    Maybe one day we’ll also see someone implement a production ready Vamana engine too, which also does really well at the billion scale.

  • by andre-z on 5/6/23, 3:14 AM

    This can be achieved with open-source solutions like Qdrant.https://github.com/qdrant/qdrant

  • by esafak on 5/5/23, 9:42 PM

    HNSW seems better in the criteria that matter.

    I think similarity search is a commodity now; I would not invest in developing an in-house solution given the abundance of good commercial solutions.