by darthShadow on 11/9/22, 8:40 PM with 373 comments
by widdershins on 11/9/22, 9:42 PM
What really excites me is that is that I'm paying them. That sounds odd, but seriously. It's incredibly refreshing to know that the company providing my search results has an incentive to make things better for _me_ and not a legion of advertisers. With Google I can't help thinking about every keypress being logged to optimize sales pitches at me. I just don't feel that with Kagi, because I'm paying them.
Sure, they might be logging every keypress (I don't actually think they are, but you never can tell) but even if they were, I could be reasonably certain they were doing it to retain my subscription, which probably means making my search better, not selling me other stuff.
It's a priveleged position to be in, and the economic argument isn't watertight, but in the "search as a brain extension" space it still _feels_ premium, because it creates trust. And that frees up brain space for other things - like where the hell was that article I was looking for?
by aabaker99 on 11/9/22, 10:14 PM
Now, I will admit that for this particular query Kagi and Google results are pretty close. But my general experience is that when I search in Google I find that I have to look farther down the search results to look past the blogspam to find the authoritative reference.
by kbyatnal on 11/9/22, 11:41 PM
For some reason, a lot of these search engines like to brag about the number of documents in their index, which never made sense to me. Maybe it was true in the past, but on the modern web, larger index !== better results. In fact, I'd argue the opposite since you're much more likely to serve SEO spam.
That was my motivation to start hacking around on CrowdView (https://crew-rho.vercel.app), a search engine specifically for forums and discussion content (e.g. forums, discords, twitter, reddit, etc). It has a curated index (today, curated by yours truly) to remove SEO junk and help you figure out "what does a real, genuine human think about this think?"
by yarg on 11/9/22, 9:28 PM
PageRank was never designed for adversarial scenarios.
It reminds me of KPIs like lines of code - it's only useful if it cannot be manipulated.
"When a measure becomes a target, it ceases to be a good measure."
by snowwrestler on 11/9/22, 10:41 PM
From a more macro perspective, I’ll believe Google is failing when a competitor starts eating their lunch. What I see right now are a bunch of would-be competitors who want to eat their lunch, including this company. The blog post is probably best understood as aspirational rather than descriptive.
As a user of search, Google results are frustrating at times, but is that because “pagerank is over?” Or because it’s an incredibly hard problem they’re working on? Google does not have to be objectively perfect to keep succeeding, they just need to be better than other search engines.
by larve on 11/9/22, 10:22 PM
I whole heartedly recommend kagi. My favourite feature is the "blast from the past" that shows results that are not online anymore, but links to web.archive.org.
by fleddr on 11/10/22, 12:24 AM
Even in a hypothetical situation where ads would no longer be the driving force of algorithms, something else will. As soon as something gets large enough, it will be gamed. If not for commercial reasons, it might be cultural/political influence.
I will end by reminding ourselves that us techies should spend some more time with ordinary folks. I agree with everybody here that Google Search has been getting worse and worse for years now, especially for our niche searches.
It's a mistake though to think that this is widely experienced as such. My mum looks up an unknown ingredient in a recipe and within a second sees a picture of it. That means it works. All kinds of personal data might be shared in the process but since you can't really see that, it didn't happen. From her point of view, Google Search works extremely well and is close to magic.
The point being, Google doesn't give a shit that you don't get the best answer for your query on JavaScript closures. Nobody searches for that, and those that do, block ads.
by akrymski on 11/9/22, 9:53 PM
Actually Google is already doing this. Results are personalized and contextual. It can't really know what I feel like eating this evening because I don't, but it can guess.
I applaud the author for trying, but I don't see an alternative to PageRank being proposed. How exactly is Kagi proposing to rank results?
And no, I don't want an AI generated summary when I search for the best tutorial to do X. I want a list of tutorials. The question is how to rank that list, and I've yet to see anyone do a better job than Google.
Google is far more than PageRank. Its an AI ranking model that has the largest training dataset (queries and clicks).
Ads may suck, but simply charging for the same service isn't really innovative. Would I pay for a version of Google without ads? Probably not. But that's just me - I actually like to know who is advertising for particular searches. A company with an ad budget to rank at the top of ads is probably more trustworthy than an anonymous website.
by candyman on 11/9/22, 9:27 PM
by krm01 on 11/9/22, 9:27 PM
It’s mostly to find an image, a video, a location on maps, etc.
I wonder if Google could turn things around by rethinking the search experience. They’re still the fastest first stop on your journey to find something on the web. But the behavior has changed dramatically that a change in their UX alone could really make a big impact. Maybe moreso than just an algorithm change.
I once redesigned the search experience of a large global ecommerce company. The UX changes alone grew their revenue quite a substantially and reduced customer complaints.
by karaterobot on 11/9/22, 10:27 PM
By the way, I've been a satisfied Kagi customer since the day the beta ended, and I have nothing bad to say about the service. Okay, it would be nice if they remembered I like to view temperature in Fahrenheit, but whatever.
by PaulHoule on 11/10/22, 12:27 AM
but it takes a huge amount of training/test data to do it and takes a lot of computational work after you've got that data. What TREC revealed is that most of the things that would obviously improve search relevance don't, and it took 5 whole years of 20 teams working on it before a useful discovery was made.
There was an article about TikTok that revealed just how wrongheaded the viewpoint of the current web is. As much as Google fetishizes data, the data collected by sites like YouTube is useless because they offer you too many things to click on so you click around like a chicken with its head cut off. There are maybe 5 things that you like out of 20 that they show you, but which one you click on is random so the signal is mostly noise and worst of all they can't come to the conclusion that you didn't like any of the other 19 things they showed you.
TikTok shows you exactly one thing at a time so the opinion that they capture is meaningful.
It reminds me of one of the first "learning to rank" papers where I talked the management at the CU Library to let the Thorsten Joachims group run our search engine and we realized just how poor of a signal you get from search engine usage and how challenging it is to feed it back to improve your results.
Really marking up judgements for all the results that turn up and using "pooling" to add new results to the original set when your search engine is essential to make really better search. Otherwise you can be just another one of those guys who posts to Medium about the "semantic search" engine he built but can't tell you if the results are any better than any other search engine.
by dageshi on 11/9/22, 9:45 PM
Plenty of excellent youtube channels that 10 years ago would've been websites.
I don't really have a problem with this, all things change.
by ineptech on 11/9/22, 11:03 PM
This is true IFF this software is open source and running on my own server. Hard to see how this can happen.
by Animats on 11/10/22, 12:00 AM
Since Google turned to the dark side on August 9, 2006[1], ranking by links weight has been hopeless. But the alternatives are not much better.
by irsagent on 11/9/22, 9:42 PM
by jmyeet on 11/9/22, 11:02 PM
This reminds me of posts from 10 years ago where "the future of search of social" (this post says "user-centric" instead). Even Google feared that outcome (given the then-meteoric rise of Facebook; oh how the turns have tabled). Personally I find the musings in this post to border on nonsencial.
But here's the interesting part: certain ideas are sticky just people want them to be true. The core one here is that "search is broken" or even the lesser "Google is getting worse". It's one many on HN like to push. I don't think theres any quantitative empirical evidence of this. It's people wanting it to be true and you'll see it on every thread about DDG (as one notable example).
Silicon Valley is littered with the corpses of "Google killers" who seem to have all fallen into different versions of this trap.
by mrkramer on 11/9/22, 11:12 PM
This is exactly what Google is trying to do....unsuccessfully. I doubt you will have the scale to outperform Google if you try to replicate Google's AI model and replace ad based search with subscription based search. My point is don't do what Google is doing, think differently.
by SevenNation on 11/9/22, 10:17 PM
> “Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is “The Effect of Cellular Phone Use Upon Driver Attention”, a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98].
A search for cellular phone now returns a page of links to plans, and one link to Wikipedia.
by aimor on 11/9/22, 10:10 PM
"And yes, the non-zero price point will mean you have to budget it with your other costs. But faster access to higher quality information will make you much more competitive globally, so you can decide if the investment will be worth it, like any other purchase you make. This will in turn incentivize these products to be even better, a positive feedback loop driven by entirely aligned incentives."
For businesses I think this effect is only driven by competition regardless of revenue model.
by matai_kolila on 11/9/22, 9:38 PM
There are so many ways to use Google to obtain the information you're looking for, and so many people do it literally on a minute-to-minute basis that it's flat absurd to call that ineffective.
It's trivially easy to learn something by typing a question into Google; maybe when writing an article try gut checking it against obvious observed reality first.
I simply cannot fathom a noble reason why the author would decide to publish this.
Edit: Oh wait I found it; the author is Vladimir Prelovac, CEO, Kagi Inc. What does Kagi Inc. do? I'll leave that as an exercise to the reader, but I bet you guess it in one try...
by pritambarhate on 11/10/22, 6:21 AM
> It costs us about $1 to process 80 searches.
This caught me by surprise. It would be good to know if this is pure hardware and bandwidth cost, or if they are factoring in all the costs associated with running a business. I wonder if costs for Google are also on the same level.
There team's plan seems to quite expensive considering we are so used to free (ad supported) search.
by andirk on 11/9/22, 11:29 PM
He then described how people will first do their usual _web searches_ via FB search bar. To my knowledge, that didn't happen. And he has since pivoted to "the future is private".
[0] https://techcrunch.com/2010/04/21/zuckerbergs-buildin-web-de...
by julienb_sea on 11/9/22, 10:49 PM
by calrain on 11/10/22, 1:38 AM
PageRank could have been fine if the intrinsic motivations were to educate, enlighten, and help people.
Just because a search technology is wrapped up to look like a personalized AI doesn't mean it is immune to the same intrinsic motivations.
Who is to say that Google won't release a 'Personalized Search AI' that is free for the user and it continues to follow the same motivations that their PageRank solution does.
I think custom AI's tuned for each person will be the defining technical foundation for the next few decades, but I'm extremely concerned that the fundamental business drivers that compete for sales, view time, and divisiveness will not be resolved.
Nothing will stop an authoritarian government, an ad driven company, an entertainment company, or a political research company from giving out 'free' personal AI's.
If the history of the Internet has anything to teach us then it is this:
The free ones will always be more successful than the paid ones.
by andromeduck on 11/9/22, 9:37 PM
It's often pretty fustrating to try and find something you know you've seen before but don't remember quite the name of or exact keywords.
by c7b on 11/9/22, 10:16 PM
It sounds all great and I agree that we should be prepared to pay for quality services instead of expecting everything to be free (so I will have a look at kagi). But this promise makes me skeptical: Twitter is apparently going to lose money on their "$8/m for half the ads" deal with US users because the ads make more money than that for them. Sure, paying for your services is going to mitigate the incentive problem, but it might not fully eliminate it. Of course, kagi has a reputation to lose with its users, so that is another line of defence, but I guess the best way to build such trust would be to be maximally transparent, even about the incentive structures.
by AussieWog93 on 11/9/22, 9:32 PM
More and more I find myself searching through moderated social media like Reddit or Facebook Groups/Marketplace for product recommendations or local services.
Gave Kagi a go, but it seemed to be even worse than Google for what I tried.
by leobg on 11/9/22, 10:23 PM
by PaulHoule on 11/9/22, 10:47 PM
It took years for peer-reviewed papers to show any benefit from PageRank at all, part of it is that a real relevance function has to balance the keyword x document influence vs the document influence and you don’t come out ahead ranking an irrelevant document highly if it has a high page rank. (E.g. a popular document that is irrelevant is… irrelevant)
If you believe the original paper, PageRank is simulating the density of a random walk over web pages and Google has been able to sample that density directly w/ Google Analytics, Chrome browser telemetry and all their other web bugs.
by JesseMReeves on 11/10/22, 12:59 PM
That being said, I'm beginning to think that having a price tag as entry barrier is a good thing for a search engine.
One of the reasons for Google's downfall is that they had to cater to dumb and easily controllable masses.
Also, the aggressive SEO that made the search results so bad made economical sense as soon as those masses flocked at the platform.
Kagi may only stay this good if it does not become too popular. I hope the developers think about the fact that their best strategy might be focusing on their current sizeable niche market of nerds and intellectually curious people.
by tren on 11/10/22, 12:24 AM
by CalChris on 11/9/22, 9:49 PM
by summerlight on 11/10/22, 7:14 AM
by entwife on 11/9/22, 10:57 PM
Two improvements that I'd like to see in search results are: (1) the ability to exclude particular domains from search results; and (2) the ability to over-weight certain referrers (i.e. a link from Encyclopedia Britannica is worth more than a link from Wikipedia, which is worth more than a link from MySpace.)
by sushiburps on 11/10/22, 9:43 AM
https://dkb.io/post/google-search-is-dying
Big discussion here and on Reddit when that article was posted. It doesn't set out to say Google search is worse/getting worse, but that people don't trust its results as much as human generated content (thus appending 'Reddit' to queries).
by endisneigh on 11/9/22, 9:47 PM
You can mitigate it with curation, but then you’re bound to niches and bias, which is true with automation as well. No victory here.
by cube2222 on 11/9/22, 10:04 PM
But in general, especially when looking for specific phrases or trying to find discussions related to a topic, it's working extremely well. Whenever I used it, I was happy with it, and the search result quality was better than in Google.
It's good to note though that iirc Kagi does actually use commercial search services by Google/Microsoft behind the scenes, in addition to their own custom components.
by birdyrooster on 11/11/22, 5:12 AM
This is a non-sequitur. Publishers could monetize their traffic without PageRank, it is not a necessary condition for the publishers to realize they could monetize the traffic. This thesis statement is the crux of the rest of their argument, which makes the rest of this not make any sense.
by bothra90 on 11/10/22, 5:58 AM
by amelius on 11/9/22, 9:44 PM
E.g. instead of training it to generate sentences given a prompt, train it to generate URLs given a query (?)
by totorovirus on 11/10/22, 8:56 AM
by narrator on 11/10/22, 2:01 AM
IMHO, Yandex feels like the old Google. It returns lots of small sites for most searches and does way better on controversial topics.
by guggalugalug on 11/10/22, 1:06 AM
unless advancement is understood as necessarily attendant upon linear time unspooling, it is no ways inevitable
by CalChris on 11/9/22, 10:16 PM
by gumby on 11/9/22, 10:31 PM
Instead they are going the other way. Sad.
by fredgrott on 11/10/22, 11:39 AM
Given Some social media platforms are moving towards some embrace of AI feeds, could those platforms retain their 2nd search engines role such as Youtube, Instagram, etc?
My own biased use case is that I started following a whole bunch of designers on Instagram to get new design news.
by Beltiras on 11/9/22, 10:36 PM
by jll29 on 11/9/22, 9:55 PM
> Yet, despite being acutely aware of the dangers of ad-supported search, selling ads was adopted as the primary business model of the new search venture just a few years later.
Google was successful as a superior search engine, pushing out AltaVista and similar, earlier alternative search engines and Yahoo!-style portals curated by humans. However, Google wasn't a successful business for quite some time. Eventually, they adopted (some people would say "stole" - there was a lawsuit) Yahoo!-owned Overture's ad model, which changed everything. Yahoo! owned Overture, and Overture had a critical patent. Yahoo! made one critical mistake: they settled the lawsuit for relatively little money. The rest is history.
Now many people complain about decaying search result quality levels. That just means there is space for a new search engine, how exciting! The good news is it has never been easier and cheaper to start a full-text index of the Web and associated search. For about 50k (a Xoogler's estimate, not mine) you should be able to get going. Sites like Gigablast show it can even be done as a one-man show, which I would not recommend (to many complexities in "small" bits even HTML to plan text conversion, load balancing, incremental inverted index updating etc. - all requiring nowadays some specialist expertise in a game where you can't afford to reinvent the wheel because you don't know the scientific literature/state of the art). The one thing that is hard to get is initial user traffic. But I think HNers will be happy to give each new engine a try!
In summary, I think there never was an "age of PageRank". But you may say Google Web search is past its prime. Perhaps Google could change that if they wanted - it may be that it isn't much of a priority at the moment, hard to say (they are (too?) big now).
Edit: Here, I've interviewed Shadi Saleh, the architect of Syria's search engine (if you think it's impossible to get up and running with a small team): https://irsg.bcs.org/informer/2019/07/syrias-first-web-searc...
by p3rls on 11/10/22, 12:58 AM
https://kagi.com/faq#Where-are-your-results-coming-from
"Our searching includes anonymized requests to traditional search indexes like Google and Bing as well"
Lol okay
by ETH_start on 11/10/22, 3:46 AM
by KaoruAoiShiho on 11/9/22, 11:01 PM
Check the history section: https://en.wikipedia.org/wiki/PageRank
by MomoXenosaga on 11/9/22, 11:07 PM
So what destroyed the relative purity of search engines and the early web? Money. How do we solve it? Money apparently.
Yeah good luck with that.
by ac130kz on 11/10/22, 3:42 AM
by kerblang on 11/9/22, 9:49 PM
by s3000 on 11/9/22, 9:50 PM
by lowbloodsugar on 11/10/22, 12:55 AM
What an optimistic perspective.
by 838405050 on 11/10/22, 8:50 PM
I’ll stick with DuckDuckGo, thank you.
by WaitWaitWha on 11/9/22, 10:12 PM
I am genuinely curious how did Kagi arrive at this cost.
by Minor49er on 11/9/22, 10:31 PM
by normalocity on 11/10/22, 3:25 AM
by nharada on 11/9/22, 10:47 PM
by andrewmcwatters on 11/9/22, 9:54 PM
Why would you not take the chance to completely reimage search into something way better than the absolute random crap we have today?
Whatever the next web discovery engine looks like, it'll look and feel entirely different.
It won't be the equivalent of someone saying, "Hey Facebook is bad, look at MY social network!" and it being the exact same layout and user experience as Facebook, with different margins, padding, fonts and colors.
Whatever comes next, it'll be like comparing Twitter to Facebook, or TikTok to Instagram.
As the kids used to say, this ain't it, chief.
by ispo on 11/9/22, 9:45 PM
by infimum on 11/10/22, 8:34 AM
by orangesite on 11/10/22, 5:39 AM
How about personalized pagerank search that assigns any URL I've bookmarked a high value for E?
Is there a reason no one has done this yet?
by ballenf on 11/9/22, 9:40 PM
Then you can check google or a mainstream engine when needed.
by jaimex2 on 11/9/22, 11:26 PM
by vagab0nd on 11/12/22, 4:41 AM
by fnordpiglet on 11/10/22, 4:36 AM
by strix_varius on 11/9/22, 10:16 PM
by aerovistae on 11/9/22, 9:35 PM
by metadaemon on 11/10/22, 5:05 AM
by pnemonic on 11/9/22, 9:50 PM
EDIT: I'm dumb. Please do not reply or I will Sylvia Plath myself.
by soared on 11/9/22, 11:03 PM
You can’t be serious with a claim like this
by beckingz on 11/9/22, 9:26 PM
Time to join a web ring.
by a-dub on 11/9/22, 9:48 PM
by harryvederci on 11/9/22, 10:21 PM
Otherwise I might have given it a shot.
by sytelus on 11/9/22, 9:35 PM