from Hacker News

Techdirt has been deleted from Bing and DuckDuckGo [fixed]

by lehi on 7/27/23, 6:37 PM with 190 comments

  • by yegg on 7/27/23, 7:11 PM

    (DuckDuckGo CEO/Founder) Just seeing this and we're looking into this now. This is not intentional.

    Update: Still investigating, but have made some progress -- Determined that on desktop there was a link to Techdirt up continuously via our About module (when you search for "Techdirt"). And now the traditional web link is back up as well (for desktop and mobile): https://duckduckgo.com/?q=techdirt&ia=web.

  • by thewataccount on 7/27/23, 7:04 PM

    I feel like this makes it clear the level of reliance DuckDuckGo has on Bing. I know it's been a bit ambigious the full extent to which they do, and they claim to have their own indexer, but there's no way this is a coincidence. Surely if you have your own indexer you would have seen techdirt before?

    At this moment "techdirt" only returns the wikipedia article, twitter, then mostly unrelated mentions of it. Surely DDG would have seen at least their homepage before?

    > Most of our search result pages feature one or more Instant Answers. To deliver Instant Answers on specific topics, DuckDuckGo leverages many sources, including specialized sources like Sportradar and crowd-sourced sites like Wikipedia. We also maintain our own crawler (DuckDuckBot) and many indexes to support our results. Of course, we have more traditional links and images in our search results too, which we largely source from Bing. Our focus is synthesizing all these sources to create a superior search experience.

    https://duckduckgo.com/duckduckgo-help-pages/results/sources...

  • by kermire on 7/27/23, 7:03 PM

    Overall search quality has been declining on all search engines. Maybe there's too much spam. Saw an entertaining video about it yesterday that echoes how I feel when I google stuff: https://www.youtube.com/watch?v=jrFv1O4dbqY. It's so hard to find content written by humans these days. Seems like only the top sites are being indexed.

  • by pdimitar on 7/27/23, 7:01 PM

    Well, the last few months DDG's search quality started dropping hard for me, some queries that I clearly remember having good results no longer have them today.

    It seems there's stuff going on behind the scenes. DDG got taken over by some vested interest, perhaps. Or Bing doing stuff and DDG never branching out of it and just blindly getting results from it.

    Either way, it's starting to rival Google in uselessness and I'm likely to stop using it fairly soon.

    Kagi seems to work pretty well.

  • by tredre3 on 7/27/23, 7:16 PM

    That can't be right, DDG's CEO has assured me that DDG isn't just a dumb proxy to Bing. DDG uses several sources as well as its own index so it cannot suffer from what is being claimed [1][2][3].

    1. https://news.ycombinator.com/item?id=32360874

    2. https://news.ycombinator.com/item?id=36149682

    3. https://news.ycombinator.com/item?id=31492631

  • by lapcat on 7/27/23, 7:44 PM

  • by flyinghamster on 7/27/23, 8:01 PM

    It's looking like 2023 is the year that search dies. Between the removal of exclusion operators (hey, we saw you were adding -pinterest to your searches, but we want to make sure you get your pinterest!) and de-indexing of sites, it's looking like it's time for the search wheel to start making its third spin.

    Guess I need to start playing more with things like Algolia or Kagi.

  • by VHRanger on 7/27/23, 6:57 PM

    As a long time DDG user, I was looking for a reason to try Kagi out, and this nonsense is finally pushing me over the fence.

    Techdirt has been an upstanding place for journalism forever, and getting censored this way is ridiculous.

  • by photonerd on 7/27/23, 7:33 PM

    Comments summary: lots of people who don’t understand how large search indexes work freaking out or pontificating about their pet favorite alternative. Occasionally both.

    Meanwhile the DDG founder calmly stating it’s being looked into, is definitely a bug, and explains that this should not generally happen… and being ignored.

  • by beej71 on 7/27/23, 7:15 PM

    Some of my stuff was in a Bing black hole for a while, and I did everything I could fix it, following their guidelines, contacting tech support, etc. It was good content, too--educational, no spam, SEO, ads, or tracking. Eventually I just gave up. Fast forward a year or so, and the search results spontaneously started working.

    So it's a black box, and you kind of just hope for the best.

  • by mutant_glofish on 7/27/23, 8:11 PM

    DuckDuckGo has a history of inheriting censorship from Bing [1] and also doing it on its own. [2]

    [1]: https://reclaimthenet.org/microsofts-bing-censors-tank-man

    [2]: https://www.washingtonexaminer.com/news/duckduckgo-slammed-f...

  • by aimor on 7/27/23, 7:02 PM

    It would be neat to see a black list of all the things Bing and Google block from search results. I don't know how to get such a list other than brute force trial and error.

  • by ty_2k on 7/27/23, 7:29 PM

    It would be really nice if DDG had their own webmaster console (like Google or Bing) to help understand why pages might be indexed or dropped. I love DDG as my primary search engine, but it's a black box from the SEO side of the desk, unless I'm missing something obvious.

  • by NayamAmarshe on 7/27/23, 7:40 PM

    Brave Search is doing pretty well.

    https://search.brave.com/search?q=techdirt

  • by oidar on 7/27/23, 8:22 PM

    techdirt is working fine on Kagi. Ever since its launch, I've been predominantly using Kagi, barely resorting to any other search engine. I estimate that only about 5% of my searches involve Google, and that's mainly when I need to utilize Scholar or Books.

    Kagi is definately worth the $10 fee. First, it allows me to block websites that I find irrelevant or annoying in my areas of interest, such as Pinterest or Quora. It also promises an environment free of advertisements and tracking. I particularly appreciate the excellent customer support, where actual human beings respond to your emails, not automated responses. Additionally, it offers 'custom lenses', which are essentially search templates. For instance, using this feature I can refine my search to target only educational sites, filter for PDFs, limit the time to the "last 48 hours", and search for specific subjects, like machine learning, along with my query.

    My only gripe with Kagi is its stringent limit on searches, set at around 10,000 per month. I've bumped against this ceiling a few times. Despite this limitation, the wealth of features and the quality of the search results make Kagi a worthwhile investment for me.

  • by butz on 7/27/23, 7:54 PM

    And that's why we need a diverse ecosystem of search engines. As indexing whole internet is pretty much impossible, maybe several startups could take a slice of it? Some overlap would be useful too.

  • by HWR_14 on 7/27/23, 6:58 PM

    I've noticed DDG dropping a lot of sites over the past month or so.

  • by jdjdjdhhd on 7/27/23, 8:03 PM

    DDG is useless, unless you really like Bing and care a bit about privacy... I'd rather use Brave for search and Yandex for image search.

  • by davb on 7/27/23, 9:25 PM

    Whatever the complaints about DDG, I still commend yegg for engaging here. I can’t imagine the head of Google Search or Bing doing the same.

  • by vancan1ty on 7/27/23, 7:19 PM

    I find that brave search is actually pretty good these days. Has plenty of results from techdirt at least... [1] https://search.brave.com/search?q=site%3Atechdirt.com&source...

  • by MrDresden on 7/27/23, 7:58 PM

    This is unrelated to the issue at hand, but in case someone from Techdirt comes here then please, allow zoom on your site.

    There is no defensible reason to disable it on a content heavy site like yours.

  • by phreack on 7/27/23, 8:55 PM

    Title says [fixed] but that's only for DDG - which despite all the flaming it's getting, somehow did fix it while Bing has not (will it?). My unfounded theory is that Techdirt is effectively banned on some country and that leaked to the global Bing index.

    https://www.bing.com/search?q=site%3Atechdirt.com

  • by WallyFunk on 7/27/23, 7:07 PM

    Kagi[0] FTW

    [0] https://kagi.com/

  • by KennyBlanken on 7/27/23, 7:44 PM

    I find it simply impossible to believe that Masnick is so incompetent at running a website after so long, that he's finding out from random friends that his site has been de-listed from Bing instead of him or his staff having their search console lighting up like a dashboard and pinging them with emails. I get emails from both bing and google if there are site indexing issues, and while they're not always super clear about why, the major stuff is pretty easy to fix.

    I find it even harder to believe that when he confirmed the problem, he then didn't do anything about it for months.

    This feels incredibly performative.

    Hey Mike: instead of bing chat, try https://www.bing.com/webmaster/tools

  • by philipov on 7/27/23, 9:36 PM

    When I do the search on bing, it doesn't just give no results, it also says "Some results have been removed" with a link to their content policy page.

  • by xnx on 7/27/23, 7:18 PM

    It's ridiculous to talk about DDG being separate from Google. In the same way Brave is Chrome with some minor doodads and settings, DDG is minor reskin of Bing.

  • by maxlin on 7/27/23, 9:45 PM

    After DDG's earlier announcements delivering the image that DDG then severed their uneasy dependency from Bing, this makes them look really bad. Retorting with mumbo jumbo about secondary stuff like flights, business listings etc that are just related to website retention rather than their core structure as an apparent search engine, just makes it worse.

    I still believe DDG has a place, in being a more privacy-focused aggregator to Bing and a few other sources, but this did shatter the previous image of DDG having a large degree of autonomy for me.

  • by thriftwy on 7/27/23, 8:03 PM

    Just checked - Yandex has recent pages from techdirt.com in search results.

  • by riffic on 7/27/23, 7:22 PM

    masnick must have pissed someone off recently lol