from Hacker News

3taps says CL blocking "all general search engines"

by sigmadelta on 8/7/12, 5:08 AM with 16 comments

homepage pop-up: "At approximately noon on Sunday August 5th, Craigslist instructed all general search engines to stop indexing CL postings -- effectively blocking 3taps and other 3rd party use of that data from these public domain sources. We are sorry that CL has chosen this course of action and are exploring options to restore service but may be down for an extended period of time unless we or CL change practices. As soon as we know more, we will share it here and on our Twitter account."
  • by storborg on 8/7/12, 9:15 AM

    I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?
  • by true_religion on 8/7/12, 5:38 AM

    Pretty brilliant, I don't think Craiglist ever needed Google traffic at all anymore. People know to go there to buy and sell.
  • by sigmadelta on 8/10/12, 8:14 PM

    http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs...

    "One data harvester, 3taps, said earlier this week that Craigslist had blocked search engines such as Google from including Craigslist pages in search results. But that report was inaccurate.

    3taps’ product and quality assurance leader, Meg Nakamura, acknowledged Wednesday in a chat with The Chronicle that something fishy was taking place, but developers there haven’t fully figured out what’s going on."

  • by sigmadelta on 8/7/12, 7:12 PM

    http://www.sfgate.com/technology/businessinsider/article/Cra...

    Not sure I agree with most the conclusions drawn in that article.

    The article does say that "sure enough, Google displays recent listings from Craigslist right now," which does seem to be true for me, too, when I try.

  • by sigmadelta on 8/10/12, 6:33 AM

    https://twitter.com/markmilian/statuses/233015694432813057

    Mark Milian ‏@markmilian 7 Aug

    Contradicting earlier statement, 3Taps spokeswoman emails to say, "Craigslist is still allowing indexing of pages." Still nothing from CL PR

  • by sigmadelta on 8/7/12, 6:28 AM

    Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
  • by sigmadelta on 8/7/12, 3:59 PM