Edgar Cervantes / Android Authority
TL;DR
- Microsoft has confirmed that its Bing search engine was blocked by Reddit.
- This comes after an investigation found alternative search engines couldn’t display results from the platform.
- Reddit signed a far-reaching deal with Google earlier this year, but the company claimed this move was unrelated to the Google deal.
A recent investigation discovered that alternative search engines are no longer able to display Reddit results. This comes a while after Google and the social news platform signed a wide-ranging deal. Now, Microsoft has confirmed that Bing is indeed blocked by Reddit.
“We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” a Microsoft representative told Search Engine Land.
The robots.txt protocol allows websites to configure which pages a web crawler (e.g. a search engine’s web crawler) can or can’t access. These changes aren’t binding and work on what is effectively an honor system. If a website updates its robot.txt file to request that certain web crawlers and search engines can’t access it, then there’s nothing technically stopping these crawlers from ignoring that request. Microsoft has accepted Reddit’s request in this case, but it recently emerged that a crawler associated with the Perplexity AI search engine ignored requests from a publisher.
Reddit’s monetization push in AI era
Reddit’s request comes just over a month after the company first confirmed plans to update its robots.txt file. It cited an “uptick in obviously commercial entities who scrape Reddit.” Reddit’s change also occurred after signing a deal earlier this year that will also allow Google to use the platform’s user-generated content for training AI.
In a statement to The Verge, a Reddit representative denied that the decision to block alternative search engines was related to the Google deal:
This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.
We seriously wonder whether Reddit would’ve blocked all search engines in the first place if it hadn’t made a deal with Google, though. The deal has also raised concerns about Google’s monopolistic practices, as being the sole home for Reddit search results is a major competitive advantage.
Got a tip? Talk to us! Email our staff at [email protected]. You can stay anonymous or get credit for the info, it's your choice.
English (US) ·