Reddit has updated its robots.txt to block all web crawlers
Reddit, one of the internet’s most popular social platforms, has updated its robots.txt file to block all web crawlers. This change builds on the company’s ongoing efforts to protect user privacy and control the misuse of its content. Announced in a blog post on May 9, 2024, Reddit introduced its new Public Content Policy to clarify its stance regarding access to public content while emphasizing user protections.
This new policy, distinct from Reddit’s existing Privacy and Content Policies, aims to address the increase in unauthorized data collection and misuse on its platform. Reddit stated, “We need to do more to restrict access to Reddit public content at scale to trusted actors who have agreed to abide by our policies.”
The latest update to the robots.txt file is part of a series of backend changes Reddit is implementing to enforce these policy changes. These changes ensure that access to Reddit content is limited to those who comply with its guidelines.
