Google's Gary Illyes has changed the way robots.txt files are placed

Gary Illyes, a Google analyst, in his message on LinkedIn, challenged the long-standing myth about the need to place the robots.txt file in the root directory of the site (for example, example.com/robots.txt). He explained that this is not a mandatory requirement and revealed a lesser-known feature of the Robot Exclusion Protocol (REP).

"The robots.txt file does not have to be in the root directory (example.com/robots.txt)."

Gary Illyes explains that it is acceptable to have two separate robots.txt files hosted on different domains - one on the main website and one on the content delivery network (CDN). In this way, websites can centralize their robots.txt file on a CDN while still controlling the indexing of their main page. 🚀

• 📌 For example, a website might have two robots.txt files: one at https://cdn.example.com/robots.txt and one at https://www.example.com/robots.txt.
• 📌 This allows them to maintain a single, comprehensive robots.txt file on their CDN and redirect requests from their main domain to this centralized file.
• 📌 Illyes notes that RFC9309-compliant indexers will follow the redirect and use the target file as the robotstxt file for the originating domain.