Google's new look at robots.txt: from bases to complex uses

Google has published a new documentation that explains how Robots.txt allows publishers and SEO specialists to manage search work and other boots that comply with Robots.txt. Documentation contains examples of locking specific pages (for example, shopping carts), limiting some bots and managing the scan behavior using simple rules.

The new documentation offers a quick introduction to what is robots.txt, and is gradually moving to an increasingly difficult description of what publishers and SEO specialists can do with robots.txt and how it uses them.

🚀 The main purpose of the first part of the document is to present robots.txt as a stable web protocol with a 30-year history, which is widely supported by search engines and other works. Google Search Console will report 404 if the robots.txt file is missing. This is normal, but if you bother you, you can wait 30 days and the warning will disappear. An alternative is to create an empty robots.txt file, which is also acceptable for Google.

Google explains: "You can leave your Robots.txt file blank (or do not have it at all) if your entire site can be scanned or you can add rules to control the scan process."

💡 From here it covers the basics, for example, its own rules to limit specific pages or sections.

• 📌 Robots.txt can target specific work with different rules.
• 📌 allows you to block URL templates, such as PDF files or search pages.
• 📌 allows you to control specific bots.
• 📌 Supports comments for internal documentation.