Thu. Aug 22nd, 2024

Google’s Gary Illyes shares an unconventional but valid method for centralizing robots.txt rules on CDNs.

Robots.txt files can be centralized on CDNs, not just root domains.

Websites can redirect robots.txt from main domain to CDN.

This unorthodox approach complies with updated standards.

Leave a Reply

Your email address will not be published. Required fields are marked *