
RFC 9309 Explained: The Robots.txt Standard Every SEO Should Actually Read
For nearly 30 years, robots.txt was just a gentleman’s agreement. Martijn Koster proposed it in 1994, everyone adopted it, but no standards body ever formally defined it. There was no official spec for how crawlers should parse the file, what happens when the server returns a 500, or how wildcard matching works. In September 2022, […]




