Google released some news about a new token that can be used to block their Bard and Vertex AI crawlers.
Google-Extended | A standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products. |
Time to edit my robots.txt file again.
(See Here’s how to block OpenAI’s bot crawlers in your robots.txt file for why I’m blocking them.)
Block Google’s AI bot
Straight from the source: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
User-agent: Google-Extended
Disallow: /That’s the important bit. It’s not even an example on the page, but at least the user-agent info is.
Happy times.
I don’t mind opting-in to things I consider helpful to the world at large. But this opting-out business is ridiculous. Businesses take intellectual property seriously when it’s other people trying to benefit from their property. But when they want to benefit commercially from other people’s property, they have no problem skipping the permission phase and hoping no one cares later.