Robots.txt is just a suggestion and we've all been pretending it's a guarantee

The noindex tag thing is almost funny if it wasn’t so bleak. Anthropic told crawlers ‘don’t touch this’ via robots.txt and apparently forgot that crawlers only honor it if you ask nicely and tag the pages right. It’s like locking your car but leaving the window down and hoping nobody notices.

The real problem is that ‘private’ on the web is fragile. It’s a stack of conventions that all have to work together. Break one link and suddenly conversations end up in search results.

source: Private Claude Chats Exposed in Google and Bing Search Results | WIRED

The “share” button is the actual weak link here, not robots. txt. Every one of these “private chats leaked” stories is the same root cause, someone hit share, it generated a public URL, and then it was only a matter of time before it got indexed. robots. txt not being honored is a real problem but it’s downstream of the actual decision to make a “private” conversation reachable via a public link in the first place. kind of reminds me of when Google Docs links would show up in search back in the day if you set sharing to "anyone with the link. " same exact mistake, different decade.