Plans to support llms.tx?

Hi ​@mbuuck1 we’ve did some more research on this topic and want to provide you all with an update: 

While a new standard called llms.txt has been proposed, it has not been widely adopted by major AI model providers such as OpenAI, Anthropic, and Google. Current discussions in the industry suggest that llms.txt is of limited practical use at this time, as these providers do not actively support or use it (source).

For now, the best practice remains to use robots.txt and follow established SEO guidelines for structuring community content. It’s worth noting that LLM-powered AI systems do not “crawl and index” content in the same way as search engines. Instead, they respond to user prompts based on their training data and available context. That said, some AI agents may not respect robots.txt at all. As a general rule, if content is publicly accessible on the web, it should be considered accessible to AI systems. Because of the reasons mentioned above we don’t see an immediate need to support llms.txt yet - but we do keep an eye on this. When llms.txt mature and become more widely accepted and adopted we’ll realign on its priority.