Skip to main content
Question

Cloudflare blocks AI crawlers accessing content without permission

  • July 11, 2025
  • 5 replies
  • 78 views
bjoern_schulze
Forum|alt.badge.img+3

Hi Gainsight team,

I’m referring to this article here:

https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/

Cloudflare, Inc. [...] today announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. Starting today, website owners can choose if they want AI crawlers to access their content, and decide how AI companies can use it. AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow.

As Gainsight is using Cloudflare, this is highly relevant to all Gainsight platforms. 

Can you give us some information about how Gainsight has set up this new configuration? In addition to that, can you give us some insights about how Gainsight will ensure its platforms to support (or even boost) AI crawlers to process the content?

See also:

Thanks! :)

Kenneth R
DannyPancratz
Saymon

5 replies

revathimenon
Forum|alt.badge.img+8
  • revathimenon
  • Gainsight Community Manager
  • July 16, 2025

Hey ​@bjoern_schulze 

I’ve raised this with the team internally and will loop back soon.

 

Where focus goes, energy flows.
aaronhatton
A
Saymon
  • Saymon
  • Contributor ⭐️⭐️
  • July 18, 2025

Interested in this too! 

Kenneth R
Forum|alt.badge.img+5
  • Kenneth R
  • Expert ⭐️⭐️
  • July 21, 2025

Hi ​@bjoern_schulze - We use Cloudfront.  I don’t believe we use Cloudflare.  

For the broader question of handling AI crawlers, in our platform it’s currently possible to block the main AI crawlers via the robots.txt file if desired.  Of course, as is the case with other web crawlers, any crawler designed specifically to ignore those instructions will do so, but robots.txt should cover most.  

And while don’t have any news right now around handling llms.txt files (we’re looking into it), the way we structure content for SEO (added rich schema markup) already helps AI models to understand our content.

DannyPancratz
A
A
  • Ambar D
  • Contributor ⭐️⭐️
  • August 7, 2025

Hi ​@bjoern_schulze, I’m interested in this topic too, thanks for adding your question here!

​@Kenneth R would you mind elaborating on your last point about schema markup helping AI models to understand our content? 

Kenneth R
Forum|alt.badge.img+5
  • Kenneth R
  • Expert ⭐️⭐️
  • August 11, 2025

​@Ambar D - I don’t know all the technical detail but what we basically mean is that the community pages are structured in a way that helps both search engine crawlers and AI models understand and make sense of the contents of the page.

Terms & ConditionsCookie settingsAccessibility statement