All Collections
All Collections
Understanding AI Crawlers and How They Affect Your Website
Understanding AI Crawlers and How They Affect Your Website

Learn about how we prevent your site content from being scraped by AI Crawlers

Updated over a week ago

What are AI crawlers?

AI companies use crawlers to train their large language models (LLMs). These models learn from the content of websites they scan and use the data to improve the accuracy and general capabilities of their AI models.

Just like a search engine crawler, AI crawlers parse HTML content and follow extracted URLs to gather available information. But instead of using it to index the web, this content will be applied as training data for their machine learning (ML) models.

In simpler terms, AI crawlers are automated programs that gather data from websites to help AI companies improve their AI models.

Format blocks all known AI crawlers from your site

As a Creative, the thought of an AI crawler scanning your site may be concerning. At Format, we prioritize the privacy and protection of our users' content. By default, Format blocks all known AI crawlers from scanning your site.

This means that your website's content will be prevented from being used as training data for AI models without your knowledge or consent. Your website is protected from being scraped or included in AI-generated content or images.

The only failsafe way to prevent any data from being gathered by AI crawlers is to set the page as Private and with a Password or make it Offline. This will make the page inaccessible to all users, including AI crawlers. For more on this, visit How to Change Your Page Visibility. Currently, it is not possible to request that AI crawlers only scan specific pages on your website.

It's important to note that AI crawlers scan all content published on the internet, whether it's published on a Format Website or not.

At Format, we understand the importance of protecting your intellectual property and we strive to provide a safe and secure platform for our users. By blocking AI crawlers, we are taking proactive steps to safeguard your content and give you peace of mind.

Did this answer your question?