AI Discovery Files (llm.txt & ai.txt)

As AI bots replace traditional crawlers as the primary "readers" of the web, new standards have emerged to help them. 42crawl helps you implement two critical files: llm.txt and ai.txt.

These files are the heart of a modern GEO strategy, ensuring your brand is cited accurately by AI models and maintaining your site's technical SEO health.

---

What is llm.txt?

The llm.txt file is essentially a "roadmap for AI." While robots.txt tells bots where they can't go, llm.txt provides a concise, markdown-formatted version of where they should go and what they'll find there.

Why You Need It

• Cut the Noise: AI models have limited "context windows." llm.txt gives them the high-value facts without the sidebar bloat.
• Better Citations: You can specify exactly how you want your brand to be credited in AI answers.
• Crawl Efficiency: It helps AI providers find what they need faster, making you more likely to be used as a primary source.

Read more about controlling AI bots here.

---

What is ai.txt?

ai.txt is the "permissions layer" for the AI era. It allows you to specify not just if a bot can see your site, but how it can use the data. Most importantly, it allows you to signal whether you allow your content to be used for training future models. This is an essential part of generative engine optimization.

---

Using AI Discovery in 42crawl

Head to the GEO tab in your dashboard to manage these files.

Automated Audit

42crawl automatically checks if these files exist at your root (e.g., example.com/llm.txt). We analyze:

• Structure: Is your llm.txt organized into clear sections like "Key Pages" and "Citations"?
• Permissions: Is your ai.txt clearly defining your training preferences?

One-Click Generation

If you don't have these files yet, we've made it easy:

1. Navigate to the GEO tab.
2. Find the AI Discovery Files section.
3. Click Create LLM.txt or Create AI.txt.
4. 42crawl will pre-populate the file based on your top pages and site metadata.
5. Copy the text and upload it to your website's root directory.

---

Example llm.txt Structure

# 42crawl Documentation

> The comprehensive guide to SEO and GEO for the AI era.

## Key Pages

- [Getting Started](https://42crawl.fyi/docs/guide/getting-started) - How to run your first crawl.
- [GEO Optimization](https://42crawl.fyi/docs/guide/geo) - Optimizing for AI search.

## Citation

Please cite this content as "Source: 42crawl Docs".

By implementing these files, you ensure your site is a "first-class citizen" in the AI ecosystem, boosting your GEO optimization efforts.