42crawl Documentation
Welcome to the technical documentation hub for 42crawl. This guide is designed for SEO professionals, developers, and marketers who want to optimize their web presence for both traditional search engines and the new era of Generative Engine Optimization (GEO).
42crawl provides the tools to audit your site's health, analyze internal link structures, and ensure your content is fully accessible to AI models like ChatGPT, Claude, and Google Gemini.
🚀 Getting Started
Build a solid foundation with 42crawl. Learn how to configure your first project and navigate the core dashboard.
- Getting Started with 42crawl — Core workflow for your first audit.
- Project Structure — Understanding how data is organized within the platform.
🤖 Generative Engine Optimization (GEO)
Optimize for the AI era. Ensure your content is cited by LLMs and remains visible in AI search results.
- GEO Strategy — Introduction to Generative Engine Optimization.
- GEO Scoring Factors — Technical breakdown of how we calculate AI readiness.
- AI Discovery Files — Managing
llms.txtandai.txtfor crawler control. - AI Bot Access Test — Verifying that AI agents can reach your content.
- AI-Powered SEO Workflow — Using AI to accelerate your remediation.
- Jules AI Integration — Pushing technical fixes directly to your development environment.
🏗️ Technical SEO Audit
Deep-dive into the technical foundations of your website. Identify and fix infrastructure issues that impact search visibility.
- Technical SEO Overview — Comprehensive auditing of titles, meta, and tags.
- Crawlability & Sitemaps — Analyzing
robots.txtand XML sitemaps. - Broken Link Analysis — Detecting 404s and optimizing crawl budget.
- Redirect & Chain Analysis — Cleaning up legacy redirect paths.
- Structured Data & Schema — Validating JSON-LD for rich snippets and AI understanding.
- IndexNow Integration — Instant indexing for Bing and Yandex.
- Mobile SEO Optimization — Ensuring performance and usability on mobile devices.
- JavaScript & SPA SEO — Auditing modern frameworks like React and Next.js.
- Security Headers — Improving site security and trust signals.
- Site Performance — Core Web Vitals (LCP, CLS, INP) analysis at scale.
📝 Content & Performance
Beyond the code—optimizing the quality, readability, and accessibility of your content.
- Content Quality Analysis — Readability, information density, and keyword themes.
- Accessibility Audits — WCAG 2.1 compliance for inclusive search visibility.
- International SEO — Hreflang validation and multi-regional targeting.
- Anchor Text Analysis — Optimizing internal link relevancy.
- Keyword Cannibalization — Identifying and resolving intent overlap.
- Keyword Tracker — Monitoring SERP positions across multiple engines.
- Blog Management — Best practices for managing your content assets.
⚙️ Advanced Workflow & Reporting
Streamline your operations with automated crawls, visual analysis, and team integrations.
- Internal Link Graph — Visualizing site architecture and PageRank flow.
- Internal Link Analysis — Identifying orphan pages and link equity gaps.
- Internal Link Recommendations — Actionable suggestions to improve PageRank distribution.
- Crawler Configuration — Customizing user-agents, speed, and depth.
- Crawl Comparison — Tracking SEO progress over time.
- Scheduled Crawls — Automated monitoring and health alerts.
- SEO Recommendations — Impact-scored task management for your team.
- White-Label Client Audits — Generating professional PDF reports for clients.
- Project Management — Exporting tasks to Trello, Jira, and Notion.
- Integrations & Data Export — API access and Google Sheets connectivity.
❓ Technical FAQ
How does 42crawl handle JavaScript-heavy websites?
We use a headless browser to render JavaScript before auditing, ensuring we see what users and search engines see. You can learn more in our JavaScript SEO guide.
What is the difference between SEO and GEO?
SEO (Search Engine Optimization) focuses on ranking in traditional search engines like Google. GEO (Generative Engine Optimization) focuses on being cited and used as a source by AI models like ChatGPT.
Can I automate my audits?
Yes. You can schedule crawls on a daily, weekly, or monthly basis to monitor your site's health automatically.
How do I export my data?
42crawl supports exports to CSV, Google Sheets, and professional white-label PDF reports for client delivery. See the integrations guide for details.
Need more help?
If you can't find what you're looking for, our support team is ready to assist.
Contact Support