Getting Started with 42crawl
Welcome to 42crawl—the modern SEO crawler built for the AI era. This guide will help you run your first audit, understand your data, and start optimizing your site for both humans and AI bots.
What Makes 42crawl Different?
Most SEO tools were built for the web of 2010. 42crawl is built for 2026. We combine traditional technical SEO (like finding 404s) with Generative Engine Optimization (GEO)—ensuring your content is ready for AI engines like ChatGPT and Google AI Overviews.
Core Capabilities
- Deep SEO Audit: Find broken links, redirect chains, and metadata errors.
- Visual Link Graph: See your site architecture in real-time. Explore the Link Graph.
- Performance Metrics: Monitor Core Web Vitals (LCP, CLS, INP) at scale.
- GEO Readiness: Validate structured data and AI Discovery files.
- Accessibility: Ensure compliance with WCAG 2.1 standards. Detailed accessibility guide.
Your First Crawl: A Step-by-Step Guide
Step 1: Enter Your URL
On the dashboard, enter your website's full URL (e.g., https://yourwebsite.com).
Step 2: Configure Your Settings
Click the "Advanced" toggle to customize the audit:
- Crawl Depth: 1-3 is great for a quick check; 5 is better for a deep dive.
- Max Pages: Start with 100-200 to get a representative sample.
- User Agent: Use Googlebot to see exactly what the search engine sees.
Step 3: Start the Audit
Click "Start Crawl." You'll see real-time progress as 42crawl discovers and analyzes your pages.
- Small sites: < 2 minutes.
- Large sites: 10-20 minutes.
Step 4: Review the Dashboard
Once finished, you'll see your Health Score and a breakdown of issues:
- 🏗️ Technical: Issues with indexing, tags, and status codes.
- 🔗 Links: Broken links and authority flow.
- ⚡ Performance: Speed and stability metrics (Core Web Vitals).
- 🤖 GEO: AI readiness and accessibility.
Deep Diving Into Your Data
The "All Issues" View
Click into any category to see a prioritized list of fixes. We don't just show you errors; we tell you how to fix them, improving your technical SEO immediately.
Page-Level Analysis
Click any URL in your results to see a "Nutrition Label" for that specific page—including its heading structure, image alt text, and internal link count.
Exporting & Sharing
Need to send a report to a developer or a client? Use the Export button to generate a branded PDF or a Google Sheet.
Next Steps: Moving from Audit to Action
- Kill the 404s: Start with broken links—they are the biggest drain on your crawl budget.
- Fix H1s: Ensure every page has one clear topic. Read our content guide.
- Optimize Images: Shrink those heavy files to boost your LCP score.
- Set Up a Schedule: Don't let your SEO decay. Set up automated weekly crawls.
- Embrace GEO: Start your generative engine optimization journey today.
Ready to go? Run your first crawl now and see what you've been missing.