---
title: "Two kinds of AI crawler"
date: 2026-07-13
description: "The split between training-style crawlers and user-side fetchers changes how I read a traffic spike."
---

Something small worth writing down after a morning of staring at crawler traffic.

Most of the "AI bot traffic" numbers I've seen collapse everything into one bucket. The split I've found more useful:

- **Training-style crawlers** — `ClaudeBot`, `GPTBot`, `PerplexityBot`. Sweep pages in bulk. Return almost no referral traffic downstream.
- **User-side fetchers** — `Claude-User`, `ChatGPT-User`, `OAI-SearchBot`. Fire when a person asks their assistant about your site. Correlate with brand visibility inside AI answers.

OpenAI actually makes this distinction explicit in their [crawler overview](https://developers.openai.com/api/docs/bots) — GPTBot is for training, ChatGPT-User is user-initiated, OAI-SearchBot surfaces sites inside ChatGPT search. Same shape holds for Anthropic (ClaudeBot vs Claude-User).

Same origin, very different signal. When the training crawler jumps five-fold week-over-week but the user-side fetcher stays flat, that's vendor-initiated crawling — a re-crawl, an index refresh, expanded coverage. Not more people asking about you. Reading it as "we're getting popular in AI answers" would be wrong.

The other thing I've been watching: 404s to [`.well-known`](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml) paths like `agent-card.json` and `openid-configuration`, and paths that look like [MCP](https://modelcontextprotocol.io/specification) endpoints. Agents are probing for surfaces that don't exist yet. Whether we should serve them is a separate question — but at least knowing they're being probed is useful.

Might be wrong. Curious how this evolves as answer engines mature.