Does your site work with AI agents?
Agents now visit sites on behalf of real customers, and many sites turn them away by accident. This corner shows how each agent identifies itself, how to let it in or keep it out, and a free check you can run on your own pages.
Checked October 20, 2026. Run the readiness check
Agents are already at your door, and they behave like customers with a robot's stamina. Muse, dots, Grok Bot and others browse, fill carts and book on behalf of a person who never loads your page. Sites have answered in three ways. Amazon and Yelp block, Walmart and Shopify partner, and Zillow allows consumer agents while its bot defenses still catch some. Decide with two questions. Does the agent bring you sales or only scrape you? Can you tell it apart from a bot? If it buys and identifies itself, allow it. If it identifies itself but you want limits, verify its signature or IP list, then rate-limit. If it hides, block on behavior. One trap: OpenAI, Perplexity, Google and Meta all say robots.txt may not bind fetches a person triggered, so a robots.txt line is a request, not a lock. Anthropic says its bots do follow it.
The readiness checklist
Ten things that decide whether an agent can finish a task on your site.
- No CAPTCHA on browse pagesKeep product, menu and pricing pages free of challenges. Anthropic says its bots will not try to get past CAPTCHAs, and Walmart blamed agent failures on a human-verification step.
- Labeled formsGive every input a visible label and a matching for attribute. Agents fill forms by reading labels, and so do screen readers.
- Key facts in plain HTMLPut prices, stock, hours and policies in the served HTML, not only in JavaScript that loads later. Fetchers that do not run scripts will miss them.
- robots.txt groups per agentWrite separate groups for Claude-User, GrokAgent, GPTBot, OAI-SearchBot and PerplexityBot so each choice is deliberate. Remember user-triggered fetchers from OpenAI, Perplexity, Google and Meta may not obey it.
- Verify signed agentsTurn on Web Bot Auth checks or Cloudflare verified-bot rules. OpenAI signs ChatGPT cloud browser requests, Google is testing signatures, and Visa's Trusted Agent Protocol uses the same approach.
- Check IP lists before trusting a user agentAnyone can fake a user-agent string. Compare the source against the lists OpenAI, Anthropic, Perplexity and Google publish.
- Rate limits instead of blanket blocksCap requests per session or account. Resy locked an account doing hundreds of bookings an hour, and a limit catches that without turning away every agent.
- Structured dataAdd schema.org markup for products, offers, events and opening hours so an agent can read them without parsing your layout.
- A clear agent policy pageSay which agents you allow and on what terms, as eBay and Yelp have. It gives makers and users something to point to.
- llms.txt, if you want itAn llms.txt file can summarize your site for language models. It is optional, it is not a formal standard, and our sources show no agent maker committing to read it.