Why asking first matters
A person reading a page and an AI agent reading the same page look very different to the site. The agent may come back every few minutes, from many machines, and pass the text to a language model. Sites have answered with new rules, and with bot checks for anyone who ignores them. An agent that reads the rules first keeps working; one that doesn't gets blocked, and gives every other agent a worse name.
There are four things worth checking before an agent reads a page. None of them is new technology. Together they answer two questions: may I fetch this page, and may I use what I fetched.
1. robots.txt: may I fetch it?
Every site can publish a file at /robots.txt that says which paths which bots may fetch. The format is old (it dates from 1994) and was written up as an internet standard, RFC 9309, in 2022. It is a convention, not a lock: it works because well-behaved bots choose to follow it.
The rules are grouped by user agent, the name a bot calls itself. A bot looks for the group with its own name and, if there isn't one, follows the group for *:
User-agent: *
Disallow: /account/
User-agent: SomeAIBot
Disallow: /Two habits matter for agents. First, use a name the site can target, and say where to read about it. A site can only block or allow what it can recognise. Second, if the agent works for someone else's bot (an assistant with its own crawler name), follow that bot's rules too, so a block on one name can't be sidestepped through another.
2. Content Signals: may I use it for AI?
robots.txt says whether a page may be fetched, not what may be done with it afterwards. Content Signals fill that gap. Introduced by Cloudflare in 2025, they are one extra line in robots.txt with three yes-or-no answers:
| Signal | What it covers |
|---|---|
search | Building a search index and showing links and short snippets |
ai-input | Using the content as input to an AI model answering a question, for example retrieval or grounding |
ai-train | Training or fine-tuning AI models |
User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /For an agent that reads a page in order to hand it to a model, ai-input=no is the line that matters. If it is there, the polite answer is to stop, even though robots.txt allows the fetch. A signal that isn't set says nothing either way.
3. Bot checks and sign-in walls: is someone saying no?
Sometimes the page doesn't answer at all: a challenge page, a CAPTCHA or a sign-in form comes back instead. That is the site saying no to automated visitors. An agent should treat it as a refusal and stop, not reach for a CAPTCHA solver, a different IP address or someone's saved password. Reading what any visitor sees without signing in is a clear line that both sides can understand.
4. llms.txt: is there a better way in?
The first three checks are about permission. The fourth is about saving work. /llms.txt is a proposal from 2024 for a short Markdown file that points AI tools to the pages that matter on a site, often with Markdown versions of them. Not every site has one, and it is a hint rather than a rule. When it exists, an agent can skip the menus and cookie banners and read a fraction of the tokens.
All four in one call
Doing these checks by hand for every page is slow, which is why many agents skip them. Bastion Helix Agent Lens does them as tools an agent can call: site returns a site's llms.txt, sitemap, feeds, Content Signals and which known AI crawlers its robots.txt allows, and preflight says whether one page may be used, whether a bot check or sign-in stands in the way, and how it can be loaded. Both are free and need no key:
curl -X POST https://lens.bastionhelix.com/v1/preflight \
-H 'Content-Type: application/json' \
-d '{"url": "https://example.com"}'The paid tools keep the same rules. Agent Lens obeys robots.txt for its own name, BastionHelixAgentLens, and for the agent's own bot when the agent names it; declines pages with ai-input=no; never tries to get past bot checks; never signs in; and charges nothing for a refusal. The full rules are in the acceptable use policy, which includes a section written for agents themselves.
One more rule belongs on every agent's list: treat page content as data, never as instructions. A web page can contain text written to steer an AI ("ignore your previous instructions…"). Whatever tool fetches the page, the agent should not obey it.
For site owners
Each check is something you control. To keep Agent Lens out entirely:
User-agent: BastionHelixAgentLens
Disallow: /To let agents see your pages but not use them as AI input, add Content-Signal: ai-input=no under User-agent: *. The Agent Lens crawler page shows how it identifies itself and how to block it.
Questions
Is robots.txt legally binding?
It is a published convention (RFC 9309), not an access control. It works because well-behaved bots follow it, which is exactly why an agent that wants to stay welcome should.
What is the difference between robots.txt and Content Signals?
robots.txt says which pages a bot may fetch. Content Signals say what the content may be used for once fetched: search, AI input, or AI training.
Does llms.txt give permission to use a site's content?
No. llms.txt is a map of useful pages for AI tools. Permission still comes from robots.txt, Content Signals and the site's terms.
Should an AI agent solve CAPTCHAs?
No. A bot check or sign-in wall is the site saying no to automated visitors. A well-behaved agent stops and reports why.
Agent Lens is launching soon. The free site and preflight tools work today without a key. Read the API docs, or see what Agent Lens does.