# robots.txt — www.anatomyit.com # # POLICY # This site is written to be read by people AND by AI agents. Everything # published here is public marketing content, so nothing is disallowed. # # The AI assistant / AI search crawlers below are named and allowed EXPLICITLY, # on purpose. Several of them treat an absent or ambiguous directive as a reason # to skip a site, and being quotable in AI answers is a stated goal of this # build — so each one gets its own group rather than relying on `User-agent: *`. # # Google-Extended and Applebot-Extended are not crawlers: they are control # tokens that govern whether already-crawled content may be used for Gemini / # Apple Intelligence grounding and training. `Allow: /` on them is a deliberate # opt-in, not a formality. Revisit with the client if that stance ever changes. # # Build-input directories that ship with the deploy (/content/, /tools/) are # kept out of search indexes with `X-Robots-Tag: noindex` in `_headers`, not # with a Disallow line here — a Disallow would advertise paths it cannot # actually protect. Do not add them below. # # Machine-readable index of the site: /llms.txt # Markdown twin of any page: same URL with `Accept: text/markdown`, # or add `.md` to the URL # API catalog (RFC 9727): /.well-known/api-catalog # API description (OpenAPI): /openapi.json # --- Content Signals (https://contentsignals.org) --- # A `Content-Signal` line states, per group, what a crawler may do with the # content it is allowed to fetch. Three signals are defined: # search - build a search index and show results (links, short excerpts) # ai-input - feed the content into an AI model at query time (grounding, # retrieval-augmented answers, AI search summaries) # ai-train - train or fine-tune an AI model # `yes` grants, `no` withholds; an absent signal says nothing either way. # # CURRENT STANCE: all three are `yes`. This mirrors the policy above - every AI # crawler and both opt-in tokens are explicitly allowed, and being quotable in # AI answers is a stated goal of the build. The line is repeated in every group # below because a crawler that matches its own named group reads only that # group and never sees the `*` group. Keep the lines identical; if the stance # changes, change all of them in the same edit. # `ai-train=yes` is the one signal that goes beyond search visibility (it # permits model training on the site's copy); `search` and `ai-input` follow # directly from the policy. Anatomy IT confirmed it in November 2026. # --- Default: every well-behaved crawler --- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Search engines --- User-agent: Googlebot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- OpenAI --- # GPTBot: training/index crawler. ChatGPT-User: fetches a page a user asked # about. OAI-SearchBot: builds the ChatGPT search index. User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Anthropic --- # ClaudeBot: index crawler. Claude-User: fetches a page for a user's request. # Claude-SearchBot: builds the Claude search index. User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Claude-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Perplexity --- # PerplexityBot: index crawler. Perplexity-User: fetches a page for a user's # question. User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Google AI (Gemini grounding / AI Overviews) — opt-in token --- User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Apple Intelligence — opt-in token --- User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Other AI assistants and AI search --- # Meta AI, Amazon (Alexa and Rufus), DuckDuckGo's DuckAssist and Mistral's # Le Chat. Same stance as every group above. User-agent: Meta-ExternalAgent Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # --- Common Crawl (the corpus many other models are built from) --- User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Sitemap: https://www.anatomyit.com/sitemap.xml Sitemap: https://www.anatomyit.com/sitemap-image.xml # ARD (Agentic Resource Discovery) capability manifest — see /.well-known/ai-catalog.json Agentmap: https://www.anatomyit.com/.well-known/ai-catalog.json