Is Blocking AI Crawlers in robots.txt Protecting Your Shopify Store or Hiding It?
Somewhere in your store's history, someone may have pasted a list of AI crawler names into robots.txt.liquid. It might have been a developer after a news story about training data, or an agency checklist from a theme migration. If nobody wrote down why, that file is making a decision on your behalf every day.
The short answer is that blocking a training crawler and blocking a search crawler are two different decisions, and many sites make the second one without meaning to. This article gives you a way to read what your file says today, decide bot by bot, and record the choice so the next person does not have to guess.
What does your robots.txt allow today?
Open yourstore.com/robots.txt in a browser and read it top to bottom. Look for named AI user agents, a "User-agent: *" group followed by "Disallow: /", and any rule touching policies, products or collections. Then check your CDN or firewall settings, because robots.txt cannot show you a block enforced there.
On a default Shopify setup the file is usually open to AI crawlers. A write-up by Craftshift (May 6, 2026) says the default rules cover paths such as admin, cart, checkout and search results, and do not block GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Claude-SearchBot. A Shopify community audit posted on October 20, 2026 points the same way: 2 of 98 stores blocked any of 14 AI crawlers, none used a wildcard block, and 0 of 52 stores fetched as GPTBot were blocked at the CDN or firewall level.
Treat that audit as a small, informal sample. Its author notes that the stores had installed an AI visibility app, so they are probably more attentive to this than average. The honest reading is that stores with a default file are usually open, and the stores that are closed tend to have been edited by someone.
Two more checks take a minute each. Load your returns and shipping policy pages in a private window (on Shopify they sit under /policies/) to confirm they open without a login or an error. Then run a free GEO check that reads your public pages, so you see what an outside reader sees rather than what your theme editor shows.
Training bot, search bot or user-triggered fetcher: what does each do?
A training crawler collects pages that may be used to build future models. A search crawler indexes pages so an assistant can surface them in answers. A user-triggered fetcher loads one page because a person asked about it. Each has a different consequence when blocked, and the third may not follow robots.txt at all.
| Role | OpenAI | Anthropic | Perplexity |
|---|---|---|---|
| Training | GPTBot | ClaudeBot | None. PerplexityBot does not collect training data. |
| Search and citation | OAI-SearchBot | Claude-SearchBot | PerplexityBot |
| Fetch on a person's request | ChatGPT-User | Claude-User | Perplexity-User |
The vendors describe the split in their own documentation. OpenAI says disallowing GPTBot signals that a site's content should not be used for training, that each bot's setting is independent, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links. For ChatGPT-User, OpenAI says robots.txt rules may not apply, because a person initiated the visit.
Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot as honoring robots.txt. It says blocking Claude-SearchBot stops indexing for search and may reduce visibility in user search results. For Perplexity, Anagram's summary of its documentation says PerplexityBot supports search results, while Perplexity-User fetches single pages on request and generally ignores robots.txt.
How many sites block one bot and forget the other?
A study by aicrawlercheck.com fetched robots.txt files from the Tranco top 10,000 domains on October 5, 2026. Of 4,962 parseable files, 710 blocked GPTBot, and 347 of those (48.9%) still allowed OAI-SearchBot. For Anthropic, 307 of 665 sites blocking ClaudeBot (46.2%) still allowed Claude-SearchBot.
The same study compared Google-Extended with Googlebot and found 477 of 597 blockers (79.9%) leaving Googlebot open. It notes that only 11 files block Googlebot deliberately, so that figure is high largely by default. The study covers large sites rather than Shopify stores, and it reads robots.txt only, not firewalls, CDN rules or meta directives. It is a snapshot of stated intent, not observed crawler behavior.
What can you edit on Shopify, and what can't you control?
You do not upload a robots.txt file to Shopify. You add a theme template named robots.txt.liquid, which replaces Shopify's generated output, and you edit the rules there. You cannot control crawlers that ignore robots.txt, rules enforced by a CDN or firewall in front of the store, or what a vendor does with pages it already holds.
Shopify's documentation recommends starting from the Liquid robots object rather than plain text. The steps are short:
- Save a copy of your current /robots.txt, so you can compare it with the new output.
- In your admin, go to Online Store, then Themes, then Edit code. Add a new file named robots.txt.liquid in the Templates folder.
- Keep the loop over robots.default_groups that Shopify's example uses, and add your own groups after it.
- Reload /robots.txt and compare it line by line with your saved copy. Shopify notes that the rendered output does not always match the generated file exactly.
Which edits do the damage?
The mistakes that show up in write-ups are consistent. Craftshift lists a wildcard group that blocks everyone except Googlebot and so catches AI bots too, a Disallow on /products/ or /collections/, and a template that was never rechecked after a theme migration. Other guides add copying a block-all list that includes search bots, and deleting the default Liquid loop. The opposite slip also happens: blocking ClaudeBot but missing Claude-SearchBot and Claude-User, so the rule does less than its author thinks.
When is blocking the right call?
Blocking is a legitimate choice when you object to your content training future models, have licensing or brand-policy reasons, or see crawl load you cannot absorb. The cost sits on the retrieval side: OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Decide each bot on its own merits.
Take an illustrative home goods brand whose 2023 theme work left a copied list of fourteen AI user agents in robots.txt.liquid. Its photography is licensed, so the owner wants GPTBot and ClaudeBot blocked. Nobody on the team wants to be absent from assistant answers, though, and the same list also blocks OAI-SearchBot, Claude-SearchBot and PerplexityBot. The owner keeps the two training blocks and removes the three search entries. OpenAI says its systems can take about 24 hours to reflect a robots.txt change for search.
That change makes the pages eligible. It does not earn a mention. In the community audit above, 65 of 94 stores with at least 10 checks were never named in an unbranded buying question, and the author's conclusion was that crawler access explains almost none of that gap. The audit's visibility checks leaned toward Gemini and were generated per store category, so a different prompt set would give different numbers. Open doors are necessary for being cited and are not sufficient.
If crawl load is the worry rather than principle, Anthropic says it supports the non-standard Crawl-delay directive, for example "Crawl-delay: 1" under a ClaudeBot group. Anthropic also advises against blocking by IP address, because that stops the crawler from reading your robots.txt in the first place.
How should you record the decision, and when should you review it?
Write the decision where the next person will find it: which bots are allowed or blocked, the reason for each, who approved it, the date and the review date. A comment inside robots.txt.liquid keeps the reason beside the rule. Review it at least twice a year and after any theme change or migration.
| Field | What to write |
|---|---|
| Rule | Each user agent and whether it is allowed or blocked |
| Reason | Content rights, brand policy, crawl load, or visibility in assistant answers |
| Approved by | One named person with authority over the brand's content |
| Date | The day the template change went live |
| Review trigger | A calendar date, plus any theme replacement or migration |
| Check | A saved copy of /robots.txt before and after, and a note of the firewall or CDN settings at the time |
The approver matters more than it looks. A rule that blocks search crawlers changes whether your returns and shipping pages can be read by the assistants shoppers ask, so it belongs to someone who owns the brand's reputation, not only to whoever has theme access.
FAQ
Does Shopify's default robots.txt block AI crawlers?
No. Third-party write-ups and a 98-store community audit report that the default rules disallow paths such as admin, cart, checkout and search results and leave the major AI crawlers alone. Check your own file anyway, because a theme edit or migration may have replaced the default.
If I block GPTBot, will ChatGPT stop mentioning my store?
OpenAI documents each bot's setting as independent. GPTBot relates to training use, while OAI-SearchBot governs whether a site is shown in ChatGPT search answers. Blocking GPTBot alone does not opt you out of search, and blocking OAI-SearchBot does.
Is robots.txt enough to keep a page away from AI assistants?
No. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity-User generally ignores them, according to Anagram's reading of Perplexity's documentation. If a page must not be fetched, enforce that at the CDN or firewall, or do not publish it.
How long does a robots.txt change take to work?
OpenAI says it can take about 24 hours for its search systems to reflect an update. I did not find a stated timeline from Anthropic or Perplexity, so reload the file to confirm it renders as intended and recheck in a few days.
