Is Blocking AI Crawlers in robots.txt Protecting Your Shopify Store or Hiding It?

Is Blocking AI Crawlers in robots.txt Protecting Your Shopify Store or Hiding It? Somewhere in your store's history, someone may have pasted a list of AI crawler names into robots.txt.liquid. It might have been a developer after a news story about training data, or an agency checklist from a theme migration. If nobody wrote down why, that file is making a decision on your behalf every day. The short answer is that blocking a training crawler and blocking a search crawler are two different decisions, and many sites make the second one without meaning to. This article gives you a way to read what your file says today, decide bot by bot, and record the choice so the next person does not have to guess. What does your robots.txt allow today? Open yourstore.com/robots.txt in a browser and read it top to bottom. Look for named AI user agents, a "User-agent: *" group followed by "Disallow: /", and any rule touching policies, products or collections. Then check your CDN or firewall settings, because robots.txt cannot show you a block enforced there. On a default Shopify setup the file is usually open to AI crawlers. A write-up by Craftshift (May 6, 2026) says the default rules cover paths such as admin, cart, checkout and search results, and do not block GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Claude-SearchBot. A Shopify community audit posted on October 20, 2026 points the same way: 2 of 98 stores blocked any of 14 AI crawlers, none used a wildcard block, and 0 of 52 stores fetched as GPTBot were blocked at the CDN or firewall level. Treat that audit as a small, informal sample. Its author notes that the stores had installed an AI visibility app, so they are probably more attentive to this than average. The honest reading is that stores with a default file are usually open, and the stores that are closed tend to have been edited by someone. Two more checks take a minute each. Load your returns and shipping policy pages in a private window (on Shopify they sit under /policies/) to confirm they open without a login or an error. Then run a free GEO check that reads your public pages, so you see what an outside reader sees rather than what your theme editor shows. Training bot, search bot or user-triggered fetcher: what does each do? A training crawler collects pages that may be used to build future models. A search crawler indexes pages so an assistant can surface them in answers. A user-triggered fetcher loads one page because a person asked about it. Each has a different consequence when blocked, and the third may not follow robots.txt at all. Role OpenAI Anthropic Perplexity Training GPTBot ClaudeBot None. PerplexityBot does not collect training data. Search and citation OAI-SearchBot Claude-SearchBot PerplexityBot Fetch on a person's request ChatGPT-User Claude-User Perplexity-User The vendors describe the split in their own documentation. OpenAI says disallowing GPTBot signals that a site's content should not be used for training, that each bot's setting is independent, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links. For ChatGPT-User, OpenAI says robots.txt rules may not apply, because a person initiated the visit. Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot as honoring robots.txt. It says blocking Claude-SearchBot stops indexing for search and may reduce visibility in user search results. For Perplexity, Anagram's summary of its documentation says PerplexityBot supports search results, while Perplexity-User fetches single pages on request and generally ignores robots.txt. How many sites block one bot and forget the other? A study by aicrawlercheck.com fetched robots.txt files from the Tranco top 10,000 domains on October 5, 2026. Of 4,962 parseable files, 710 blocked GPTBot, and 347 of those (48.9%) still allowed OAI-SearchBot. For Anthropic, 307 of 665 sites blocking ClaudeBot (46.2%) still allowed Claude-SearchBot. The same study compared Google-Extended with Googlebot and found 477 of 597 blockers (79.9%) leaving Googlebot open. It notes that only 11 files block Googlebot deliberately, so that figure is high largely by default. The study covers large sites rather than Shopify stores, and it reads robots.txt only, not firewalls, CDN rules or meta directives. It is a snapshot of stated intent, not observed crawler behavior. What can you edit on Shopify, and what can't you control? You do not upload a robots.txt file to Shopify. You add a theme template named robots.txt.liquid, which replaces Shopify's generated output, and you edit the rules there. You cannot control crawlers that ignore robots.txt, rules enforced by a CDN or firewall in front of the store, or what a vendor does with pages it already holds. Shopify's documentation recommends starting from the Liquid robots object ra

Is Blocking AI Crawlers in robots.txt Protecting Your Shopify Store or Hiding It?

相关文章

llms.txt, agents.md or Neither: What Shopify Already Publishes for You and What Is Worth Editing

From Draft to Live: Following One Product Page Change Through a Managed GEO Engagement

Will It Arrive by Christmas? How to Write the Shipping and Returns Answers AI Assistants Give Your Shoppers