Shopify
Shopify robots.txt.liquid: edit it safely
Shopify generates robots.txt from a Liquid template you can add. What the default disallows, how to extend it without breaking it, and how to undo.
Shopify generates a default robots.txt and adding a "robots.txt.liquid" template to the theme replaces that output. Extend the default by looping over "robots.default_groups" and adding or skipping rules inside the loop; never hand-write the whole file. Deleting the template restores Shopify's generated file, and checkout, cart and admin Disallows should stay.
Why Shopify does this
Shopify's developer docs: "Shopify generates a default robots.txt file that works for most stores. Adding the robots.txt.liquid template replaces Shopify's generated robots.txt output." The safe pattern they document is to output the defaults and modify inside the loop: "{% for group in robots.default_groups %}", printing "group.user_agent" and each "rule", then adding a rule conditionally, for example "{%- if group.user_agent.value == '*' -%} {{ 'Disallow: /*?q=*' }} {%- endif -%}". Rules can be removed the same way with an unless check, such as skipping the Disallow on "/policies/". The docs also recommend: "Before you publish changes, save a copy of your current /robots.txt output and compare it with the rendered template output." The template supports only the robots, group, rule, user_agent, sitemap and request Liquid objects. Related context: the store access setting, not the template, controls the site-wide Disallow that private mode serves, and Shopify's default file already disallows /checkout, /cart/ and /admin on purpose.
Check it right now
Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.
How to fix it
- Save a copy of the live /robots.txt output today, as Shopify's docs recommend, so you can compare and roll back.
- In the theme code editor, right-click the Templates folder and create a new file named robots.txt.liquid.
- Paste Shopify's documented default loop, which outputs robots.default_groups with each group's user_agent, rules and sitemap.
- Add your change inside the loop: a new Disallow under the star group, an unless block to skip a default rule, or a separate custom group below the loop.
- Save, then open /robots.txt on the store domain and diff it against your saved copy. Only your intended lines should differ.
- Run the robots.txt tester on key URLs, including /checkout and one product URL, to confirm intended blocks and intended allows both hold.
Why it happens again
The template lives in a theme, so a theme update, a new theme, or a copy-paste from an old store can silently drop or duplicate it, and the rendered output changes shape whenever Shopify revises its default groups. The undo is cheap, which is also the risk: deleting the template reverts everything at once, including rules someone added a year ago and forgot about.
stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 30-day free trial, 3 sites, no card required, and a 30 day refund after that.
Catching it next time
Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.
Other ways Shopify loses pages
- A page or product is live in the store but missing from Google, or a whole store is invisible after launch.
- my Shopify robots.txt is blocking pages that should be crawled
- A store page in Google shows the wrong URL as its preferred version, and the canonical tag in the page source points somewhere unexpected.
- my Shopify migration lost pages from search
- A Shopify store reaches its final URL after two hops: one domain redirect from the platform and a second from a manual URL redirect.
- A Shopify storefront shows a connection warning because the automatic TLS certificate is stuck pending or failed.
- A Shopify store launched but Google shows nothing, or robots.txt fills with Disallow rules nobody wrote, and products do not appear in search.
robots.txt blocking pages that should be crawled, on other platforms
How Search Console reports it
Sources
Every claim about Shopify above is from their own documentation, read on 2026-10-25. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.