robots.txt
Definition
robots.txt is a plain text file at a site's root that tells crawlers which paths they may or may not fetch. It controls crawling, not indexing: a blocked page can still appear in results if other sites link to it, so a noindex tag is needed to keep a page out. Stores use it to keep cart, checkout, account, and internal search pages out of the crawl budget.
Why it matters for your store
Crawlers have a budget per site. Spending it on ten thousand filter combinations, cart pages, and search result pages means product pages get crawled less often. robots.txt is the first tool for pointing the budget at what matters.
It is also a common source of accidents. A single Disallow line copied from a staging site can hide an entire store from search. The reverse mistake, leaving a test site open, gets the test address indexed and competing with the real one. Both are avoidable when the file is generated from the store's state rather than edited by hand.
The file is now where sites declare their position on AI crawlers, too.
How Kambloo handles it
robots.txt is generated per store and per hostname, not stored as a file. Test stores on their kambloo.com address always get a full Disallow, plus a noindex header and meta tag on every response, and this cannot be turned off by the store. When the custom domain is verified, the live hostname gets an open policy automatically: everything allowed except cart, checkout, account, search, and admin paths, with the sitemap location included. You can add your own rules in the SEO settings.
The main AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) are allowed by default on live stores, because a store that wants to be cited must let them in. Each store has a switch to deny them.
Questions about robots.txt
Does blocking a page in robots.txt remove it from Google?
No. It stops Google from fetching the page, but the URL can still be indexed from links elsewhere. To keep a page out of results, allow crawling and add a noindex tag, or use both a noindex header and the block.
Should I block AI crawlers in robots.txt?
Only if you do not want your products cited in AI answers. Kambloo allows the main AI crawlers by default on live stores and gives each store a switch to block them.