A product page published in Shopify is not automatically accessible to every crawler. robots.txt, server responses and storefront restrictions affect access. For AI crawlers, first understand each agent’s purpose before changing your rules.
Separate crawling from indexing
robots.txt gives crawling instructions to compliant crawlers. It is not a confidentiality control: a blocked URL can sometimes be discovered from other links. Protect private data with appropriate authentication, not robots.txt alone.
Google also notes that an accessible, technically valid page may still not be indexed. Check availability, indexable content, internal links and noindex directives alongside robots.txt.
- Check that pages intended for discovery are public and return a successful response.
- Review robots rules and robots meta tags in your Shopify templates.
- Use Search Console crawl and indexing reports for Google pages.
OAI-SearchBot and GPTBot serve different purposes
OpenAI documents OAI-SearchBot for discovering sites in ChatGPT search results. GPTBot relates to crawling content that may be used to train models. Their directives are independent: allowing one does not require allowing the other.
Choose based on your preferences for discovery and reuse, then apply rules to the correct agent. Do not copy a generic rule without checking that it matches your publishing policy and intended paths.
Make a measured change
Before changing rules, save the current configuration and identify the exact path involved. After publishing, check the robots.txt file served on the storefront domain and review server logs or provider tools where available.
Allowing a crawler does not promise indexing or citation. At most, it removes an access obstacle; quality, relevance, availability and system policies still matter.
Sources and references
Frequently asked questions
Do I need to allow GPTBot to appear in ChatGPT search?
OpenAI documentation distinguishes OAI-SearchBot, used for search, from GPTBot, associated with training. Check the official documentation and choose each rule independently.
Is robots.txt enough to protect a private page?
No. robots.txt gives crawlers instructions; it does not protect access. Use authentication and server-side controls for private content.