Free tool
Generate clean, validated robots.txt directives tailored for Shopify, WooCommerce, and custom stores to optimize search crawler access.
Unknown calculator.
The robots.txt file is a plain text document placed in your web server's root directory that provides crawl directives to automated search engine bots (like Googlebot and Bingbot). For e-commerce stores with extensive catalogs, robots.txt prevents search bots from wasting crawl budget on checkout endpoints, dynamic cart sessions, user account dashboards, and internal search results.
A frequent architectural mistake is using robots.txt Disallow in an attempt to de-index a page. If external websites link to a disallowed URL, Google may still index the URL without snippet content. To completely remove thin pages from search results, allow bots to crawl the page and serve a 'noindex' robots meta tag instead.
Shopify automatically manages robots.txt.liquid, disallowing /cart, /checkout, /orders, and internal search queries (/search). WooCommerce stores on Nginx or Apache must ensure wp-admin, wp-login, and checkout endpoints are disallowed while ensuring wp-content/uploads and static theme CSS/JS remain fully accessible for rendering.
The robots.txt file must reside in the exact root directory of your website domain (e.g., https://example.com/robots.txt). Search engine crawlers will not recognize robots.txt files placed in subdirectories.
Never block CSS, JavaScript, or image directories in robots.txt. Googlebot must render your pages just like a human browser to evaluate mobile friendliness, visual layout, and dynamically injected content.
Yes, always include a full absolute Sitemap directive at the bottom of your robots.txt file (e.g., 'Sitemap: https://example.com/sitemap.xml') to accelerate crawler discovery of your indexable catalog.
User-agent: * applies rules globally to all compliant web crawlers. You can create specialized override blocks for specific crawlers, such as Googlebot, Bingbot, or AI scraping bots like GPTBot.
Format, validate, and structure bulk catalog URL lists into clean XML sitemap formats with priority tags and lastmod timestamps.
Generate clean, SEO-friendly URL slugs from article titles and product names, stripping stop words, symbols, and formatting errors.
Simulate desktop and mobile Google search engine result snippets including title tags, meta descriptions, and clean URL breadcrumb displays.
A practical map of e-commerce SEO: information architecture, keyword research, product and category pages, technical indexation, schema, Core Web Vitals and measurement.
SEO for Shopify stores: themes, collections, products, apps, duplicate URLs, speed and a high-level look at international — including constraints you cannot theme away.
SEO for WooCommerce: permalinks, plugins, hosting and performance, indexation control and schema — conceptually, without treating any paid plugin as required.
Technical SEO for stores is duplicates, canonicals, robots, sitemaps, JavaScript rendering and parameter URLs. Fix the catalog graph before you rewrite copy.
Shopify SEO is theme structure, collections, duplicate URLs and apps. Online Store 2.0 helps, but canonicals, robots and thin collection copy still matter.
Product, Offer, BreadcrumbList and visible FAQ markup help machines parse the page. Never fake reviews or ratings. Schema follows what shoppers can see.
Stores need collection–product links, breadcrumbs and a path to orphan SKUs. Related-product widgets are merchandising; SEO links need crawlable HTML.
