Guiding the crawlers
A robots.txt file tells search-engine crawlers which parts of your site they may fetch. It is the first file a crawler looks for, and a malformed one can accidentally hide your whole site or expose paths you meant to keep quiet. This generator writes a valid file from a few simple choices.
Allow, block, or fine-tune
Open the whole site, block everything while you are building, or list specific paths to keep crawlers out of — admin, cart, checkout — and add your sitemap URL so it is easy to find. Place the result at your domain root. It is all generated on your device.