Technical SEO
Robots Policy
A robots policy is the set of crawl and indexation rules that controls which parts of a website bots may crawl, index or ignore.
What it means
A robots policy usually combines Robots.txt rules, robots meta tags, X-Robots-Tag headers, canonical tags and sitemap inclusion rules. It should separate useful public pages from utility URLs, private pages, duplicate filters and technical paths.
Why it matters
Robots policy matters because inconsistent crawl and indexation rules create waste and confusion. A clean policy helps search engines focus on pages that deserve visibility while keeping low-value or sensitive URLs out of search.
Example
A WordPress ecommerce site may allow product and category pages, Noindex cart and checkout pages, block selected internal search paths and keep canonical product URLs inside the sitemap.
Common mistakes
Common mistakes include using Robots.txt as a privacy tool, blocking pages that need Noindex evaluation, contradicting canonical and noindex signals or leaving staging URLs crawlable.
Written by Marius Dosinescu. Reviewed by AYSA SEO Editorial Team.
SEO execution campaign
Less SEO work. More organic growth.
AYSA monitors your website, finds opportunities, prepares the work, asks for approval and executes accepted changes so you can grow without living in SEO tools.
