A robots.txt file is a plain text file sitting in a website’s root directory that tells search engine crawlers which parts of the site they’re welcome to visit and which parts to leave alone. It’s one of the oldest pieces of technical SEO still in daily use, dating back to a voluntary standard web crawlers agreed to respect long before search engines looked anything like they do now.
The file itself is short, usually just a handful of lines, and every major crawler checks for it automatically before crawling anything else on the domain.
How Robots.txt Directives Work
A robots.txt file is built from a small set of directives, each one doing a specific job. User-agent names which crawler a rule applies to, whether that’s Googlebot specifically or an asterisk standing in for every bot at once. Disallow blocks a crawler from a folder or a page, while Allow carves out an exception inside a directory that’s otherwise off-limits, useful when one file inside a blocked folder still needs to stay visible. A Sitemap line, separate from the blocking rules, simply points crawlers toward the site’s XML sitemap so they know where to look for the full list of pages.
Put together, a few lines of these directives can steer crawlers away from admin panels, duplicate filtered product views, internal search result pages, or staging subdomains, the kind of content that adds nothing for a searcher and just eats into a site’s crawl budget.
Robots.txt and Crawl Budget
For most small sites, a search engine can crawl every page without breaking a sweat, so robots.txt does little heavy lifting. Large sites are a different story. An e-commerce catalogue generating thousands of filtered URL combinations, or a forum spinning out endless tag pages, can burn through a search engine’s patience for crawling that one domain before it ever reaches the pages actually worth ranking.
That’s the real job robots.txt does well: steering limited crawl attention toward what matters. What it doesn’t do is guarantee removal from search results. A disallowed page can still appear in search listings if other sites link to it, just without a proper snippet describing what’s on it, since the crawler was never allowed in to read the content. A noindex tag or genuine password protection is what actually keeps something out of Google, and the same protection matters for anything truly sensitive. A robots.txt rule alone won’t do either job. Blocking a folder in a public, plain text file advertises its location rather than hiding it; anyone can open the file and see exactly where to look.