How to Block Access to Your Website Content

Not every page on your site is meant for search. Staging URLs, admin sections, thank-you pages, internal search results, low-value tag archives, PDFs meant for customers only, content that’s syndicated from third parties — all of it benefits from being kept out of Google (and, in 2026, out of ChatGPT’s and Claude’s training sets). But “blocking” means different things depending on the tool, and picking the wrong one can leak content you thought was private or fail to hide content you thought was blocked.

This guide covers the five methods that actually work — what each one does, when to use it, and the common mistakes that leave content exposed despite best efforts.

Why Block Content at All?

Several legitimate reasons to keep content out of search results:

• Privacy. Member contact info, internal directories, customer order pages — anything that shouldn’t be public. These pages need real access controls, not just SEO directives.
• Duplicate content. If you syndicate third-party feeds, have printer-friendly variants, or run heavy faceted navigation, Google can end up indexing dozens of near-duplicate URLs. Blocking the duplicates keeps the canonical URL clean.
• Thin or low-value pages. Tag archives, author pages, paginated list views, site-search results — these often add no unique content but get indexed and dilute rankings for real pages.
• Pre-launch content. Staging sites and draft pages shouldn’t be in search before launch. Indexed staging URLs are one of the most common SEO leaks.
• AI training. A distinct 2026 concern: keeping your content out of AI crawler datasets used to train LLMs, separate from keeping it out of search indexes.

The Five Main Blocking Methods

Each method does a different thing. Picking the right one starts with understanding what you actually want to stop:

• robots.txt — blocks crawling. Doesn’t reliably block indexing.
• noindex meta tag — blocks indexing in search results. Page can still be crawled.
• X-Robots-Tag HTTP header — noindex for non-HTML files (PDFs, images, videos) or server-level blocking.
• Password protection — blocks access entirely. The only way to truly hide content.
• Search Console Removals — temporarily hides a URL from Google search results while you implement a permanent fix.

For blocking AI crawlers specifically, robots.txt with user-agent-specific directives is the standard approach — covered below.

robots.txt — Control Crawling, Not Indexing

The robots.txt file at the root of your domain tells crawlers which paths they can and can’t visit. It uses two main directives — User-agent and Disallow — plus the optional Allow for overriding specific subpaths:

User-agent: *
Disallow: /private/
Disallow: /admin/
Allow: /private/public-page.html

Save the file as plain text at the root of your domain (https://example.com/robots.txt), submit the URL in Google Search Console’s robots.txt report, and Google will fetch it and respect the directives.

The critical thing to understand: robots.txt blocks crawling, not indexing. If another site links to a URL you’ve blocked in robots.txt, Google can still index it — just without being able to fetch the page content. The result is a search listing with the URL but no description. If your goal is to keep a URL out of search entirely, use noindex instead (or in combination).

Other limitations to know:

• Directives are advisory, not enforced. Googlebot respects them; malicious crawlers often don’t.
• Each crawler interprets syntax independently. Stick to the widely-supported directives.
• Don’t block CSS or JavaScript files — Google needs them to render your pages.
• A typo in robots.txt can silently expose your site or silently block the whole thing. Test carefully.

Search Console includes a robots.txt report (under Settings → Crawling) that shows Google’s current fetched version, flags syntax errors, and lets you preview how a specific URL would be treated.

noindex Meta Tag — Keep URLs Out of Search

The noindex meta tag is the authoritative way to tell Google (and most other search engines) to drop a page from their index. Add it to the <head> of any page you want excluded:

<meta name="robots" content="noindex">

To target Googlebot specifically, use name="googlebot". Other useful robots meta values you can combine:

• noindex — don’t include in search results
• nofollow — don’t follow links on this page (unrelated to the link-level rel="nofollow")
• noarchive — don’t show a cached version in search results
• nosnippet — don’t generate a snippet for this page
• max-snippet:120, max-image-preview:small, max-video-preview:30 — limit how much content Google can show in snippets or AI Overviews

A critical caveat: noindex only works if Google can actually crawl the page and read the tag. If you’ve also blocked the URL in robots.txt, Google can’t fetch the page, can’t see the noindex, and might still index the URL based on external links. The combination “Disallow in robots.txt + noindex in meta” is one of the most common contradictions in SEO and it defaults to the worse outcome for blocking.

Rule: for any page you want out of the index, let Google crawl it (don’t block in robots.txt) and rely on the noindex meta tag.

X-Robots-Tag HTTP Header — noindex for Non-HTML Files

The meta tag only works for HTML pages — it can’t help if you need to noindex a PDF, image, video, or any other binary file. For those, use the X-Robots-Tag HTTP response header instead:

X-Robots-Tag: noindex

You configure this at the server level — typically in your Apache .htaccess, Nginx server block, or via a CDN’s response-header rules. Example for Apache to noindex all PDFs in a directory:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

X-Robots-Tag supports the same directives as the meta tag (noindex, nofollow, noarchive, nosnippet, max-snippet, etc.) but applies at the HTTP layer. It’s also a good choice when you want site-wide indexing control without editing every HTML page.

Password Protection — The Real Lockout

If the content is genuinely private — staging sites, customer-only documents, internal tools — robots.txt and noindex aren’t enough. Anyone who guesses or discovers the URL can still read the content. The only real block is access control.

Options:

• HTTP Basic Authentication via .htaccess and .htpasswd on Apache (see our htaccess password guide), or the Nginx auth_basic directive.
• CMS-level access gating — WordPress’s built-in per-page password, or plugins like Password Protected for entire-site protection.
• Application-level authentication — login walls, paywalls, member-only sections built into the application.
• Cloudflare Access and similar zero-trust services — identity-based gating that requires Google/Microsoft/SSO login. Better than shared passwords for team-accessed content.

Google Search gracefully returns a 401 Unauthorized or 403 Forbidden response on protected URLs and drops them from the index — usually exactly the intended behavior for staging and private content.

Blocking AI Crawlers

New in the last two years: blocking crawlers that collect training data for large language models, separately from blocking search-engine crawlers. Most major AI companies publicly respect robots.txt with their specific user-agent:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

What each one does:

• GPTBot — OpenAI’s training crawler (ChatGPT, GPT models)
• ClaudeBot — Anthropic’s training crawler (Claude models). Anthropic also runs Claude-SearchBot for search retrieval and Claude-User for user-triggered fetches — block those separately if you want granular control
• Google-Extended — Google’s crawler for training Gemini and other AI products (separate from Googlebot, which handles Search)
• CCBot — Common Crawl, whose data is widely used to train open-source models
• PerplexityBot — Perplexity AI’s crawler
• Bytespider — ByteDance’s crawler (TikTok, Doubao)

A key choice: blocking Google-Extended stops your content from training Gemini, but your pages still appear in Google Search results because Googlebot is a separate user-agent. The same applies to ClaudeBot (training) vs. Claude-SearchBot (retrieval). Think about whether you want your site out of AI training data, out of AI search results, or both — and block the corresponding user-agents.

Not every AI crawler honors robots.txt. For hard enforcement you need rate limiting, Cloudflare Bot Management, or a WAF rule.

Search Console Removals — Temporary Hiding

Sometimes a URL needs to disappear from Google search now, while you implement the proper long-term fix (delete the page, add noindex, change the content). The Removals tool in Google Search Console temporarily hides a URL for about six months.

To use it: open Search Console → Removals → New Request → paste the URL → choose “Remove this URL only” or “Remove all URLs with this prefix”. Google hides the URL from search within hours, but the page itself remains online and crawlable. Use this for accidental leaks (a staging URL that got indexed, a draft that shipped too early, outdated content showing wrong information), not as a long-term blocking strategy — the removal expires.

For permanent removal, pair the Removals tool with a real fix: delete the page, add a noindex meta tag, or password-protect it. The Removals tool is the bandage; the meta tag or 410 response is the permanent solution.

Common Mistakes

• Blocking a page in robots.txt AND adding noindex. Google can’t see the noindex if it can’t crawl the page. External links to the URL may keep it in the index anyway. Use one or the other.
• Using robots.txt for sensitive content. Public robots.txt files literally tell anyone which URLs you don’t want crawled — an invitation for bad actors to visit exactly those URLs. For sensitive content, use password protection.
• Blocking CSS/JS in robots.txt. Google renders pages using those resources to evaluate them properly. Blocking them hurts rankings.
• Forgetting noindex on staging sites. The most common SEO leak — a staging site gets linked to or submitted to Google and shows up in search results next to the production site, with debug URLs indexed.
• Thinking canonical tags block indexing. rel="canonical" tells Google which URL to prefer among duplicates; it doesn’t block indexing of the non-canonical version. If you want a URL out of the index, use noindex.
• Assuming every crawler respects robots.txt. Well-behaved crawlers do. Malicious scrapers, some AI systems, and many specialized crawlers don’t. See our guide to ethical scraping for context on what crawlers are expected to do.
• Not re-checking blocked URLs after migration. Site migrations frequently change robots.txt, move pages, or change URL structures. Verify after every migration that intended blocks are still in place and accidental blocks are removed.

Opt-Out from Google Verticals

Beyond the main web index, Google runs several vertical products — Google Business Profile (formerly Google My Business), Google Shopping, Google Hotels, Google Flights, Google Images, and others — that can display your content. Each has its own opt-out path:

• Google Business Profile listings — managed directly in your Business Profile dashboard; you can mark a listing as permanently closed or request suspension for sensitive scenarios.
• Google Shopping — controlled via Google Merchant Center; don’t submit the product feed, or use the feed’s excluded_destination field to keep items out of specific surfaces.
• Google Images — use X-Robots-Tag with noimageindex or max-image-preview:none at the page or server level.
• AI Overviews — blocking Google-Extended in robots.txt stops your content from training Gemini; separately, nosnippet in your meta robots or X-Robots-Tag prevents Google from showing your content in AI-generated snippets and summaries.

Vertical-specific opt-outs usually take effect within days to a few weeks after the next successful crawl.

Frequently Asked Questions

What’s the difference between robots.txt Disallow and noindex?

Disallow in robots.txt tells crawlers not to fetch a URL. Noindex in the meta tag tells search engines not to display a URL in search results even if they can fetch it. For keeping URLs out of search, noindex is the right tool — robots.txt alone leaves URLs eligible to appear in results based on external links.

Does blocking Google-Extended affect my search rankings?

No. Google-Extended is a separate user-agent used for AI training (Gemini and related products). Googlebot, which crawls for Search, is unaffected. You can block Google-Extended to keep your content out of AI training data while continuing to rank in Google Search.

How do I block AI crawlers like ChatGPT and Claude?

Add user-agent-specific Disallow rules to robots.txt for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI training), PerplexityBot, CCBot (Common Crawl), and Bytespider (ByteDance). Most of these companies have publicly committed to respecting robots.txt for their AI crawlers. For crawlers that don’t respect robots.txt, use Cloudflare Bot Management or a WAF rule.

Is password protection enough to hide content from Google?

Yes, as long as Google can’t guess the password. A 401 Unauthorized or 403 Forbidden response drops URLs from the index and keeps them out. That’s why password protection via .htaccess or Cloudflare Access is the gold standard for private content — not SEO directives, which are advisory.

Bottom Line

Pick the method that matches the goal. robots.txt controls crawling; noindex controls indexing; password protection controls access; the Removals tool handles emergencies; AI-crawler user-agents control training data. Don’t combine conflicting directives (robots.txt block + noindex fails); don’t rely on SEO directives for genuinely private content; and don’t leave staging sites indexable. Used correctly, these tools give you precise control over which pages show up in Google Search, which end up in AI training sets, and which stay completely private.