Crawl Traps from URL Parameters: How to Find and Fix Them

Short answer: A parameter crawl trap is a part of your site that can generate new URLs forever: filters, sort orders, calendars, session IDs. Crawlers follow the links and keep finding "new" pages that show the same products or posts. To fix it, decide which parameter URLs deserve to exist in search. Block the rest from crawling (robots.txt is Google's first recommendation), keep the ones you want clean and consistent, and return a real 404 for empty or nonsensical combinations. Then re-crawl and prove the URL count dropped.

What a crawl trap is, in plain terms

A crawler can't tell whether a URL is useful until it fetches it. A filtered category page like this one looks as new to Googlebot as any other page:

https://example.com/shoes?color=red&size=9&sort=price-asc

Now add four colors, six sizes, three sort orders and a brand filter. Each filter multiplies the others, so one category can turn into thousands of URLs. Add a "next month" link on a calendar and the URL space has no end.

Google's documentation describes two costs. The first is overcrawling: crawlers typically fetch a very large number of faceted URLs before deciding they're useless. The second is slower discovery: crawl time spent on useless URLs is time not spent on new, useful ones.

Note where the trap comes from. Googlebot doesn't invent these URLs. Your own pages link to them.

Does it matter for your site?

Be honest about size.

• Small and medium sites: Google's crawl budget guide says it is an advanced guide for very large sites (1 million+ unique pages that change weekly, or 10,000+ pages that change daily) and for sites where a large share of URLs sit in "Discovered - currently not indexed". It also says that if your pages are crawled the day they're published, you don't need the guide. For you, the main cost is usually noise: duplicates, a cluttered index and an audit that nobody can read.
• Large shops and publishers: the cost is real. The same guide says that without guidance Google tries to crawl all or most of the URLs it knows about, and that "perceived inventory" is the factor you can influence most.
• Parameters aren't always bad. Google also says its crawlers have become better at guessing which parameters are useful, and it retired the Search Console URL Parameters tool in 2022 for that reason. Don't count on that guessing for an unbounded URL space, though. Google's own advice for that case is robots.txt.

Google doesn't document a penalty for having many parameter URLs, and we won't invent one.

The usual suspects

How to find it

Count what your site really links to

You need a list of URLs: a crawl export, your sitemap, or Googlebot hits from server logs. The first step is simply to count parameters. This one-liner lists the parameter names that appear most often:

grep -o '[?&][^=&#]*=' urls.txt | tr -d '?&=' | sort | uniq -c | sort -rn | head -20

If one name (sort, color, sid) appears in thousands of URLs, you've found a candidate. Then check which paths have the most variants, and whether the same filters appear in a different order:

# trap_report.py  ->  python3 trap_report.py urls.txt
import sys
from collections import Counter, defaultdict
from urllib.parse import urlsplit, parse_qsl

names = Counter()
variants = defaultdict(set)   # path -> URLs seen
same_filters = defaultdict(set)  # (path, sorted params) -> URLs seen

for line in open(sys.argv[1]):
    url = line.strip()
    if not url:
        continue
    u = urlsplit(url)
    pairs = parse_qsl(u.query, keep_blank_values=True)
    for key, _ in pairs:
        names[key] += 1
    variants[u.path].add(url)
    same_filters[(u.path, tuple(sorted(pairs)))].add(url)

print("Most common parameters:")
for key, n in names.most_common(10):
    print(f"{n:>8}  {key}")

print("\nPaths with the most URL variants:")
for path, urls in sorted(variants.items(), key=lambda x: -len(x[1]))[:10]:
    print(f"{len(urls):>8}  {path}")

print("\nSame filters, more than one URL (order or repeats):")
dupes = [(k, v) for k, v in same_filters.items() if len(v) > 1]
for (path, _), urls in sorted(dupes, key=lambda x: -len(x[1]))[:10]:
    print(f"{len(urls):>8}  {path}  e.g. {sorted(urls)[0]}")

It's a screen, not a verdict. It tells you where to look; you still decide whether a parameter is a trap or a page you want to rank.

In Search Console

Look at two things. In the Page indexing report, a large share of URLs in "Discovered - currently not indexed" is one of the signs Google's crawl budget guide lists for sites that need to care. In the Crawl stats report, compare how many requests Googlebot makes per day with how many real pages you have. If requests are many times your page count, find out where they go. Both reports show what Google saw, not the current state of your site.

In server logs

Logs are the only place where you see what Googlebot actually requested. Filter by the Googlebot user agent (verify the IP, because the user agent can be faked), extract the URLs, and feed them to the script above. This answers "is Google really crawling these?" instead of "could it?".

With a crawler

A crawler follows your internal links the way a bot would. If parameter URLs fill the sample, that is evidence, and the crawl shows which page linked to each one. SiteOptima covers parameters and crawl traps among its checks, keeps the evidence for each URL, and groups related findings into likely causes. One limit to know: an audit covers up to 50 pages on Starter and up to 100 on Pro and Agency. A capped crawl can't list an endless URL space. It shows the pattern in what it reached, and your logs or sitemap give you the full count.

Common causes

1. Filters rendered as plain links. Each checkbox is an <a href> to a new URL.
2. No fixed parameter order. Links use whatever order the visitor clicked.
3. Filters that append instead of toggle. Click "blue" twice and the URL gets longer.
4. Empty combinations that return 200. Green fish, size tiny, none in stock, and the page still loads with "no results".
5. Sort, view and tracking parameters in internal links. Tracking belongs on external links, not on your own navigation.
6. Calendars and pagination without an end. The "next" link exists even when there's nothing after it.
7. Session IDs in the URL.

How to fix it

Start by deciding. For every parameter, answer one question: should this URL be able to show up in search?

Block what you don't need crawled

Google's faceted navigation guide recommends robots.txt to disallow these URLs, allowing the individual product pages and one listing page without filters. Its example looks like this:

user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$

Be careful: /*?*color= blocks every URL whose query string contains color=. Before you deploy, take a list of URLs that must stay crawlable and test them against the new rules with a parser that understands Google's wildcard syntax (for example Google's open-source robots.txt parser). If your real category pages use a parameter, a broad rule will hide them.

Two details from Google's documents:

• Prefer robots.txt to noindex for crawl waste. The crawl budget guide says noindex still costs a request, because Google fetches the page and only then drops it. Blocked URLs, on the other hand, stay in the crawl queue longer than 404 pages and are recrawled when the block is lifted.
• Already-indexed parameter URLs need a different order. If Google can't crawl a URL because robots.txt blocks it, it can't see a noindex tag on it. Our reading of Google's noindex documentation: if junk URLs are already in the index, let Google crawl them and see noindex or a 404 first, then add the robots.txt block once they've dropped out. That's our interpretation, not a step Google spells out.

Or use fragments

Filters placed after a # don't change what Google crawls. Google says Search generally doesn't support URL fragments in crawling and indexing, so a filter based on fragments has no impact on crawling, positive or negative:

https://example.com/items#products=fish&color=green

This is a development task. It also means filtered views can't rank, so it only suits filters you don't want indexed anyway.

If you want some filter pages indexed

Google's guidance for faceted URLs you want crawled:

• Use the standard & separator. Commas, semicolons and brackets are hard for crawlers to recognize as separators.
• If filters live in the path (/products/fish/green/tiny), keep their order fixed and don't allow duplicate filters.
• Return a real 404 when a combination has no results. Do the same for duplicate filters, nonsensical combinations and pagination that doesn't exist. Don't redirect them to a shared error page; serve the 404 at the URL that was requested. (Single-page apps can't always do this. Google has separate guidance for them.)

Fixing the order at the source is the cheap win. Generate every link in one order, and if you want to be thorough, 301 the other order to it. That redirect is a common approach, not a Google requirement. Keep it to one hop, as described in How to Fix Redirect Chains.

What about canonical and nofollow?

Google describes both as less effective in the long term than blocking crawling. A rel="canonical" from a filtered URL to the unfiltered one may, over time, reduce crawling of the filtered versions. rel="nofollow" helps only if every anchor pointing to that URL carries it. If you already use canonicals on filtered pages, check that each one points where you mean it to.

Clean up the leftovers

Remove parameter URLs from your sitemap (see Sitemap Contains Redirected or noindex URLs), strip tracking parameters from internal links, and give calendars and pagination an end.

How to check it's fixed

This is the step most guides skip. You made rules; now prove they do what you meant.

1. Re-count. Get a fresh list of URLs (new crawl, new sitemap, last week of logs) and run the same commands. Expected result: the top parameter counts fall, "URL variants per path" shrinks, and the "same filters, more than one URL" section is empty or short.
2. Test the traps directly. Request a few sample trap URLs. Expected: empty combinations return 404, and the second parameter order either redirects once or no longer appears in internal links. ``bash curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" "https://example.com/shoes?color=green&size=tiny" `` curl doesn't read robots.txt, so check blocked patterns with your robots.txt parser instead.
3. Test the pages you want to keep. Your real category and product URLs must still return 200, be their own canonical, carry no noindex and sit in the sitemap. A blocking rule that's too broad is the most common way this fix goes wrong.
4. Audit again and compare. Run a new audit and compare it with the one you made before the change: which findings are resolved, which are still open, and whether new ones appeared. SiteOptima is built around this loop. You change the site outside the tool, audit again and see what moved. The result covers the pages the crawl reached, up to the plan limit.
5. Check Search Console later. Crawl stats and the Page indexing report change on Google's schedule, not yours. We can't say how long it takes, and fewer wasted requests don't guarantee more traffic or better rankings. Google makes no such promise.

Checklist

• List of URLs collected (crawl, sitemap, logs)
• Parameters counted; each one marked "indexable" or "not needed in search"
• Filter, sort, session and tracking parameters removed from internal links where possible
• robots.txt rules written for parameters that don't need crawling
• Rules tested against URLs that must stay crawlable
• Indexed junk URLs given a 404 or noindex before being blocked
• Filters you do keep use &, one fixed order and no duplicates
• Empty and nonsensical combinations return 404
• Calendars and pagination end
• Parameter URLs removed from the sitemap
• Re-crawled, counts compared, and the site re-audited

FAQ

Is a crawl trap the same as crawl budget waste? They're related. A trap is the cause (an unlimited supply of URLs) and wasted crawling is the effect. Google's crawl budget guide says that for most sites, keeping the sitemap current and checking the Page indexing report is enough.

Should I use canonical tags or robots.txt for faceted navigation? Google lists robots.txt first when you don't need the URLs in search, and says canonical and nofollow are generally less effective in the long term. Use canonicals as a supporting signal, not as the main control.

Can I set parameters in Search Console? No. Google retired the URL Parameters tool in 2022 and says its crawlers learn how to deal with parameters automatically. For more control it points to robots.txt.

Will blocking parameter URLs make Google crawl my other pages more? Not necessarily. Google says it won't shift the freed-up crawl budget to other pages unless it's already hitting your site's crawl capacity limit.

Does SiteOptima fix the trap? No. SiteOptima finds the pattern, shows the evidence, suggests likely causes and verifies after you change the site. The change itself is made by you, your developer or your agency.

Sources

Checked on 29 October 2026.

• Google Crawling infrastructure, Managing crawling of faceted navigation URLs (last updated 2025-12-18): overcrawling and slower discovery, robots.txt example, URL fragments, canonical and nofollow, separators, 404 for empty combinations.
• Google Crawling infrastructure, Optimize your crawl budget (last updated 2026-07-22): who the guide is for, perceived inventory, robots.txt vs noindex, 404 vs blocked URLs, redirect chains, freed crawl budget.
• Google Search Central, Block Search indexing with noindex: why a robots.txt block hides the noindex rule.
• Google Search Central Blog, Spring cleaning: the URL Parameters tool (28 March 2022): the tool's deprecation, automatic handling of parameters, robots.txt for more control.