Sitemap Contains Redirected or noindex URLs: How to Fix

Short answer: Your sitemap should list only the URLs you want people to find in Google: pages that return 200, have no noindex, and are their own canonical. If a sitemap URL redirects, returns 404, carries noindex or canonicalizes to another page, the sitemap is contradicting the site. Fix it by changing the sitemap source (or the page, if the page is the mistake), then re-test every URL to confirm.

What this problem is, in plain terms

A sitemap is a list of suggestions. Google's documentation says to include the URLs you want to see in search results, and notes that Google generally shows canonical URLs, which a sitemap can influence. It also says that all pages in a sitemap are suggested as canonicals.

So every entry is a small claim: "this URL is a real page, it is the preferred version, please consider it." Four things break that claim:

• The URL redirects (301, 302, 307, 308). The sitemap names a URL that is not the final page.
• The URL returns 404 or 410. The page is gone.
• The URL has noindex, in a robots meta tag or an X-Robots-Tag header. The page says "do not index me" while the sitemap says "please consider me".
• The URL's canonical points somewhere else. The sitemap suggests one URL as canonical and the page names a different one.

None of these breaks the site for visitors. They make your signals disagree, and you end up with a sitemap nobody can trust, including you.

Does it matter?

Be precise here.

• Google does not document a penalty for any of these. The pages we cite below do not describe one, and we won't invent one.
• Conflicting signals make the outcome harder to predict. Google's canonicalization guide ranks the methods by strength: redirects are a strong signal, rel="canonical" is a strong signal, and sitemap inclusion is a weak signal. When a weak signal disagrees with a strong one, the strong one is more likely to decide. That last step is our reading of the guide, not a Google statement about sitemaps with redirects.
• noindex is honored. Google's documentation says that when Googlebot crawls a page and extracts the tag or header, Google drops that page from search results. If the page was meant to rank, noindex is the bug. If it was not, the sitemap entry is the bug.
• The sitemap is your own inventory. After a migration, a clean sitemap is the quickest way to see what you believe the site contains. A dirty one hides real problems under noise.

How to find them

By hand, in bulk with curl

Pull the URLs out of the sitemap and test each one without following redirects, so a 301 shows up as a 301. These commands use GNU grep (-P). On macOS, install GNU grep or use another extractor. If you have a sitemap index, run step 1 on each child sitemap.

# 1. Extract every <loc> from the sitemap
curl -s https://www.example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' > urls.txt
# 2. Status code and redirect target for each URL (no -L, so redirects stay visible)
while read -r url; do
  curl -s -o /dev/null -w "%{http_code} $url -> %{redirect_url}\n" "$url"
done < urls.txt | grep -v '^200'

Every line it prints is a sitemap URL that does not return 200. A 301 or 308 line shows the destination. A 404 or 410 line is a dead page.

Then check noindex and canonical on the URLs that do return 200:

while read -r url; do
  header=$(curl -sI "$url" | grep -i '^x-robots-tag:.*noindex')
  meta=$(curl -s "$url" | grep -ioE '<meta[^>]*(robots|googlebot)[^>]*>' | grep -i noindex)
  canon=$(curl -s "$url" | grep -ioE '<link[^>]*rel="canonical"[^>]*>' | grep -oE 'href="[^"]+"' | sed 's/href="//;s/"$//')
  [ -n "$header$meta" ] && echo "noindex: $url"
  [ -n "$canon" ] && [ "$canon" != "$url" ] && echo "canonical differs: $url -> $canon"
done < urls.txt

This is a quick screen, not a full parser. It assumes the canonical tag uses double quotes with rel before href, and it reads the raw HTML, so a noindex added by JavaScript will not show up. For those, open DevTools and check the rendered page.

In Search Console

Open the Page indexing report and look at the reasons pages are not indexed. Statuses worth checking against your sitemap include "Page with redirect", "Not found (404)", "Excluded by 'noindex' tag" and "Submitted URL marked 'noindex'". Names change over time, so match them by meaning, not by exact label. Search Console shows what Google saw on its last crawl, not the current state of your site.

With a crawler

A script tests the URLs in one file. A crawler also reads each page's HTML and compares it with what the sitemap lists. SiteOptima reads sitemaps together with response codes, redirects, canonicals and indexability, and keeps the evidence for each URL. One limit to know: an audit covers up to 50 pages on Starter and up to 100 on Pro and Agency. On a big site, the result is only the part that was crawled. Use the curl approach for the full sitemap, and a crawler to explain a sample from each template.

Common causes

1. The sitemap was not updated after a URL change. The redirect was added, the sitemap entry stayed.
2. Static or cached sitemap files. A file generated once, or cached for a long time, keeps deleted and moved URLs.
3. Separate systems. One plugin sets noindex, while a different plugin or script builds the sitemap and does not know about it.
4. Templates. One template (tag pages, filtered listings, thin archives) is set to noindex but still feeds the sitemap.
5. Staging settings that went live. A site-wide noindex left over from development.
6. Canonical rules that disagree. The sitemap lists parameter or variant URLs, while the canonical on each page points to the clean version.
7. Protocol and host mismatches. The sitemap lists http or non-www URLs, and the server redirects them.

How to fix it

Decide what each URL should be, then change the one source that is wrong.

Three details that trip people up:

• Fix the generator, not the file. If a plugin or script builds your sitemap, editing the XML by hand is undone on the next rebuild. Change the setting or the template.
• Do not block a noindex page in robots.txt. Google's documentation says that if robots.txt blocks the page, the crawler never sees the noindex rule, and the page can still appear in results, for example if other pages link to it. Google also does not support a noindex rule inside robots.txt.
• List URLs exactly as they should be crawled. Google says it attempts to crawl URLs exactly as listed, and asks for fully qualified absolute URLs, such as https://www.example.com/mypage.html rather than /mypage.html. A single sitemap is limited to 50 MB uncompressed or 50,000 URLs, so large sites split it and use an index file.

If an old URL moved, make sure the redirect goes in one hop. Chains are covered in How to Fix Redirect Chains. If a page's canonical disagrees with the sitemap, see Canonical Pointing to Another URL.

How to check it is fixed

This is the step most guides skip.

1. Rebuild or redeploy the sitemap, then open it in a browser and confirm the old URLs are gone.
2. Re-run the bulk status check on the new sitemap. Expected result: the command prints nothing, because every URL returns 200.
3. Re-run the noindex and canonical check. Expected result: no noindex: lines and no canonical differs: lines.
4. Re-crawl and compare. Run a new audit and compare it with the one you made before the fix: which findings are resolved, which are still open, and whether any new ones appeared. SiteOptima is built for this loop. You change the site outside the tool, audit again and see what moved. Health Score reflects technical state, not traffic or rankings.
5. Check Search Console later. Google recrawls on its own schedule, so the statuses will change over time. We can't say how long it takes.

Checklist

• Sitemap URLs extracted into one list (every sitemap file, if you use an index)
• Every URL returns 200 without following redirects
• No listed URL has a robots meta noindex or an X-Robots-Tag noindex
• Each listed URL is its own canonical
• URLs that should not rank are out of the sitemap, not blocked in robots.txt
• Sitemap generator or template fixed, not just the file
• All URLs are absolute and use the final protocol and host
• Redirects for moved URLs go in one hop
• Sitemap re-tested and the site re-audited, with before and after compared

FAQ

Should a sitemap contain redirected URLs?

No. List the final URL. Google's documentation says to include the URLs you want to see in search results, and a redirecting URL is not the page you want shown.

Can I keep noindex pages in my sitemap?

There is no reason to. The page asks not to be indexed and the sitemap asks for it to be considered. Remove one of the two.

Will cleaning my sitemap improve my rankings?

Google does not say so, and we won't either. A clean sitemap removes conflicting signals and makes your site easier to audit.

Does Google read noindex in robots.txt?

No. Google's documentation states that the noindex rule in robots.txt is not supported. Use a robots meta tag or an X-Robots-Tag header, and let Google crawl the page.

My sitemap is generated by a plugin. Why does it still list these URLs?

Often because the redirect, the noindex or the canonical is set somewhere the plugin does not read, such as the server, a CDN or a different plugin. Check the HTTP response, not only the plugin settings.

Sources

• Google Search Central, Build and submit a sitemap: which URLs to include, absolute URLs, size limits.
• Google Search Central, How to specify a canonical URL with rel="canonical" and other methods: strength of redirects, rel="canonical" and sitemap inclusion as signals.
• Google Search Central, Block Search indexing with noindex: meta tag and X-Robots-Tag, robots.txt blocking, noindex not supported in robots.txt.
• Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications: X-Robots-Tag header syntax.