Cannibalization at Scale: Finding and Killing Duplicate Intent Across 10k Programmatic Pages

Programmatic SEO breeds cannibalization the way a warm server room breeds fan noise. Once you cross a few thousand URLs generated from the same template family, at least a handful will start competing with each other for the same query — and Google will punish the whole cluster by rotating rankings, suppressing impressions, or quietly demoting the lot. This is the audit and fix pipeline we use when a client's programmatic set has stopped growing traffic despite growing URL count.

What cannibalization actually looks like at scale

At small scale, cannibalization is two blog posts fighting over "best CRM for startups". You spot it in an afternoon. At programmatic scale, it hides.

The symptoms we see most often:

• A template that ranked well at 500 URLs plateaus or regresses after you push it to 5,000.
• Google Search Console shows a query where the landing page URL rotates weekly between three or four near-identical pages.
• Average position for a cluster hovers between 8 and 15 and refuses to move, even after content quality work.
• Click-through rate drops on pages that used to convert, because the wrong URL is now surfacing for the query.

The underlying cause is almost always the same: your template generated pages whose intent overlap is higher than their content differentiation. City pages that all read the same. Long-tail modifier pages ("cheap", "affordable", "budget") that Google correctly considers synonyms. Category × attribute pages where the attribute doesn't actually change the answer.

Why the classic "one query per page" rule breaks here

The old-school fix — map each page to one primary keyword — assumes you wrote the pages by hand. Programmatic pages are generated from a data model, and the model itself is often the thing producing the overlap. You cannot fix cannibalization at the page level if the template is the source. You have to audit at the cluster level and push the fix back into the data model.

Step 1: Pull the raw signal from GSC

Start with 90 days of GSC performance data at the query + page granularity. The Search Console UI truncates aggressively, so use the API or a BigQuery export. You want every row where a query has more than one URL earning impressions.

-- BigQuery on the GSC bulk export
SELECT
  query,
  url,
  SUM(impressions) AS impressions,
  SUM(clicks)      AS clicks,
  SAFE_DIVIDE(SUM(clicks), SUM(impressions)) AS ctr,
  AVG(sum_position) AS avg_position
FROM `project.searchconsole.searchdata_url_impression`
WHERE data_date BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY) AND CURRENT_DATE()
  AND is_anonymized_query = FALSE
GROUP BY query, url
HAVING impressions >= 20

Now flag queries where at least two URLs have meaningful impressions:

WITH q AS (
  SELECT query, COUNT(DISTINCT url) AS url_count,
         SUM(impressions) AS total_impr
  FROM base
  GROUP BY query
)
SELECT * FROM q
WHERE url_count >= 2 AND total_impr >= 100
ORDER BY total_impr DESC;

Don't panic at the row count. Multi-URL queries are normal on branded terms and legitimate hub-vs-spoke setups. You are looking for patterns, not individual pairs yet.

Step 2: Separate real cannibalization from noise

Not every multi-URL query is a problem. In our experience, four buckets emerge:

1. Legitimate hub/spoke. A category page and a subcategory page both rank for a broad query. Fine, expected, often desirable.
2. Brand + modifier overlap. "acme pricing" and "acme cost" pulling different URLs. Usually harmless.
3. Template-on-template cannibalization. Two pages from the same template competing. This is the expensive kind.
4. Cross-template cannibalization. A blog post competing with a programmatic landing page. Almost always the blog post is winning the click and losing the conversion.

Bucket 3 and 4 are what you're hunting. To isolate them, join your GSC data against a URL classification table you should already have from your content model:

SELECT c.query,
       u1.template AS template_a,
       u2.template AS template_b,
       COUNT(*)    AS query_count
FROM cannibal_queries c
JOIN url_meta u1 ON c.url_a = u1.url
JOIN url_meta u2 ON c.url_b = u2.url
WHERE u1.url < u2.url
GROUP BY 1,2,3
ORDER BY query_count DESC;

If you see a template pair with hundreds of overlapping queries, that's a design bug, not a page bug.

Step 3: Cluster by intent, not by string match

String-level query grouping ("best crm" vs "top crm") misses semantic duplicates and over-splits typos. We embed queries with a lightweight model and cluster with HDBSCAN or plain agglomerative clustering at a cosine distance around 0.15 – 0.25.

from sentence_transformers import SentenceTransformer
import hdbscan

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(queries, normalize_embeddings=True)

clusterer = hdbscan.HDBSCAN(
    min_cluster_size=3,
    metric="euclidean",  # embeddings are L2-normalized
    cluster_selection_epsilon=0.2,
)
labels = clusterer.fit_predict(embeddings)

Now within each intent cluster, look at how many distinct URLs Google is trying. A cluster of 40 semantically identical queries served by 12 URLs from the same template? That's your fix list.

Sanity-check with a click-weighted "winner"

For each cluster, compute a winner score per URL:

score = 0.6 * clicks_share + 0.3 * impressions_share + 0.1 * (1 / avg_position)

If one URL has a score above ~0.55, that's your keeper. If the scores are flat across four URLs, Google genuinely can't decide, and you have to.

Step 4: Decide — consolidate, differentiate, or prune

Three valid outcomes per cluster. Pick one, do not do all three.

• Consolidate. Merge the competing pages into the winner, 301 the losers. Best for near-duplicate intent with no user-visible reason for separate pages.
• Differentiate. Rewrite the template so the pages actually answer different sub-questions. Best when the underlying data is different but the template hid it (e.g. city pages that should surface local pricing but currently show the same national table).
• Prune. Noindex or 410 the losers with no redirect. Best when the losing pages have no backlinks, no conversions, and no rescue value. Yes, deleting programmatic pages is a valid growth move. We've seen 15 – 30% organic click lifts from pruning 40% of a bloated template set, though your mileage will vary with authority and topic.

Step 5: Push the fix into the data model, not just the pages

This is the step most teams skip and why cannibalization comes back six months later. If your audit found that city × service pages cannibalize whenever two cities are within the same metro, that rule needs to live in the generator:

def should_generate(city, service):
    metro = metro_for(city)
    primary_city = primary_city_for(metro)
    if city != primary_city and not has_unique_local_data(city, service):
        return False
    return True

Bake the anti-cannibalization rules into the same code that decides which pages exist. Otherwise the next data import re-creates the mess.

Step 6: Monitor with a standing dashboard

Once you've cleaned up, ship a weekly job that recomputes the cannibalization index per template:

cannibalization_index = (# queries with 2+ competing URLs from same template) / (# queries with impressions from that template)

Alert when the index rises more than a few points week-over-week. That's your early warning that a new content push or a template change reopened the wound.

If you want the full data model discussion behind this, our programmatic SEO services page walks through how we set the generator up from day one.

Where we'd start

If you're staring at a stalled programmatic set right now: pull 90 days of GSC data tonight, filter to queries with 2+ URLs and 100+ impressions, and join against your template metadata. You don't need embeddings on day one — a spreadsheet of the top 50 same-template cannibalized queries will already tell you whether this is your problem. If it is, budget a two-week sprint: one week to audit and cluster, one week to consolidate, prune, and update the generator rules. Then put the monitoring dashboard in place before you write another line of template code.