All articles
SEO & GrowthNovember 17, 2026 6 min read

Query Clustering with Embeddings: Building the Topic Map That Killed Our Cannibalization

Keyword cannibalization on a 12k-page site isn't a content problem — it's a clustering problem. Here's the embeddings pipeline we built to merge, redirect, and consolidate pages that were fighting each other in Google.

A client came to us with a 12,000-page programmatic site and a familiar story: traffic plateaued, new pages weren't ranking, and old pages were slipping. The usual suspects — thin content, backlinks, Core Web Vitals — all checked out. The real issue was that roughly 18% of their indexed URLs were competing against each other for overlapping intents. This is how we found them, grouped them, and decided what to merge.

Why string-match cannibalization audits miss the real problem

Most cannibalization audits are embarrassingly primitive. You export Google Search Console queries, pivot by query, and flag any query that has more than one URL with impressions. That catches the obvious cases — two pages both ranking for best running shoes for flat feet — but it misses the ones that actually hurt.

The damaging cannibalization is semantic, not lexical. Two pages targeting running shoes for overpronation and stability shoes for flat arches are fighting for the same intent even though they share almost no tokens. Google knows they're the same topic. Your pivot table doesn't.

On the flip side, string matching produces a flood of false positives. python list sort and python sort list are the same query, but they're also the same URL ranking for both — that's not cannibalization, that's working as intended.

We needed a method that understood intent, not strings.

The pipeline, end to end

Here's the shape of what we built. It runs weekly on a scheduled job and produces a consolidation queue for the content team.

  1. Pull 90 days of GSC data at the query + URL granularity
  2. Filter to queries with at least 10 impressions across the site
  3. Embed each query with a sentence-transformer model
  4. Cluster the embeddings with HDBSCAN
  5. For each cluster, list the URLs competing on it
  6. Score severity by lost impressions and rank volatility
  7. Emit a ranked action list: merge, redirect, or differentiate

The whole thing fits in a single Python service with a Postgres cache. No ML infrastructure needed beyond a decent CPU.

Why HDBSCAN and not k-means

k-means forces every query into a cluster and makes you pick k up front. Neither of those is what we want. Most queries on a large site are long-tail singletons that shouldn't be clustered with anything. HDBSCAN lets points stay unclustered as noise, which is exactly right — those are the queries doing their job on a single URL.

It also handles varying cluster densities, which matters when your site covers both broad commercial topics (dense clusters) and narrow informational ones (sparse clusters).

The code that does the heavy lifting

This is a trimmed version of the clustering step. The GSC pull and the Postgres writes are the boring parts — this is where the decisions happen.

import numpy as np
import pandas as pd
from sentence_transformers import SentenceTransformer
import hdbscan

def cluster_queries(gsc_df: pd.DataFrame, min_impressions: int = 10):
    # gsc_df columns: query, page, clicks, impressions, position
    agg = (gsc_df
           .groupby('query')
           .agg(impressions=('impressions', 'sum'),
                pages=('page', lambda s: list(set(s))))
           .reset_index())
    agg = agg[agg['impressions'] >= min_impressions]

    model = SentenceTransformer('all-MiniLM-L6-v2')
    embeddings = model.encode(
        agg['query'].tolist(),
        batch_size=128,
        show_progress_bar=False,
        normalize_embeddings=True,
    )

    clusterer = hdbscan.HDBSCAN(
        min_cluster_size=4,
        min_samples=2,
        metric='euclidean',
        cluster_selection_method='eom',
    )
    agg['cluster'] = clusterer.fit_predict(embeddings.astype(np.float64))

    # -1 is noise in HDBSCAN; keep those out of the cannibalization report
    clustered = agg[agg['cluster'] != -1].copy()
    return clustered, embeddings

A few things worth knowing:

  • all-MiniLM-L6-v2 is small, fast, and good enough for query-level semantics. We tried larger models and the clustering quality barely moved. Save the GPU budget.
  • normalize_embeddings=True lets you use euclidean distance as a proxy for cosine, which HDBSCAN handles well.
  • min_cluster_size=4 is aggressive on purpose. On a 12k-page site we want tight, actionable clusters, not sprawling topics.

Turning clusters into a cannibalization score

A cluster by itself doesn't mean cannibalization. What matters is how many distinct URLs are competing within it and how much that competition is costing you. We score each cluster on three dimensions:

URL spread

For every query in the cluster, we look at which URL Google picked. If one URL dominates across the cluster, there's no cannibalization — the site has a clear winner. If five different URLs each take a share, that's the fire.

We use a normalized Herfindahl index on URL share within the cluster. Anything below 0.4 flags as high-spread.

Rank volatility

We pull position data by week and compute the standard deviation of the ranking URL for each query. If the URL serving a query keeps flipping week to week, Google is confused. That's a stronger signal than any single-snapshot audit.

Lost opportunity

For each query, we estimate the clicks you'd get at position 3 (using a flat CTR curve — we don't trust industry CTR tables on programmatic sites) and subtract actual clicks. Sum across the cluster and you have a dollar-able number for the content team.

The final severity is a weighted product of these three. We tune the weights per client based on what their content ops team can actually action.

From score to action: merge, redirect, or differentiate

A high-severity cluster gets one of three labels:

Merge. Two or more URLs are serving nearly identical intent and neither has a decisive authority advantage. We consolidate content into whichever URL has more backlinks and 301 the others. This is the most common outcome, maybe 60% of flagged clusters.

Redirect. One URL is clearly stronger (links, age, better content) and the others are vestigial — often old blog posts that got overtaken by a newer category page. Straight 301s to the strong URL.

Differentiate. The URLs are targeting genuinely different sub-intents that happen to embed close together. We rewrite titles, H1s, and intro copy to sharpen the distinction, and update internal anchors to reinforce the split. This is the hardest call and the one that benefits most from a human review.

We never let the pipeline act autonomously. It opens tickets with a recommendation and the evidence. A strategist approves.

What actually changed after rollout

In our experience across three programmatic sites that ran this pipeline, the pattern is consistent: a messy first quarter where consolidation causes short-term ranking dips on the redirected URLs, followed by net gains on the surviving URLs as Google reconsolidates signals. The sites that stayed the course saw meaningful organic growth within two quarters. The one that panicked and reverted redirects after four weeks got the worst of both worlds.

Two lessons from the war stories:

  • Always batch redirects by cluster, not by URL. Shipping 40 redirects across 12 clusters on a Tuesday gives you a clean before/after to measure. Dribbling them out destroys your ability to attribute recovery.
  • Keep a redirect log with the cluster ID, the reasoning, and the baseline metrics. Six months later when someone asks why /guides/old-thing 301s to /hub/new-thing, you'll want the receipts.

Where this breaks

The pipeline assumes GSC data is representative. On very new sites or sections with low impression volume, clusters are too sparse to form and you'll get mostly noise. We don't bother running it on sections with fewer than ~500 queries clearing the impression threshold.

It also doesn't know about your business. A cluster might flag enterprise pricing and small business pricing as one topic — semantically they're close — but you may have deliberate reasons to keep them split. That's why the human review step isn't optional.

Finally, embedding models drift. If you lock in a model version today and the world moves to a better one in 18 months, your historical cluster IDs stop being comparable. We stamp the model version on every cluster record and re-embed everything when we upgrade.

Where we'd start

If you're staring at a programmatic site that feels stuck, don't start with a full pipeline. Start with a one-off notebook: pull 90 days of GSC, embed the top 2,000 queries, run HDBSCAN once, and eyeball the biggest clusters. You'll know within an afternoon whether cannibalization is your problem. If it is, productionize the loop. If it isn't, you've saved yourself a quarter of engineering time and can go chase the real bottleneck instead. Either way, the embeddings-first view of your query universe is worth having — it changes how you think about every content decision after it. If you want help wiring this into a content ops workflow, that's the kind of work our growth engineering team does day in, day out.

#SEO#Programmatic SEO#Embeddings#GSC#Content Strategy

Want a team like ours?

72Technologies builds production software for the kind of teams who actually read this blog.

Start a project