Handling Near-Duplicate Programmatic Pages: A Similarity Pipeline That Actually Ships

Every programmatic SEO site hits the same wall eventually: two templates, one data source, and a long tail of pages that are technically unique but functionally the same. Google's Helpful Content systems have gotten aggressive about this, and "unique title tag" is not the defense it was in 2020. Here's the similarity pipeline we run on client sites before pages ever hit the sitemap.

Why string-level uniqueness stopped being enough

A classic programmatic template — say, "Plumbers in {city}" — generates uniqueness through variables: city name, population, a few local landmarks, maybe a weather snippet. On paper each page is different. In practice, if 80% of the DOM is identical boilerplate and the variable content is short, you've built a near-duplicate cluster.

We've seen three failure modes on audits:

• Thin variable payload. The template has 1,200 words of shared scaffolding and 90 words of unique data. Google clusters them and picks one canonical, ignoring the rest.
• Semantic collisions. "Best pizza in Camden" and "Top pizza restaurants Camden" render as distinct URLs from different data joins, but the SERP intent and the body copy are indistinguishable.
• Data sparsity tail. The top 5,000 cities have rich data; the bottom 40,000 fall back to the same default paragraphs. That tail is where index bloat and manual actions start.

You can't eyeball this at 100k URLs. You need a pipeline.

The four-stage similarity pipeline

We run four stages, cheapest to most expensive, and short-circuit as soon as a page fails a threshold. This keeps the whole thing runnable nightly on commodity hardware.

1. Normalisation — strip chrome, boilerplate, nav, footer.
2. Lexical similarity — MinHash over shingles for pairwise near-duplicate detection.
3. Semantic similarity — sentence embeddings for intent-level collisions.
4. Decision layer — rules that map a similarity score to an action (merge, noindex, keep, regenerate).

Stage 1: extract the content that actually matters

Boilerplate is the enemy of every similarity score. If your header, footer, and sidebar are 60% of the DOM, every page looks 60% similar before you've measured anything real.

We use a main-content extractor (trafilatura works well, so does a custom readability-style pass) and then strip:

• Any element inside <nav>, <footer>, <aside>
• Repeated blocks that appear on more than ~30% of sampled pages
• Structured data blocks (JSON-LD) — those get compared separately

import trafilatura
from bs4 import BeautifulSoup

def extract_main(html: str) -> str:
    text = trafilatura.extract(
        html,
        include_comments=False,
        include_tables=False,
        favor_precision=True,
    )
    return text or ""

What you feed the next stage should be the copy a human would actually read on the page — nothing else.

Stage 2: MinHash for lexical near-duplicates

MinHash with LSH (locality-sensitive hashing) is the workhorse. It gives you a Jaccard-similarity estimate over word shingles without pairwise comparison, so it scales to hundreds of thousands of documents.

We use 5-word shingles and 128 permutations. Empirically that's a good balance between recall and index size:

from datasketch import MinHash, MinHashLSH

def shingles(text: str, k: int = 5):
    words = text.lower().split()
    return {" ".join(words[i:i+k]) for i in range(len(words)-k+1)}

def minhash(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    for s in shingles(text):
        m.update(s.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.7, num_perm=128)
for url, text in corpus.items():
    lsh.insert(url, minhash(text))

# For any page, find its neighbours
candidates = lsh.query(minhash(target_text))

Our working thresholds, adjust for your corpus:

• Jaccard ≥ 0.85 — treat as duplicate. Merge or noindex the weaker URL.
• 0.70 – 0.85 — flag for stage 3.
• < 0.70 — lexically distinct, but not necessarily semantically.

Stage 3: embeddings for semantic collisions

MinHash catches copy-paste and template overlap. It misses two pages that say the same thing in different words — the "best pizza" vs "top pizza restaurants" case. For that, you need vector similarity.

We generate sentence embeddings (a small local model like all-MiniLM-L6-v2 is plenty for this — you're not doing generation, just comparison) and compute cosine similarity between candidate pairs from stage 2, plus a sampled cross-cluster check.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

def embed(text: str) -> np.ndarray:
    return model.encode(text[:2000], normalize_embeddings=True)

def cosine(a, b):
    return float(np.dot(a, b))

Thresholds we've landed on for programmatic corpora:

• Cosine ≥ 0.92 — semantic duplicate even if lexically different. Same intent, same answer.
• 0.85 – 0.92 — same topic, different angle. Usually keep both but review internal linking.
• < 0.85 — genuinely different.

One trick: also embed the H1 and meta description separately and compare those. A pair that scores 0.98 on titles and 0.88 on body is almost always an intent collision worth merging, regardless of body similarity.

The decision layer: what to actually do with a match

Detection is the easy part. The hard part is deciding which URL survives and how. We codify this as a rule table rather than case-by-case judgement, because at scale you can't afford judgement.

Picking the survivor

When a cluster of near-duplicates is detected, we rank candidates by a weighted score:

• Clicks in the last 90 days (from GSC) — weight 0.4
• Impressions in the last 90 days — weight 0.2
• Internal links pointing in — weight 0.2
• Content depth (unique-tokens count) — weight 0.1
• URL depth (shorter wins on ties) — weight 0.1

Highest score keeps its URL and gets any additional unique content merged in. Losers get 301'd to the survivor. If none of the cluster has meaningful traffic yet, we consolidate to the shortest URL and leave the rest as noindex, follow for a crawl cycle before redirecting — that way we don't burn crawl budget on dead ends but we also don't lose the internal link equity overnight.

When to regenerate instead of merge

Sometimes the right answer isn't consolidation — it's fixing the template. If stage 2 flags a systematic issue (say, 8,000 pages all above 0.8 Jaccard because your "local landmarks" API is returning the same generic paragraph for small towns), merging is treating the symptom.

Go back to the data model. Either enrich the data source, raise the minimum-data threshold for page generation, or accept that the tail shouldn't exist. We've written about the underlying tradeoff in our programmatic SEO consulting work — the honest answer for most sites is that the bottom 30% of the URL set shouldn't have been published in the first place.

Running it in production

A few practical notes from actually operating this:

• Run it pre-publish, not just post-hoc. The pipeline should gate new page generation. If a candidate page's nearest neighbour scores above 0.85 Jaccard or 0.92 cosine, don't publish — either enrich the data or route to the existing URL.
• Persist the MinHash index. Rebuilding from scratch on 200k pages takes hours. Store the signatures alongside your page records and update incrementally.
• Sample, don't audit everything, every night. For established pages, a rolling 10% sample plus any page that changed is enough to catch drift.
• Track the false-positive rate. Every quarter, pull 50 flagged pairs and human-review them. If you're above ~10% false positives, your thresholds are too loose.
• Log decisions. When you noindex or redirect a page, log the cluster, the scores, and the survivor. When traffic moves, you want to be able to reconstruct why.

The metrics that tell you it's working

Three numbers we watch in GSC and GA4 after a deduplication pass:

• Indexed-URL count should drop, then stabilise. If it keeps climbing, your template is still generating duplicates.
• Clicks per indexed URL should rise. This is the real health metric — you want fewer, better pages.
• Impressions on survivor URLs should absorb most of the impressions the merged URLs used to get. If they don't, your redirects or canonicals are wrong.

We've covered the query-to-revenue side of this in a previous post on GA4 and GSC joins; the deduplication metrics slot into the same dashboard.

Where we'd start

If you're staring at a programmatic site that feels bloated: skip the fancy embeddings for week one. Run stage 1 and stage 2 only, on a sample of 5,000 URLs, and look at the distribution of pairwise Jaccard scores. If more than 15% of pairs are above 0.7, you have a template problem, not a page problem — fix the data model before you fix any individual URL. Bring in embeddings once the lexical noise is gone and you're hunting the subtler intent collisions.