Internal Linking at Scale: The Graph-Based Approach That Beat Our Rule-Based System

Most programmatic sites ship internal linking as a WHERE category = ? LIMIT 10 query and call it done. That worked for us for about eighteen months, then stopped working in a way that was hard to see from inside the codebase. This is the story of ripping it out and rebuilding it as a graph scoring problem, plus the parts we'd do differently.

Why rule-based internal linking quietly rots

The first internal linking system we built on a ~400k-page programmatic property was embarrassingly simple. Each page template had a "Related" block that fired a query like "same category, same region, order by popularity". A sidebar block pulled "other items in this city". A footer block linked to parent and sibling taxonomy nodes.

It looked fine. It passed code review. It even looked fine in Screaming Frog for the first year, because the site was small enough that most pages were two or three clicks from the homepage anyway.

Then the catalogue grew. And three things happened at once:

• Orphan clusters appeared. Entire long-tail segments — think "[service] in [small town]" — had no inbound internal links except from their own category hub, which itself was buried four clicks deep.
• Link equity concentrated on the wrong pages. Our "popular" ordering meant the same 200 pages got linked from everywhere. They ranked. Everything else starved.
• Crawl depth exploded. Googlebot's average crawl depth, measured from log files, drifted from ~3.1 to ~5.8 over nine months. Pages at depth 6+ barely got recrawled.

The rule-based system wasn't wrong. It was just blind to the thing that actually matters: the shape of the link graph it was producing.

The symptom that forced the rewrite

What finally made us act wasn't a ranking drop — it was a GSC report showing that about 22% of our indexed pages hadn't been crawled in 90+ days. Cross-referencing with our link graph (which we built retroactively, more on that below), nearly all of them had fewer than 3 internal inbound links, and the ones they did have came from pages that themselves were deep in the tree.

We were pumping out content velocity and the crawler was ignoring half of it.

Reframing internal linking as a graph problem

The mental shift is small but important. Stop thinking about "what should this page link to?" and start thinking about "what graph am I producing across the whole site, and does it have the properties I want?"

The properties we wanted:

1. Low average crawl depth — ideally under 4 for anything we care about ranking.
2. No orphan pages — every indexable URL has at least 3 internal inbound links from relevant pages.
3. Link equity distributed by commercial intent, not by whatever ORDER BY happened to be in the query.
4. Topical relevance preserved — linking a plumber-in-Leeds page from a dentist-in-Bristol page is worse than not linking at all.

Once you write those down, it's obvious this is a constrained optimisation problem, not a template question.

The pipeline we actually built

We run this as a nightly batch job. It writes a links table that templates read from at render time — the templates themselves don't do any ranking logic anymore.

Step 1: Build the full page graph

First, materialise every indexable page as a node, with features attached:

# nodes.py
node = {
    "url": "/services/plumbing/leeds",
    "type": "service_city",
    "topic_vec": embed("plumbing leeds emergency repair"),
    "commercial_score": 0.78,  # from GSC impressions * CTR * value
    "current_depth": 4,
    "inbound_count": 2,
}

The topic_vec is a cheap sentence embedding of the page's h1 plus its primary keyword set. We use a small open model and cache aggressively — this is not the place for API calls per page.

Step 2: Score candidate edges

For every page, we generate candidate link targets and score them. The scoring function is the whole game:

def edge_score(src, dst):
    if src["url"] == dst["url"]:
        return 0

    relevance = cosine(src["topic_vec"], dst["topic_vec"])
    if relevance < 0.35:
        return 0  # hard floor, no off-topic links

    # reward linking to pages that need help
    depth_boost = max(0, dst["current_depth"] - 3) * 0.15
    orphan_boost = 0.4 if dst["inbound_count"] < 3 else 0

    # reward linking to commercially valuable pages
    commercial = dst["commercial_score"] * 0.3

    return relevance + depth_boost + orphan_boost + commercial

Notice what's missing: popularity. Popular pages already have links. The scoring function is biased toward pages that are relevant and currently under-linked.

Step 3: Constrain the output

Unconstrained, the scorer will happily give you 50 links per page pointing at every orphan on the site. That's spammy and dilutes everything. We cap:

• Max 8 contextual links per page
• Max 20 inbound links per destination from a single template type
• At least 2 of the 8 must be "peer" links (same template, same taxonomy level) for navigability

The last constraint matters more than it sounds. Pure optimisation gives you weird link patterns that confuse users. Peer links keep the site feeling like a site.

Step 4: Write and version

The output is a page_links table: (source_url, target_url, anchor_text, slot, score, generated_at). Templates query this at render time. We keep the previous version around for a week so we can diff and roll back if something looks wrong.

What changed after we shipped

I'm going to be careful with numbers here because every site is different and we only have one data point. In our case, over roughly 10 weeks:

• Average crawl depth (from log analysis) dropped from ~5.8 to ~3.4.
• Pages crawled in the last 30 days rose from ~61% to ~89% of indexable URLs.
• The long-tail cluster that had been starving — about 40k pages — went from near-zero clicks to being responsible for a meaningful chunk of non-brand traffic. Not a hockey stick, but a visible, sustained lift.
• Top-of-funnel pages (the ones that used to get all the links under the old system) dropped slightly in rankings for a few weeks, then recovered. Equity got redistributed; it didn't disappear.

The thing we didn't expect: our AdSense RPM on the previously-orphaned pages was higher than the site average, because they were more specific and attracted more intent-heavy queries. The old linking system had been actively suppressing our best inventory.

What we got wrong on the first pass

Two mistakes worth flagging:

We under-weighted anchor text diversity. The first version used the destination page's h1 as anchor text almost every time. Google doesn't penalise this directly, but it reads as templated and it's wasteful — anchor text is a ranking signal and we were throwing it away. We now rotate between h1, primary keyword, and a short descriptive variant generated from the page's intro.

We didn't account for the pages being removed from the link graph. When our scorer stopped linking to the old popular pages from everywhere, some of them lost 70%+ of their internal inbound links overnight. A couple of commercially important pages dipped before we added a floor rule guaranteeing a minimum of inbound links to anything above a commercial threshold.

When this is overkill

If your site is under ~5,000 pages, don't do this. A well-designed hub-and-spoke taxonomy with human-curated related links will outperform any graph scoring you build. Graph-based linking pays off when (a) you have too many pages for humans to curate and (b) your templates are already producing broadly similar sets of candidate links.

The other place to be careful is if your topical clusters are poorly defined. The relevance floor in the scoring function is only as good as your embeddings. If your taxonomy is a mess, fix that first — a scoring function won't rescue incoherent content.

Where we'd start

If you're sitting on a programmatic site with declining crawl coverage and a rule-based internal linking system, the lowest-risk first move is diagnostic, not architectural. Pull your sitemap, crawl your own site, and build the link graph as it exists today. Then overlay GSC crawl stats and look at the correlation between inbound internal link count and last-crawled date. If the correlation is strong (ours was brutal), you have a graph problem, not a content problem.

From there, you can prototype the scoring approach on a single template type before touching the whole site. Pick the template that's producing the most orphans. Ship it to that template only. Measure for four weeks. Then decide whether to roll it out further.

If you'd rather have someone else build this pipeline against your stack, that's the kind of work our team does under content engineering and SEO services — and we've written more on the measurement side in our programmatic SEO posts.