Indexing Budget Math: Deciding Which Programmatic Pages Deserve to Exist

Every programmatic SEO project we've inherited has the same rot: 200k URLs generated, 12k indexed, 3k getting impressions, and maybe 400 doing anything commercially useful. The team is proud of the 200k. Google is not. The fix isn't better content — it's deciding, before generation, which pages have any business existing at all.

This is a framework we use on client audits and on our own properties. It's not glamorous. It's spreadsheet work with a scoring model and a hard cutoff. But it's the single highest-leverage lever in programmatic SEO once your templates are decent.

The problem with "generate everything"

The default programmatic workflow is: take a dataset, cross-join it with a template, publish the cartesian product. City × service. Product × attribute. Job title × location × seniority. You end up with a URL count that looks impressive in a deck.

The issue is that Google now aggressively demotes and de-indexes pages it considers thin, redundant, or low-demand. Since the Helpful Content updates through 2024 and the site-wide quality signals rolled forward into 2025 and 2026, a bloated long tail actively drags down the pages that would have ranked. We've watched sites recover 30–60% of organic traffic (in our experience, on mid-size programmatic properties) purely by removing pages, not adding them.

So the question stops being "what can we generate?" and becomes "what should we generate, and what proof do we have?"

The three-signal scoring model

Before a page template ships to production, every candidate URL gets scored on three axes. Everything else is noise.

1. Demand signal

Does anyone actually search for this? Not the head term — the specific cross-section this URL represents.

Sources we trust, in order:

• Google Search Console impressions on existing URLs that partially match the pattern
• Clickstream data if you have access (Similarweb, Semrush, Ahrefs — treat as directional only)
• Google Suggest and People Also Ask expansion via the autocomplete endpoint
• Internal site search logs if the site has meaningful search traffic

What we don't trust: keyword tools reporting "10 searches/month" for a five-word tail. That's usually rounding noise. We require either a real GSC signal on a neighbouring URL, or a confirmed autocomplete presence, or both.

2. Content sufficiency signal

Do we have enough unique, useful data to make this page materially different from its siblings?

This is the one teams lie to themselves about. If your "Plumbers in Peoria" page differs from "Plumbers in Springfield" only in the city name and a swapped-in weather API blurb, you don't have content sufficiency. You have a mail merge.

We score sufficiency by counting unique, structured data points per URL:

• Under 5 unique facts: kill it
• 5–10: borderline, requires editorial review or aggregation
• 10+: eligible for programmatic generation

3. Commercial signal

Does this page connect to something the business actually sells or monetises? Pages that rank but sit three clicks from any conversion path or ad unit are a tax, not an asset. This matters especially for AdSense-monetised properties where brand safety and content quality get evaluated site-wide.

Turning the model into SQL

Here's roughly what the scoring query looks like against a warehouse where you've joined GSC, your CMS content table, and a keyword research dump. Adapt column names to your stack.

WITH candidate_urls AS (
  SELECT
    c.url_pattern,
    c.dimension_1,
    c.dimension_2,
    c.unique_facts_count,
    c.has_cta,
    c.has_ad_slot
  FROM programmatic_candidates c
),
demand AS (
  SELECT
    url_pattern,
    SUM(impressions) AS gsc_impressions_90d,
    COUNT(DISTINCT query) AS distinct_queries
  FROM gsc_neighbour_matches
  WHERE date >= CURRENT_DATE - INTERVAL '90 days'
  GROUP BY url_pattern
),
scored AS (
  SELECT
    c.url_pattern,
    -- Demand score, capped
    LEAST(d.gsc_impressions_90d / 50.0, 40) AS demand_score,
    -- Sufficiency score
    CASE
      WHEN c.unique_facts_count >= 10 THEN 30
      WHEN c.unique_facts_count >= 5  THEN 15
      ELSE 0
    END AS sufficiency_score,
    -- Commercial score
    (CASE WHEN c.has_cta THEN 15 ELSE 0 END) +
    (CASE WHEN c.has_ad_slot THEN 15 ELSE 0 END) AS commercial_score
  FROM candidate_urls c
  LEFT JOIN demand d USING (url_pattern)
)
SELECT
  url_pattern,
  demand_score + sufficiency_score + commercial_score AS total_score,
  CASE
    WHEN demand_score + sufficiency_score + commercial_score >= 60 THEN 'generate_and_index'
    WHEN demand_score + sufficiency_score + commercial_score >= 35 THEN 'generate_noindex'
    ELSE 'do_not_generate'
  END AS decision
FROM scored
ORDER BY total_score DESC;

The thresholds (60, 35) are the interesting part. Tune them by looking at where your existing indexed-and-ranking pages fall on the same scoring, then draw the line just below that cohort. Don't invent thresholds in a vacuum.

Three decision buckets, not two

The common mistake is treating this as binary: publish or don't. In practice you want three states.

Generate and index

High confidence across all three signals. These get the full treatment: unique title, structured data, internal links from hub pages, sitemap inclusion, priority in your refresh queue.

Generate but noindex

Useful for logged-in users, internal search, or as content that supports faceted navigation, but not something you're asking Google to rank. These get <meta name="robots" content="noindex,follow"> and stay out of the sitemap. Follow is important — they can still pass link equity to the pages that matter.

This bucket is where most teams fail. They either index everything and get demoted, or they noindex too aggressively and lose the internal linking benefit.

Do not generate

The page doesn't exist. No URL, no template render, no soft 404, no redirect. If someone crafts the URL by hand they get a genuine 404. This matters because even a redirected URL is a signal to Google that you once thought this page should exist.

Handling the "but what if it ranks" objection

Someone on the team will always argue: "we should generate it anyway, just in case." This is how you got 200k URLs.

The honest answer: pages that scored below your threshold and would have ranked are rare, and the aggregate quality drag from publishing 50k borderline pages costs you more than the handful of surprise wins. If you're worried, run a controlled experiment — generate a random 5% sample of the "do not generate" bucket, index them, wait 90 days, and measure. We've run this test on three different properties. The results, in our experience, have never justified opening the floodgates.

Operationalising the model

A scoring model that lives in one analyst's notebook is worthless. It has to be part of the pipeline.

1. Score at generation time. The build step that produces programmatic pages should call the scoring function and refuse to write files below the threshold.
2. Re-score quarterly. Demand shifts. A page that scored 30 last year might score 70 now if a related trend emerged. Move pages between buckets accordingly.
3. Track the deleted. Keep a table of URLs you've decided not to generate and why. When someone asks "why don't we have a page for X" you have an answer with a date and a score.
4. Tie it to Search Console. Pull the Coverage report weekly. If "Crawled — currently not indexed" is growing faster than your indexed page count, your thresholds are too loose.

What we'd do first

If you're staring at a bloated programmatic site right now, don't start by building the scoring model. Start by pulling a GSC export for the last 90 days, filtering to URLs with zero impressions, and sampling 50 of them by hand. You'll immediately see the pattern of what should never have shipped. That gives you the sufficiency and demand thresholds for your specific site — everything else is instrumentation around that judgement.

Then build the model, wire it into the pipeline, and — the hard part — actually delete the pages that fail it. 301 the ones with backlinks, 410 the rest, resubmit the sitemap, and give it 6–8 weeks. If you want a second set of eyes on the audit, that's the kind of work our SEO and growth engineering team does regularly.