Embeddings and Vector Analysis for SEO

An embedding is a vector that represents the meaning of a piece of text. Similar meanings sit close together.
Cosine similarity measures that closeness. Scores are only comparable within the same model.
AI search retrieves passages, not whole pages, often by combining vector and keyword matching.
Practical uses: cannibalisation checks, content gaps, internal link suggestions, migration redirect mapping and off-topic page detection.
Screaming Frog can generate embeddings during a crawl, and a short Python script covers the rest.

Keyword tools compare words. Embeddings compare meaning. An embedding model reads a piece of text and returns a long list of numbers, a vector, positioned so that texts about similar things land near each other. "How to fix hreflang return errors" and "why my alternate language tags are being ignored" share few words but sit close together. Modern search and AI retrieval work this way, and the same maths gives SEOs a much sharper toolkit than keyword overlap.

What an embedding is

A model maps text to a point in a space with hundreds or thousands of dimensions. Nobody can picture that space, but you do not need to. What matters is distance. The usual measure is cosine similarity, the angle between two vectors. Scores run up to 1, where 1 means pointing in exactly the same direction. Google's machine learning crash course has a friendly visual introduction.

Two caveats from the start. First, scores depend on the model. A 0.85 from one provider is not a 0.85 from another, so never mix models within an analysis. Second, similarity is not quality. Two equally thin pages can be very similar.

How search and AI retrieval use embeddings

Google has used vector representations of meaning for over a decade, and dense retrieval is now standard across search. AI search systems typically:

Split documents into passages and embed each one.
Expand the user's question into several related queries, the query fan-out described in our guide to how AI search works.
Retrieve the passages closest to those queries, very often mixing vector similarity with classic keyword scoring such as BM25.
Re-rank the shortlist and hand the best passages to the language model to write the answer.

The implication is simple. Each section of a page competes on its own. A passage that directly and completely answers one sub-question has a better chance of being retrieved than a meandering one, however strong the page is overall.

Getting embeddings for your site

Since version 22, Screaming Frog's SEO Spider can request embeddings from OpenAI, Gemini or a local Ollama model as it crawls. Its semantic similarity tutorial covers the setup: connect a provider, store the page HTML, then enable embeddings under the content configuration. It flags near-duplicate pages (0.95 and above by default, adjustable), identifies low-relevance pages by comparing each one to the average of the whole site, and offers a semantic search tab for finding the pages closest to a query.

For anything bespoke, export the vectors or call an embeddings API directly, such as OpenAI's embeddings or Gemini's embeddings, and do the comparisons yourself.

Practical uses

Cannibalisation: pairs of pages with very high similarity that target the same intent. Cross-check with Search Console to see whether they swap rankings for the same queries, then merge or differentiate.
Content gaps: embed a list of target questions, including fan-out style sub-questions, and find the ones whose best-matching page scores low. Those are topics you do not really cover.
Internal linking: for each page, list the most similar pages that do not already link to it. It beats keyword matching for spotting relevant links.
Migration redirect mapping: match old URLs to their closest new equivalents, then check the borderline matches by hand. Screaming Frog publishes a tutorial for this.
Off-topic content: pages far from the site's centre of gravity are candidates for pruning or for moving somewhere more appropriate.
Keyword clustering: group large keyword lists by meaning rather than by shared words.
Passage checks: embed each section of a key page and compare it to the questions it should answer, to find sections that wander.

A minimal comparison in Python

 import numpy as np

def cosine(a, b):
    a, b = np.array(a), np.array(b)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

# page_vectors: {url: vector} from your crawl export or an API
query_vector = embed("how to verify googlebot")
scores = sorted(
    ((url, cosine(query_vector, v)) for url, v in page_vectors.items()),
    key=lambda x: x[1], reverse=True
)
print(scores[:10])

Here embed() is whichever provider call you use. For a few thousand pages this runs in seconds. For hundreds of thousands, a vector database or a library such as FAISS keeps lookups quick.

Getting reliable results

Embed the main content only. Navigation, footers and cookie text make every page look alike. Configure the content area or strip boilerplate first.
Use one model throughout and record which one, so later runs are comparable.
Chunk long pages into sections. One vector for a 4,000 word page blurs everything together.
Treat thresholds as relative. Calibrate on pairs you already know are duplicates or unrelated before trusting a cut-off.
Mind cost and confidentiality. API calls cost money and send content to a third party. A local model avoids both, at some loss of quality.

Common mistakes

Reading similarity scores as rankings. Your model is not Google's.
Merging pages purely because they score as similar, without checking intent and performance.
Comparing vectors from different models or versions.
Ignoring the keyword side. Retrieval is usually hybrid, so exact terms still matter.

The GEO angle

AI answers are assembled from retrieved passages, so passage-level relevance is the unit that matters. Embeddings let you check, before publishing, whether each section of a page sits close to the questions you want it to answer. That turns "write for AI citation" from a style guide into something you can measure.