Definition and Overview

Cosine similarity is a mathematical metric that quantifies the similarity between two non-zero vectors by computing the cosine of the angle between them in a multi-dimensional vector space. The resulting value ranges from -1 (diametrically opposite direction) through 0 (orthogonal, no similarity) to +1 (identical direction), providing an intuitive and bounded measure of directional alignment. Unlike Euclidean distance, which measures absolute spatial separation, cosine similarity is magnitude-invariant -- it compares only the orientation of vectors, making it robust to differences in vector scale or document length.

The mathematical formulation is straightforward. For two vectors A and B, cosine similarity is defined as:

cos(theta) = (A . B) / (||A|| * ||B||)

where A . B is the dot product, and ||A|| and ||B|| are the L2 norms (magnitudes) of the respective vectors. When vectors are pre-normalized to unit length, the computation simplifies to a single dot product, which modern hardware can execute extremely efficiently through SIMD (Single Instruction, Multiple Data) instructions.

Cosine similarity has become the de facto standard metric in natural language processing, information retrieval, and recommendation systems. Its dominance in these fields stems from a key property: when applied to text embeddings generated by transformer models, cosine similarity correlates strongly with human judgments of semantic relatedness. Two sentences about the same topic will produce embedding vectors pointing in similar directions, regardless of length or phrasing, resulting in high cosine similarity scores.

Technical Deep Dive

Mathematical Properties

Understanding the mathematical properties of cosine similarity is essential for correct application and interpretation:

Property	Description	Implication
Bounded range	Output always in [-1, 1]	Directly interpretable as similarity percentage
Magnitude invariance	Only direction matters	Robust to document length differences
Triangle inequality	Does NOT satisfy	Cannot be used directly as a distance metric in metric spaces
Symmetry	cos(A, B) = cos(B, A)	Order of comparison is irrelevant
Identity	cos(A, A) = 1.0	Perfect self-similarity guaranteed

The triangle inequality violation is significant: cosine similarity is technically a similarity measure, not a distance metric. To convert it to a proper distance metric for algorithms requiring metric space properties (e.g., k-d trees, ball trees), the cosine distance 1 - cos(A, B) or angular distance arccos(cos(A, B)) / pi is used instead.

Embedding Space Geometry

Modern embedding models (BERT, GPT, sentence-transformers) map text into high-dimensional vector spaces (typically 384 to 4,096 dimensions). In these spaces, semantic relationships manifest as geometric patterns:

Synonyms: Near-parallel vectors (cosine similarity > 0.90)
Related concepts: Acute angle (cosine similarity 0.60-0.90)
Unrelated content: Near-orthogonal (cosine similarity 0.10-0.40)
Antonyms: Can be surprisingly similar (cosine similarity 0.40-0.70) because antonyms share topical context

The curse of dimensionality introduces a subtle effect in high-dimensional spaces: randomly sampled vectors tend to become nearly orthogonal. In a 768-dimensional space, random vector pairs produce cosine similarities clustered around 0. This concentration of measure means that even modest similarity scores (0.3-0.4) can indicate meaningful relationships, while truly similar items typically score above 0.7.

Comparison with Alternative Metrics

Metric	Formula	Use Case	Advantage	Disadvantage
Cosine similarity	dot(A,B) / (norm(A)*norm(B))	Text similarity, recommendations	Magnitude-invariant	Not a true metric
Euclidean distance	sqrt(sum((A-B)^2))	Spatial proximity, clustering	True metric, intuitive	Sensitive to magnitude
Manhattan distance	sum(abs(A-B))	High-dimensional sparse data	Robust to outliers	Less geometric intuition
Dot product	sum(A*B)	Maximum inner product search	Fastest computation	Not normalized
Jaccard similarity	intersection/union	Set overlap	Works with binary features	Ignores frequency/magnitude

Efficient Computation Strategies

For large-scale similarity search (millions of vectors), brute-force pairwise comparison is prohibitive. Approximate Nearest Neighbor (ANN) algorithms trade exactness for speed:

HNSW (Hierarchical Navigable Small World): Graph-based index with O(log n) query time. Used by Meilisearch and Qdrant.
IVF (Inverted File Index): Partitions vectors into clusters, searching only relevant partitions. Common in FAISS.
Product Quantization: Compresses vectors while preserving approximate cosine similarity. Reduces memory by 8-64x.
LSH (Locality-Sensitive Hashing): Hashes similar vectors to the same bucket with high probability.

Architecture and Implementation

Vector Similarity Pipeline

A production similarity system follows a pipeline architecture:

Input Text
    |
    v
Embedding Model (Ollama/sentence-transformers)
    |
    v
Vector (float32[768])
    |
    v
Normalization (L2 norm -> unit vector)
    |
    v
Index Insertion / Query
    |
    v
Top-K Results by Cosine Similarity
    |
    v
Re-ranking (optional cross-encoder)
    |
    v
Final Results

The normalization step is critical: by pre-normalizing all vectors to unit length at insertion time, runtime similarity computation reduces to a simple dot product, eliminating the expensive norm computations during query time. This optimization is standard practice in production vector search systems.

Threshold Calibration

Selecting appropriate similarity thresholds is domain-specific and requires empirical calibration:

Application	High Confidence	Medium Confidence	Low Confidence
Duplicate detection	> 0.95	0.90 - 0.95	0.85 - 0.90
Entity resolution	> 0.85	0.75 - 0.85	0.65 - 0.75
Semantic search	> 0.75	0.60 - 0.75	0.45 - 0.60
Topic clustering	> 0.70	0.55 - 0.70	0.40 - 0.55
Recommendation	> 0.60	0.45 - 0.60	0.30 - 0.45

These thresholds vary significantly based on the embedding model, domain vocabulary, and error tolerance. Production systems typically establish thresholds through evaluation against labeled datasets with precision-recall analysis.

Usage in Prismatic Platform

The Prismatic Platform uses cosine similarity as the primary semantic comparison metric across multiple subsystems, from OSINT entity resolution to intelligent agent routing.

Meilisearch Vector Search

The Meilisearch integration uses cosine similarity for semantic ranking of search results alongside traditional keyword matching:

defmodule PrismaticSearch.VectorQuery do
  @moduledoc """
  Vector similarity search using Meilisearch hybrid mode.
  """

  @similarity_threshold 0.65

  @spec semantic_search(String.t(), keyword()) :: {:ok, list(map())} | {:error, term()}
  def semantic_search(query_text, opts \\ []) do
    threshold = Keyword.get(opts, :threshold, @similarity_threshold)
    limit = Keyword.get(opts, :limit, 20)

    with {:ok, embedding} <- generate_embedding(query_text),
         {:ok, results} <- search_by_vector(embedding, limit) do
      filtered =
        results
        |> Enum.filter(fn result -> result.similarity_score >= threshold end)
        |> Enum.sort_by(& &1.similarity_score, :desc)

      {:ok, filtered}
    end
  end

  defp generate_embedding(text) do
    PrismaticOllama.embed(text, model: "nomic-embed-text")
  end

  defp search_by_vector(embedding, limit) do
    PrismaticStorageMeilisearch.vector_search(
      index: "documents",
      vector: embedding,
      limit: limit,
      hybrid: %{semantic_ratio: 0.7}
    )
  end
end

OSINT Entity Resolution

The entity resolution pipeline computes pairwise cosine similarity between entity name embeddings to identify potential matches across disparate data sources:

defmodule PrismaticOsint.EntityResolver do
  @moduledoc """
  Resolves entity identities across OSINT sources using embedding similarity.
  """

  @match_threshold 0.85
  @review_threshold 0.70

  @spec resolve(map(), list(map())) :: {:ok, list(match_result())}
  def resolve(source_entity, candidate_entities) do
    {:ok, source_embedding} = embed_entity(source_entity)

    matches =
      candidate_entities
      |> Enum.map(fn candidate ->
        {:ok, candidate_embedding} = embed_entity(candidate)
        similarity = cosine_similarity(source_embedding, candidate_embedding)

        %{
          candidate: candidate,
          similarity: similarity,
          confidence: classify_confidence(similarity)
        }
      end)
      |> Enum.filter(fn match -> match.similarity >= @review_threshold end)
      |> Enum.sort_by(& &1.similarity, :desc)

    {:ok, matches}
  end

  defp cosine_similarity(vec_a, vec_b) do
    dot_product = Enum.zip(vec_a, vec_b) |> Enum.reduce(0.0, fn {a, b}, acc -> acc + a * b end)
    norm_a = :math.sqrt(Enum.reduce(vec_a, 0.0, fn x, acc -> acc + x * x end))
    norm_b = :math.sqrt(Enum.reduce(vec_b, 0.0, fn x, acc -> acc + x * x end))

    case norm_a * norm_b do
      0.0 -> 0.0
      denominator -> dot_product / denominator
    end
  end

  defp classify_confidence(similarity) when similarity >= @match_threshold, do: :high
  defp classify_confidence(similarity) when similarity >= @review_threshold, do: :medium
  defp classify_confidence(_similarity), do: :low

  defp embed_entity(entity) do
    text = "#{entity.name} #{entity.description} #{entity.aliases |> Enum.join(" ")}"
    PrismaticOllama.embed(text, model: "nomic-embed-text")
  end
end

Agent Routing

The AIAD agent system uses cosine similarity to match user queries to the most relevant agent among the 434 available agents:

defmodule PrismaticAgents.Router do
  @moduledoc """
  Routes user queries to the most capable agent using semantic similarity.
  """

  @routing_threshold 0.55

  @spec route_query(String.t()) :: {:ok, atom()} | {:error, :no_matching_agent}
  def route_query(query) do
    {:ok, query_embedding} = PrismaticOllama.embed(query)
    agent_embeddings = PrismaticAgents.Registry.all_embeddings()

    best_match =
      agent_embeddings
      |> Enum.map(fn {agent_id, embedding} ->
        {agent_id, cosine_similarity(query_embedding, embedding)}
      end)
      |> Enum.max_by(fn {_id, score} -> score end)

    case best_match do
      {agent_id, score} when score >= @routing_threshold -> {:ok, agent_id}
      _ -> {:error, :no_matching_agent}
    end
  end
end

Best Practices

Always normalize vectors at insertion time. Pre-normalizing to unit length converts cosine similarity to a simple dot product at query time, reducing computational cost by approximately 40%. Store both the original and normalized vectors if magnitude information is needed for other purposes.

Calibrate thresholds empirically. Never use arbitrary threshold values. Build an evaluation dataset with known positive and negative pairs, compute precision-recall curves at different thresholds, and select the operating point that matches your application's error tolerance. Document the evaluation methodology and revisit thresholds when switching embedding models.

Use approximate nearest neighbor for scale. Exact cosine similarity computation is O(n*d) per query against n vectors of dimension d. For collections exceeding 100,000 vectors, use ANN indexes (HNSW, IVF) that provide 95-99% recall at 10-100x speedup. The recall-speed tradeoff is tunable via index parameters.

Consider hybrid retrieval. Combining cosine similarity (semantic) with BM25 (keyword) retrieval consistently outperforms either method alone. The semantic component captures meaning while the keyword component handles exact terms, acronyms, and domain-specific jargon that embedding models may not encode well.

Monitor embedding model drift. When the embedding model is updated or replaced, all precomputed vectors become incompatible. Implement versioned vector storage and re-embedding pipelines to handle model transitions without service disruption.

Common Pitfalls

Comparing embeddings from different models. Vectors from different embedding models occupy different vector spaces -- their cosine similarity is meaningless. Ensure all vectors in a comparison use the same model and version.

Ignoring the magnitude information. While cosine similarity intentionally discards magnitude, sometimes magnitude carries signal. In TF-IDF vectors, longer documents have larger magnitudes, which may be relevant for ranking. Consider whether magnitude invariance is actually desired for your use case.

Assuming linearity of similarity scores. The relationship between cosine similarity values and human-perceived similarity is not linear. The difference between 0.90 and 0.95 is often perceptually much larger than between 0.50 and 0.55. Avoid treating similarity scores as linear percentages.

Neglecting dimensionality effects. In very high-dimensional spaces (>1000 dimensions), most random vector pairs have cosine similarity near zero. This concentration means that small absolute differences in similarity (e.g., 0.25 vs 0.30) can represent meaningful distinctions, contrary to intuition from low-dimensional geometry.

Using cosine similarity for sparse vectors. For extremely sparse vectors (e.g., one-hot encodings, bag-of-words with large vocabularies), most dot product terms are zero, making cosine similarity degenerate. Use Jaccard similarity or specialized sparse metrics instead.

Meilisearch -- Search engine applying cosine similarity for semantic ranking
Entity Resolution -- Identity matching using cosine similarity between embeddings
Knowledge Graph -- Graph enrichment driven by cosine similarity between node vectors
Ollama -- AI runtime producing embeddings compared via cosine similarity
OSINT -- Intelligence discipline using cosine similarity for entity correlation
Vector Database -- Storage system using cosine similarity for nearest-neighbor search
Prompt Engineering -- Techniques for optimizing embedding quality through input design

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

GitHub | GitLab | LinkedIn | Contact
Developer Portal | Architecture | Meet the Creator

Cosine Similarity

Definition and Overview

Technical Deep Dive

Mathematical Properties

Embedding Space Geometry

Comparison with Alternative Metrics

Efficient Computation Strategies

Architecture and Implementation

Vector Similarity Pipeline

Threshold Calibration

Usage in Prismatic Platform

Meilisearch Vector Search

OSINT Entity Resolution

Agent Routing

Best Practices

Common Pitfalls

See Also

Connect & Contribute

Glossary

Related Terms

Cosine Similarity

Definition and Overview

Technical Deep Dive

Mathematical Properties

Embedding Space Geometry

Comparison with Alternative Metrics

Efficient Computation Strategies

Architecture and Implementation

Vector Similarity Pipeline

Threshold Calibration

Usage in Prismatic Platform

Meilisearch Vector Search

OSINT Entity Resolution

Agent Routing

Best Practices

Common Pitfalls

Related Concepts

See Also

Connect & Contribute

Glossary

Related Terms