Prismatic Meilisearch

Overview

Prismatic Meilisearch integrates the Meilisearch full-text search engine into the Prismatic Platform, providing lightning-fast, typo-tolerant search across all indexed data. Meilisearch delivers sub-50ms search responses with built-in relevancy ranking, faceted search, and filtering -- essential for searching across the platform's vast OSINT intelligence data, entity records, and agent outputs. The search engine handles over 500,000 documents across six specialized indexes, supporting query patterns that range from simple keyword searches to complex faceted analytics with multi-attribute filtering.

As part of the Prismatic Storage adapter ecosystem, Meilisearch is accessed through the Indexable and Queryable traits, allowing transparent search operations across the platform. The actual application is prismatic_storage_meilisearch, implementing the storage adapter protocol for consistent access patterns alongside ETS, Ecto, KuzuDB, and DuckDB adapters. This uniform interface means that application code never interacts with Meilisearch's HTTP API directly -- all queries flow through the trait system, enabling future backend substitution without consumer changes.

The integration goes beyond simple search indexing. Meilisearch serves as the platform's primary discovery layer for human-facing interfaces: when an analyst searches for an entity by name in any LiveView dashboard, the query routes through this adapter. Typo tolerance ensures that misspelled company names, transliterated personal names, and approximate domain matches still return relevant results -- a critical capability when dealing with international intelligence data where name spellings vary across sources and languages.

Architecture

PrismaticStorageMeilisearch.Application
+-- PrismaticStorageMeilisearch.Supervisor (:one_for_one)
    +-- PrismaticStorageMeilisearch.Client (GenServer)
    |   +-- HTTP client for Meilisearch API (port 7700)
    +-- PrismaticStorageMeilisearch.IndexManager (GenServer)
    |   +-- Index lifecycle: create, configure, delete
    +-- PrismaticStorageMeilisearch.SearchEngine (GenServer)
    |   +-- Query builder with facets, filters, sorting
    +-- PrismaticStorageMeilisearch.Sync (GenServer)
    |   +-- Data synchronization from other storage adapters
    +-- PrismaticStorageMeilisearch.Adapter
        +-- Indexable + Queryable trait implementation

The architecture follows a supervised GenServer pattern where each concern -- HTTP communication, index management, query building, and data synchronization -- runs as an independent process under OTP supervision. This isolation ensures that a failure in synchronization cannot affect search query processing, and that index management operations do not block ongoing searches.

Application Code --> PrismaticStorage.search(:meilisearch_adapter, query)
                        |
                   PrismaticStorageMeilisearch.Adapter (Indexable trait)
                        |
                   Meilisearch Engine (port 7700)
                        |
                   Search Results --> Application Code

Key Modules

Key Features

Typo-Tolerant Search

Meilisearch's built-in typo tolerance uses a prefix-tree-based algorithm that automatically handles misspellings up to two characters for words of eight or more characters, and one character for words of four or more. This is configurable per index:

defmodule PrismaticStorageMeilisearch.SearchEngine do
  @spec search(String.t(), String.t(), keyword()) :: {:ok, map()} | {:error, term()}
  def search(index, query, opts \\ []) do
    params = %{
      q: query,
      limit: Keyword.get(opts, :limit, 20),
      offset: Keyword.get(opts, :offset, 0),
      filter: Keyword.get(opts, :filter),
      facets: Keyword.get(opts, :facets),
      sort: Keyword.get(opts, :sort),
      matchingStrategy: Keyword.get(opts, :strategy, "last"),
      attributesToHighlight: Keyword.get(opts, :highlight, ["*"])
    }

    case Client.post("/indexes/#{index}/search", params) do
      {:ok, %{status: 200, body: body}} ->
        {:ok, %{
          hits: body["hits"],
          estimated_total: body["estimatedTotalHits"],
          processing_time_ms: body["processingTimeMs"],
          facet_distribution: body["facetDistribution"]
        }}

      {:error, reason} ->
        {:error, {:search_failed, reason}}
    end
  end
end

Multi-Index Search

The multi-search capability enables simultaneous queries across multiple indexes, returning unified results. This is critical for entity searches where a name might match companies, persons, or domains:

# Multi-index search implementation
def multi_search(queries) when is_list(queries) do
  payload = %{queries: Enum.map(queries, &build_search_params/1)}

  case Client.post("/multi-search", payload) do
    {:ok, %{status: 200, body: %{"results" => results}}} ->
      {:ok, Enum.zip(queries, results) |> Enum.into(%{}, fn {q, r} ->
        {q.index, %{hits: r["hits"], total: r["estimatedTotalHits"]}}
      end)}

    {:error, reason} ->
      {:error, {:multi_search_failed, reason}}
  end
end

Index Synchronization

The Sync module maintains consistency between the platform's primary storage backends (PostgreSQL, ETS) and Meilisearch indexes through a configurable synchronization pipeline:

Faceted Search and Filtering

The platform uses Meilisearch's faceted search extensively for intelligence analysis interfaces, enabling analysts to drill down through large result sets by source, severity, category, and time period without issuing new queries.

Configuration

config :prismatic_storage_meilisearch,
  # Meilisearch connection
  url: System.get_env("MEILISEARCH_URL", "http://localhost:7700"),
  api_key: System.get_env("MEILISEARCH_API_KEY"),

  # Index defaults
  default_ranking_rules: ["words", "typo", "proximity", "attribute", "sort", "exactness"],
  typo_tolerance: %{enabled: true, min_word_size_for_typos: %{one_typo: 4, two_typos: 8}},

  # Sync settings
  sync_batch_size: 1000,
  sync_interval_ms: 60_000,

  # HTTP client
  http_timeout_ms: 5_000,
  http_pool_size: 10,

  # Health check
  health_check_interval_ms: 30_000,
  health_check_timeout_ms: 2_000

Indexed Data

Usage

# Search across all indexed documents
{:ok, results} = PrismaticMeilisearch.search("companies", "Prismatic s.r.o.",
  limit: 20,
  offset: 0,
  filter: "country = 'CZ' AND status = 'active'",
  sort: ["relevancy:desc", "created_at:desc"]
)
# => %{hits: [...], estimated_total_hits: 3, processing_time_ms: 12}

# Typo-tolerant search (finds "Prismtic" -> "Prismatic")
{:ok, results} = PrismaticMeilisearch.search("companies", "Prismtic")

# Faceted search with filters
{:ok, results} = PrismaticMeilisearch.search("osint_reports",
  "vulnerability assessment",
  facets: ["source", "severity", "category"],
  filter: "severity IN ['critical', 'high']"
)

# Multi-index search
{:ok, results} = PrismaticMeilisearch.multi_search([
  %{index: "companies", query: "Novak"},
  %{index: "persons", query: "Novak"},
  %{index: "domains", query: "novak"}
])

# Create an index with configuration
{:ok, index} = PrismaticMeilisearch.create_index("companies",
  primary_key: "ico",
  searchable_attributes: ["name", "address", "business_activities"],
  filterable_attributes: ["country", "legal_form", "status"],
  sortable_attributes: ["name", "created_at"]
)

# Add documents in bulk
{:ok, task} = PrismaticMeilisearch.add_documents("companies", company_list)

# Get index statistics
{:ok, stats} = PrismaticMeilisearch.stats("companies")
# => %{number_of_documents: 45_230, is_indexing: false, field_distribution: %{...}}

NABLA Compliance

Meilisearch serves as a retrieval layer rather than a signal source. NABLA compliance is enforced at the indexing stage by upstream components that populate search indices with provenance-tagged documents. The adapter ensures that provenance metadata is preserved through the indexing pipeline and returned with search results.

Testing

Search accuracy tests verify typo tolerance thresholds, faceted filtering correctness, multi-index query result merging, and relevancy ranking stability across document distributions. Adapter contract tests verify trait compliance against the PrismaticStorage.AdapterContractTest suite, ensuring consistent behavior with other storage backends.

Synchronization tests exercise the batch, real-time, and delta sync strategies against a running Meilisearch instance, verifying document consistency after concurrent update operations. Performance tests validate sub-50ms search latency under load with realistic document volumes.

# Run all Meilisearch adapter tests
cd apps/prismatic_storage_meilisearch && mix test

# Run with coverage
mix test --cover

# Run search engine tests (requires running Meilisearch instance)
mix test test/prismatic_storage_meilisearch/search_engine_test.exs

# Run adapter contract tests
mix test test/prismatic_storage_meilisearch/adapter_test.exs

Integration Points

Performance

Telemetry events: [:prismatic, :meilisearch, :search], [:prismatic, :meilisearch, :index], [:prismatic, :meilisearch, :sync].

Related Resources

• Prismatic Storage Core -- Storage adapter trait system defining Indexable and Queryable contracts
• Prismatic Storage ETS -- In-memory storage complement for sub-millisecond lookups
• Prismatic Storage KuzuDB -- Graph storage for relationship queries complementing text search
• Consolidation Architect -- Data deduplication across indexes preventing duplicate search results
• Cross-Domain Flexibility -- Search spanning all data domains through multi-index queries
• Real-Time Monitoring -- Search latency and index health monitoring through telemetry
• Intelligence Synthesis -- Multi-source data fusion populating search indexes

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator