AI Model

technology

AI Model

Definition

An AI model is a mathematical construct, typically parameterized by millions to trillions of weights, that has been trained on data to perform specific tasks such as prediction, generation, classification, or reasoning. In the context of modern software platforms, AI models serve as the computational engines that transform raw inputs into structured, actionable outputs. Within the Prismatic Platform, AI models operate through both local inference via Ollama and remote API calls to Anthropic's Claude, providing a dual-track architecture for intelligence-driven operations across 530+ autonomous agents.

Overview

The concept of a trained mathematical model predating modern neural networks stretches back to the earliest days of statistics and operations research. Linear regression models from the 19th century, decision trees from the 1960s, and support vector machines from the 1990s all represent forms of AI models in their broadest sense. However, the contemporary meaning of "AI model" has been profoundly shaped by the deep learning revolution beginning around 2012, when convolutional neural networks demonstrated superhuman performance on image classification tasks.

The landscape shifted dramatically with the introduction of the Transformer architecture in 2017 (Vaswani et al., "Attention Is All You Need"), which enabled the creation of large language models (LLMs) capable of understanding and generating human language with remarkable fluency. Models such as GPT, Claude, LLaMA, Qwen, and DeepSeek represent the current frontier, with parameter counts ranging from 1 billion to over 1 trillion.

AI models can be categorized along several axes:

AxisCategoriesExamples
ArchitectureTransformer, CNN, RNN, GAN, DiffusionClaude (Transformer), ResNet (CNN)
ModalityText, Image, Audio, MultimodalGPT-4 (multimodal), Whisper (audio)
Training ParadigmSupervised, Unsupervised, Reinforcement, Self-supervisedRLHF (reinforcement), BERT (self-supervised)
SizeSmall (<1B), Medium (1-10B), Large (10-100B), Frontier (100B+)Qwen3 7B (medium), Claude Opus (frontier)
DeploymentCloud API, Local inference, Edge, EmbeddedClaude API (cloud), Ollama (local)
Task TypeGenerative, Discriminative, Embedding, ReasoningClaude (generative), BERT (discriminative)

The significance of AI models in modern software engineering cannot be overstated. They have moved from research curiosities to production infrastructure components that drive core business logic, automate complex decision-making, and enable capabilities that were previously impossible through traditional programming alone.

Technical Details

Model Architecture Fundamentals

Modern AI models, particularly large language models, are built on the Transformer architecture. The core mechanism is self-attention, which allows the model to weigh the relevance of different parts of an input sequence when producing each output element.

The fundamental computation in a Transformer layer involves:

  1. Multi-Head Attention: Parallel attention computations that capture different aspects of token relationships
  2. Feed-Forward Networks: Position-wise dense layers that transform attention outputs
  3. Layer Normalization: Stabilization of activations between layers
  4. Residual Connections: Skip connections that enable training of very deep networks

Model Serving and Inference

Serving AI models in production involves several critical considerations:

ConcernDescriptionPrismatic Approach
LatencyTime from request to first token<3s local (Ollama), <5s cloud (Claude)
ThroughputRequests per second capacityBatched inference, connection pooling
MemoryGPU/CPU RAM requirements7B models <8GB, quantization for efficiency
CostPer-token or per-request pricingLocal models for high-volume, cloud for quality
ReliabilityUptime and error handlingCircuit breaker + automatic fallback chain
QualityOutput accuracy and coherenceModel selection based on task complexity

Quantization and Optimization

For local deployment, models undergo quantization to reduce memory footprint while preserving quality:

Model Size Comparison (Qwen3 7B):
- FP32 (full precision):  ~28 GB
- FP16 (half precision):  ~14 GB
- INT8 (8-bit quantized): ~7 GB
- INT4 (4-bit quantized): ~3.5 GB

Token Economics

AI models process text as tokens (subword units), and understanding token budgets is essential for cost management and context window optimization:

ModelContext WindowInput CostOutput Cost
Claude Opus200K tokens$$$$$$
Claude Sonnet200K tokens$$$$
Qwen3 7B (local)32K tokensFreeFree
DeepSeek Coder 6.7B (local)16K tokensFreeFree

Implementation in Prismatic Platform

The Prismatic Platform implements a sophisticated dual-track AI model infrastructure that combines local Ollama models for high-volume, low-latency operations with cloud-based Claude API calls for tasks requiring frontier-level reasoning.

Ollama Local Model Integration

defmodule PrismaticOllama.ModelManager do
  @moduledoc """
  Manages local AI model lifecycle including loading, inference, and health monitoring.
  Supports automatic model selection based on task requirements and available resources.
  """

  @spec generate(String.t(), String.t(), keyword()) ::
          {:ok, String.t()} | {:error, term()}
  def generate(model, prompt, opts \\ []) do
    timeout = Keyword.get(opts, :timeout, 30_000)
    temperature = Keyword.get(opts, :temperature, 0.7)

    case check_model_availability(model) do
      {:ok, _} ->
        do_generate(model, prompt, temperature, timeout)

      {:error, :not_loaded} ->
        with {:ok, _} <- pull_model(model),
             {:ok, response} <- do_generate(model, prompt, temperature, timeout) do
          {:ok, response}
        end
    end
  end

  @spec list_available_models() :: {:ok, [map()]} | {:error, term()}
  def list_available_models do
    case Ollama.API.list() do
      {:ok, %{models: models}} ->
        {:ok, Enum.map(models, &format_model_info/1)}

      {:error, reason} ->
        {:error, {:ollama_unavailable, reason}}
    end
  end
end

Cloud API Fallback Chain

The platform implements an automatic fallback chain that routes requests to progressively more capable (and expensive) models based on task complexity and local model availability:

defmodule PrismaticClaude.FallbackChain do
  @moduledoc """
  Implements automatic model fallback with circuit breaker protection.
  Routes to local models first, falling back to cloud API when needed.
  """

  @fallback_order [
    {:local, "qwen3-coder"},
    {:local, "deepseek-coder"},
    {:cloud, "claude-sonnet"},
    {:cloud, "claude-opus"}
  ]

  @spec complete(String.t(), keyword()) ::
          {:ok, map()} | {:error, :all_models_exhausted}
  def complete(prompt, opts \\ []) do
    Enum.reduce_while(@fallback_order, {:error, :all_models_exhausted}, fn
      {provider, model}, _acc ->
        case invoke_model(provider, model, prompt, opts) do
          {:ok, response} -> {:halt, {:ok, response}}
          {:error, _reason} -> {:cont, {:error, :all_models_exhausted}}
        end
    end)
  end
end

Agent-Model Integration

Each of the 530+ AIAD agents in the platform can specify model preferences based on their operational requirements:

# Example agent model configuration
agent:
  name: "code-analyzer"
  model_preferences:
    primary: "qwen3-coder"        # Fast local model for code analysis
    fallback: "claude-sonnet"      # Cloud fallback for complex cases
    max_tokens: 4096
    temperature: 0.3               # Low temperature for deterministic analysis
    timeout_ms: 30000

Performance Metrics

The platform tracks detailed telemetry for all model interactions:

MetricLocal (Ollama)Cloud (Claude)
P50 Latency1.2s2.8s
P95 Latency2.8s6.5s
P99 Latency4.5s12.0s
Availability99.2%99.9%
Error Rate0.8%0.1%
Memory Usage<8GB per modelN/A

Comparison with Alternatives

ApproachLatencyCostPrivacyQualityOfflineCustomization
Local Ollama (Prismatic)LowFreeFullGoodYesHigh
Cloud API (Prismatic)MediumPer-tokenLimitedExcellentNoMedium
OpenAI APIMediumPer-tokenLimitedExcellentNoMedium
AWS BedrockMediumPer-tokenAWS-scopedGoodNoLow
Self-hosted vLLMLowInfrastructureFullGoodYesHigh
Hugging Face InferenceVariableFree/PaidSharedVariableNoHigh
Google Vertex AIMediumPer-tokenGCP-scopedGoodNoMedium

The Prismatic approach of combining local and cloud models provides the best balance of latency, cost, privacy, and quality. Local models handle routine operations at zero marginal cost, while frontier cloud models handle complex reasoning tasks that justify their per-token pricing.

Best Practices

  1. Select models based on task complexity: Use small local models (7B parameters) for routine tasks like code formatting, classification, and extraction. Reserve frontier models for complex reasoning, creative generation, and multi-step analysis. This optimization reduces costs by 80-90% without sacrificing quality where it matters.

  2. Implement circuit breakers for model calls: Wrap all model inference calls in circuit breaker patterns that automatically fail open after consecutive failures. This prevents cascading failures when a model endpoint becomes unavailable and enables automatic recovery when service is restored.

  3. Set explicit token budgets per agent: Define maximum input and output token counts for each agent's model interactions. This prevents runaway costs from overly verbose prompts or responses and ensures predictable resource consumption across the platform.

  4. Version-pin model deployments: Always specify exact model versions (e.g., qwen3-coder:7b-q4_K_M) rather than floating tags. Model updates can introduce behavioral changes that break downstream expectations, and version pinning ensures reproducibility.

  5. Monitor inference quality continuously: Implement automated quality checks on model outputs using structured evaluation criteria. Track quality metrics over time to detect model degradation, prompt drift, or changes in model behavior after updates.

  6. Use structured output formats: Request JSON, YAML, or other structured formats from models rather than free-form text. This reduces parsing errors, enables automated validation, and improves downstream processing reliability.

  7. Implement graceful degradation: Design systems so that model unavailability degrades functionality rather than causing failures. Cached responses, fallback heuristics, and human-in-the-loop escalation ensure continuous operation.

Common Pitfalls

  • Over-reliance on a single model provider: Coupling your entire system to one model provider creates a single point of failure and vendor lock-in. The Prismatic dual-track approach (local + cloud) mitigates this risk, but teams should still design for model-agnostic interfaces.

  • Ignoring token context window limits: Sending prompts that exceed a model's context window results in silent truncation or errors. Always calculate prompt token counts before submission and implement chunking strategies for large inputs.

  • Treating model outputs as ground truth: AI models generate plausible outputs that may contain factual errors, logical inconsistencies, or hallucinated information. All model outputs in production systems must pass through validation layers before being acted upon.

  • Neglecting prompt engineering as a discipline: The quality of model outputs is directly proportional to the quality of prompts. Ad-hoc prompting leads to inconsistent results. Invest in prompt templates, few-shot examples, and systematic prompt evaluation.

  • Underestimating local model resource requirements: Running multiple local models simultaneously can exhaust GPU memory and CPU resources. Profile actual resource consumption under load and implement model scheduling to prevent resource contention.

  • Skipping evaluation before deployment: Deploying models or prompts without systematic evaluation against a representative test set leads to production surprises. Maintain evaluation datasets and run benchmarks before any model change reaches production.

Use Cases

Autonomous Agent Reasoning

The 530+ AIAD agents in the Prismatic Platform use AI models as their core reasoning engine. Each agent receives structured context about its domain, applies model-driven reasoning to analyze situations, and produces actionable outputs. For example, the code-analyzer agent uses local Qwen3 models to perform rapid code quality assessment, while the archer-supreme orchestrator uses Claude Opus for complex multi-step strategic planning.

OSINT Intelligence Analysis

The platform's 120 OSINT tools generate raw intelligence data from diverse sources (ARES, Shodan, VirusTotal, etc.). AI models synthesize this raw data into coherent intelligence reports, identify patterns across disparate data sources, and assess confidence levels for analytical conclusions. The model's ability to reason about incomplete or contradictory information makes it essential for the NABLA Infinity epistemic framework.

Automated Code Generation and Review

AI models power the platform's code generation capabilities, producing Elixir modules, test suites, and documentation that conform to the platform's strict quality standards. The model understands the umbrella architecture, OTP conventions, and the NO MERCY/NO DOUBTS doctrine, generating code that passes all 13 quality domains on first submission.

Security Rating Computation

In the Prismatic Perimeter EASM module, AI models assist with interpreting security scan results, correlating vulnerability data across assets, and generating human-readable security assessment narratives. The models translate technical findings into A-F security ratings with supporting evidence chains that satisfy NIS2 and ZKB compliance auditors.

  • LLM - Large Language Models are the most prominent category of AI models in modern platforms, specializing in text understanding and generation
  • Ollama - Local model serving infrastructure that enables privacy-preserving AI inference without cloud dependencies
  • Fine-Tuning - The process of adapting a pre-trained AI model to specific tasks or domains through additional training
  • Embedding - Dense vector representations produced by AI models that capture semantic meaning for similarity search and clustering
  • AI Inference - The runtime execution of a trained AI model to produce predictions or generations from new inputs
  • Prompt Engineering - The discipline of crafting effective inputs to maximize AI model output quality and reliability
  • Neural Network - The underlying computational architecture that powers most modern AI models
  • Machine Learning - The broader field encompassing the training and application of AI models across diverse problem domains

See Also

Connect & Contribute

Created by Tomas Korcak (korczis) | Open Source under GHL

Back to Glossary