Ollama

Overview

Ollama is the local AI inference engine that enables the Prismatic Platform to run AI models entirely on-premises, ensuring complete data privacy and eliminating cloud API dependencies for routine AI operations. It provides a simple interface for downloading, running, and managing large language models on local hardware, with GPU acceleration support for both NVIDIA CUDA and Apple Metal. Ollama transforms what would otherwise require complex model serving infrastructure into a single-binary solution.

The Prismatic Platform integrates Ollama as a first-class AI provider, running models like qwen3-coder (7B), gpt-oss (20B), and deepseek-coder (6.7B) for code generation, analysis, and natural language processing tasks. Ollama's response times of under 3 seconds for 7B models and under 5 seconds for 20B models meet the platform's interactive usage requirements. These response times are achieved on consumer-grade hardware (Apple M-series with 16GB+ RAM or NVIDIA GPUs with 8GB+ VRAM), making local AI inference accessible without datacenter hardware.

Ollama operates as the primary AI backend for development and testing, with automatic fallback to cloud providers (Claude, OpenAI) when higher capability is needed. This hybrid approach maximizes privacy while maintaining access to frontier model capabilities. Sensitive data -- proprietary code, security findings, OSINT intelligence -- can be processed locally through Ollama without ever leaving the developer's machine.

Key Features

• Local Execution: Complete data privacy with no cloud dependencies or network requirements
• Model Library: Access to Llama, Mistral, CodeLlama, Qwen, DeepSeek, Phi, and dozens of other open-source models
• GPU Acceleration: NVIDIA CUDA and Apple Metal support for fast inference on consumer hardware
• REST API: Simple HTTP interface at localhost:11434 compatible with OpenAI API format for drop-in replacement
• Model Management: Pull, list, copy, customize, and remove models with simple CLI commands
• Concurrent Inference: Multiple model serving and request queuing for parallel AI operations
• Modelfile Customization: Custom system prompts, parameters, and model configurations through Modelfile definitions
• Memory Efficiency: Automatic model loading/unloading based on available memory and usage patterns

Platform Integration

Ollama serves as the default local AI provider for the platform through a unified provider abstraction.

defmodule PrismaticClaude.Providers.Ollama do
  @moduledoc """
  Local AI provider using Ollama for privacy-first inference.
  Implements the unified provider behaviour for seamless
  integration with the multi-provider AI architecture.
  """
  @behaviour PrismaticClaude.Providers.Behaviour

  @base_url "http://localhost:11434"

  @impl true
  def generate(prompt, opts \\ []) do
    model = Keyword.get(opts, :model, "qwen3-coder")
    body = %{
      model: model,
      prompt: prompt,
      stream: Keyword.get(opts, :stream, false),
      options: %{
        temperature: Keyword.get(opts, :temperature, 0.7),
        num_predict: Keyword.get(opts, :max_tokens, 2048),
        top_p: Keyword.get(opts, :top_p, 0.9)
      }
    }

    case HTTPoison.post("#{@base_url}/api/generate", Jason.encode!(body),
           [{"Content-Type", "application/json"}],
           recv_timeout: 60_000) do
      {:ok, %{status_code: 200, body: resp_body}} ->
        %{"response" => response} = Jason.decode!(resp_body)
        {:ok, response}

      {:ok, %{status_code: status}} ->
        {:error, {:http_error, status}}

      {:error, %HTTPoison.Error{reason: :timeout}} ->
        {:error, :timeout}

      {:error, reason} ->
        {:error, reason}
    end
  end

  @impl true
  def chat(messages, opts \\ []) do
    model = Keyword.get(opts, :model, "qwen3-coder")
    body = %{
      model: model,
      messages: messages,
      stream: false,
      options: %{
        temperature: Keyword.get(opts, :temperature, 0.7),
        num_predict: Keyword.get(opts, :max_tokens, 2048)
      }
    }

    case HTTPoison.post("#{@base_url}/api/chat", Jason.encode!(body),
           [{"Content-Type", "application/json"}],
           recv_timeout: 60_000) do
      {:ok, %{status_code: 200, body: resp_body}} ->
        %{"message" => %{"content" => content}} = Jason.decode!(resp_body)
        {:ok, content}

      {:error, reason} ->
        {:error, reason}
    end
  end

  @impl true
  def available? do
    case HTTPoison.get("#{@base_url}/api/tags", [], recv_timeout: 5_000) do
      {:ok, %{status_code: 200}} -> true
      _ -> false
    end
  end
end

Architecture

Ollama integrates into the platform's multi-provider AI architecture as the local inference option.

The provider selection follows a priority-based routing strategy.

Model Configuration

The platform standardizes on specific models for different task types, balancing capability against resource usage.

Custom Modelfiles enable task-specific model configurations.

# Modelfile for Prismatic code analysis
FROM qwen3-coder

SYSTEM """
You are an expert Elixir developer working on the Prismatic Platform.
Follow OTP conventions, use GenServer patterns, and ensure all code
includes @spec annotations and @moduledoc documentation.
Never use Manager/Handler/Utils naming patterns.
"""

PARAMETER temperature 0.3
PARAMETER num_predict 4096
PARAMETER top_p 0.9

Performance Characteristics

Ollama performance depends on the available hardware and selected model.

Configuration

Ollama integration is configured through environment variables and the platform's configuration system.

# Ollama environment configuration
export OLLAMA_HOST="http://localhost:11434"
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=2

# Model management
ollama pull qwen3-coder
ollama pull deepseek-coder:6.7b
ollama pull nomic-embed-text
ollama list

# Elixir configuration for Ollama integration
config :prismatic_claude, :ollama,
  base_url: System.get_env("OLLAMA_HOST", "http://localhost:11434"),
  default_model: "qwen3-coder",
  embedding_model: "nomic-embed-text",
  timeout: 60_000,
  max_retries: 3,
  fallback_to_cloud: true

# Claude Code integration
config :prismatic_claude, :claude_code,
  auth_token: "ollama",
  base_url: "http://localhost:11434"

Best Practices

• Use 7B models for routine tasks -- 7B parameter models provide the best balance of speed and quality for code generation and analysis
• Reserve 20B models for complex reasoning -- larger models are significantly slower; use them only when smaller models produce insufficient results
• Configure OLLAMA_MAX_LOADED_MODELS -- limit concurrent model loading to prevent memory exhaustion on constrained hardware
• Use streaming for interactive use -- enable stream: true for chat interfaces to provide progressive output
• Implement health checks -- the provider's available?/0 function should be called before routing to ensure Ollama is responsive
• Cache model outputs -- store generated embeddings and repeated query results in ETS to avoid redundant inference
• Create custom Modelfiles -- tailor system prompts and parameters to the platform's coding conventions for better output quality
• Monitor memory usage -- Ollama can consume significant RAM with large models; ensure the system has adequate free memory

Comparison with Alternatives

Ollama was chosen for its minimal setup complexity, built-in model management, and broad hardware support. The single-binary installation and OpenAI-compatible API make it the easiest local AI solution to integrate with the platform's existing HTTP-based provider architecture.

Related Technologies

• Claude - Cloud AI provider for complex reasoning tasks
• OpenAI - Cloud AI provider for embeddings and fallback
• ETS - In-memory cache for Ollama response caching
• Elixir - Host language providing the HTTP client for Ollama integration

Related Apps

• prismatic_claude - Multi-provider AI layer hosting the Ollama adapter
• prismatic_agents - AI-powered agent operations using local inference
• prismatic_ollama - Ollama-specific integration and model management

Connect & Contribute

Created by Tomáš Korcak (korczis) | Open Source under GHL

• GitHub | GitLab | LinkedIn | Contact
• Developer Portal | Architecture | Meet the Creator