Observer in Production: The BEAM Introspection Nobody Talks About

The first time you attach observer to a running production node and see every process, every mailbox, every ETS table, every linked supervisor — live, updating, interactive — you realize how much production debugging you have been doing blind. This is BEAM introspection, and it is arguably the single biggest operational advantage Elixir has over other runtimes.

Attaching to a remote node

# On your laptop, start a local node with the same cookie:
iex --sname debug --cookie $COOKIE

# Connect:
iex(debug@laptop)> Node.connect(:"prismatic@prod-01")
true

# Open observer scoped to the remote node:
iex(debug@laptop)> :observer.start()
# In Observer: Nodes → prismatic@prod-01

The GUI now shows the remote node's processes, applications, ETS tables, and load averages. Every tab is live. Drill into a process and see its mailbox depth, stack trace, links, monitors, and current function.

What to look at first

Three tabs, three things:

1. Processes — sort by reductions desc. Top 10 tells you where the CPU went. Sort by message queue desc. Top 10 tells you which GenServer is overloaded.
2. Applications — the supervision tree visualized. Crashed children are obvious. Restart intensity is visible per supervisor.
3. Table Viewer — every ETS table with row counts and memory. A table that keeps growing is a leak.

Most production problems show up in one of these three tabs in under a minute.

Safety rules

Observer on prod je dost mocné na to, aby ti ublížilo. Tři pravidla, žádná výjimka:

The textual alternative

If GUI over SSH is too slow, :runtime_tools and :recon give you the same information in IEx:

:recon.proc_count(:memory, 10)      # top 10 memory-hungry processes
:recon.proc_count(:message_queue_len, 10)  # top 10 overloaded mailboxes
:recon.bin_leak(10)                 # find binary memory leaks

:recon is the tool to reach for on locked-down hosts where Observer is a non-starter. Same information, text-only, scriptable.

Where to go next