Ops
Ops and finance agents
The agent reports the invoice was created. If it wasn't, you find out at month-end close.
- invoice.create
- ledger.update
t = 0:00.0 / 0:39.8
1.0×
{ query: "status?" }user_id: u_83a1
{ status: "shipped" }tokens: 1,284
Your agent said it worked.Did it?
Your agent marks the run complete. The invoice was never created. Omium checks every write against your database and re-runs the ones that never landed.
Drops into
- LangChain
- LangGraph
- CrewAI
- AutoGen
- OpenAI Agents SDK
Sits downstream of the tracing you already have.
/FailuresFailures/All agents
24h·Filter·Export
Real-time diagnostics and anomaly detection for your agent swarm.
42
last 24h
3
open
88%
this week
1,284
total
Actionable Insights3 automations
AllPatternsSuggested
- 2m agoSchema guard added to lookup_order74 paused runs resumed · 0 dupes
- 14m agoRetry wrapper on tool.searchFlaky rate dropped from 4.2% → 0.3%
- 1h agoContext dropped in llm.chatSuggest trim(MAX_TOKENS) before plan step
Incident Contribution
Failures over the last 4 months
Less
More
Top 3 Offenders
billingAgent.retry
us-east-1
supportAgent.tool
eu-west-1
researchAgent.plan
ap-south-1
Cost of Failure
$247.50−62%
Saved vs last week via auto-recovery.
00:0012:0024:00
Silent failure
Your agent reported success. The row was never written.
Tracing logs what the agent said it did. Evals run in staging. Neither one opens your database to check that the row is there.
Other tools
tracingLOG✓
run.completed
4.2s · no exception thrown
LOG✓
status = success
as reported by the agent
LOG—
orders.row
never checked
LOG—
on_call.paged
nothing threw
Nothing broke. That's the problem.
Omium/ground truthrun.status
success, as reported
orders.row
no matching row exists
run.reclassified
silent failure
alert.sent
with the step that broke
One check no production tool runs. Does the row exist?run_9af2
How it works
From silent failure to proof in four steps.
The SDK goes in beside your agent. Nothing about the agent changes.
- 01
Connect
Point our SDK at your agent. On LangGraph it's a drop-in. Everything else takes a 30-line adapter.
- 02
Detect
We replay the run with variations to find the step that broke. Runs marked complete get replayed too.
- 03
Verify
We open your database and look for the orders row. If it isn't there, the fix is rejected.
- 04
Prove
Every week, one page. It puts your current failure rate against your Week 1 baseline.
The checkpoint
We keep enough of the run to run it again.
Four layers, captured at every step. This is what sits between the failed run and the fix.
- 01
Run state
Every tool call, with the arguments it used
- 02
Side effects
What the run wrote, and to which table
- 03
The failing step
The step that broke, found by replay
- 04
The re-run
Restarts from the last good checkpoint
1.0Detect
The step that broke, not a wall of logs.
When a run fails we replay it, changing one thing at a time, until only one step is left standing. Counterfactual replay, and it points at a line number.
Support Agent
recoveredhealthyckpt_0a3f
Billing Agent
InternalPublic Beta
Research Agent · status signals
Alpha
2.0Verify
A recovery only counts if your database agrees.
The check reads your own tables. If the row the run was supposed to write isn't there, the recovery is rejected.
supportAgent·run_8f2aOn it. Tracing live execution of supportAgent.
Captured checkpoint ckpt_0a4c · tool_callbefore tool invocation.
Running lookup_order in prod/us-east-1.
supportAgent/run_8f2a$ trace --follow --tool lookup_order
→ resolved order #A-29481 in 312ms
→ latency p95 ok · retries 0
Tool returned non-JSON payload. Schema validation failed.
Pausing before downstream call. Snapshot captured: ckpt_0a4d · pre_llm
Analyzing failure mode
Partner result
Nimbus AI went from about 2,500 silent failures a day to about 108.
Same agent, same workload. The difference is that every run now gets checked against their own database.
~2,500/ day
Silent failures at the start
~108/ day
Silent failures at iteration 47
47iterations
Of detect, verify, repair
One partner, one workload, measured over 47 iterations.
Where it pays
Three workloads where a wrong write costs money.
The more expensive a wrong write is to undo, the less a green run is worth.
Support
Customer support agents
A refund issued twice still closes the ticket. The customer finds it before your dashboard does.
- refund.issue
- ticket.close
- r_9af2supportAgent3:44schema-miss
- r_9ad7ingestAgent3:41hallucination
- r_9ac1supportAgent3:38tool-error
- r_9ab6plannerAgent3:36infinite-loop
- r_9a9esupportAgent3:31context-drop
Code
Coding agents
CI went green and the regression shipped anyway. Someone reverts it seven days later.
- pr.merge
- migration.run
handle_request
1840ms
intent.classify
312ms
retrieve.docs
412ms
db.query
268ms
llm.call
720ms
format.response
128ms
Integrations
Attaches to the framework you already run.
Everything below is picked up when the SDK initialises. Anything else goes through the adapter.
LangChain
LlamaIndex
CrewAI
LangGraph
Pydantic AI
Mastra
Vercel AI
OpenAI
AnthropicLangChainLlamaIndexCrewAILangGraphPydantic AIMastraVercel AIOpenAIAnthropic
Google
Mistral
Cohere
BAML
Guardrails
AutoGen
HaystackGoogleMistralCohereBAMLGuardrailsAutoGenHaystackInstall
Three lines, then it watches every write.
One import, one init, one instrument call. Nothing else in your agent changes, and the first silent failure usually surfaces within the hour.
In the dashboard
The screen your on-call engineer keeps open.
One row per run. The checkpoint count is how far back a retry can pick up.
Recovery stream
Last 5 runs
FAQ
What people ask on the first call.
The six questions that come up before anyone signs anything.
How does the money-back guarantee work?
By Day 60 we either cut your failure rate by 40% or land 30 verified recoveries. If neither happens we refund the full $15,000 within 5 business days and you keep everything we built.
What frameworks do you support?
LangChain, LangGraph, CrewAI, AutoGen and the OpenAI Agents SDK are picked up when the SDK initialises. Anything else goes through the adapter.
Do you host our data?
Yes, today. Traces you send land in Omium's managed store, and that includes raw prompt and response text — we're not going to tell you otherwise. Bring-your-own-bucket and a collector that runs inside your VPC are both designed and on the roadmap; neither has shipped, so don't buy on them. If raw text leaving your account is a blocker, say so on the design-partner call. That's a conversation to have before a contract, not after.
What if our agent doesn't have ground-truth checks?
If nothing in your system can tell whether a run was right, no reliability platform can help you yet. On the design-partner call we'll find the signal with you. It's usually a row your agent was supposed to write.
Are you SOC 2 compliant?
No, and we won't dress it up. We hold no SOC 2 report of any type and we have not yet engaged an auditor. Type I is on our roadmap, not on a contract. Same answer for HIPAA and ISO 27001. If a current report is a hard requirement for you, we are not the right fit yet — we would rather tell you now than lose your legal review later.
What if we already use LangSmith or Langfuse?
Keep them. They record what your agent reported. Omium checks whether it was true and re-runs what wasn't. We ingest LangSmith traces natively.
Pick one agent. We'll check every write it made.
48 hours, read-only, free. You keep the report either way.