The Self-Driving Product | Merge × PostHog × Redis
The road to self-driving agents
Using product context & LLMs to go from observability to self-driving
Marco G
PM, AI Observability · PostHog
The case
Self-driving compresses the feedback loop
06
PostHog's Bet
The whole picture
Observe
AI observability
Every agent step: prompts, tools, reasoning, cost.
Watch
Session replay
Replay the real session, now read by AI vision.
Catch
Error tracking
The exact exception the agent hit, with its stack.
Ask
Surveys
A 👍 / 👎 on every answer, captured as data.
08
PostHog's Bet
The insight is in the overlap
AI observability + Surveys
Every trace carries the user's 👍 / 👎, so you see which answers actually landed.
AI observability + Session replay
The trace looks done, but the replay shows rage-clicks on a frozen response.
AI observability + Error tracking
The real exception from inside the agent loop, tied to the step that threw it.
PostHog's Bet
Signals and scouts
Reactive
Signals
Something happened: a pattern in the data becomes a signal you can act on.
Online Evaluations · error spikes · survey drops
Proactive
Scouts
Nothing's broken yet: scouts go looking for the users and behavior worth a closer look.
costly or unusual users
Zooming in
One signal, one product
PostHog
many products
→
AI observability
one product
→
Online Evaluations
one signal
→
This example
end to end
AI Observability · the data model
The data model, in plain terms
Structure
In plain terms
Session
chat / conversation
└─ Trace
one message
├─ Span
a non-LLM call, or a grouping of work
└─ Generation
a single LLM call
Online Evaluations: a signal end to end
Distilling actions from data
Traces
10k events
Evals
200 fails
Signals
2 signals
LLM introduced
Reports
1 report
Actions
1 PR
Step 1 · Traces
Tracing captures every step
User"Summarize the top papers on RLHF."
ReasoningI already know these — no need to search.
Toolsearch_papers(…) · never called
Assistant"The top papers are PPO, InstructGPT, and DPO."
10
Step 2 · Evals
Evals score production traces
✓
What's new in RLHF lately?
Tool search_papers("RLHF") → 3 results
Three recent ones: DPO, RLAIF, and process rewards.
Tool use · pass
✕
Summarize the top papers on RLHF.
Tool search_papers(…) · not called
The top papers are PPO, InstructGPT, and DPO.
Tool use · fail — no tool call
11
Step 3 · Signals
Evals are grouped into signals
Missing answered without searching
Missing no search_papers call
Broken search_papers → 500, 7 retries
Cost a hair over average — fine
+ 196 more eval runs
→
Signal
Missing tool call
194 failing runs
Signal
Cost
6 flagged runs
12
Step 4 · Reports
Not every signal is worth a report
Signal · Missing tool call
missed tool call78%
tool errored (broken)12%
other10%
One clear cause → report created
Signal · Cost
99%
of runs within budget
Flat and well within budget — nothing unusual.
Nothing to act on → no report
Step 5 · Actions
The report becomes a fix
Report
194 sessions answered without calling the tool
→
→
Fix · PR #4212
+ agent: require search_papers before answering
no more answers straight from memory
no more answers straight from memory
13
Takeaways
What to take home
- Self-driving is about compressing the feedback loop between an insight and its action.
- It runs on context: data across every product in one place.
- Each step strips noise: traces → evals → signals → reports → actions.
Thank youQuestions?
Read more
- AI observability · posthog.com/ai-observability
- Slides · posthog.github.io/presentations/self-driving-agents
Find me
- marco.gancitano.com
- github.com/mgancita
