PostHog × One | Rust meetup
PostHog

From Snakes to Crabs

Rewriting PostHog’s feature flag platform in Rust.
Dylan Martin
Dylan Martin
@dmarticus Tech lead, PostHog
PostHog
Who I am

Dylan Martin

(Former) tech lead, Feature Flags @ PostHog
LinkedIn recommendation: A bit smart or whatever.
02
PostHog
Mental model · what our service does
The old Django decide service architecture
03
PostHog
Section 01

The problem.

What was structurally wrong with the old Python service.
PostHog
Section 01 / The problem

Three things were wrong with the old service.

01
The ORM
02
The runtime
03
Connection pooling
The old Django decide service architecture
01
05
PostHog
Section 01 / The problem · The ORM

Evaluating all flags was one big mega-query

-- ONE query per request. Conditions for EVERY flag,
-- evaluated inline by Postgres:
SELECT person.id, person.properties,
  (person.properties->>'email' LIKE '%@acme.com' AND
   (person.properties->>'plan')::text = 'enterprise') AS flag_a,
  (cohort.id = ANY($2)) AS flag_b,
  -- ... one expression per flag, for every flag
FROM person
  LEFT JOIN cohortpeople ON cohortpeople.person_id = person.id
  LEFT JOIN cohort ON cohort.id = cohortpeople.cohort_id
WHERE person.team_id = $1 AND person.id = $3;
06
PostHog
Section 01 / The problem · The ORM

Why that one query didn't scale.

01
JSONB matching can't use an index
02
Query cost grows with every flag
03
Operators with no SQL form
07
PostHog
Section 01 / The problem

Three things were wrong with the old service.

01
The ORM
02
The runtime
03
Connection pooling
The old Django decide service architecture
01
02
08
PostHog
Section 01 / The problem · The runtime

Python couldn't use the box.

Pod · 8 cores
Sync workers block a whole process on the DB, and the GIL pins Python to one core per worker
09
PostHog
Section 01 / The problem

Three things were wrong with the old service.

01
The ORM
02
The runtime
03
Connection pooling
The old Django decide service architecture
01
02
03
10
PostHog
Section 01 / The problem · Connection pooling

Timeouts trapped in PgBouncer.

# Postgres caps out in the low hundreds, so thousands of
# workers funnel through PgBouncer. Our deadlines live there too:
query_timeout      = 15
query_wait_timeout = 10

# in the Django request handler:
# …no real per-request deadline we could reach.
A separate system, clunky to tune
11
PostHog
Section 02

Choosing Rust.

What the replacement had to do, and why Rust cleared the bar.
PostHog
Section 02 / Choosing Rust

What the replacement had to solve.

01
Evaluate flags outside the database
02
A genuinely fast async runtime
03
Connection pooling without PgBouncer
13
PostHog
Section 02 / Choosing Rust

Why Rust won.

Async PythonFastAPI NodeHyperExpress RustTokio
Eval off the ORM No Yes Yes
Fast async runtime Async, still slow Async, slower than Rust Fast and async
Pool without PgBouncer No Needs a proxy Native, via sqlx
Team can ship it Yes Yes Yes
Verdict Same problems, deferred Forced into a proxy Won or tied on everything
14
PostHog
Section 03

Build it without losing what we knew.

Tests were my best friend.
PostHog
Section 03 / Implementation · The loop

One module at a time.

01
Stand up the new endpoint
→
02
Rewrite a module
↑
↺
↓
04
Add integration tests
←
03
Port its tests
16
PostHog
Section 03 / Implementation · Synthetic test framework

Then, test it with real data.

Local build
All prod flags + a test-bed of users
users built to match across every flag
──▶
Old
Django, local
New
Rust, local
──▶
Diff
Fix edge cases
↺ repeat
17
PostHog
Section 03 / Implementation

What changed in the new service.

01
Evaluation in app memory
Flag matching moved out of Postgres SQL — fetched raw, matched in parallel.
02
Cohorts cached in-memory
No re-joining membership on every request.
03
Direct pool management
Pools managed through sqlx at the app level. PgBouncer gone.
New Rust flags architecture with the key changes highlighted
01
02
03
18
PostHog
Section 03 / Implementation · The Rust that mattered

Pooling moves in-process.

// sqlx owns the pool — no PgBouncer in the path.
let pool = PgPoolOptions::new()
    .max_connections(20)
    .acquire_timeout(Duration::from_millis(50))
    .connect(&url).await?;

// And the deadline lives in our code, per request:
let row = timeout(Duration::from_millis(100), q.fetch_one(&pool)).await?;
One pool per pod, bounded and shared across tasks. PgBouncer gone, timeouts back where we can reach them.
19
PostHog
Section 03 / Implementation · The Rust that mattered

Match every flag in parallel.

// Fetch raw properties once, then fan the matching
// out across cores — the thing the GIL never allowed.
let results: Vec<FlagMatch> = flags
    .par_iter()                       // rayon
    .map(|flag| flag.evaluate(&props))
    .collect();
Postgres stops doing application work; the CPU-bound match runs on all the cores, in memory.
20
PostHog
Section 04

Shipping it without breaking production.

Taking advantage of an existing service.
PostHog
Section 04 / Rollout

Roll out by hashing the API token.

1%
wk 1
10%
wk 2
35%
wk 3
70%
wk 4
100%
wk 5
22
PostHog
Section 05

The results.

What the rewrite actually bought us.
PostHog
Section 05 / Results

Latency.

p99
904ms 85ms
10.6× faster · 90.5% reduction
Response time chart across cutover
Response time across the rollout. Cutover ≈ 21:15.
p95381 → 43 ms
p90160 → 31 ms
p5022 → 12 ms
24
PostHog
Section 05 / Results

Cost.

Pods
300
→
90
−70%
Compute spend
$8.8k
→
$2.8k
−68%
Traffic
500k rpm
→
730k rpm
+46%
25
PostHog
Section 05 / Results · where it stands now

Bonus: here's now

p99 today
25ms
Current latency dashboard
Live dashboard · May 2026.
26
PostHog
SECTION 06

Reflection.

What I'd do differently, and what to take home.
PostHog
Section 06 / Reflection

What I'd do differently.

Property-based testing.
Load testing infrastructure.
Leverage AI more heavily.
28
PostHog
Section 06 / Reflection

What I'd do differently.

Property-based testing.
Load testing infrastructure.
Leverage AI more heavily.
29
PostHog
Section 06 / Reflection

What I'd do differently.

Property-based testing.
Load testing infrastructure.
Leverage AI more heavily.
30
PostHog
End

Thank you.Questions?

Read more

  • Part 1 · posthog.com/blog/even-faster-more-reliable-flags
  • Part 2 · posthog.com/blog/untangling-tokio-and-rayon

Find me

  • dylanamartin.com
  • github.com/dmarticus
  • linkedin.com/in/dmarticus