Tacnode™

Read replicas that fall behind at peak

A prediction market moves its reads onto its failover replicas at the peak, the replicas fall behind and fail, and the site goes down.

Photo: Lukenn Sabellano / Unsplash
→→AtWeek 1
The story

Four people decide to build a prediction market.

  • They don’t have a database administrator and can’t afford one, so they go with a managed Postgres service from their cloud provider.
  • That’s a good choice. It’s what I’d pick too.
  • The provider runs it for them. It’s Postgres, so the four of them can get on with the product.
What traders see
What’s happening underneath
POSTGRES CLUSTERwritesreadssame storageApp tierWriter instancetakes every writeReplicareplicaReplicareplicaShared storage layersix copies, three availability zones8% busy2%2%promote a replica if the writer dies
1 / 16

Illustrative scenario. Timings show the shape of the problem, not a measurement of any one company. Story by Boyd Stowe.

Your stack today, and with the Tacnode Context Lake™

Today

App
Trading app
Writer
Postgres
Read replicas
Postgres read replicas
Failover copies now serve every read
Traders’ screens
Web and mobile app

Moving reads to replicas is the standard advice, and the founders made sensible choices at every step. A replica’s main job is to take over when the writer fails. Once the product reads from it, it has a second job. At the peak, held connections put most of the traffic on one replica, it falls behind, and the app shows a book that is seconds old while every health check stays green. When that replica gives out, its traffic moves to the next one. The only way to add read capacity was to add a copy, and the copies are what failed.

With the Tacnode Context Lake™

App
Trading app
Tacnode Context Lake™
Compute scales over one shared store
Traders’ screens
Web and mobile app
  • Read capacity without a copyCompute scales out over one shared store, so there is no replica to fall behind and the book on screen is current to the second.
  • Heavy reads off the order pathRisk, settlement and search run on the same tables as the orders without slowing confirmations.
  • Durability without the failover trapData is erasure-coded across the storage pool, so no standby copy gets used up serving reads.

Questions to ask your team

  1. 1Do your production reads run on replicas that exist for failover?
  2. 2At your peak, how far behind the writer is your busiest replica, and does anyone see it?
  3. 3When an order is rejected for a moved price, can you tell whether the price was really gone?