Fraud detection architectures converge on the same canonical stack — Kafka → Flink → feature store → model serving → rules engine — and fail at three predictable seams under concurrent load: velocity counter staleness, feature-store / rules-engine divergence, and cross-channel retrieval gap. Sub-50ms p99 on each component doesn’t fix any of these.
TL;DR: The canonical real-time fraud stack (Kafka → Flink → feature store → model serving → rules engine) has three predictable failure seams under concurrent load: (1) velocity counter staleness during bursts, (2) feature-store / rules-engine divergence from independently propagating pipelines, (3) cross-channel retrieval gap when fraud signals live in per-channel services. Sub-50ms p99 on each component doesn’t fix any of these, because the structural problem is that derived context has a propagation delay that exceeds the decision’s validity window under load. The fix is a serving layer where all derived context is maintained continuously under one coherent snapshot, not a stack of caches each at a different propagation stage.
Every fraud detection architecture diagram has the same boxes. A transaction lands on a payment API. An event fires into Kafka. A Flink job maintains velocity counters and writes them to Redis. A model server reads features from a store, scores the transaction, returns a risk number. A rules engine composes the number with hard checks and commits the decision. Each box looks fast in isolation. The diagram looks right.
Production incidents don’t come from a box being slow. They come from the seams between boxes being inconsistent at the moment a decision commits. This post walks the canonical fraud detection architecture, names the three seams where it fails under concurrent load, and describes what a decision-coherent architecture has to do differently.
Real-time fraud detection is the architectural category where every transaction is scored and decided before it commits, against derived context — velocity counters, exposure aggregates, model features, session state — that is being updated in parallel by other transactions. A fraud detection pipeline is a multi-stage system that processes transaction data within milliseconds, composing anomaly detection, machine learning models, and rule-based checks into a single approve/decline outcome.
Fraud detection architectures are typically categorized into rule-based, machine learning, and hybrid systems, implemented across both real-time and retrospective timelines. Modern real-time deployments use an event-driven, modular microservices architecture to hit sub-second response times: authorization decisions must commit in under 100 milliseconds to avoid declining legitimate transactions and forfeiting revenue. Most teams converge on the same reference stack because each step solves a recognizable problem.
The data ingestion layer captures transaction streams via Kafka or AWS Kinesis — durable, ordered event streams are table-stakes. A stream processor (Flink or Kafka Streams) subscribes to the relevant topics, maintains windowed aggregates (velocity counters, rolling sums, distinct merchant counts, behavioral patterns), and emits updates to a serving store. A feature engineering pipeline transforms raw transaction data into model-ready features — historical behavior aggregates, cross-device signals, embeddings — which land in a feature store (Redis, DynamoDB, Tecton, Feast, or a hand-rolled equivalent) alongside entity attributes like customer risk tier, merchant MCC, and device fingerprint reputation.
A model server reads features at inference time and returns a risk score using machine learning models trained on labeled fraud outcomes. Real-time fraud detection systems must process and analyze vast volumes of structured and unstructured transaction data to identify anomalies and suspicious activities within the authorization window. A rules engine composes the score with hard checks (`decline if kyc_status != verified`, `decline if velocity_1min > 5`) and commits the final decision. The decision writes back to the system of record and the event is logged for offline model training and post-hoc analysis.
The architecture is correct for the problem as originally framed: turn an event into a decision as quickly as possible, with the best signals available. Every piece has a measurable latency budget, and in isolation every piece meets it. The failure mode is not latency. It’s what happens when concurrent events against shared state hit the composition before the pipeline has caught up.
Each component was designed against a specific set of assumptions. The assumptions are defensible in isolation. They compound under concurrent load.
Kafka is optimized for durable, ordered throughput. It assumes downstream consumers tolerate propagation lag — the time between a producer write and a consumer read is bounded but not zero, and the contract does not require the consumer to see writes inside any particular time window relative to the producer.
Stream processors like Flink hold state well — RocksDB, checkpoints, exactly-once delivery semantics — but the state is operator-local. A Flink job that maintains a velocity counter for card X does not inherently hold state for card Y in the same snapshot, and reading the counter from outside Flink into a serving store introduces a propagation delay that Flink doesn’t manage. See the decision-time system model for why a stream processor’s state is not a serving layer for decisions.
Feature stores are optimized for low-latency reads of pre-materialized features. The refresh cadence — how often the materialized feature is recomputed from upstream events — is typically a seconds-to-minutes batch job or a scheduled cache refresh. For decisions whose validity window is sub-second, the feature store’s refresh cadence is already outside the window before a decision arrives.
Model servers are optimized for high-throughput inference. They assume inputs are fresh and consistent — the model doesn’t know that the feature vector it scored against was recomputed two seconds ago and has since been invalidated by three intervening events. Concept drift detection, continuous learning pipelines, automated retraining, and Explainable AI (XAI) address a different problem (model accuracy over time and analyst trust for regulatory compliance); none of them fix stale inputs at decision time. A model that has drifted less is still wrong if the feature vector it scores is from before the decision’s window opened.
Rules engines are optimized for correct logical composition. They assume inputs are internally coherent — that the velocity counter and the exposure aggregate and the session flag all describe the same entity at the same moment. Rule semantics are evaluated against whatever the engine reads; the engine does not verify that its reads came from a consistent snapshot.
Each of these assumptions holds under design-time analysis. Each breaks under concurrent production load in a different, compounding way.
Fraud patterns continuously evolve and fraudsters adapt their tactics to evade detection, which is why real-time transaction monitoring leverages both anomaly detection and predefined rules to flag suspicious activities as they occur. Three failure patterns recur across production real-time fraud systems regardless of how adaptive the models are. They differ in mechanism but are identical in structure: a pipeline’s propagation delay exceeds the decision’s validity window at the moment the decision commits. Machine learning can learn new fraud strategies; it cannot learn around context that wasn’t readable when the decision fired.
The first reflex when these seams show up is to tune. Shorter TTLs. More replicas. Pre-warming. Smarter invalidation. These moves help at the margin; none of them change the structural problem.
Shorter TTLs make the preparation gap visible more often, not smaller. A TTL forces the cache to re-fetch on expiry. If the preparation pipeline takes 400ms to recompute the aggregate, a 100ms TTL produces more cache misses, each of which still waits on the same upstream pipeline. The aggregate does not arrive faster because you invalidated the cache more aggressively.
More replicas scale reads, not writes. If the problem is that the feature pipeline can’t emit updates fast enough, adding read replicas makes stale state available in more places. The underlying write rate doesn’t change.
Pre-warming is a one-time win. You can load the cache with the current state at startup; within seconds of traffic arriving, concurrent events update the underlying state and the pre-warmed cache is already behind.
Smarter invalidation requires knowing what to invalidate. Pattern-based invalidation — “invalidate this feature vector when that event lands” — requires a data model that maps every event to every dependent derivation. That mapping is rarely maintained precisely in production, and when it isn’t, the cache quietly serves stale entries nobody invalidated.
Multi-cache architectures are immune to per-cache tuning. The retrieval gap is an architectural property of having multiple backing stores that propagate independently. You can tune any single store as tightly as you like. The composite a decision reads is still assembled from pipelines at different stages.
Every failure seam above fits one pattern. The decision has a validity window — the duration within which the decision must commit, and within which the context it reads has to remain correct for the decision to be correct. Card authorization validity windows are typically 100–400ms. Live transaction fraud scoring sits in the same range. Agent-driven commerce decisions can stretch to 1–2 seconds.
A real-time fraud decision fails when the preparation gap (how long the pipeline takes to reflect an event in a readable derivation) or the retrieval gap (how inconsistent the multiple backing stores are at read time) exceeds the validity window. Every concrete failure above is exactly this: the Flink job’s propagation delay exceeded the authorization’s validity window; the feature-store and rules-cache gap exceeded the same; the cross-channel event bus’s propagation delay exceeded the wallet decision’s window.
Framing the problem as “context freshness and coherence inside the validity window” rather than “each component’s latency” makes two things clear. First, it’s a per-decision spec — the same cache behavior might be fine for an overnight batch scoring job and broken for a live authorization. Second, it’s the thing you can actually commit to as an SLA: not “Redis returns in 2ms” but “state committed to the SoR is reflected in the serving layer within Y milliseconds, under concurrency rate Z, coherent across features and rules checks.” See why real-time decisions fail for the full taxonomy.
If tuning caches doesn’t fix the structural problem, what does? The answer looks less like “a better cache” and more like “one serving layer that replaces the feature store, the rules-engine cache, and the per-channel state, maintained continuously under concurrent load” — what Tacnode calls a Context Lake.
Four properties define it.
One read path, not several. When a fraud decision needs a velocity counter, an exposure aggregate, a model feature vector, and a cross-channel flag, it reads all four from the same serving layer under one logical snapshot. There is no retrieval gap because there are not multiple backing stores to drift from each other. Every composite read is internally coherent because it comes from the same set of ingested events.
Incremental maintenance, not scheduled refresh. Derived state — velocity counters, exposure aggregates, cross-channel risk flags — updates as events arrive, not on a cadence. When a transaction commits upstream, the aggregates that depend on it update incrementally, in sub-second time, without waiting for a Flink checkpoint interval or a feature-store batch refresh. The preparation gap collapses to the propagation latency of the change, which is bounded and much smaller than a scheduled refresh.
On-demand computation against coherent data. For derivations that can’t be pre-maintained — arbitrary window aggregations, cross-entity joins, vector similarity, LLM-derived signals — the query runs against state that is itself coherent at query time. The on-demand computation doesn’t add a new divergent snapshot; it runs against the same snapshot the rest of the decision is reading.
Read behavior that matches the decision’s concurrency model. Concurrent decisions that read the same state read it under the same logical snapshot. The system surfaces conflicts that matter (two authorizations for the same card within the exposure window) rather than hiding them (two decisions each reading a stale counter and approving independently).
Compliance constraints shape the serving layer as much as the architectural ones. PCI DSS requires network segmentation and encryption of sensitive fields in transit and at rest; comprehensive audit trails and row-level access controls are non-negotiable for regulatory compliance across card, ACH, and wallet channels. A unified serving layer makes these easier, not harder — one boundary to audit, one encryption configuration to maintain, one access-control policy that applies across every decision path.
The shape of the architecture changes per vertical, but the coherence requirement does not. Each vertical has a canonical failure mode, and the features involved (device fingerprinting, device history, behavioral patterns, fraud risk scores, manual review workflows) interact with the coherence gap in vertical-specific ways.
Card authorization needs coherence across the exposure counter, per-merchant velocity, and the active fraud case flag, all within the 100–400ms authorization window. The risk score that drives the decision is built from device fingerprinting, device history, behavioral patterns, and cross-merchant signals — each of which lives in the feature store at its own propagation stage. The canonical failure is concurrent merchants racing against a stale exposure total, approving a sequence of transactions that individually look legitimate and collectively exceed the card’s limit.
BNPL and per-transaction credit needs coherence across exposure (shared across merchants), velocity (across channels), and the risk model’s features (recent behavioral signals, user behavior deltas, suspicious-behavior flags). Flagged transactions route to manual review by fraud analysts, who investigate using visualization dashboards and relationship graphs — but the real-time approve/decline commits long before the analyst sees anything. The canonical failure is two merchants each approving against an exposure counter that hadn’t absorbed the other’s transaction, pushing the customer past their limit before either decision had visibility into the other. See real-time credit decisioning architecture for the credit-specific treatment of the same underlying failure mode.
Digital wallet and ACH need coherence with the card channel’s fraud state, because fraudsters pivot across channels when one declines. These paths integrate with payment processors, financial institutions, and mobile apps at high transaction volume, and each channel generates its own fraud risk score feeding its own local decision. The canonical failure is the cross-channel retrieval gap — card declines at 2:14pm, wallet approves the same credential at 2:14:30pm because the decline signal hadn’t propagated yet.
Agent-driven commerce adds a new timescale. The validity window stretches to 1–2 seconds, but agents compose multiple tool calls against a coherent customer state. The canonical failure is an agent reading balance, credit, fraud score, and inventory in parallel and committing an action against a composite that never existed as a real-world snapshot.
Graph neural networks (GNNs) are increasingly used across card, BNPL, ACH, wallet, and agent-driven fraud detection to reveal relationships between users, devices, merchants, and accounts — surfacing organized fraud rings that any single per-entity signal would miss. A GNN trained on the transaction graph catches coordinated fraud that passes velocity rules and single-transaction scoring because the signal lives in the graph topology, not the individual edges. NVIDIA’s financial fraud detection blueprint, AWS’s graph-neural-network reference architecture, and Monzo’s production deployment all center on this pattern.
The coherence requirement intensifies with graph features. A graph embedding computed five minutes ago against a pre-burst state is less useful than one computed against the current graph — but graph feature recomputation is expensive and typically runs on its own propagation schedule. Production stacks maintain graph embeddings in a separate vector store or graph database, adding another independently-propagating stage to the retrieval gap. The fraud decision now reads from a velocity counter at one stage, a feature store at another, a rules cache at a third, and a graph embedding at a fourth. Four snapshots, none of them synchronized.
A decision-coherent architecture treats graph features like any other derived context: maintained incrementally as the graph changes, served from the same snapshot as the other context the decision reads. When the transaction graph updates, the embeddings that depend on it update with sub-second convergence, not on the graph processor’s refresh cycle. The decision reads the embedding under the same snapshot as the velocity counter and the exposure aggregate — not from a separately-propagating graph pipeline.
The fraud detection vendor landscape is crowded — Feedzai, FICO Falcon, Featurespace, SAS, and others build scoring engines and rules platforms on top of whatever infrastructure the operator provides. Streaming platforms like Hazelcast and Materialize serve real-time features. Each of these products solves a specific problem: model scoring, rules evaluation, stream processing. None of them solve the coherence problem across the full composition.
A scoring vendor returns a risk score against whatever feature vector you send it. The vector still comes from your feature store, still lags the stream, still diverges from your rules engine’s cache. A streaming platform serves fresh data, but it doesn’t collapse the preparation and retrieval gaps if the rules engine, feature store, and per-channel state remain separate. Monzo’s fraud detection architecture, Microsoft Fabric’s real-time intelligence reference, and Snowflake’s banking fraud blueprint are all variations on the same composed stack with the same structural seams.
A decision-coherent architecture doesn’t replace the scoring vendor or the streaming pipeline. It replaces the serving layer they all read from. The model still produces scores, the rules engine still composes them, but every composite read comes from one snapshot. The vendor stack gets more coherent context, not less integrated.
If the architecture is designed against the validity window, the SLAs should be too. Measuring each component’s p99 and declaring the architecture real-time is the diagnostic that misleads most teams. A better dashboard measures fraud detection accuracy, detection rate, model performance, and two decision-path metrics most stacks never expose.
Propagation freshness. The time between an event committing to the SoR and the corresponding derived state being readable by the decision path, measured under peak concurrency, not average. This number should live inside the validity window with headroom.
Cross-read coherence. For every decision that composes multiple reads, the maximum time skew between the reads — “the velocity counter is from T-200ms and the exposure aggregate is from T-1200ms” quantifies the retrieval gap that a rules engine cannot see. In a decision-coherent architecture this number is zero by construction because all reads come from the same snapshot. In a composed stack it’s the right telemetry to expose.
Track false positive rates explicitly. High false positive cost erodes merchant trust and declines legitimate transactions, costing revenue in lost sales and customer retention. Cost-sensitive learning in fraud detection models helps align model objectives with business economics, but cost-sensitive models are still wrong if the features they score were stale at decision time — the problem framing stays architectural, not just model-tuning.
Fraud detection architectures built on the canonical stack work well for the problem as originally framed — score transactions fast, keep costs manageable, tolerate some marginal freshness gap in exchange for scale. The problem has moved. Fraudsters operate across channels on agentic timescales, velocity patterns compress into sub-second bursts, and the cost of a missed approval has grown faster than the cost of infrastructure. The architectures that succeed at this scale treat decision context as a first-class serving concern — one coherent snapshot across velocity, exposure, features, and cross-channel state, updated continuously under load.
The question isn’t whether your Kafka is fast or your Redis returns in two milliseconds. It’s whether your serving layer keeps every fraud decision coherent while state is changing under it. Those are different questions. They have different answers. And they have different architectures — one canonical, widely deployed, showing its seams under concurrency; the other designed against the validity window from the start.
For the cross-domain framing, see the decision coherence pillar. For the fraud-specific angle, see the fraud detection solution page.
Former Meta and Microsoft. Built distributed query engines at petabyte scale. Author of the Composition Impossibility Theorem (arXiv:2601.17019).
Book a demo and discover how Tacnode can power your AI-native applications.
Book a Demo