The Late Arrival
The nightly join ran at 1:30. One of its sources showed up at 2:10.
Source A lands.
Where this shows up
Why the ETL job gets it wrong
A batch job picks a cutoff and runs once. In a distributed system some data always arrives late, and the job has no way to know it’s missing anything. Late rows stay out until the next run, assuming the next run picks them up.
By the time the decision ran, its context was incomplete. We call that a context gap.
Pick up late rows on the next refresh.
In Tacnode a derived total is a view that refreshes on a schedule you set, typically every few seconds, applying only what changed since the last refresh. When Source B lands at 2:10, the next refresh puts it in the total. Where that still isn’t soon enough, the total can be computed on demand against everything that has arrived.
If an engineer asks how
Incrementally refreshed views apply only the rows that changed since the last refresh, on a schedule set per view in SQL.
Questions to put to your team
- What happens to a row that lands after the join ran?
- How would you know a total was missing a source?
- Has late data ever changed a number after someone acted on it?
Is your settlement total telling this lie?
Bring it to a demo, and we’ll trace what it reads and how old that data is by the time it runs.
Book a demo →