Tacnode™
Pipelies2 min

The Late Arrival: Settlement Released Short After a Source Landed Late

Daily load complete. But Source B landed forty minutes after the join ran. At 1 a.m. the first source, Source A, lands on time. Half an hour later the nightly join runs. Source B hasn't landed yet, and the job has no way to know anything is missing. Forty minutes after that, Source B lands. The join doesn't run again, so the total still holds Source A only. At 6 a.m. the total is published and used to release settlement. The reserve is sized to a total that never saw Source B, so it goes out short. Then the same night with the total kept in Tacnode. When Source B lands, the next refresh puts it in the total, hours before settlement goes out, and it's released in full. In Tacnode the total is a view that refreshes as often as you need, applying only what changed since the last refresh. Where that isn't soon enough, the total can be computed on demand. The release is written back as context, and your sources keep sending on their own schedules.

Overview

One of the fourteen Pipelies lies: how a source that lands after the join leaves a total incomplete, and the same night with the total kept in Tacnode.

Topics Covered

  • •Source A landing on time
  • •A nightly join that runs before Source B
  • •A total that holds Source A only
  • •Settlement released on that total
  • •A view that picks up late rows on the next refresh

Key Takeaways

  • ✓The join had no way to know anything was missing
  • ✓The load was complete and the total wasn't
  • ✓When Source B lands, the next refresh puts it in the total, hours before settlement
  • ✓Your sources keep sending on their own schedules

Ready to get started?

Book a demo to see how Tacnode can power your real-time data infrastructure.