Tacnode™
Pipelies2 min

The Silent Skip: The Job Succeeded and 412 Rows Went Missing

Job status: succeeded. 412 rows never made it through. At 2 a.m. the nightly load starts, and every row is arriving. A few minutes in, a source adds a column and 412 rows fail to parse. The job logs a warning and exits clean. At 6 a.m. the dashboard shows the load as succeeded. Nobody reads the warning, and the rows are still missing. At nine an underwriting rule reads that account history to decide on a credit limit increase. It looks complete, so the increase is approved. The account's recent late payments were among the dropped rows. Then the same morning with the rule reading from Tacnode: same rule, same request. This time the late payments are in the history, so the rule sends it to a person. Source changes come into Tacnode through one path, and the numbers your rules read are computed there, over the same data. A new derived number is a view you write, not one more copy job reporting its own status. The decision is written back as context, and your source systems keep their schemas.

Overview

One of the fourteen Pipelies lies: how a job that exits clean hides the rows it dropped, and the same morning with the underwriting rule reading from Tacnode.

Topics Covered

  • •A source that adds a column mid-load
  • •412 rows that fail to parse
  • •A warning nobody reads and a status of succeeded
  • •A credit limit increase approved on an incomplete history
  • •Source changes arriving through one path

Key Takeaways

  • ✓The job's status described the job, not the data
  • ✓The dropped rows held the account's recent late payments
  • ✓With the late payments in the history, the rule sends the request to a person
  • ✓A new derived number is a view you write, not one more copy job reporting its own status

Ready to get started?

Book a demo to see how Tacnode can power your real-time data infrastructure.