A data pipeline should detect incompatible source changes, identify affected outputs and keep incomplete results distinguishable from current trusted data. A job finishing without an exception does not prove the incoming fields still mean the same thing. Define schema and business-rule checks alongside transport success.
Let’s plan your data pipeline checks.
Bring your workflow, constraints and open questions. We’ll help you define the requirements and work through a practical next step.
Record the source contract
List required fields, types, identifiers, units, allowed values and update behaviour. Assign an owner for clarifying meaning. A field remaining numeric does not establish that its unit or population stayed unchanged.
For a hypothetical sales feed, an amount changing from rupees to paise can pass a basic numeric check while corrupting reporting. Include business-level validation where the consequence justifies it.
Create a change-response matrix
| Source change | Detection to consider | Output treatment |
|---|---|---|
| Required column removed | Contract validation | Stop affected processing visibly |
| Type changes | Parsing and schema check | Quarantine incompatible records |
| New optional field | Controlled compatibility rule | Accept or review under the contract |
| Meaning or unit changes | Business validation and owner communication | Version the transformation |
| Identifier behaviour changes | Uniqueness and relationship checks | Investigate duplicate or orphan records |
| Partial extract | Population and completeness checks | Mark output incomplete |
Choose whether to reject a whole batch or isolate affected records from the workflow's consistency needs. Explain partial processing in downstream status.
Preserve evidence for repair
Keep permitted source references, extraction time and transformation version. Protect sensitive content in quarantine and diagnostic stores. An unrestricted failed-row bucket can become a second data exposure.
Do not silently coerce every invalid value into a default. A missing amount treated as zero may be harder to detect than an explicit failed record. Record any approved transformation and its intended meaning.
Verify recovery and downstream communication
Introduce controlled test changes and confirm the alert reaches an accountable owner. Repair the mapping and rerun the appropriate population without duplicating already accepted work.
Tell report consumers when the latest dataset is unavailable or incomplete. Retaining the last valid result can be useful, but display its age rather than presenting it as a fresh successful run.
Schema-change questions
Should a pipeline accept every new column automatically?
Use a defined compatibility policy. Unexpected additions can be harmless, sensitive or evidence of a wider contract change.
Does schema validation establish data correctness?
No. Structure is one check; meaning, completeness and relationships need their own acceptance rules.
Use data acceptance checks, and review source contracts with Drema's data engineering team.