A vendor added a field called loyaltyTier to their orders API. No changelog. No deprecation notice. Just a new column showing up in the payload, and a schema-on-write job fails.
Let’s call it the quiet tax of building on third-party APIs: you don't control the contract. A vendor adds a field, renames a field, - and somewhere downstream, a pipeline that was working perfectly yesterday throws a schema mismatch. If we catch the exception, log it, and let the batch through anyway. That "fixes" the outage. It also means bad data is now sitting in a warehouse that finance is about to run numbers off of.
We think that's backwards. The right pipeline doesn't treat every schema change as a crisis, and it doesn't treat no schema change as safe either. It tells the two apart - automatically - and reacts differently to each
The pattern that actually works starts with treating "what fields exist and what type they are" as data, not as something baked into code. We keep a registry - call it an entity schema table - that says, for every source system and every table: here's every column we know about, and here's the type it's supposed to be.
That single design choice changes everything about how the pipeline reacts to change. When a batch comes in, every column gets checked against that registry instead of against a schema hard-coded in a script somewhere. A column the registry has never seen isn't an error condition anymore - it's just a lookup that comes back empty. And an empty lookup is something you can have an opinion about, instead of something that has to crash the job.
But what would the opinion be?
A registry table lists every column expected for each source table, and what type it should be. Nothing is hard-coded into a script, so nothing has to be edited by hand when a vendor changes something.
If a field isn't in that registry, it gets cast to a string, let through, and logged as a new entry to review later. Nobody guesses what it means yet. That decision gets made deliberately, not automatically.
If a column already has a registered type and the incoming value can't be cast into it, that's not drift. That's the data going bad. The row gets pulled, logged with the exact bad value and the record it came from, and the batch fails loudly.
Bad dates get a flagged placeholder instead of dropping the whole record. Losing an entire order over one malformed timestamp costs more than it save. Every other type mismatch still stops the batch.
And voila- you managed to save the day without spamming the stakeholder’s mailbox with production failure emails.
We've built this pattern into client pipelines that ingest from CRMs, fintech systems, and marketing platforms that all change their APIs on their own timeline, with zero coordination with the teams depending on them. The result isn't a pipeline that never has problems - it's a pipeline where the problems that surface are the ones actually worth an engineer's attention, and everything else just gets absorbed and logged.
If your ingestion layer is still one unannounced vendor field away from a 2 a.m. page, that's a design problem, not a bad-luck problem -and it's a solvable one.
From architecture review to production hardening - we help
1. Doesn't auto-registering unknown columns as strings mean we lose type safety?
Only temporarily, and on purpose. A new column starts as a string because the pipeline has no basis yet for assuming anything stronger -guessing wrong is worse than being conservative. Promoting it to a real type is a deliberate, reviewed decision, not something the pipeline should infer on its own.
2. Why hard-fail the whole batch instead of just dropping the bad row and moving on?
It does drop the bad row - Â but it also stops the run, because a type mismatch on a known column usually means something upstream is actually broken, not just different. Silently continuing would let that condition run indefinitely without anyone noticing.
3. Isn't stopping the pipeline on any type mismatch overly strict?
It would be, if it applied uniformly. The one deliberate exception -malformed dates -shows the design isn't "strict for strictness's sake." It's calibrated per failure mode, based on which ones are common, well-understood, and safe to absorb versus which ones need a human's attention.
4. Does this replace data quality monitoring?
No - it feeds it. Every substitution and every dropped row gets logged with the exact column, value, and record it came from, so the log itself becomes the starting point for whatever monitoring or alerting sits on top.
.svg)
.svg)