Why pipelines fail, eight practices that make them reliable, and how to tell when scripts are no longer enough.
In short
A data pipeline that does not break is one that can be rerun safely, fails loudly instead of silently, and has a named owner. In practice that means a written contract with each source system, raw data landed unchanged, idempotent loads, automated data tests, freshness monitoring, version-controlled deployment and a runbook for the person who gets the alert.
A data pipeline is the chain of jobs that moves data from source systems, such as ERP, CRM and finance tools, to the place where it is reported on or used by AI. This guide is written by Devies, a Gothenburg-based product and engineering consultancy, for data leads and CTOs whose dashboards go blank on Monday morning more often than they should. It explains why pipelines break, the eight practices that prevent most failures, and what to fix first.
Why do data pipelines break?
Most pipelines break because something upstream changed without warning, not because the code was wrong on the day it was written. A pipeline is a set of assumptions about other systems, and those systems keep moving.
The common causes are familiar to anyone in data engineering:
- Schema changes. Someone renames a field in the CRM or adds a new status value, and a transformation downstream either crashes or quietly drops rows.
- Late or partial data. A source export arrives after the load has run, or only half of it arrives, and the job processes what it finds.
- Unsafe reruns. A job fails halfway, someone restarts it, and the rows that were already written are written again.
- Expired credentials and changed APIs. A service account password rotates, or a vendor deprecates an endpoint.
- Logic hidden in the wrong place. Business rules live in a dashboard or a spreadsheet, so nobody sees them when the pipeline changes.
The difference that matters most is between loud and silent failures. A loud failure stops the job and someone notices. A silent failure lets the job succeed with wrong numbers, and leaders find out in a meeting. Reliability work is mostly about turning silent failures into loud ones.
What does a reliable data pipeline look like?
A reliable data pipeline produces the same result every time it runs on the same input, tells you when it cannot, and can be understood by someone other than its author. Four properties make that possible, and each can be checked.
An idempotent load is one that gives the same end state whether it runs once or five times. A data contract is a written agreement between a source system owner and the data team about structure, keys, update frequency and how changes are announced. Data freshness is the age of the newest record compared with what consumers expect. Observability is the ability to see what ran, what it processed and what failed, without reading the code.
Property
What it protects against
How to check it
Idempotent loads
What it protects against
Duplicates after reruns
How to check it
Run the same load twice in test and compare row counts
Data contracts
What it protects against
Surprise schema changes
How to check it
A documented schema per source, with a named contact
Data tests
What it protects against
Silent wrong numbers
How to check it
Automated tests on keys, nulls, allowed values and references
Freshness and volume monitoring
What it protects against
Late, partial or missing loads
How to check it
Alerts when data is older or smaller than expected
Ownership and runbooks
What it protects against
Slow recovery
How to check it
Every dataset has an owner and a written first response
Eight steps to data pipelines that do not break
The eight steps below are ordered so that each one makes the next one easier, starting with ownership and ending with recovery. You do not need a new platform to begin; most of them apply to the pipelines you already have.
- Name an owner and a consumer for every dataset. Write down who maintains each table and who depends on it. When a dataset has no consumer, consider retiring it rather than making it reliable.
- Agree a contract with each source system. Record the fields you use, the keys, the expected update frequency and who tells you about changes. Even a one-page contract per source turns surprise schema changes into planned work.
- Land raw data unchanged first. Store what the source delivered, with a load timestamp, before you transform it. If a transformation is wrong, you can rebuild from the raw layer instead of asking the source for a new export. The medallion architecture describes this as the bronze layer.
- Make every load idempotent. The Apache Airflow best-practice guide says tasks should be treated like database transactions and produce the same outcome on every re-run, for example by replacing plain inserts with upserts and reading a fixed time partition rather than "now". The principle applies whatever orchestrator you use.
- Test the data, not only the code. Unit tests check your logic; data tests check what actually arrived. dbt's data tests ship with four generic checks: unique, not null, accepted values and relationships between tables. In Databricks pipelines, expectations let you choose to warn, drop invalid records or fail the update. Decide per rule which response fits the risk.
- Monitor freshness and volume. Alert when a table is older than its agreed refresh time or when row counts fall outside a normal range. These two checks catch most partial and late loads before a user does.
- Version-control and deploy through a test environment. Keep pipeline code and transformation logic in Git, review changes, and run the data tests against a copy of production data before you deploy.
- Write a runbook and alert the right person. For each critical dataset, document how to rerun it, how to backfill a missing period and who to tell. Send alerts to the owner, not to a shared channel nobody reads.
Which practices should you fix first?
Fix first whatever protects the numbers leaders act on: tests and freshness alerts on the few datasets behind executive and finance reporting. A full rebuild can wait; the first goal is that nobody makes a decision on stale or duplicated data without knowing it.
A short audit is usually enough to set priorities. List the datasets behind your most important reports, note who uses each one, when it last failed and whether anyone noticed. Then match symptoms to first fixes:
Symptom
Likely cause
First fix
Totals jump after a failed night
Likely cause
Non-idempotent reruns
First fix
Switch to upserts or partition overwrites
A report is empty or stale on Monday
Likely cause
Late source or failed job with no alert
First fix
Freshness alert to the dataset owner
Numbers differ between two reports
Likely cause
Logic duplicated in reports
First fix
Move the rule into one shared transformation
Pipeline crashes after a system update
Likely cause
Unannounced schema change
First fix
Source contract plus schema test
When is a set of scripts no longer enough?
A set of scripts stops being enough when several teams depend on the same data and nobody can say which version of a number is correct. At that point the problem is no longer one broken job but data integration across the business.
Signs that you have reached that point include more sources each quarter, the same customer or product defined differently in different reports, plans to connect AI to company data, and knowledge of the pipelines concentrated in one or two people. If the only data engineer (dataingenjör in Swedish job titles) is on holiday and nobody else can rerun the finance load, the risk is organisational as much as technical.
A shared platform helps because it makes the eight practices the default rather than a discipline each script author has to remember. It does not remove the need for owners, contracts and tests; it gives them one place to live.
Keep reading
From our work
Client Case: HedinFaster loads and clean code: bringing modern software habits to Hedin’s BI platform.Read the caseDevies
Want a second opinion?
Devies, a Gothenburg-based product and engineering consultancy, works on questions like this with Nordic product and IT teams. Tell us where you are and we will suggest a sensible first step.
Frequently asked questions
What is a data pipeline?
A data pipeline is a sequence of automated steps that extracts data from source systems, transforms it into a consistent shape and loads it where it can be used, for example in a data warehouse, a lakehouse, a report or an AI application. A good pipeline also records what it processed, tests the result and alerts someone when data is late, missing or wrong.
What is the difference between ETL and ELT?
ETL means extract, transform, load: data is cleaned before it is stored in the target. ELT means extract, load, transform: raw data is stored first and transformed inside the target platform. ELT has become common with cloud data platforms because it keeps the raw data, which makes it easier to rebuild a table when a transformation turns out to be wrong.
How many data tests does a pipeline need?
Start with tests on the columns that would cause visible harm if wrong: primary keys should be unique and not empty, status fields should only contain known values, and foreign keys should match a record in the related table. Add a freshness and a volume check per critical table. Expand from there when an incident shows a gap, rather than testing every column on day one.
Do we need a data engineer or can analysts maintain pipelines?
Analysts can maintain simple, well-tested transformations, especially in SQL-based tools. Someone with data engineering skills is usually needed for source integrations, orchestration, idempotent loading, security and deployment. Many mid-sized companies combine a small internal team with outside support for the platform, so that knowledge of critical pipelines does not sit with one person.
Sources
- Apache Airflow: Best Practices
- dbt Developer Hub: Add data tests to your DAG
- Databricks: Manage data quality with pipeline expectations
- Devies: Data Foundation Platform
- Devies: DataHub
