What a data catalog and data lineage are, how they work together, what they are often confused with, and when they are worth the effort.
In short
A data catalog is a searchable inventory of an organisation's data with descriptions, owners and access information. Data lineage is the record of where each piece of data comes from, how it is transformed and where it is used. The catalog answers "what do we have and can I trust it?"; lineage answers "where did this number come from, and what breaks if I change it?".
As a data estate grows, the hardest question stops being "how do we store this?" and becomes "which of these forty tables called customer is the right one?". A data catalog (datakatalog) and data lineage are the two tools built for that question. This explainer from Devies, a Gothenburg-based product and engineering consultancy, is for data leads, architects and governance owners deciding whether they need them and what to expect.
What is a data catalog?
A data catalog is a searchable inventory of data assets, such as tables, files, reports and semantic models, enriched with business descriptions, owners, classifications and access rules. It works like a library catalogue for data: you search for a concept, find the relevant assets and see whether they are certified, who to ask and how to get access.
A useful catalog holds three kinds of metadata:
- Technical metadata: schemas, column types, locations and refresh times, usually harvested automatically by scanning sources.
- Business metadata: descriptions, glossary terms, owners, stewards and certification status, added by people.
- Operational metadata: usage, quality scores and lineage, produced as data moves.
Vendors package these differently. Microsoft Purview Unified Catalog, for example, organises data into governance domains and data products, with glossary terms, critical data elements, access policies and data quality scores on top of an inventory of assets, metadata and lineage. Databricks Unity Catalog and the catalogs in Snowflake and other platforms cover similar ground inside their own ecosystems.
What is data lineage?
Data lineage is the record of how data flows from its sources through transformations to the reports, models and applications that consume it. It shows dependencies upstream (where did this come from?) and downstream (who uses this?).
Lineage comes at different levels of detail:
- Table-level lineage: table A feeds table B, which feeds report C.
- Column-level lineage: the revenue column in the report is derived from these three source columns via this calculation.
- Process lineage: which job, notebook or pipeline run performed each step, and when.
The practical uses are concrete. Databricks documents four of them for Unity Catalog lineage: impact analysis before changing a table or column, root-cause investigation when a report looks wrong, tracking sensitive data for compliance and understanding dependencies between teams. Unity Catalog captures lineage automatically for queries it runs, down to column level, but only for assets it governs; sources and tools outside the platform must be registered separately.
For lineage across tools from different vendors, OpenLineage is an open specification for collecting lineage events. It models datasets, jobs and runs, and is a graduate project of the LF AI & Data Foundation.
How are a data catalog and lineage different from related things?
A data catalog and data lineage are metadata tools: they describe data, but they neither store it nor define its meaning by themselves. The table separates them from things they are often confused with.
Concept
What it is
How it differs
Data catalog
What it is
Searchable inventory of data assets with business context
How it differs
Describes and points to data; does not hold it
Data lineage
What it is
Map of data flows and dependencies
How it differs
Shows how data moves; a catalog shows what exists
Business glossary
What it is
Agreed definitions of business terms
How it differs
Defines words; a catalog links those words to actual tables
Data warehouse or lakehouse
What it is
Storage and processing of the data itself
How it differs
Holds the data; the catalog describes it
Semantic layer
What it is
Shared definitions of metrics and dimensions for analysis
How it differs
Calculates numbers consistently; lineage shows where its inputs came from
Data governance
What it is
Roles, rules and decisions about data
How it differs
The operating model; catalog and lineage are tools that support it
Data dictionary
What it is
Technical list of fields and types for one system
How it differs
A narrow subset of what a catalog holds
The most common misunderstanding is to treat the catalog as the goal. A catalog full of undocumented tables is a more expensive way of not knowing what you have. It becomes valuable only when owners maintain descriptions and certify the assets people should use, which is a data governance task, not a tooling one.
When does a company need a data catalog and lineage?
A company needs a data catalog when people regularly cannot find data, cannot tell which version to trust or depend on a few individuals who know where everything is. It needs lineage when changes break reports unexpectedly, when finding the cause of a wrong number takes days or when regulators or auditors ask where data came from.
Typical signals:
- Analysts ask the same "where is the data for X?" questions every week.
- Several reports with similar names show different numbers, and the team spends time on reconciliation rather than analysis; a single source of truth is the underlying goal.
- A schema change in an ERP integration breaks reports nobody knew about.
- Personal or regulated data flows through the platform and you must show where it goes.
- An AI assistant is about to use internal data, and someone has to decide which sources are fit for it.
Lineage also matters when a platform changes hands. When Devies took over Stena Line's data platform after the original team left, the first step was to map how data moved into the warehouse and which transformations held the business rules, before changing anything. Automated lineage makes that kind of mapping faster and keeps it current.
How do you start without a big catalog project?
Start with the data people use most and grow the catalog from there, instead of scanning everything and hoping someone documents it. A practical sequence:
- Use what your platform already offers. Fabric, Databricks, Snowflake and Purview all include catalog and lineage features; a separate tool is rarely the first step.
- Pick one domain, such as customers or finance, and the ten to twenty assets that matter most.
- Assign owners and write short descriptions in business language, including known limitations.
- Certify the trusted assets and mark duplicates or legacy tables as deprecated.
- Turn on automated lineage within the platform and register key external sources and reports.
- Link quality scores so users see whether data passed its checks; see our guide to measuring and fixing data quality.
- Review usage after three months and expand to the next domain only when the first one is maintained.
Expect gaps. Automated lineage rarely covers spreadsheets, manual exports or transformations in tools outside the platform, so record those dependencies by hand until they can be brought into a governed pipeline.
Keep reading
From our work
Client Case: Stena LineStena Line modernised its Azure data platform - improving Power BI reliability, reducing pipeline runtimes and scaling with medallion architecture.Read the caseDevies
Want a second opinion?
Devies, a Gothenburg-based product and engineering consultancy, works on questions like this with Nordic product and IT teams. Tell us where you are and we will suggest a sensible first step.
Frequently asked questions
What is the difference between a data catalog and data lineage?
A data catalog is an inventory: it lists data assets with descriptions, owners and access rules so people can find and judge them. Data lineage is a map of flows: it shows where data came from, how it was transformed and which reports or applications depend on it. Most modern catalog tools include lineage, and the two are most useful together.
Do we need a separate data catalog tool?
Often not at first. Microsoft Purview, Databricks Unity Catalog, Snowflake and Microsoft Fabric all include catalog and lineage features for the data they manage. A separate, cross-platform tool becomes relevant when data is spread across several platforms and vendors, when many teams need one shared search, or when the built-in features lack the governance workflow you need.
Can data lineage be fully automated?
Partly. Platforms capture lineage automatically for queries and pipelines they run, often down to column level. They cannot see transformations in spreadsheets, manual file exports or tools outside the platform unless these are registered or emit lineage events, for example through the OpenLineage specification. Expect automated lineage for the governed core and manual documentation for the edges.
Who maintains a data catalog?
Data owners and stewards maintain the business content: descriptions, glossary links, certification and known limitations. The data platform team maintains the technical side: scanning, lineage capture, access integration and tool administration. If no named person is responsible for a domain's entries, the catalog will go out of date within months, however good the tool is.
