The Metadata Lifecycle: How Knowledge Is Created, Changed and Kept Current

Understanding Metadata Systems

This article is part of my Understanding Metadata Systems series, where I explore metadata platforms from first principles—from why metadata exists to how these systems are modelled, implemented and operated.

Previous: Modelling Metadata: From Data Assets to Entities, Identity and Relationships

In the previous article, we explored how a metadata system can represent the things it needs to know about.

We started with entities, then introduced identity, type, metadata components, extensibility and relationships.

Using our familiar customer_orders example, we gradually moved from a simple data asset:

customer_orders

to a richer representation:

customer_orders
      │
      ├── is a → Dataset
      ├── belongs to → Commerce Domain
      ├── owned by → Commerce Team
      ├── produced by → order_pipeline
      ├── consumed by → sales_dashboard
      ├── contains → customer_id
      │                  └── classified as → PII
      │
      ├── freshness → 15 minutes
      └── status → Active

At this point, our metadata system has a structured representation of the data ecosystem.

But there is a problem.

The ecosystem it represents does not remain static.

Schemas change. Pipelines change. Owners change. Classifications change. New assets appear, while others are deprecated or removed.

So representing metadata correctly at one point in time is not enough.

This leads us to another architectural question:

If the systems, data assets and organizations represented by a metadata platform are constantly changing, how does the platform keep its knowledge sufficiently aligned with reality?

This is the problem of the metadata lifecycle.

A Correct Model Can Still Become Wrong

Suppose that at 10:00 AM our metadata platform knows the following about customer_orders:

Entity:          customer_orders
Type:            Dataset
Owner:           Commerce Team
Schema:          customer_id, product_id, quantity, price
Classification:  customer_id → PII
Freshness:       15 minutes
Status:          Active

Everything is correct.

Now imagine that during the day several things happen.

10:15 AM
A new column is added:
discount_amount DECIMAL(10,2)
11:00 AM
Owner changes:
Commerce Team → Customer Platform Team
12:30 PM
A security classification changes
2:00 PM
Pipeline schedule changes:
15 minutes → 30 minutes
Next month
customer_orders → Deprecated

Nothing unusual has happened. This is simply how real production environments evolve.

But unless those changes also reach the metadata platform, its representation gradually diverges from the environment it is supposed to describe.

At 3:00 PM the actual environment might say:

REAL WORLD
Owner:
Customer Platform Team
Schema:
customer_id
product_id
quantity
price
discount_amount
Freshness:
30 minutes

while the metadata platform still says:

METADATA PLATFORM
Owner:
Commerce Team
Schema:
customer_id
product_id
quantity
price
Freshness:
15 minutes

Both representations may have been correct at different moments. The problem is that one of them is now stale.

This gives us an important principle:

Metadata correctness is not only about representing knowledge correctly. It is also about keeping that representation sufficiently aligned with the world it describes.

Where Does Metadata Come From?

Before asking how metadata remains current, we need to ask where it comes from in the first place.

As we saw earlier in this series, there is rarely one universal source of metadata.

Different kinds of knowledge originate in different places.

System-Generated Metadata

Some metadata can be discovered directly from technical systems.

For example:

Data Warehouse
      ↓
Tables
Columns
Data types
Views
Orchestrator
      ↓
Schedules
Execution status
Last successful run
Processing Systems
      ↓
Dependencies
Lineage
Data Quality System
      ↓
Quality rules
Quality results

These systems already know these facts as part of their normal operation.

The metadata platform does not necessarily create this knowledge. It acquires and represents it.

Human-Supplied Metadata

Other knowledge may not exist in a form that can reliably be discovered from technical systems.

For example:

Business description
Business meaning
Ownership context
Usage guidance
Domain information
Glossary associations

A database can tell us that price is a DECIMAL(10,2).

It probably cannot tell us that:

price represents the unit selling price after discount but before tax.

That knowledge may need to come from the people or teams who understand the business context.

Derived Metadata

There is also knowledge that may be calculated or inferred from other information.

For example:

Usage statistics
Impact information
Aggregated quality signals
Derived relationships
Popularity indicators

So even before discussing ingestion technologies, we can identify three broad origins:

System-generated
Human-supplied
Derived
       ↓
Metadata Knowledge

Their origins differ, and therefore their lifecycle characteristics may differ as well.

How Does Knowledge Enter the Metadata Platform?

Once metadata exists somewhere, the platform needs a mechanism to acquire it.

There is no single universal approach.

Several patterns naturally emerge.

Polling or Pull

The metadata platform periodically asks a source:

What do you know now?

For example, it might inspect a warehouse every few hours to discover databases, schemas, tables, columns and views.

Conceptually:

Metadata Platform
       │
       │ periodically queries
       ▼
   Source System
       │
       │ returns metadata
       ▼
Metadata Platform

This can be relatively straightforward because the source does not necessarily need to actively participate in the metadata platform.

But there is an obvious trade-off.

If discovery runs every six hours, a change may remain unknown to the platform for several hours.

Metadata freshness therefore depends partly on the synchronization frequency.

Push or API-Based Updates

Instead of waiting for the metadata platform to discover a change, another system can actively send metadata to it.

Producer
   │
   │ sends metadata
   ▼
Metadata Platform

This can reduce the delay between a change and its representation in the platform.

But now the producer needs to know how and when to communicate with the metadata system.

Event-Driven Updates

Changes can also be communicated through events.

Source / Producer
       │
       │ metadata event
       ▼
   Event Channel
       │
       ▼
Metadata Platform

This can support more continuous propagation of metadata changes.

However, once events and asynchronous processing are introduced, we also introduce familiar distributed-systems questions:

• What happens if an event is delayed?
• What if processing fails?
• What if events arrive out of order?
• What if the consumer is temporarily unavailable?
• How do we know whether the final state is correct?
• We will return to these questions later in the series when we look inside the architecture of metadata platforms.

Human Input

Some metadata will continue to require human contribution.

• A domain owner may update a description.
• A steward may associate an asset with a glossary term.
• A security team may review a classification.
• A data owner may document intended usage.

So a metadata platform may simultaneously receive knowledge through:

Polling
Push / APIs
Events
Human Input
Derived Processing
        │
        ▼
 Metadata Platform

The important point is:

A metadata platform does not necessarily have one ingestion mechanism. Different kinds of knowledge may enter through different paths.

Full or Incremental Synchronization?

Once metadata can be acquired, another question appears.

Should we repeatedly retrieve everything, or only retrieve what changed?

Full Synchronization

A full synchronization effectively asks:

Tell me everything you currently know.

Suppose the warehouse contains 10,000 tables.

A full discovery process may inspect all 10,000 again even if only ten changed since the previous run.

The advantage is that the platform periodically gets another opportunity to compare its understanding with the source’s current state.

The disadvantage is cost.

At large scale, repeatedly rediscovering everything can become expensive in time, compute and source-system load.

Incremental Synchronization

An incremental approach instead asks:

What changed since the last successful synchronization?

Perhaps only these events need to be processed:

Table created
Column added
Owner changed
Pipeline updated
Dataset deprecated

This can be much more efficient.

But it creates another requirement:

The system must reliably determine what changed. If a change is missed, the platform may continue carrying stale state.

The trade-off therefore looks roughly like this:

FULL SYNCHRONIZATION
+ Easier to reconstruct current state
+ Useful for reconciliation
- More expensive
- More source-system work
INCREMENTAL SYNCHRONIZATION
+ Efficient
+ Faster propagation of changes
- Requires reliable change tracking
- Missed changes can create divergence

Many real systems therefore combine incremental processing with periodic reconciliation.

Change Is More Difficult Than Creation

Creating metadata is relatively easy to understand.

Yesterday there was no entity.

Today the source contains:

customer_orders

The metadata platform creates a representation.

But updates raise more interesting questions.

Suppose:

Commerce Team
      ↓
Customer Platform Team

Did the entity change?

No.

Our knowledge about the entity changed.

Suppose a column is added:

customer_id
product_id
quantity
price
discount_amount

Again, the dataset may retain the same identity even though one of its metadata components has changed.

This is why the identity concept from the previous article matters.

The metadata system needs to distinguish between the identity of an entity and the knowledge that changes around that identity.

Without stable identity, an update could accidentally be interpreted as a new entity.

What Does Disappearance Mean?

Deletion introduces an even more subtle problem.

Suppose yesterday the source contained:

customer_orders

Today the metadata discovery process cannot see it.

Does that mean the dataset was deleted?…..Maybe.

But there are other possibilities, such as:

• The service account may have lost permission.
• The source may be temporarily unavailable.
• The discovery process may have failed.
• A filter may have changed.
• The object may have been renamed.
• The schema may no longer be visible to that particular integration.

Therefore:

Not seeing an entity is not necessarily the same as knowing that the entity no longer exists.

This distinction matters.

Immediately deleting metadata because an object disappeared from one discovery run could remove valid knowledge because of a temporary technical failure.

On the other hand, never removing anything would eventually fill the platform with stale assets.

Metadata systems therefore need some way to reason about lifecycle states such as:

Active
Deprecated
Removed
Unknown

The exact states and behaviour depend on the platform and organization, but the underlying problem exists regardless of technology.

When the Platform and Reality Disagree

We have now reached one of the most important problems in the metadata lifecycle.

Imagine:

AUTHORITATIVE SOURCE             METADATA PLATFORM
Schema v7                        Schema v6
Owner:
Customer Platform Team          Commerce Team
Freshness:
30 minutes                      15 minutes
Status:
Deprecated                      Active

The metadata platform itself may be perfectly healthy.

• Its API responds.
• Its database is available.
• Search works.
• The UI loads.
• CPU and memory are normal.

From an infrastructure perspective, everything may appear healthy. Yet the platform is providing incorrect knowledge.

This gives us another important distinction:

Platform availability does not guarantee metadata correctness.

A healthy metadata service can still contain stale or incomplete knowledge.

This is where reconciliation becomes important.

Reconciliation: Comparing Knowledge With Reality

Reconciliation is the process of detecting and correcting divergence between the metadata platform’s representation and the sources that are authoritative for particular facts.

Conceptually:

Authoritative Source
        │
        │ current state
        ▼
     Compare
        ▲
        │ represented state
        │
Metadata Platform
        │
        ▼
 Detect Difference
        │
        ▼
 Correct / Reconcile

For example:

Source:
Owner = Customer Platform Team
Platform:
Owner = Commerce Team
Difference detected
        ↓
Platform representation updated

Reconciliation becomes particularly valuable when metadata is propagated incrementally or asynchronously.

Individual changes may be delayed or missed.

A periodic comparison with authoritative state provides another mechanism for detecting divergence.

This does not mean every metadata component must constantly be compared with every source. That may be operationally impractical.

The appropriate frequency and strategy depend on factors such as:

• Importance of the metadata
• Rate of change
• Cost of discovery
• Acceptable staleness
• Source-system capabilities
• Business and governance requirements

The architectural objective is not necessarily instantaneous synchronization. It is maintaining an acceptable relationship between the represented state and the authoritative state.

Metadata Freshness Is a System Property

We often talk about the freshness of data. But metadata itself also has freshness.

Consider a business definition that was reviewed two years ago.

Or an owner who left the team six months ago.

Or lineage captured before a pipeline redesign.

Or a classification that no longer reflects the current regulatory interpretation.

The metadata may still physically exist. The metadata platform may still serve it successfully. But its usefulness has degraded.

So metadata freshness cannot be considered only an ingestion concern.

It depends on the entire lifecycle:

Source of Knowledge
        ↓
Production
        ↓
Acquisition
        ↓
Representation
        ↓
Updates
        ↓
Consumption
        ↓
Reconciliation
        ↓
Deprecation / Removal

A failure anywhere in this lifecycle can create stale or misleading knowledge.

A Metadata Lifecycle Emerges

We began this article with a simple question:

How does a metadata platform remain aligned with a world that keeps changing?

Without starting from any particular metadata product, several lifecycle stages have emerged.

Produce

Some system, person or process knows something about an entity.

Acquire

The metadata platform receives or discovers that knowledge.

Represent

The knowledge is associated with the appropriate entity and metadata component.

Update

As the underlying environment changes, the represented knowledge changes.

Consume

Humans and systems discover and use the metadata.

Reconcile

The platform detects and corrects divergence from authoritative sources.

Deprecate or Remove

Knowledge and entities eventually reach the end of their useful lifecycle.

Conceptually:

PRODUCE
   ↓
ACQUIRE
   ↓
REPRESENT
   ↓
UPDATE
   ↓
CONSUME
   ↓
RECONCILE
   │
   └───────────┐
               │
               ▼
             UPDATE
               │
               ▼
       DEPRECATE / REMOVE

This should not be interpreted as a rigid industry-standard workflow.

It is a reasoning model.

Real metadata platforms may implement these stages differently, combine some of them, or introduce additional mechanisms.

What matters is the underlying principle:

Metadata is not static documentation. It is knowledge with its own lifecycle.

From Lifecycle to Platform Implementation

Across the series so far, we have deliberately avoided beginning with a metadata product.

Instead,

• we started with the problems. We asked why metadata is necessary.
• We explored why distributed knowledge eventually creates the need for a shared metadata capability.
• We separated ownership, authority and permission.
• We developed a generic model based on entities, identity, types, metadata components, extensibility and relationships.

And now we have explored how that knowledge is created, changed, synchronized, reconciled and eventually retired.

At this point we have enough of a conceptual foundation to ask a different question.

How do modern metadata platforms actually implement these ideas?

That is:

• How do they represent entities?
• How do they create stable identities?
• How do they divide metadata into independently changing components?
• How do they represent relationships?
• How do they allow organizations to extend the model?
• And how closely do different platforms follow the generic model we have derived?

That takes us from the conceptual model into real platform architecture.

Continue the Series

Previous: Modelling Metadata: From Data Assets to Entities, Identity and Relationships

Next: From Model to Platform: How Modern Metadata Systems Represent Knowledge — Coming soon

Understanding Metadata Systems — Series

This article is part of my Understanding Metadata Systems series, where I explore metadata platforms from first principles—from why metadata exists to how these systems are modelled, implemented and operated.

1. Beyond Infrastructure: Seeing the Data & AI Platform as a System
2. Why Metadata? When Data Alone Is Not Enough
3. Why a Metadata Platform? When Distributed Knowledge Stops Scaling
4. Who Owns Metadata? Authority, Accountability and the Operating Model
5. Modelling Metadata: From Data Assets to Entities, Identity and Relationships
6. The Metadata Lifecycle: How Knowledge Is Created, Changed and Kept Current
7. From Model to Platform: How Modern Metadata Systems Represent Knowledge — Coming soon
8. Inside a Metadata Platform: Storage, Events, Search and Distributed State — Coming soon
9. Operating Metadata at Scale: Reliability, Consistency and Observability — Coming soon
10. Metadata for AI: Giving Machines the Context to Understand and Govern Data — Coming soon