Modelling Metadata: From Data Assets to Entities, Identity and Relationships

Understanding Metadata Systems

This article is part of a series exploring metadata systems from first principles—from why metadata exists to how modern metadata platforms are designed, implemented and operated.

Previous: Who Owns Metadata? Authority, Accountability and the Operating Model

In the previous article, we explored an important distinction: bringing metadata together into a shared platform does not mean that the platform becomes authoritative for every piece of knowledge it contains.

A database may remain authoritative for its physical schema. A business domain may define business meaning. Security or governance may determine classification, while pipelines and operational systems provide information about freshness and execution.

This leaves us with another architectural question.

If a metadata platform needs to bring together knowledge about datasets, pipelines, dashboards, domains, models and many other objects, how should it represent all these different things consistently?

Before thinking about databases, APIs, search engines or metadata products, we first need to understand what information the system actually needs to represent.

Before deciding how metadata should be stored, we first need to decide what the metadata system needs to know.

What Does the Metadata System Need to Know About?

Let us return to the customer_orders example we have been using throughout this series.

Initially, we may think that our metadata system only needs to know about the dataset:

customer_orders

But understanding this dataset may require knowledge about several other things:

customer_orders
order_pipeline
sales_dashboard
Commerce domain
customer_id

These are clearly not all the same kind of thing.

• customer_orders may be a dataset.
• order_pipeline may be a data-processing pipeline.
• sales_dashboard may be an analytical dashboard.
• Commerce may represent a business domain.
• customer_id may be a field within a dataset.

Yet our metadata system needs some way to represent and reason about all of them.

For now, let us use the informal word thing to describe anything in the data ecosystem that we want the metadata system to identify, describe or relate to something else.

The word thing is not intended as a formal metadata-management term. It is simply a useful starting point for our reasoning.

Once that thing is represented inside the metadata system, we can think of that representation as an entity.

This gives us our first building block.

Thing in the data ecosystem
          ↓
       Entity

Identity: Which Exact Thing Are We Talking About?

Creating an entity introduces another problem.

Suppose our metadata platform contains:

customer_orders

Is the name enough to identify it? ….Probably not.

We might have:

PRODUCTION.COMMERCE.customer_orders
TEST.COMMERCE.customer_orders

Or two completely different systems might independently contain an object called customer_orders.

The human-readable name tells us something about the object, but it may not uniquely identify it.

This leads to another requirement:

Every entity needs an identity that allows the metadata system to distinguish it unambiguously from other entities.

Conceptually, an identity might include information such as:

Platform:     Data Warehouse
Environment:  Production
Database:     Commerce
Schema:       Sales
Object:       customer_orders

The exact implementation is not important yet.

Different metadata platforms may use URIs, URNs, keys, paths, internal IDs or other mechanisms.

The architectural requirement comes first:

Name and identity are not necessarily the same thing.

A name is often intended for humans.

Identity allows the system to know precisely which entity we mean.

Type: What Kind of Thing Is It?

Knowing the identity still does not tell us what kind of entity we are dealing with.

Consider:

customer_orders
order_pipeline
sales_dashboard
customer_churn_model
Commerce

The metadata system may need to treat each differently.

• A dataset may have columns and schemas.
• A pipeline may have execution information and dependencies.
• A dashboard may contain charts and consume datasets.
• An ML model may have training data, features and evaluation information.
• A domain may group assets around a business capability.

So an entity also needs a type.

Conceptually:

Entity                    Type
customer_orders           Dataset
order_pipeline            Pipeline
sales_dashboard           Dashboard
customer_churn_model      ML Model
Commerce                   Domain

The type tells the metadata system what kind of thing an entity represents and what kinds of knowledge may reasonably describe it.

This distinction between an entity and an entity type is important.

customer_orders is an entity.
Dataset is its type.

What Do We Know About the Entity?

Now we can identify customer_orders and know that it is a dataset. But we still know very little about it.

Consumers may want to know:

• What does it mean?
• Who owns it?
• What fields does it contain?
• Does it contain sensitive information?
• How frequently is it refreshed?
• Is it reliable?
• Is it still active?

These are different pieces of knowledge describing the same entity.

For example:

Entity:
customer_orders
Identity:
production / commerce / customer_orders
Type:
Dataset
Description:
Customer orders placed through commerce channels
Schema:
customer_id STRING
price DECIMAL(10,2)
Owner:
Commerce Team
Classification:
customer_id → PII
Freshness:
15 minutes
Quality:
PASS
Status:
Active

Instead of treating metadata as one large undifferentiated document, it is useful to think of these as different metadata components describing different characteristics of the entity.

Conceptually:

                 customer_orders
                       │
        ┌──────────────┼───────────────┐
        │              │               │
      Schema        Ownership     Classification
        │              │               │
    Freshness        Quality          Status

This becomes particularly useful as metadata grows because different components may have different producers, authorities and lifecycles—as we discussed in the previous article.

What If Our Organization Needs Something More?

No generic metadata model can predict every piece of knowledge that every organization will eventually need.

Suppose our organization wants to capture:

Criticality: Tier 1
Data Residency: EU
Business Review Cycle: Quarterly
Regulatory Scope: GDPR
Cost Centre: CC-204

Another organization may not care about any of these attributes but may require completely different information.

Does that mean the core metadata model must be redesigned every time a new requirement appears?…….Ideally, No.

A useful metadata model therefore needs some form of extensibility.

It should provide a stable core model while allowing organizations to attach additional structured knowledge appropriate to their own business, governance and operational requirements.

This gives us another architectural requirement:

The model should be structured enough to remain consistent, but extensible enough to represent knowledge that was not anticipated when the platform was first designed.

How individual products implement that extensibility is an implementation decision we can examine later.

Relationships: Metadata Is Not a Collection of Isolated Objects

So far, we have represented individual entities and the knowledge describing them. But data systems do not consist of isolated objects.

Consider:

order_service
      ↓
order_pipeline
      ↓
customer_orders
      ↓
sales_dashboard

These relationships tell us something important that the individual entities cannot tell us alone.

• The pipeline produces the dataset.
• The dashboard consumes the dataset.
• The dataset may belong to the Commerce domain.
• A team may own the data product.
• A model may use the dataset for training.
• Another dataset may be derived from it.

Once relationships are represented, the metadata system can begin answering much richer questions:

• Where did this data come from?
• What consumes it?
• What could be affected if it changes?
• Which datasets belong to this business domain?
• Which pipelines produce data used by this dashboard?

The system is no longer merely describing individual assets. It is representing connected knowledge about the data ecosystem.

Does This Mean We Need a Graph Database?

At this point, the model naturally begins to look graph-shaped.

We have entities:

order_service
order_pipeline
customer_orders
sales_dashboard
Commerce

and relationships:

order_service
      │ produces
      ▼
order_pipeline
      │ produces
      ▼
customer_orders
      │ consumed by
      ▼
sales_dashboard
customer_orders
      │ belongs to
      ▼
Commerce

It is tempting to immediately conclude:

We need a graph database.

But that would mix two different architecture decisions.

The logical information model tells us that entities are connected through relationships.

The physical storage model determines how those entities and relationships are actually persisted.

A graph-shaped information model could potentially be implemented using relational storage, document storage, graph technology, search indexes or a combination of technologies.

Therefore:

A graph-shaped metadata model does not automatically require a graph database.

We should first model the information correctly. Storage technology comes later.

A Generic Metadata Model Begins to Emerge

We started without assuming any metadata product. We simply asked what the system needs to know.

That reasoning has given us a relatively small set of concepts:

Together, these concepts give us a generic mental model:

This is not intended to prescribe how every metadata platform must be implemented.

It gives us something more useful: a product-independent way to reason about what a metadata system needs to represent.

Later, when we examine real metadata platforms, we can ask how each one implements these concepts rather than learning its terminology in isolation.

From Metadata Objects to Connected Knowledge

Let us return one final time to customer_orders.

At the beginning, it was simply a table name.

Now our metadata system can understand considerably more:

The original data has not changed. What has changed is the structure of the knowledge surrounding it.

And that is an important part of what makes a metadata platform useful: not simply storing descriptions of assets, but building a structured and connected representation of the data ecosystem.

But the World We Model Does Not Stand Still

We now have a way to think about what a metadata platform needs to represent.

But there is another problem.

Tomorrow:

Commerce Team → Customer Platform Team
price DECIMAL(10,2) → DECIMAL(12,2)
Refresh 15 minutes → 30 minutes
customer_orders → Deprecated

The entity may be the same, but our knowledge about it has changed.

• A new dataset may appear.
• A pipeline may be replaced.
• A classification may change.
• An owner may move to another team.
• A relationship may disappear.
• A previously active asset may eventually be deleted.

If the metadata platform continues to represent yesterday’s state, its structured model may be technically elegant but operationally misleading.

So our next question becomes:

How does metadata enter the platform, change over time and remain sufficiently aligned with the systems and organizations it describes?

That takes us to the next article in the series.

Continue the Series

Previous: Who Owns Metadata? Authority, Accountability and the Operating Model

Next: The Metadata Lifecycle: How Knowledge Is Created, Changed and Kept Current

Understanding Metadata Systems — Series

This article is part of my Understanding Metadata Systems series, where I explore metadata platforms from first principles—from why metadata exists to how these systems are modelled, implemented and operated.

1. Beyond Infrastructure: Seeing the Data & AI Platform as a System
2. Why Metadata? When Data Alone Is Not Enough
3. Why a Metadata Platform? When Distributed Knowledge Stops Scaling
4. Who Owns Metadata? Authority, Accountability and the Operating Model
5. Modelling Metadata: From Data Assets to Entities, Identity and Relationships
6. The Metadata Lifecycle: How Knowledge Is Created, Changed and Kept Current
7. From Model to Platform: How Modern Metadata Systems Represent Knowledge — Coming soon
8. Inside a Metadata Platform: Storage, Events, Search and Distributed State — Coming soon
9. Operating Metadata at Scale: Reliability, Consistency and Observability — Coming soon
10. Metadata for AI: Giving Machines the Context to Understand and Govern Data — Coming soon