For much of my career, infrastructure architecture meant solving problems familiar to any cloud, DevOps, or platform engineer:

1. Where should workloads run, and how should the network be segmented?
2. How do systems authenticate and authorize communication?
3. How do we ensure high availability and resilient failover?
4. How do we automate provisioning through code?
5. How do we monitor health and latency?

In traditional platform engineering, architecture focuses heavily on operational durability—specifically compute sizing, network topology, identity federation, automated provisioning, and failure domains.

These questions remain foundational. However, as enterprise architectures shift towards modern Data & AI platforms and agentic AI systems, a different set of challenges emerges that infrastructure primitives alone cannot address.

When building Data & AI platforms, infrastructure integrity does not guarantee data usability. A platform can achieve 99.99% uptime, with no security drift, while functionally remaining broken for consumers asking fundamental questions:

1. Context: Where is the authoritative customer dataset, and what do the underlying metrics represent?
2. Provenance: Who owns the upstream pipeline and what downstream models break if an upstream schema changes?
3. Integrity: Does the data contain unmasked PII, and is its quality predictable enough for an AI system to consume reliably without exposing sensitive information?

A green dashboard on your clusters tells you the host environment is healthy; it tells you nothing about whether the data flowing through it is reliable, compliant or comprehensible.

Treating a Data & AI platform as a system means recognising that compute and network are part of the foundation rather than the entire architectural surface. The architectural boundary extends beyond infrastructure into data semantics, governance, and trust.

Follow the Data, Not the Infrastructure

Let’s consider a customer dataset in an e-commerce platform, originating from an operational application and stored in a transactional database. The data may first be published to a streaming platform, processed through one or more pipelines, persisted in a data warehouse, transformed into analytical datasets, and eventually consumed by dashboards, data products, or AI applications.

From an infrastructure perspective, every component in this path may be healthy. The streaming platform is available, pipelines are completing successfully, the warehouse is responding within its SLOs, and the consuming applications can establish connectivity.

Yet a consumer discovering a dataset called customer_orders may still be unable to determine whether it is the right dataset to use.

For example:

customer_id	product_id	quantity	price
C10234	P779	2	499.00

1. What exactly does customer_id represent -> a customer, an account, or a household?
2. Is 499.00 the unit price or the total transaction value? Which currency does it represent? Is it before or after tax and discount?
3. Which operational system produced this record?
4. Who owns this dataset?
5. Is customer_id classified as PII?
6. How fresh is the data?
7. What transformations were applied before it reached this dataset?
8. Which dashboards, data products, or AI models depend on it?

The data is available. The infrastructure is healthy. Yet the data is not necessarily understandable or usable.

An Example Data Flow

The Missing Layer: Context

The problem is therefore not that the data does not exist. Nor is it necessarily that the infrastructure hosting the data is unavailable. The missing element is context.

A value such as 499.00 becomes useful only when we know what it represents. A dataset becomes discoverable only when we know that it exists and how it should be described. A consumer can make an informed decision only when ownership, provenance, quality, classification, and relationships are visible.

Where does this context live?

Expanding the Architectural Boundary

This changes how I look at the architecture of a Data & AI platform.

Compute, networking, storage, identity, automation and observability remain foundational. Without them, there is no reliable platform on which workloads can operate. But they represent only one part of the system.

Once we follow the data across producers, processing platforms and consumers, the architectural boundary begins to expand. We now have to consider not only where the data is stored or how it moves, but also what it means, where it originated, who owns it, how it is governed, whether it can be trusted and how other systems depend on it.

The architecture therefore needs to connect two views that are sometimes considered separately: the operational platform that runs the workloads and the information system that makes the data understandable and usable.

This is where I find systems thinking particularly useful. Rather than treating Kubernetes, Kafka, databases, data warehouses, catalogs, pipelines and AI services as independent technologies, we can ask how each capability contributes to the outcome expected from the overall platform.

From Tools to Capabilities

This also changes the way we evaluate technologies. If we start with products, the architecture can quickly become a collection of boxes: Kafka for streaming, Kubernetes for workloads, a data warehouse for analytics, a catalog for discovery and an AI platform for models.

But the existence of those products does not explain the architecture.

A more useful starting point is to ask what capabilities the system requires.

Seeing the platform as a system therefore means starting with the outcome and working backwards towards the capabilities required to achieve it. The technologies still matter, but they become implementation choices within a broader architecture rather than the architecture itself.

Why I Started Looking Deeper

My own journey into metadata management and data catalog architecture did not begin by deciding to become a metadata or data catalog specialist. It emerged while solving a broader Data & AI platform architecture problem.

Coming from an infrastructure, cloud and platform engineering background, my initial instinct was naturally to understand the technologies, integrations and operational architecture. But as the problem expanded into data management and discoverability, understanding how to deploy and integrate the tools was no longer sufficient.

I needed to understand why metadata existed, what exactly was being ingested, how different metadata related to the underlying data, where authority resided, how ownership worked, and why a separate catalog or metadata platform was required when the source systems already maintained metadata of their own.

Those questions forced me to move above the individual tools and look at the environment as an integrated system.

Before we can reason about data catalogs, ingestion architectures or metadata platforms, however, there is a more fundamental question:

Why do we need metadata at all?

If the data already exists in databases, warehouses, streaming systems and applications, what additional problem are we trying to solve?