Data Understanding Layer Design

Status

Magda v7 core design. The core normalized distribution metadata is implemented by #3807 / #3820 (data-dictionary) and #3808 / #3828 (distribution-contract). The Magda v8 AI Agent Platform (#3810) is developed on next and is a downstream consumer of these v7 capabilities.

This document defines the overall architecture and user experience for helping people and machine clients understand what a dataset contains, how its distributions can be accessed, and how existing dataset-level catalogue metadata contributes context. Detailed core contracts are defined in:

The former dataset-usage proposal is retained as a deferred, use-case-driven extension rather than a required part of the v7 core. See dataset-usage-design.md.

Problem

Magda’s current dataset and distribution pages are strongest when a distribution can be previewed directly. The distribution page renders source/access information and then delegates understanding largely to specialised preview components such as tabular visualisation and map preview.

That model works well for some resources, but it does not generalise to the full range of data that a catalogue may contain:

A universal visualisation layer is therefore neither necessary nor realistic. Magda needs a more general data understanding layer that can explain a resource independently of whether a specialised preview exists.

Goal

Enable a researcher, analyst, developer or software agent to answer the following questions from Magda metadata:

  1. What is this dataset about, and what dataset-level context already exists?
  2. What information or fields does each distribution contain?
  3. How can a distribution be accessed or queried?
  4. What relevant constraints, coverage and provenance information is available?
  5. How could it potentially combine with other datasets?

Dataset-level analytical fitness remains a domain judgement. General context belongs in existing dataset description/coverage/source metadata; a future structured fitness extension should be introduced only when a concrete authoritative use case requires machine-readable claims.

The same machine-readable metadata should support both human-facing catalogue UX and future machine use such as agent tools, federated query adapters, validation and schema-drift detection.

Non-goals

This design does not:

Design principles

1. Explain before visualising

A useful catalogue entry should remain understandable even if no map, table or chart renderer exists. Visualisation remains a valuable progressive enhancement, not the primary data-understanding contract.

2. Keep the layers separate

The v7 core uses two normalized distribution-level contracts, complemented by Magda’s existing dataset-level metadata:

Layer Scope Primary question
existing dataset metadata Dataset What is this dataset about, and what spatial/temporal/source context is known?
distribution-contract Distribution How can I access or query this representation?
data-dictionary Distribution What information does this representation contain?

The same dataset can have multiple distributions with different interfaces and schemas, so structure/access contracts belong at distribution level. Dataset description, spatial/temporal coverage, provenance/source metadata, themes and keywords remain the default home for dataset-level context.

A future dataset-usage extension may add authoritative, purpose-specific fitness or non-inference claims, but it is not a required third core layer. See Dataset Usage / Fitness-for-Use Extension Design.

3. Preserve native specifications as authorities

When a source already exposes a machine-readable contract, Magda should reference it and normalise the subset required for discovery and interaction.

Examples include:

Magda should not become a competing full specification language for those ecosystems.

4. Provenance is part of the meaning

Every generated or imported description must make its origin clear. A field definition supplied by a custodian is materially different from a type inferred from five sampled rows.

Consumers must be able to distinguish, at minimum:

5. Descriptive metadata is not access authority

The data-understanding layer may describe authentication mechanisms, access modes and source-side limitations, but existing Magda access/access-control policy remains authoritative for catalogue access. A future query gateway may have its own execution policy.

6. Normalise semantics without requiring full standardisation

The design should make heterogeneous sources easier to compare without forcing all custodians into one source schema. Normalised field descriptions, semantic concept identifiers, units and stable paths provide a standards-lite mediation layer.

7. Align with established schema standards without adopting one universal schema language

No single existing standard covers all target sources. The normalized aspects should reuse/mirror mature concepts where practical:

Magda remains a mediation layer: native specifications stay authoritative and the normalized aspects expose the common subset needed by people and software agents.

Architecture

External source / native specification
        |
        | crawl, connector, minion, agent-assisted authoring or manual authoring
        v
+----------------------------------+
| Magda metadata                   |
|                                  |
| existing dataset metadata        |
| distribution-contract            |
| data-dictionary                  |
|                                  |
| optional future: dataset-usage   |
+----------------------------------+
        |
        +---------------------+----------------------+-------------------+
        |                     |                      |                   |
        v                     v                      v                   v
Dataset/distribution     Search / semantic      LLM / agent        Future federated
page UX                  indexing               consumption        query adapters
        |
        v
Optional specialised previews
(map / table / chart / other plugins)

Harvesting is intentionally decoupled from rendering. A source adapter can populate normalised metadata even when the web client has no specialised preview implementation for that source type.

The two core aspect contracts, typed client access and read-only UI must also be useful without automatic harvesting. A deployment/project may initially populate them manually through Registry APIs, mgd, project tooling or an agent, then add automatic producers incrementally.

Registry model

The v7 core aspect IDs are:

Dataset-level context continues to use existing catalogue aspects such as dcat-dataset-strings, spatial-coverage, temporal-coverage, provenance/source metadata, themes and keywords.

The core aspect JSON Schemas are shipped as built-in schemas in magda-registry-aspects so they are available to connectors, minions, mgd, the web client and external consumers in the same way as other first-party Magda aspects.

The web client should use explicit typed access to the two normalized contracts rather than depending on ad-hoc reads from rawData in view components.

Each core aspect payload carries its own schema version and provenance/review metadata where appropriate. Consumers must ignore unknown additive fields so a v7.x deployment can extend the contracts without coordinated upgrades of every reader.

If a future dataset-usage aspect is introduced, it should follow the same built-in/typed/provenance conventions only after the revisit criteria in its design are met.

User experience

The dataset/distribution pages should evolve from a preview-first presentation to a consistent explanation-first presentation.

A target distribution experience is:

  1. About this data — title, description, spatial/temporal coverage, source/provenance and other existing dataset-level context.
  2. Structure — normalised data dictionary.
  3. How to use — distribution contract, operations, parameters and native documentation/specification links.
  4. Example — bounded sample rows, response example or safe query result when available.
  5. Access & limitations — access notes, descriptive source constraints, provenance and review status.
  6. Preview — map/table/chart/plugin visualisation where a compatible renderer exists.

The exact visual layout can evolve, but the information architecture should not require a preview to make sections 1-5 useful.

The numbered list describes the information architecture, not the on-page order for distributions that already have a preview. See Placement relative to existing previews.

Placement relative to existing previews

New Data Understanding sections (Structure, How to use, and later sections) are added below the existing preview on the distribution page rather than inserted above it:

On the distribution page, the Data Understanding sections are presented as tabs in one panel (Structure, How to use, and later sections in information-architecture order), so users can switch between them without scrolling:

The same additive principle applies to other pages, such as the dataset page’s Using this dataset section: new sections should extend the page without displacing content users already rely on, unless a later UX review deliberately changes the layout.

Progressive enhancement examples

CSV / Excel

REST API

ArcGIS Feature Service

NetCDF

Core aspect implementation convention established by Data Dictionary

The completed v7 Data Dictionary core (#3807 / #3820) established a reusable implementation pattern that was then applied to Distribution Contract (#3808 / #3828) and should be reused by any future first-party Data Understanding aspect:

#3808 / #3828 already follows this pattern. A future dataset-usage implementation should reuse it where applicable rather than independently rediscovering these conventions.

Metadata production paths

The same aspects may be populated through multiple paths.

Core-first delivery

The first usable v7 milestone should establish the built-in aspect contracts, typed client/model support and read-only UI independently of source-specific automation. Manual/custodian authoring is a first-class production path, not merely a temporary fallback.

Automatic producers can then be added incrementally according to project/source needs. Complete CSV/Excel/OpenAPI/ArcGIS/OGC/NetCDF coverage is not a prerequisite for the normalized contracts to be useful to people, mgd or agents.

Authoritative import

A connector or harvester reads a native specification and produces normalised metadata. Examples: OpenAPI, ArcGIS layer JSON, JSON Schema or Frictionless Table Schema.

Deterministic inspection

A component reads file metadata or a bounded portion of a file and derives structure. Examples: CSV headers and types or NetCDF variable metadata.

Agent-assisted generation

An agent may inspect a resource or documentation and propose descriptions, semantic mappings or usage notes. Agent-generated content must be marked as such and must not silently replace authoritative metadata.

Manual authoring/review

Custodians or catalogue editors can correct or approve generated metadata. The design must preserve review state rather than flattening reviewed and unreviewed content into an indistinguishable result.

Relationship to current Magda components

Registry aspects

magda-registry-aspects is the source of built-in JSON Schemas. The new aspect schemas belong there.

Minions/connectors

Magda already uses connectors to transform external metadata into aspects and minions to enrich records. Automatic data-understanding producers should follow those patterns rather than placing protocol-specific crawling logic in the React web client.

Preferred production patterns are:

A source-specific producer may emit more than one normalized aspect in a single pass. For example, an OpenAPI or ArcGIS adapter may populate both distribution-contract and data-dictionary rather than fetching/parsing the same source twice.

Shared normalization, validation and merge helpers may live in the main Magda repository, while source-specific producers remain independently evolvable.

Web client

DistributionDetails.tsx currently presents source/access information and then mounts DataPreviewVis, DataPreviewMap and plugin visualisation sections. The new renderer should sit alongside those components and work even when the distribution is not previewable.

The current preview block is gated by distribution.downloadURL || distribution.accessURL. The new data-dictionary and distribution-contract sections must not be placed inside that gate: query-only/restricted/service distributions may have useful Data Understanding metadata even when the existing preview path cannot run. They are rendered after the preview block (including plugin visualisation sections), as described in Placement relative to existing previews.

Implementation map for the v7 core

The following current code paths define the v7 core and should be treated as part of the core contracts rather than source-specific producer work:

There is no v7-core requirement to add a dataset-usage schema, fetch path, typed model or dataset-page UI. Those become implementation work only if the deferred extension is reactivated from a concrete use case.

Preview plugins

Existing preview systems remain useful. They consume the same distribution but are not made responsible for explaining generic structure, provenance or API usage.

Search and semantic indexing

A later phase may index selected human-readable portions of existing dataset description/coverage metadata, distribution-contract and data-dictionary so users can search for concepts such as fields and supported operations. Large raw aspect payloads should not be indiscriminately copied into search documents.

If a future dataset-usage extension is implemented, selected authoritative fitness/limitation claims may also be indexed, but ranking must take provenance/review state into account.

Cross-dataset interoperability

The first release should not attempt automatic federation planning. It should, however, record enough semantics to enable later mediation.

Useful building blocks include:

A future service or agent can use these signals to propose relationships such as:

Any such relationship should remain a proposal unless backed by explicit mappings or validated rules.

Delivery sequence

The recommended implementation sequence is now:

  1. complete data-dictionary core (#3807 / #3820);
  2. complete distribution-contract core (#3808 / #3828);
  3. add automatic source-specific producers incrementally through minions/connectors as needs arise; one producer may populate both dictionary and contract where appropriate;
  4. add bounded API/service examples and query exploration with explicit safety policy;
  5. add semantic indexing/search integration for selected existing dataset metadata and normalized contract fields;
  6. expand machine consumption for agents and future query adapters;
  7. revisit dataset-usage only when a concrete authoritative fitness/limitation use case meets the criteria in its deferred design.

This sequence keeps the v7 core useful without coupling it to complete automatic source coverage or to a speculative dataset-level fitness model. The v8 Agent Platform (#3810) can consume existing dataset metadata plus manually or automatically populated distribution contracts as they become available.

Safety and trust boundaries

Compatibility

The feature is additive. Existing datasets and distributions without these aspects continue to render using current behaviour.

The web client should render each section only when the relevant aspect is available. Existing preview-map and tabular preview behaviour remains intact.

Acceptance criteria for the overall capability