Unified Data Dictionary Design

Status

Proposed design for a Magda v7 capability. Implementation targets the main branch.

This document defines the data-dictionary distribution aspect introduced by the data understanding layer.

Purpose

Different data sources expose their structure in different ways:

Magda needs one normalized representation that allows users and software clients to answer:

The data-dictionary aspect provides that representation.

Goals

The design should:

Non-goals

The data dictionary does not:

Standards alignment

There is no single existing schema standard that cleanly covers all of Magda’s target sources. The data-dictionary should therefore remain a compact normalised logical projection, but its concepts should align with established standards so source adapters can map deterministically rather than inventing new semantics.

Relevant standards/conventions include:

The normalized aspect does not claim conformance to all of these standards. Instead, producers should map authoritative source metadata into common Magda concepts and retain the native source/specification reference for details that cannot be represented losslessly.

Compatibility contract

For v1, compatibility means that Magda can represent a useful logical description of a source for discovery, UI and agent/query planning. It does not mean lossless round-trip reconstruction of every native schema.

The v1 model should be able to represent, at a useful logical level:

Unknown/native details should remain available through source metadata rather than forcing the normalized contract to become a replacement for every source schema language.

Aspect ID and placement

Aspect ID:

data-dictionary

Record type:

distribution

Built-in JSON Schema location:

magda-registry-aspects/data-dictionary.schema.json

Core model

The dictionary is organized around entities. An entity is a logical record shape such as:

Each entity contains fields addressed by stable field paths.

Proposed v1 shape

{
  "schemaVersion": "1.0",
  "source": {
    "type": "arcgis-layer-definition",
    "url": "https://example.org/FeatureServer/0?f=pjson",
    "retrievedAt": "2026-09-14T10:00:00Z",
    "fingerprint": "sha256:..."
  },
  "provenance": {
    "method": "authoritative-import",
    "generator": "magda-data-dictionary-harvester/1.0",
    "generatedAt": "2026-09-14T10:00:00Z",
    "reviewStatus": "unreviewed"
  },
  "entities": [
    {
      "id": "tree-crop-feature",
      "name": "Tree crop feature",
      "role": "response-record",
      "description": "One mapped tree-crop feature",
      "fields": [
        {
          "path": "commodity",
          "name": "commodity",
          "type": "string",
          "description": "Mapped tree-crop commodity",
          "semanticConcept": "crop-type"
        },
        {
          "path": "year",
          "name": "year",
          "type": "string",
          "description": "Feature-level mapping currency"
        },
        {
          "path": "hectares",
          "name": "hectares",
          "type": "number",
          "unit": "ha",
          "description": "Mapped feature area"
        }
      ]
    }
  ]
}

schemaVersion

Required string for the normalized dictionary contract.

Initial value:

1.0

Consumers should tolerate unknown additive properties in the same major version.

Minimum v1 structural requirements

For schema version 1:

These requirements keep the machine contract predictable without requiring every optional descriptive/semantic property to be known.

Source metadata

The source object describes the schema/metadata source used to build the dictionary.

Suggested properties:

Example source types:

Provenance metadata

The provenance object explains how the normalized dictionary was produced or curated and carries the dictionary-level review state.

Suggested fields:

{
  "method": "inferred",
  "generator": "magda-csv-inspector/1.0",
  "generatedAt": "2026-09-14T10:00:00Z",
  "sample": {
    "rows": 100,
    "bytes": 65536
  },
  "reviewStatus": "unreviewed"
}

Initial method values:

Initial reviewStatus values:

Where inference is based on samples, the sample extent must be recorded so the UI does not present it as complete-source authority.

Entities

Each entity should have:

Initial role examples:

The role is descriptive and extensible rather than a closed list.

Fields

A normalized field should support the following core properties.

Identity

path is required because nested JSON cannot be represented reliably by name alone.

Examples:

scientificName
occurrences[].scientificName
properties.location.latitude
items[].measurements[].value

The first version should use a documented lightweight path notation suitable for display and machine matching. It does not need to implement the full JSONPath language.

Normalized path rules

path is a stable normalized schema path, not an executable JSONPath/query expression.

For v1:

Examples:

scientificName                    # simple field
properties.location.latitude      # nested object
items[].measurements[].value      # nested arrays
a\.b                              # literal flat/source field named "a.b"
items[].a\.b                      # nested property literally named "a.b"
field\[\]                         # literal field named "field[]"

sourcePath may preserve a native path/addressing form where useful. For flat tabular resources, name remains the exact source column name even when path must escape reserved characters.

The normalized path grammar exists for identity/matching and UI/agent consumption. A query adapter is responsible for translating it into JSONPath, SQL identifiers, ArcGIS field references or another source-specific expression.

Type and format

Suggested normalized values for type:

format may preserve useful source-specific refinement such as:

The normalized model should avoid pretending that all source type systems map perfectly. A sourceType property may retain the original native type where useful.

Description and semantics

semanticConcept may be a stable URI or another controlled identifier, for example a Darwin Core term URI.

This field is the main bridge toward later semantic mediation, but the first implementation must not require semantic concepts for ordinary dictionary rendering.

Units and values

Useful optional properties:

For tabular resources, these concepts intentionally align with Frictionless Table Schema where practical.

Value domains and codelists

A plain enum is sufficient when only machine codes matter. Some sources, especially ArcGIS/OGC and domain datasets, expose richer code lists with labels/descriptions or an authoritative external vocabulary.

An optional valueDomain may therefore contain:

{
  "type": "codelist",
  "uri": "https://example.org/codelists/status",
  "values": [
    { "value": "A", "label": "Active" },
    { "value": "I", "label": "Inactive" }
  ]
}

values may be omitted when uri is authoritative and enumeration would be impractical. Producers should preserve native coded-value domains rather than reducing them to unlabeled values when labels are available.

Nullability/requiredness

The dictionary may record:

These properties should reflect the native schema when authoritative. When inferred from samples, the provenance must make that distinction visible.

Node- and property-level provenance overrides

Entities and fields inherit dictionary-level provenance. An entity or field may override it when the node as a whole came from a different source or has a different review state.

A node may also carry optional propertyProvenance when only particular properties have a different origin/review state. This is important for mixed-origin metadata: for example, a field’s type/nullability may come from an authoritative ArcGIS definition while its description or semantic concept is manually curated.

For v1, propertyProvenance keys are direct property names on the node. Nested objects can either carry their own provenance where the model supports it, or be treated as one property for override/merge purposes.

Example:

{
  "path": "hectares",
  "name": "hectares",
  "type": "number",
  "unit": "ha",
  "description": "Mapped feature area",
  "semanticConcept": "area",
  "propertyProvenance": {
    "description": {
      "method": "manual",
      "reviewStatus": "custodian-approved"
    },
    "semanticConcept": {
      "method": "manual",
      "reviewStatus": "custodian-approved"
    }
  }
}

This allows Magda to express that structural facts such as type and nullable remain source-derived while selected descriptive/semantic properties are human-curated. A later producer can update source-derived structure without silently replacing the reviewed properties.

API request and response modelling

The boundary between distribution-contract and data-dictionary is important.

Scalar invocation parameters

Path/query/header/cookie parameters belong in distribution-contract.operations[].parameters because they explain how an operation is invoked.

Complex request bodies

A structured request body belongs in the data dictionary and is referenced from the operation:

{
  "request": {
    "mediaTypes": ["application/json"],
    "dictionaryEntity": "search-request"
  }
}

Response bodies

Response record/envelope structures belong in the dictionary and are referenced from the operation:

{
  "response": {
    "mediaTypes": ["application/json"],
    "recordsPath": "occurrences",
    "dictionaryEntity": "occurrence-record"
  }
}

This design gives the UI a natural Request fields / Response fields presentation without storing the same field definitions in two aspects.

Multiple tables, sheets and layers

One distribution can contain more than one logical entity.

Examples:

Excel workbook

entity: Sheet1 / Observations
entity: Sheet2 / Sites

ArcGIS service root

entity: layer-0 / Orchards
entity: layer-1 / Packing sheds

API response

entity: response-envelope
entity: occurrence-record
entity: facet-count

The UI should group fields by entity rather than flattening everything into one table.

Geometry metadata

For feature-oriented entities, optional geometry metadata may include:

Example:

{
  "geometry": {
    "type": "Polygon",
    "crs": "EPSG:4326",
    "fieldPath": "geometry"
  }
}

This information may later help cross-dataset spatial mediation.

Keys and identifiers

Optional field/entity metadata may mark:

The initial UI should present this information descriptively. It must not assume that two similarly named identifier fields are join-compatible without explicit evidence.

Explicit relationships between entities

Multiple entities alone are not enough to describe a workbook/database/table group when relationships are explicitly known.

The dictionary may therefore contain optional top-level relationships. The initial form should support declared references/foreign keys without implying that arbitrary same-named fields are join-compatible.

Example:

{
  "relationships": [
    {
      "id": "observations-site",
      "type": "foreign-key",
      "source": {
        "entity": "observations",
        "fields": ["site_id"]
      },
      "target": {
        "entity": "sites",
        "fields": ["id"]
      }
    }
  ]
}

Composite references use multiple field paths in the same order. Relationships may carry provenance/review metadata. This aligns with CSVW/Frictionless foreign-key concepts and is also useful for relational databases and multi-sheet Excel workbooks.

A relationship should be recorded only when supported by authoritative metadata, explicit user input or reviewed evidence. Similar names alone are not enough.

Multidimensional arrays and axes

For array-oriented scientific data, an entity may define named dimensions:

{
  "id": "climate",
  "name": "Climate variables",
  "role": "variable-set",
  "dimensions": [
    { "id": "time", "name": "time", "size": 365, "unlimited": true },
    { "id": "lat", "name": "latitude", "size": 720 },
    { "id": "lon", "name": "longitude", "size": 1440 }
  ],
  "fields": [
    {
      "path": "temperature",
      "name": "temperature",
      "type": "number",
      "sourceType": "float32",
      "dimensions": ["time", "lat", "lon"],
      "unit": "K",
      "roles": ["data-variable"]
    },
    {
      "path": "lat",
      "name": "lat",
      "type": "number",
      "dimensions": ["lat"],
      "roles": ["coordinate"],
      "semanticConcept": "latitude"
    }
  ]
}

A dimension descriptor may include id, name, optional fixed size, unlimited, description/semantic metadata and provenance. A field may reference dimension IDs in source order.

This is sufficient for a useful NetCDF/Zarr/raster logical dictionary while leaving storage layout, chunking/compression and other physical details in the authoritative native metadata.

Producer mapping rules

Source adapters should follow the same mapping rules so equivalent source concepts produce equivalent Magda metadata:

  1. Prefer authoritative machine-readable schema metadata over inference. Use bounded inspection only when stronger metadata is unavailable.
  2. Normalize common logical facts such as field identity, type, nullability, units, codelists, keys, relationships, geometry, dimensions and semantics.
  3. Preserve source-specific precision through sourceType, sourceIdentifier, sourcePath, source/specification metadata and provenance rather than expanding the normalized vocabulary for every native type.
  4. Do not copy whole native specifications into the aspect. OpenAPI/JSON Schema combinators, Parquet physical encoding, NetCDF chunking/compression and similar source-specific details remain in the authoritative native schema.
  5. Do not infer relationships, identifiers or semantic equivalence from names alone. Record them only when supported by authoritative metadata, explicit manual input or reviewed evidence.
  6. Keep invocation/access concerns in distribution-contract. General dataset-level context belongs in existing dataset description/coverage/source metadata; only authoritative purpose-specific fitness/limitation claims should move to a future dataset-usage extension if that capability is reactivated.
  7. Prefer stable source/native identifiers for entity identity. Where none exist, use a deterministic producer-specific ID and preserve the original source name/identifier separately.
  8. When information is unknown, omit optional properties or use the documented unknown logical type rather than inventing certainty.

Normalized versus native/source metadata

Put in data-dictionary Keep in native/source metadata or another aspect
Logical field/entity identity and stable paths Complete OpenAPI/JSON Schema/ArcGIS/CF/etc. document
Normalized type + useful sourceType Every source-specific type-system detail
Description, semantic concept, unit Full domain ontology/specification
Required/nullability, simple constraints Arbitrary validation/programming logic
Primary/composite keys and explicit relationships Inferred joins based only on similar names
Geometry/CRS summary Full rendering/style definition
Named dimensions/axes Physical array chunks/compression/storage layout
Codelist values/labels or authoritative URI Large duplicated external vocabularies
Provenance/review state Credentials or execution authority
Request/response body entities Scalar API invocation parameters (distribution-contract)

Worked source-family examples

These examples are illustrative normalized projections, not lossless copies of the native schemas. They show how different source families map into the same v1 concepts and are intended to double as the basis for compatibility fixtures/tests.

CSV / simple tabular data

A CSV with an authoritative table schema or a bounded inferred schema maps to a single table entity:

{
  "schemaVersion": "1.0",
  "source": {
    "type": "frictionless-table-schema",
    "url": "https://example.org/observations.schema.json"
  },
  "provenance": {
    "method": "authoritative-import",
    "reviewStatus": "unreviewed"
  },
  "entities": [
    {
      "id": "observations",
      "name": "Observations",
      "role": "table",
      "primaryKey": ["observation_id"],
      "fields": [
        {
          "path": "observation_id",
          "name": "observation_id",
          "type": "string",
          "roles": ["identifier"],
          "required": true
        },
        {
          "path": "observed_at",
          "name": "observed_at",
          "type": "datetime"
        },
        {
          "path": "temperature",
          "name": "temperature",
          "type": "number",
          "sourceType": "number",
          "unit": "Cel",
          "unitSystem": "UCUM",
          "nullable": true
        }
      ]
    }
  ]
}

If the schema is inferred from rows instead of supplied authoritatively, the same logical shape can be used with provenance.method: "inferred" and a bounded sample description.

Manual enrichment of harvested structure

The final aspect may combine authoritative/harvested structure with human-curated properties without losing which values are protected from automatic refresh:

{
  "schemaVersion": "1.0",
  "provenance": {
    "method": "authoritative-import",
    "sourceType": "arcgis-layer-definition",
    "reviewStatus": "unreviewed"
  },
  "entities": [
    {
      "id": "layer-0",
      "name": "Tree crops",
      "role": "feature",
      "fields": [
        {
          "path": "commodity",
          "name": "COMMODITY",
          "type": "string",
          "sourceType": "esriFieldTypeString",
          "description": "Commercial crop classification used by the survey",
          "semanticConcept": "https://example.org/concepts/crop-type",
          "propertyProvenance": {
            "description": {
              "method": "manual",
              "reviewStatus": "custodian-approved"
            },
            "semanticConcept": {
              "method": "manual",
              "reviewStatus": "custodian-approved"
            }
          }
        }
      ]
    }
  ]
}

If the ArcGIS field later changes type, an automatic producer may refresh the source-derived type/sourceType. It must preserve the reviewed description and semanticConcept unless a reviewer explicitly replaces them. If a source change invalidates a reviewed property, record/surface a conflict rather than silently choosing one side.

Multi-sheet Excel workbook

Each sheet becomes an entity. Relationships are recorded only when explicitly known or reviewed:

{
  "schemaVersion": "1.0",
  "source": {
    "type": "file-inspection"
  },
  "provenance": {
    "method": "manual",
    "reviewStatus": "reviewed"
  },
  "entities": [
    {
      "id": "observations",
      "name": "Observations",
      "sourceIdentifier": "Observations",
      "role": "sheet",
      "primaryKey": ["observation_id"],
      "fields": [
        {
          "path": "observation_id",
          "name": "observation_id",
          "type": "string",
          "roles": ["identifier"]
        },
        {
          "path": "site_id",
          "name": "site_id",
          "type": "string",
          "roles": ["reference"]
        },
        {
          "path": "value",
          "name": "value",
          "type": "number"
        }
      ]
    },
    {
      "id": "sites",
      "name": "Sites",
      "sourceIdentifier": "Sites",
      "role": "sheet",
      "primaryKey": ["id"],
      "fields": [
        {
          "path": "id",
          "name": "id",
          "type": "string",
          "roles": ["identifier"]
        },
        {
          "path": "name",
          "name": "name",
          "type": "string"
        }
      ]
    }
  ],
  "relationships": [
    {
      "id": "observations-site",
      "type": "foreign-key",
      "source": {
        "entity": "observations",
        "fields": ["site_id"]
      },
      "target": {
        "entity": "sites",
        "fields": ["id"]
      }
    }
  ]
}

The same representation works for a relational database or a Frictionless/CSVW table group.

ArcGIS feature layer

ArcGIS layer metadata maps native field types, geometry, identifiers and coded-value domains without downloading feature records:

{
  "schemaVersion": "1.0",
  "source": {
    "type": "arcgis-layer-definition",
    "url": "https://example.org/arcgis/rest/services/TreeCrops/FeatureServer/0?f=pjson"
  },
  "provenance": {
    "method": "authoritative-import",
    "reviewStatus": "unreviewed"
  },
  "entities": [
    {
      "id": "layer-0",
      "name": "Tree crops",
      "sourceIdentifier": "0",
      "role": "feature",
      "primaryKey": ["OBJECTID"],
      "geometry": {
        "type": "Polygon",
        "crs": "EPSG:4326",
        "fieldPath": "geometry"
      },
      "fields": [
        {
          "path": "OBJECTID",
          "name": "OBJECTID",
          "type": "integer",
          "sourceType": "esriFieldTypeOID",
          "roles": ["identifier"],
          "required": true
        },
        {
          "path": "status",
          "name": "STATUS",
          "title": "Status",
          "type": "string",
          "sourceType": "esriFieldTypeString",
          "valueDomain": {
            "type": "codelist",
            "values": [
              { "value": "A", "label": "Active" },
              { "value": "I", "label": "Inactive" }
            ]
          }
        },
        {
          "path": "geometry",
          "name": "geometry",
          "type": "geometry",
          "format": "polygon",
          "sourceType": "esriGeometryPolygon",
          "roles": ["primary-geometry"]
        }
      ]
    }
  ]
}

Service/query operations themselves belong in the companion distribution-contract.

OpenAPI / nested JSON API

Structured request and response bodies become dictionary entities. Scalar query/path/header parameters remain in distribution-contract.

{
  "schemaVersion": "1.0",
  "source": {
    "type": "openapi",
    "url": "https://example.org/openapi.json"
  },
  "provenance": {
    "method": "authoritative-import",
    "reviewStatus": "unreviewed"
  },
  "entities": [
    {
      "id": "search-request",
      "name": "Search request",
      "role": "request-body",
      "fields": [
        {
          "path": "filters.species",
          "name": "species",
          "type": "string"
        },
        {
          "path": "filters.boundingBox[]",
          "name": "boundingBox",
          "type": "number",
          "format": "coordinate"
        }
      ]
    },
    {
      "id": "occurrence-record",
      "name": "Occurrence",
      "role": "response-record",
      "fields": [
        {
          "path": "id",
          "name": "id",
          "type": "string",
          "roles": ["identifier"]
        },
        {
          "path": "scientificName",
          "name": "scientificName",
          "type": "string"
        },
        {
          "path": "location.latitude",
          "name": "latitude",
          "type": "number",
          "semanticConcept": "latitude"
        },
        {
          "path": "location.longitude",
          "name": "longitude",
          "type": "number",
          "semanticConcept": "longitude"
        },
        {
          "path": "tags[]",
          "name": "tags",
          "type": "string"
        }
      ]
    }
  ]
}

A companion operation in distribution-contract might reference the entities as:

{
  "request": {
    "mediaTypes": ["application/json"],
    "dictionaryEntity": "search-request"
  },
  "response": {
    "mediaTypes": ["application/json"],
    "recordsPath": "items",
    "dictionaryEntity": "occurrence-record"
  }
}

NetCDF / Zarr multidimensional data

Named dimensions and coordinate/data-variable roles describe scientific arrays without copying physical storage/chunking metadata:

{
  "schemaVersion": "1.0",
  "source": {
    "type": "netcdf-metadata"
  },
  "provenance": {
    "method": "authoritative-import",
    "reviewStatus": "unreviewed"
  },
  "entities": [
    {
      "id": "climate",
      "name": "Climate variables",
      "role": "variable-set",
      "dimensions": [
        { "id": "time", "name": "time", "size": 365 },
        { "id": "lat", "name": "latitude", "size": 720 },
        { "id": "lon", "name": "longitude", "size": 1440 }
      ],
      "fields": [
        {
          "path": "time",
          "name": "time",
          "type": "number",
          "sourceType": "float64",
          "dimensions": ["time"],
          "roles": ["coordinate"],
          "semanticConcept": "time"
        },
        {
          "path": "lat",
          "name": "lat",
          "type": "number",
          "sourceType": "float32",
          "dimensions": ["lat"],
          "roles": ["coordinate"],
          "semanticConcept": "latitude",
          "unit": "degrees_north"
        },
        {
          "path": "lon",
          "name": "lon",
          "type": "number",
          "sourceType": "float32",
          "dimensions": ["lon"],
          "roles": ["coordinate"],
          "semanticConcept": "longitude",
          "unit": "degrees_east"
        },
        {
          "path": "temperature",
          "name": "temperature",
          "type": "number",
          "sourceType": "float32",
          "dimensions": ["time", "lat", "lon"],
          "roles": ["data-variable"],
          "unit": "K"
        }
      ]
    }
  ]
}

CF standard names or other authoritative semantic identifiers should be retained in semanticConcept or source metadata where available.

Arrow / Parquet nested columnar data

The normalized model captures logical structure while preserving the native logical type in sourceType:

{
  "schemaVersion": "1.0",
  "entities": [
    {
      "id": "records",
      "name": "Records",
      "role": "table",
      "fields": [
        {
          "path": "customer.id",
          "name": "id",
          "type": "string",
          "sourceType": "arrow:utf8"
        },
        {
          "path": "amount",
          "name": "amount",
          "type": "number",
          "format": "decimal",
          "sourceType": "parquet:DECIMAL(18,2)"
        },
        {
          "path": "events[].occurredAt",
          "name": "occurredAt",
          "type": "datetime",
          "sourceType": "arrow:timestamp[us,UTC]"
        }
      ]
    }
  ]
}

Physical encoding, row groups, compression, dictionary encoding and chunk layout stay in Parquet/Arrow metadata rather than the normalized dictionary.

When no useful data dictionary exists

The aspect is optional. For example, a pure WMS distribution may expose layer/rendering capabilities but no useful row/feature schema. In that case it is better to omit data-dictionary (or provide only genuinely useful structural metadata) than to invent an empty or misleading field model.

Delivery and metadata production model

The core v7 Data Dictionary capability consists of:

Automatic harvesting is deliberately not a prerequisite for this core capability. Projects can author useful dictionaries immediately and add source-specific automation later. Automatic producer work is tracked separately from the core implementation so source coverage can evolve incrementally.

Manual/custodian metadata is a first-class source. Users may need to correct or enrich harvested structure with descriptions, semantic concepts, units or other annotations. A later producer run must not blindly replace those human-authored/reviewed values.

Automatic production should be pluggable:

A source adapter may populate both data-dictionary and distribution-contract in one pass where the same native source provides both structure and interface metadata, for example OpenAPI or ArcGIS.

Manual population with mgd

The generic aspect commands are sufficient for the core/manual workflow, for example:

mgd dataset aspect set <distribution-id> data-dictionary @data-dictionary.json
mgd dataset aspect get <distribution-id> data-dictionary --json
mgd dataset aspect patch <distribution-id> data-dictionary @patch.json

set replaces the aspect; patch is useful for partial object updates. Source-specific automatic producers remain a separate concern.

Source-specific automatic producers

CSV

Harvesting should:

If a Frictionless Table Schema is supplied, prefer it over inference and record it as the source.

Excel

Harvesting should:

JSON / REST APIs

Preferred sources, in order:

  1. explicit JSON Schema/OpenAPI schema;
  2. another authoritative native schema;
  3. bounded response sample inference.

Nested paths must be retained.

OpenAPI

Harvesting should map reusable/request/response schemas into entities and connect operation request/response references through distribution-contract.

The dictionary should not embed the whole OpenAPI document.

ArcGIS

Layer ?f=pjson metadata can populate:

Field structure can therefore often be harvested without reading actual feature records.

OGC API / STAC

Use exposed collection/schema metadata where available. If a schema is not available, bounded example inference may be used with explicit provenance.

WMS

WMS capabilities primarily describe layers and rendering metadata, not row-level feature fields. The dictionary may therefore be absent or limited for pure WMS distributions. The UI should not treat that as an error.

WFS

Feature type/schema metadata should be used when available and represented as feature entities.

NetCDF

Harvesting should extract metadata without loading full arrays into memory where possible:

CF convention identifiers should be retained when available.

Manual/agent-assisted sources

Manual or agent-assisted population is supported even when automatic producers also exist. An editor/agent may create a dictionary from documentation or bounded inspection and may add/correct annotations that a producer cannot infer reliably. Provenance must state that origin clearly.

Automatic producer output must be merged with existing human-authored/reviewed metadata rather than treated as an unconditional whole-aspect replacement.

Updating and drift

If the source exposes a stable schema/specification, the harvester should retain a fingerprint.

On change:

  1. produce a newly harvested candidate dictionary;
  2. compare it with the stored dictionary using stable entity IDs/field paths and native identifiers where available;
  3. preserve manual/custodian-reviewed nodes and propertyProvenance values where they still map unambiguously;
  4. update machine-derived structural facts/properties where the mapping is safe and no protected property override applies;
  5. mark changed/conflicting/ambiguous entries for review rather than silently replacing human input or presenting stale annotations as current;
  6. update generated metadata timestamps/fingerprints while retaining provenance/review state.

The first automatic producer should establish and test a consistent merge policy for later producers. Detailed merge/review UX can be implemented later, but the storage design must not make provenance or human overrides impossible to preserve.

Web-client design

The distribution page should render a Structure or Data dictionary section whenever the aspect exists.

The section is placed after the existing previews and outside the preview gate, following the layer design’s placement rule: distributions with a preview keep their familiar layout, and non-previewable distributions still show the section.

The first version should support:

Nested fields should be understandable without dumping raw JSON Schema.

For APIs, the operation view can link to the relevant request/response entities, producing a user experience such as:

Search occurrences
  Parameters
    q              string   required
    pageSize       integer  optional

  Response fields
    scientificName string
    eventDate       datetime
    latitude        number
    longitude       number

Search and semantic indexing

A later phase may index selected dictionary text such as:

This would allow searches such as “datasets containing scientific name and observation date”.

Large dictionaries should be chunked/represented deliberately rather than copied wholesale into one search document.

Agent consumption

An agent should be able to use the dictionary to:

Generated/inferred metadata remains advisory unless backed by an authoritative source or review state.

Compatibility fixtures

The v1 compatibility claim should be verified with representative aspect fixtures validated against the built-in JSON Schema and exercised through typed parsing/UI tests.

At minimum, fixtures should cover:

  1. simple CSV/tabular fields;
  2. a multi-sheet Excel-style dictionary with multiple entities and an explicit relationship;
  3. nested JSON/OpenAPI request/response entities;
  4. an ArcGIS feature layer with geometry, identifier role and coded-value domain;
  5. relational-style composite primary and foreign keys;
  6. Arrow/Parquet-style nested/native logical type preservation;
  7. NetCDF/Zarr-style named dimensions with coordinate and data variables;
  8. reserved-character path handling (for example a literal flat field named a.b);
  9. absence of the aspect on existing distributions.

These fixtures are not intended to prove lossless support for every native schema feature. They define the minimum normalized v1 interoperability envelope and prevent later schema tightening from accidentally breaking one of the documented source families.

Implementation notes (v7 core)

The core capability is implemented by:

Additive details settled during implementation:

User-facing authoring guidance: Data Dictionary guide.

Compatibility and evolution

The aspect is optional and additive.

Existing visualization-info remains responsible for current preview-specific hints and should not be overloaded into the new dictionary. A future migration may reuse overlapping observations, but preview compatibility should not depend on completing such a migration.

Within schema version 1:

Acceptance criteria