Get the most out of your data
Government agencies possess a wealth of valuable data, but it often sits hidden inside spreadsheets and on shared drives, making it impossible to discover or navigate through. Data that could improve efficiency, transparency and accountability is trapped in silos, and the opportunity that it offers is lost.
Even if data is already opened and published to an open data portal, it’s still difficult to discover. Because current open data portals have been focused around hosting files they possess simplistic search functionality, poor metadata authoring tools and no ability to easily preview datasets before downloading, making even the best datasets difficult to find and use.
A set of tools for every part of the data landscape
Magda is an open-source software platform that has been designed to make publishing, searching and using government data easier, for both government agency staff using private data internally and citizens using open data publically. It consists of a set of tools that can be combined into a number of configurations:
- An internal portal for hosting data within agencies, which will eventually have robust mechanisms for controlling which users can see or use which data
- An internal/external portal that both hosts internal data as above, but also provides workflows for safely sharing datasets between agencies or releasing some datasets as open data, and making those available to the public
- A public aggregator that takes datasets from many sources and makes it all easily searchable in one place
… or any combination thereof. Its capabilities include:
The easiest way to find a dataset is by searching for it, and Magda puts its search functionality front and centre. Magda is able to rate datasets by their quality and return the best ones at the top of the search, understand synonyms and acronyms, as well as provide a number of advanced filters including the ability to search for datasets that affect a certain spatial region or a time period.
Seamlessly combine self-hosted datasets with those from external sources into one, easily-searchable repository.
Ensure that your users can quickly determine if a dataset is useful for them with charting, spatial preview with TerriaJS and table previews.
Automatic metadata enhancement
The metadata that describes datasets is often poorly formatted or completely absent, making them difficult to search for and understand. Magda is able to enhance the metadata of both locally hosted and external datasets in by checking for broken links, normalising formats, calculating quality and determining the best means of visualisation.
Magda is designed as a set of microservices that allow extension by simply adding more services into the mix. Extensions to collect data from different data sources or enhance metadata in new ways can be written in any language and added or removed from a running deployment with zero downtime and no effect on upgrades of the core product.
Easy set up and upgrades
Magda uses Kubernetes and Helm to allow for simple installation and zero-downtime upgrades with a single command.
Want to run Magda inside your agency?
We’d love to talk to you! Please get in contact with us at firstname.lastname@example.org.
See it in action
Magda currently powers the new data.gov.au beta at search.data.gov.au, serving 2000 users per week.
Magda is currently in development, and we’re currently working on adding more features to make Magda useful as an internal data portal like publishing, dataset hosting and access control. You can see our roadmap and current progress here.
Want to get it running yourself?
Try the latest version, or build and run from source
v0.0.50, released at 2019-01-11 05:04:40 UTC
Magda is fully open source, licensed under the Apache License 2.0. Thanks to all our open source contributors so far:
We welcome new contributors too! please check out our Contributor’s Guide.
The project was started by CSIRO Data61 and Australia’s Department of Prime Minister and Cabinet. It’s progressing thanks to Data61 and the Digital Transformation Agency