In our institution we have a large database of observational data consisting in precipitation, temperature, elevation and a large number of other variables, coming from different projects and in different formats. We currently have this in a single folder, with each dataset having a subfolder. Due to the amount of data, a need has arised for storing and retrieving the data based on its metadata (version, years covered, variable, resolution etc.) on a new archive.
We are looking for the best application to implement this, based on our required upload/retrieve workflows, which are listed below.
A) Uploading data to the archive
The following workflow is what we would like to implement to upload a new dataset to the archive:
- The user prepares the data (in any format) and uploads it to a folder of his/her choice in our cluster
- The end user compiles a metadata form online. The form includes a requirement for the source directory (see previous point) where data is residing
- Once submitted, the system validates the metadata (checking everything is present and consistent) and copies the data from the user-specified folder to a separate location where all of the observations are to be stored
B) Retrieving data from the archive
To retrieve data, users would need to:
- access the database via a web form
- filter by the necessary metadata
- obtain a download link (or location in the archive)
Possible applications / backends?
This system is to be used mainly for internal use. We have started looking around for applications to simplify this (we would prefer not to code everything from scratch).
Kit Data Manager,
Open Data Kit are currently being evaluated. I've heard of
W3C-DCAT , which might be related. What are some possible applications or backends usually employed with this kind of workflows?