2

In our institution we have a large database of observational data consisting in precipitation, temperature, elevation and a large number of other variables, coming from different projects and in different formats. We currently have this in a single folder, with each dataset having a subfolder. Due to the amount of data, a need has arised for storing and retrieving the data based on its metadata (version, years covered, variable, resolution etc.) on a new archive.

We are looking for the best application to implement this, based on our required upload/retrieve workflows, which are listed below.

A) Uploading data to the archive

The following workflow is what we would like to implement to upload a new dataset to the archive:

  1. The user prepares the data (in any format) and uploads it to a folder of his/her choice in our cluster
  2. The end user compiles a metadata form online. The form includes a requirement for the source directory (see previous point) where data is residing
  3. Once submitted, the system validates the metadata (checking everything is present and consistent) and copies the data from the user-specified folder to a separate location where all of the observations are to be stored

B) Retrieving data from the archive

To retrieve data, users would need to:

  1. access the database via a web form
  2. filter by the necessary metadata
  3. obtain a download link (or location in the archive)

Possible applications / backends?

This system is to be used mainly for internal use. We have started looking around for applications to simplify this (we would prefer not to code everything from scratch). Kit Data Manager, OpenRosa and Open Data Kit are currently being evaluated. I've heard of OPeNDAP and W3C-DCAT , which might be related. What are some possible applications or backends usually employed with this kind of workflows?

Improve this question
6
  • At step A3, if I understand correctly the user uploads a big data file via the HTTP form? Is any data format OK, for instance CSV or TSV or XLS or XYZ?
    – Nicolas Raoul ♦
    Commented Mar 8, 2019 at 12:38
  • @NicolasRaoul the data will already be stored in our system, under another folder, possibly a user home path, a scratch disc, or an archive. These shares are all mounted on all local systems. Thus no uplodading is necessary, the application only needs to copy the data from the indicated folder to its internal folder structure, and add the metadata to the database.
    – AF7
    Commented Mar 9, 2019 at 13:52
  • That's a very big specificity, so please edit your question to explain it, thanks! I guess all data formats are OK?
    – Nicolas Raoul ♦
    Commented Mar 10, 2019 at 8:38
  • @NicolasRaoul I have done so, thanks for the suggestion. Yes, we trust our users to use proper data formats. An admin can then validate the files form time to time.
    – AF7
    Commented Mar 10, 2019 at 11:08
  • @AF7 I'm curious if you ever found something? Or did you "roll your own" in the end?
    – Saaru Lindestøkke
    Commented Nov 30, 2020 at 23:59

1 Answer 1

Reset to default
0

AFAIK there is no ready to use software which handles your particular case.

I would use this software stack:

  • PostgreSQL for the meta data
  • Django/Python for a web interface
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.