Data catalog for unstructured data for search/discovery and access provisioning (for non data engineer/science users)?

Question

Are there any data catalogs for unstructured data (eg. loose CSV, PDF, .log, .sas, and other text files) for search/discovery and access provisioning (for non data engineer/science users)?

Have seen lots of data catalogs like Alation, Erwin, Collibra, Amundsen (it's open source if any DEs are looking for a internal catalog), but all of these seem to be geared more towards data engineers / scientists and only focus on handling relational / tabled data.

In our case, we have...

• Teams of non data science users (eg. medical researchers / biostatisticians, PhD students working on projects, etc. (our org is a teaching hospital))
• working in siloed teams with their own private data sets (that other teams may not even have permission to know exist, eg. sensitive data from outside orgs granted to only certain individuals within a team specifically)
• but do have other data assets that they can share (eg. de-identified data that they themselves produce)
• (which we want other teams to be able to find/search and request access to if they want)
• all of which are managed by our org, so we need to know about all of the data we are supporting in terms of
• • what data assets there are (including metadata like size and "location" within our system)
• • and who has access (or is requesting access) to those data assets
• And much of this data (in addition to being in SQL tables) may be in the form of unstructured data stored in files servers in our system

Answer 1

You can take a look at Spectrum Discover. https://www.ibm.com/products/spectrum-discover. It has been used by some universities.

Answer 2

After a short exploratory meeting with Alation, was pointed towards Data Cookbook which looks promising (and apparently is used by many other universities). Will look into further and may update this answer.

https://www.datacookbook.com/videos-2/ (as a DE, I'd recommend watching the videos in order: 1, 2, 5, 4, 3)

---

1. After watching the pre-recorded demo videos, I think it actually looks pretty good and could be what we need for our data cataloging needs (unlike the other tech/datasci. focused catalog systems I've seen with Erwin, Alation, Collibra, Amundsen, et al which focus more on structured data for more tech-centric users). Appears to be used by many other universities (and MSU, PSU, Birmingham U, etc.) (confirming what I had been told when initially pointed to them by the Alation people I met with). More details can be found here in this evaluation by the University of Birmingham on Data Cookbook for data governance, cataloging, and reporting: https://intranet.birmingham.ac.uk/it/innovation/documents/public/Experiments/TheDataCookBook-Evaluation-v0.1.pdf

2. The marketed pricing (https://www.datacookbook.com/pricing/) appears not as expensive as some other price tags I've seen

3. I'd also note that Lyft's Amundsen could be a good open source internal data catalog system for our own data analysts for cataloging and making-searchable our internal structured/tabled data assets.

4. A major question that I was still not able to discern was whether certain Defs/Specs could be hidden from certain users / only visible to a subset of catalog users. It appears so

"You can grant user groups permissions for definitions, specifications, and technical information based on functional area. For specifications and definitions, these permissions are manager, editor, and viewer"

but not as totally sure at the moment (would have to see that in action).

Preamble:

Def vs Spec? From watching the videos, seem to refer to...

• "Definition" = an individual data asset definition/entry (eg. a table or set of unstructured data)
• "Specification" = A deliverables/report, includes description and what data system it is for/from (eg. is a basic report, PowerBI dashboard?). Can have multiple Definitions linked to them (eg. so users can see what data assets are used in this report).

This link kinda illustrates the differences / relations: https://youtu.be/5SQR0NmccMo

A good description of the general workflow that Data Cookbook aims to support: https://youtu.be/jA-z9r8RnzA

Users:

Analysts:

• Works with requesting users on Specs (ie. reports)
• Day-to-day, can log in a see new requests ("information requests") by end users
• or work on Specs and Defs that they are working on
• Request info from requesting end users
• Approve Specs to publish to catalog (can include link to report if reporting system supports that (eg. a PowerBI link))

Data stewards:

• Works with requesting users on Defs (ie. data asset entries)
• Can be assigned to a subset of "functional areas"
• Get updates when new data asset Definitions are added / updated in the catalog
• Can review Defs, request input from creator/editor (eg. for clarification or reasoning) (who then gets alerted)
• Send email link to data asset definition creation/changes proposal to relevant users to request additional comments on proposed creation/changes
• Approve or reject changes to publish to catalog

End users:

• Can log into a web UI
• Search for data
• Leave comments about a Def or Spec (for data stewards or analysts to be notified of)
• Make ticket requests in general if can't find the data they need (for a data "librarian" to either direct them to or start process of new Def or Spec creation for that data)

---

To a much lesser extent, CKAN or Magda could also work. Magda looks like a good candidate for this, though I question the current state of the user access restriction capabilities and CKAN currently lacks any moderation features like iData has. Note that magda requires either using Google Cloud services to host (uses Terraform) or having your own kubernetes cluster to host it on. Also, I see that the AU gov uses it, but when you try to sign in or create an account, it takes you to a site that uses CKAN. I wonder what the difference is between CKAN and Magda?