After a short exploratory meeting with Alation, was pointed towards Data Cookbook which looks promising (and apparently is used by many other universities). Will look into further and may update this answer.
http://www.datacookbook.com/videos-2/ (as a DE, I'd recommend watching the videos in order: 1, 2, 5, 4, 3)
After watching the pre-recorded demo videos, I think it actually looks pretty good and could be what we need for our data cataloging needs (unlike the other tech/datasci. focused catalog systems I've seen with Erwin, Alation, Collibra, Amundsen, et al which focus more on structured data for more tech-centric users). Appears to be used by many other universities (and MSU, PSU, Birmingham U, etc.) (confirming what I had been told when initially pointed to them by the Alation people I met with). More details can be found here in this evaluation by the University of Birmingham on Data Cookbook for data governance, cataloging, and reporting: https://intranet.birmingham.ac.uk/it/innovation/documents/public/Experiments/TheDataCookBook-Evaluation-v0.1.pdf
The marketed pricing (http://www.datacookbook.com/pricing/) appears not as expensive as some other price tags I've seen
I'd also note that Lyft's Amundsen could be a good open source internal data catalog system for our own data analysts for cataloging and making-searchable our internal structured/tabled data assets.
A major question that I was still not able to discern was whether certain Defs/Specs could be hidden from certain users / only visible to a subset of catalog users. It appears so
"You can grant user groups permissions for definitions, specifications, and technical information based on functional area. For specifications and definitions, these permissions are manager, editor, and viewer"
but not as totally sure at the moment (would have to see that in action).
Def vs Spec?
From watching the videos, seem to refer to...
- "Definition" = an individual data asset definition/entry (eg. a table or set of unstructured data)
- "Specification" = A deliverables/report, includes description and what data system it is for/from (eg. is a basic report, PowerBI dashboard?). Can have multiple Definitions linked to them (eg. so users can see what data assets are used in this report).
This link kinda illustrates the differences / relations: https://youtu.be/5SQR0NmccMo
A good description of the general workflow that Data Cookbook aims to support: https://youtu.be/jA-z9r8RnzA
- Works with requesting users on Specs (ie. reports)
- Day-to-day, can log in a see new requests ("information requests") by end users
- or work on Specs and Defs that they are working on
- Request info from requesting end users
- Approve Specs to publish to catalog (can include link to report if reporting system supports that (eg. a PowerBI link))
- Works with requesting users on Defs (ie. data asset entries)
- Can be assigned to a subset of "functional areas"
- Get updates when new data asset Definitions are added / updated in the catalog
- Can review Defs, request input from creator/editor (eg. for clarification or reasoning) (who then gets alerted)
- Send email link to data asset definition creation/changes proposal to relevant users to request additional comments on proposed creation/changes
- Approve or reject changes to publish to catalog
- Can log into a web UI
- Search for data
- Leave comments about a Def or Spec (for data stewards or analysts to be notified of)
- Make ticket requests in general if can't find the data they need (for a data "librarian" to either direct them to or start process of new Def or Spec creation for that data)
To a much lesser extent, CKAN or Magda could also work. Magda looks like a good candidate for this, though I question the current state of the user access restriction capabilities and CKAN currently lacks any moderation features like iData has. Note that magda requires either using Google Cloud services to host (uses Terraform) or having your own kubernetes cluster to host it on. Also, I see that the AU gov uses it, but when you try to sign in or create an account, it takes you to a site that uses CKAN. I wonder what the difference is between CKAN and Magda?