I have several flat file data sources (CSVs) that are extracted, transformed and loaded into relational tables. At some point in time, when I look at a table, I often have a hard time figuring out where some of the fields in that table actually came from, since multiple flat files would have contributed to a single table.

  1. I am looking for a visualization tool that can help me map fields in my CSVs to fields in my databases so that I can trace back and forth when required.

  2. The tool also needs to have a search functionality which lets me search for a field by name to help me identify which tables and/or CSVs have that field.

I found Amundsen to be promising but it does not support requirement 1.

  • 1
    Would it be possible to just add a "source_csv" column to the relational tables and fill it with the filename upon import? Nov 24 '19 at 17:40
  • That doesn't solve the problem completely. I still need a way to visualize the mappings to get a sense of which file is the major contributor to the fields in a given table and so forth. Also, without a lot of rework, I cannot enforce the source_csv field to be correct since it's just a text field. Nov 24 '19 at 19:40
  • It's not clear to me the environment and the constraints you have. You should provide more details about what you can do and what you can't. Moreover, assuming the tool you are after actually existed, I suspect it won't be able to enforce anything on the filesystem, unless that tool is the whole OS. Nov 24 '19 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.