3

I have several flat file data sources (CSVs) that are extracted, transformed and loaded into relational tables. At some point in time, when I look at a table, I often have a hard time figuring out where some of the fields in that table actually came from, since multiple flat files would have contributed to a single table.

  1. I am looking for a visualization tool that can help me map fields in my CSVs to fields in my databases so that I can trace back and forth when required.

  2. The tool also needs to have a search functionality which lets me search for a field by name to help me identify which tables and/or CSVs have that field.

I found Amundsen to be promising but it does not support requirement 1.

  • Would it be possible to just add a "source_csv" column to the relational tables and fill it with the filename upon import? – Lucio Crusca Nov 24 at 17:40
  • That doesn't solve the problem completely. I still need a way to visualize the mappings to get a sense of which file is the major contributor to the fields in a given table and so forth. Also, without a lot of rework, I cannot enforce the source_csv field to be correct since it's just a text field. – Abhilash Kishore Nov 24 at 19:40
  • It's not clear to me the environment and the constraints you have. You should provide more details about what you can do and what you can't. Moreover, assuming the tool you are after actually existed, I suspect it won't be able to enforce anything on the filesystem, unless that tool is the whole OS. – Lucio Crusca Nov 24 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.