Big amount of measurement files. How to handle it? Kind of library software?

Question

We have a big amount of files (especially .csv files). These measurement files are archived in different folders with some information about the measurement and date.

The problem is that we lost the overview about all these files. Sometimes nobody knows if there was some measurement with similar conditions.

And if you want to compare the measurement, you have to search it in the deep folder structures of archives and then you have to search the other file = time consumption is really high.

I need your opinion and the view from different experiences.

Does self-made software makes any sense, with the job to create only an overview (with search functionality) about all files with their measurement characteristics?

Is there already a ready-to-go solution?

Answer 1

At least for the CSV files I would suggest importing all of those into a sql database such as mariadb(formerly mysql).

select * from table
where (condition1 > 5) and (condition1 <10 ) and
(condition2 > 100) and (condition2 <1000 )
(condition3 > 5) and (condition3 <10 )

Writing these out graphically can be helped by many web based tools like phpmyadmin or adminer.

You can even create indexes for columns that are slow to query. Many more optimization exist if you are will to learn them.

LibreOffice has librebase built-in and Microsoft Office has MS Access built-in