I have a dataset consisting of CSV files (weather data). My issue is to clean the wrong values or empty values before using the dataset for machine learning. Right now I have to copy by hand the data from an external CSV using Excel (and some built-in functions). My objectives are to replace data and merge CSV files.
To automate this process, is it a must to write a script that manipulates CSV or does a software like that already exist?
I tried Apache Nifi but then I need to transform the CSV to a database, do SQL on the database and transform back to CSV again... Not the best solution I guess.