We recently deployed a new 'data lake' on the Azure platform. The decision-making process had been messy and I cannot shake the feeling that we have bought an expensive, unwieldy database while we had much better and cheaper options.
Our requirements are pretty basic: we have about 7 GB of data, most of it comes from a daily database dump (CSV).
The data is used for reporting (currently query results are exported to excel, something like PowerBi will be used in the future) and analysis. With the amount of data we currently have, a full Big Data solution like Cloudera or MapR is unneccessary I think (but maybe I'm wrong).
In our current solution the ETL process that reads in the CSV dump and the database are managed by a third party. Using MSSQL management studio or SquirrelSQL, we can run queries and export the results. We have write-access to part of the database and I need to be able to design ETL processes to supplement the data already there.
This is the first glitch: I have not been able to automatically/programatically read in a flat file into the DB. In addition, the DB is not particulary fast and we are having trouble implementing logging/audit trail.
In my experience Postgres on CentOS is fast and logs DB access pretty much out-of-the box. Automating the import of flat files is no problem either.
What are better solutions than our current Azure setup?