I'm working on a big data project and have several old on-prem servers running a mix of operating systems (Ubuntu, CentOS, Windows 2012, Windows 10). One of the big reasons I would like a distributed storage solution is that I'm gathering large amounts of data at a very fast pace, and attempting to write all the data to a single disk is overwhelming the disk, which is leading to data loss. The data gathering algorithm is 100% Python. Ultimately, I want to use the data to train machine learning models in TensorFlow.
I'm overwhelmed with the distributed data storage options out there and it seems like the space has evolved rapidly over the last couple years.
It seems like Hadoop has been the go-to solution for big data, but it looks like it's difficult to configure, particularly on Windows, and does not appear to work natively in Python. Databricks seems interesting, but it's not clear to me whether it is as good as Hadoop in distributed data storage (or if that's even a capability of Databricks).
I would greatly appreciate any recommendations of tools and frameworks to explore (I'm open to paid and free tools) and any corrections on my understandings of Hadoop and Databricks above.