0

I'm working on a big data project and have several old on-prem servers running a mix of operating systems (Ubuntu, CentOS, Windows 2012, Windows 10). One of the big reasons I would like a distributed storage solution is that I'm gathering large amounts of data at a very fast pace, and attempting to write all the data to a single disk is overwhelming the disk, which is leading to data loss. The data gathering algorithm is 100% Python. Ultimately, I want to use the data to train machine learning models in TensorFlow.

I'm overwhelmed with the distributed data storage options out there and it seems like the space has evolved rapidly over the last couple years.

It seems like Hadoop has been the go-to solution for big data, but it looks like it's difficult to configure, particularly on Windows, and does not appear to work natively in Python. Databricks seems interesting, but it's not clear to me whether it is as good as Hadoop in distributed data storage (or if that's even a capability of Databricks).

I would greatly appreciate any recommendations of tools and frameworks to explore (I'm open to paid and free tools) and any corrections on my understandings of Hadoop and Databricks above.

0

PySpark with Amazon EMR go hand in hand and may suit your needs for worrying less about configuration and having storage reliability by spinning up Amazon EC2 to keep your data.

Here's an Amazon article on setting it up. Mind you'll need to pay some fees if you wanna try out Amazon EMR out the gate, but it should be relatively cheap to try before implementing it for real.

If you use Jupyter Notebook give this a read.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.