Distributed Database Systems

Question

I am looking for an open source distributed SQL database system which will distribute the data across nodes but so that there is data redundancy (if one node fails, the data is still available without having to to backup all of the data on each node).

It has to be able to store at least 1 petabyte of data and should have the ability for multiple simultaneous connections, unlike Apache Hadoop.

I searched online for such solutions but most of the distributed databases are noSQL with document structure such as Apache Cassandra but a few options are not open source and free, like CrateDB and CockroachDB.

Please help me with this.

Answer 1

Are you not considering cloud service options? I assume you must have heard of AWS. Doing exactly this is a big part of what they offer, although a PB of data would be a huge expense.

If you prefer to use free software, most commercial-grade databases have a way to do this. Look up "master slave replication" plus your database of choice and you'll probably find several tutorials. For example, here is the manual for doing it with MySQL 8.0.

I should clarify that master-slave replication will provide redundancy and make your data more accessible, but it's not a truly distributed system. Every machine imitates the master, and slaves cannot write to the master. So every replication node is read-only. And only one host at a time can be the master. From your description, it sounds like you might be ok with that, but it's worth noting.

Answer 2

Since another user mentioned cloud platform AWS:

You could try Google Cloud Spanner which is non-free and not open-source, but you could experiment with its distributed-computing capabilities. There are a few research papers about the technology, see Wikipedia Article. Perhaps some groups are re-implementing it according to the specs (just like it happened in the 2000s-2010s with MapReduce, Hadoop and Spark)

(Google Bigquery is another alternative to check out.)