Open source SQL database scalable on commodity hardware?

Question

Is there an open source SQL database specifically designed or optimized for scaling on clusters built from commodity hardware? In other words, something that does for data what Beowulf clusters did for computation. Bonus if there's an EF Core provider for it.

I'm talking about something like MySQL Cluster CGE, which is not open source. It wouldn't exist as a commercial product if its capabilities weren't special. But is there any open source competition for it?

Answer 1

I had to search Beowulf cluster on Google because I never heard about it.

This is a problem you had in the '90 and that doesn't exists any more. That problem existed when you home lab was sort of this:

But today your home lab is like this:

So there are no more problems of scaling since 20 years because your Hard Drive is a SAN that takes multiple hard drives and you don't have to care any more about managing the drives.

You just keep adding disks if you want and SQL Server, PostgreSQL, MySQL, Redis, etc... they all handle that without problems.

The reply to your question "Is there an open source SQL database specifically designed or optimized for scaling on clusters" is "YES, all of them".

Every single database engine that was on the market for the last 20 years can do that.

...as long as you create a SAN with your commodity hardware.

And by the way, what is an EF Core provider, Marty McFly?

Answer 2

Open Source Distributed SQL

Is there an open source SQL database specifically designed or optimized for scaling on clusters built from commodity hardware?

Why yes! There are. :)

Here are a handful of open source SQL-compliant distributed database systems. This list is not exhaustive, I suggest looking into each to compare the feature sets.

CockroachDB

https://github.com/cockroachdb/cockroach

CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

• Open Source, Business Source License, CockroachDB Community License, MIT
• Uses the PostgreSQL wire protocol.
  • This means you can use Npgsql with EF Core
• Allows for globally distributed, multi-site, multi-master topologies (www.cockroachlabs.com/docs/v21.2/topology-patterns.html)
• Commercial support available
• Cloud support available

PostgreSQL pglogical 2 extension

https://github.com/2ndQuadrant/pglogical

The pglogical 2 extension provides logical streaming replication for PostgreSQL, using a publish/subscribe model. It is based on technology developed as part of the BDR project (2ndquadrant.com/BDR).

• Open Source, PostgreSQL License (similar to BSD or MIT)
• Native PostgreSQL Extension, works anywhere you can install extensions (e.g. Cloud DBaaS services)
• You can use Npgsql with EF Core
• Supports bidirectional replication
• Limited to single-database-at-atime configurations (meaning, it can't replicate all databases on a host)
• Commercial support available (via 2ndQuadrant / enterprisedb.com)
• Cloud support available

Galera Cluster

https://github.com/codership/galera

Galera Cluster for MySQL is a true Multi-Master Cluster based on synchronous replication. It’s an easy-to-use, high-availability solution, which provides high system up-time, no data loss and scalability for future growth.

• Open Source, GPL2
• MySQL and MariaDB Compatibility
• Use github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql with EF Core
• Generic Replication layer via the Write Set Replication API (github.com/codership/wsrep-API)
• Commercial Support available via galeracluster.com
• Comparable to Amazon Aurora and Google Cloud Spanner

YugabyteDB

https://github.com/yugabyte/yugabyte-db

YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features. It is best to fit for cloud-native OLTP (i.e. real-time, business-critical) applications that need absolute data correctness and require at least one of the following: scalability, high tolerance to failures, or globally-distributed deployments.

• Open Source, Apache 2.0
• PostgreSQL compatible with custom extensions
• Horizontally Scalable
• Allows for globally distributed, multi-site, multi-master topologies (docs.yugabyte.com/latest/deploy/multi-dc/3dc-deployment/)
• Comparable to Amazon Aurora and Google Cloud Spanner

CovenantSQL

https://github.com/CovenantSQL/CovenantSQL

CovenantSQL(CQL) is a decentralized, GDPR-compliant, trusted, SQL database with blockchain features:

• ServerLess: Free, High Available, Auto Sync Database Service for Serverless App
• GDPR-compliant: Zero pain to be GDPR-compliant.
• SQL: Most SQL-92 support.
• Decentralize: Running on Open Internet without Central Coordination.
• Privacy: Access with Granted Permission and Encryption Pass.
• Immutable: Query History in CQL is Immutable and Trackable.
• Permission: Column Level ACL and SQL Pattern Whitelist.

• Open Source, Apache 2.0
• SQLite Based
• Distributed by default
• Immutable ledger
• No commercial support currently available

Jepsen Tests

Distributed Database Correctness Analysis testing!

I highly recommend taking at look at jepsen.io/analyses. They have tested and verified (or debunked!) the claims most of these distributed database systems - and more.

From jepsen.io/analyses:

Since 2013, Jepsen has analyzed over two dozen databases, coordination services, and queues—and we’ve found replica divergence, data loss, stale reads, read skew, lock conflicts, and much more.