0

Is there an open source SQL database specifically designed or optimized for scaling on clusters built from commodity hardware? In other words, something that does for data what Beowulf clusters did for computation. Bonus if there's an EF Core provider for it.

I'm talking about something like MySQL Cluster CGE, which is not open source. It wouldn't exist as a commercial product if its capabilities weren't special. But is there any open source competition for it?

2 Answers 2

1

I had to search Beowulf cluster on Google because I never heard about it.

This is a problem you had in the '90 and that doesn't exists any more. That problem existed when you home lab was sort of this:

enter image description here

But today your home lab is like this:

enter image description here

So there are no more problems of scaling since 20 years because your Hard Drive is a SAN that takes multiple hard drives and you don't have to care any more about managing the drives.

You just keep adding disks if you want and SQL Server, PostgreSQL, MySQL, Redis, etc... they all handle that without problems.

The reply to your question "Is there an open source SQL database specifically designed or optimized for scaling on clusters" is "YES, all of them".

Every single database engine that was on the market for the last 20 years can do that.

...as long as you create a SAN with your commodity hardware.

And by the way, what is an EF Core provider, Marty McFly?

8
  • From Wiki: "Since 2017, every system on the Top500 list of the world's fastest supercomputers has used Beowulf. . ." Probably a different question, but if commodity hardware can provide scaling and redundancy, why do SMBs still drop $30K on server hardware to run a LAMP stack? Are they even older-school than me? Dec 22, 2021 at 15:02
  • From the same link: "[...] however, most are by no means just assemblages of commodity hardware". We are talking about Supercomputers which are heavily customized, of course they have 1000 developers and they setup their custom solution. Probably they don't even use VMWare or Hyper-V, they have their own virtualization. They don't use product that you find on the market, they build them! You cannot compare Supercomputers with commodity hardware. You cannot use Beowulf cluster with commodity hardware today Dec 22, 2021 at 15:39
  • I don't know what SMBs stands for but a medium company size (100-500 employees) still drop $30K or $300K on hardware.... because is reliable. Otherwise if they have to setup Beowulf cluster they have to hire 100 engineers (paid $100K each) and build their own solution. From this point of view $30K or $300K is cheaper than several millions dollars. Because only Top500 which have lots of $$$ can afford to build their own Beowulf cluster Dec 22, 2021 at 15:42
  • Reliability through redundancy is one of the reasons to build clusters of commodity hardware. And it never took 100 engineers to set up a Beowulf cluster; individual hobbyists are still doing it with hardware like Raspberry Pis. It's hard to take your answer seriously if you don't know what Beowulf, EF Core, and SMBs are. Dec 22, 2021 at 15:47
  • So don't be cryptic and tell me what those letters stands for, please Dec 22, 2021 at 16:12
1

Open Source Distributed SQL

Is there an open source SQL database specifically designed or optimized for scaling on clusters built from commodity hardware?

Why yes! There are. :)

Here are a handful of open source SQL-compliant distributed database systems. This list is not exhaustive, I suggest looking into each to compare the feature sets.

CockroachDB

https://github.com/cockroachdb/cockroach

CockroachDB is a distributed SQL database built on a transactional and strongly-consistent key-value store. It scales horizontally; survives disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention; supports strongly-consistent ACID transactions; and provides a familiar SQL API for structuring, manipulating, and querying data.

  • Open Source, Business Source License, CockroachDB Community License, MIT
  • Uses the PostgreSQL wire protocol.
    • This means you can use Npgsql with EF Core
  • Allows for globally distributed, multi-site, multi-master topologies (www.cockroachlabs.com/docs/v21.2/topology-patterns.html)
  • Commercial support available
  • Cloud support available

PostgreSQL pglogical 2 extension

https://github.com/2ndQuadrant/pglogical

The pglogical 2 extension provides logical streaming replication for PostgreSQL, using a publish/subscribe model. It is based on technology developed as part of the BDR project (2ndquadrant.com/BDR).

  • Open Source, PostgreSQL License (similar to BSD or MIT)
  • Native PostgreSQL Extension, works anywhere you can install extensions (e.g. Cloud DBaaS services)
  • You can use Npgsql with EF Core
  • Supports bidirectional replication
  • Limited to single-database-at-atime configurations (meaning, it can't replicate all databases on a host)
  • Commercial support available (via 2ndQuadrant / enterprisedb.com)
  • Cloud support available

Galera Cluster

https://github.com/codership/galera

Galera Cluster for MySQL is a true Multi-Master Cluster based on synchronous replication. It’s an easy-to-use, high-availability solution, which provides high system up-time, no data loss and scalability for future growth.

  • Open Source, GPL2
  • MySQL and MariaDB Compatibility
  • Use github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql with EF Core
  • Generic Replication layer via the Write Set Replication API (github.com/codership/wsrep-API)
  • Commercial Support available via galeracluster.com
  • Comparable to Amazon Aurora and Google Cloud Spanner

YugabyteDB

https://github.com/yugabyte/yugabyte-db

YugabyteDB is a high-performance, cloud-native distributed SQL database that aims to support all PostgreSQL features. It is best to fit for cloud-native OLTP (i.e. real-time, business-critical) applications that need absolute data correctness and require at least one of the following: scalability, high tolerance to failures, or globally-distributed deployments.

  • Open Source, Apache 2.0
  • PostgreSQL compatible with custom extensions
  • Horizontally Scalable
  • Allows for globally distributed, multi-site, multi-master topologies (docs.yugabyte.com/latest/deploy/multi-dc/3dc-deployment/)
  • Comparable to Amazon Aurora and Google Cloud Spanner

CovenantSQL

https://github.com/CovenantSQL/CovenantSQL

CovenantSQL(CQL) is a decentralized, GDPR-compliant, trusted, SQL database with blockchain features:

  • ServerLess: Free, High Available, Auto Sync Database Service for Serverless App
  • GDPR-compliant: Zero pain to be GDPR-compliant.
  • SQL: Most SQL-92 support.
  • Decentralize: Running on Open Internet without Central Coordination.
  • Privacy: Access with Granted Permission and Encryption Pass.
  • Immutable: Query History in CQL is Immutable and Trackable.
  • Permission: Column Level ACL and SQL Pattern Whitelist.
  • Open Source, Apache 2.0
  • SQLite Based
  • Distributed by default
  • Immutable ledger
  • No commercial support currently available

Jepsen Tests

Distributed Database Correctness Analysis testing!

I highly recommend taking at look at jepsen.io/analyses. They have tested and verified (or debunked!) the claims most of these distributed database systems - and more.

From jepsen.io/analyses:

Since 2013, Jepsen has analyzed over two dozen databases, coordination services, and queues—and we’ve found replica divergence, data loss, stale reads, read skew, lock conflicts, and much more.

3
  • I would have added more... but I apparently don't have enough rep to make a more substantial answer. Also, yes, there are more!
    – Rick B
    Dec 22, 2021 at 19:46
  • That is six answers. Please post each separately so that users may vote on each separately.
    – Chenmunka
    Dec 22, 2021 at 21:46
  • @Chenmunka, sorry I didn't even know I could post multiple answers. I'll do that instead. Still learning how2StackExchange. Thanks for the tip!
    – Rick B
    Dec 25, 2021 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.