In-memory database for streaming market data

Question

I'm looking for recommendations for an in-memory database for capturing streaming market data state. While the transaction rate may reach into millions/sec, most of these transactions will be updates to existing records, so the database is not expected to grow much throughout the day. The database will be reset each business day.

Ideally:

• runs on windows environment, resides inside my .NET process, but I suppose an unmanaged process is not out of the question
• offers the ability for other processes to connect and run queries, preferrably using SQL or I'm open to Linq
• Allows me to embed triggers inside the database so that a row update can trigger another row update
• Allows some way to push notifications to clients that are connected and subscribed to receive notifications when a particular row is changed.
• gratis or very inexpensive

So far, I believe MemSQL fits the bill on all points except the last one.

Answer 1

You might consider VoltDB. It meets most of your criteria.

It doesn't run on Windows, only on 64-bit Linux, but that could be in a VM on Windows or in the cloud.

Rather than triggers, it uses stored procedures written in plain Java. You could have a procedure that performs the first update and conditionally performs the second update. It could also check a subscription table and conditionally insert notification records in an export table, which could stream automatically to Kafka or RabbitMQ using an export connector.

I wrote an example app that performs a real-time national best bid & offer calculation on simulated streaming market data. The majority of the code is actually the simulated data generation, the database schema is very simple and the stored procedure that does the calculation isn't a lot of code and may give you a quick idea at how simple an application like this could be to build in VoltDB.

We have many customers from Fortune 500 to very small startups and cost has never been an obstacle.

Answer 2

H2 Database Engine

Not sure from your description if running Java is an option. If so, the H2 Database Engine has an option for entirely in-memory databases. And H2 supports triggers. Free-of-cost, and open-source. Supports SQL through JDBC.

A similar product is Apache Derby, though I find H2 easier to deal with.

Database with sophisticated caching

Another option is using a database that writes to storage but maintains a sophisticated cache in memory. This avoids the problem of running out of memory unexpectedly as the database will continue to operate successfully. And when you do have sufficient memory for all your data, good caching will give you similar performance as in-memory.

Postgres springs to mind as a superior database with good caching that also offers powerful and flexible triggers. See this page on dba.StackExchange.com about loading Postgres data into memory. Postgres also has a NOTIFY feature for pushing changes to a client, as mentioned in the Question. Postgres is open-source and free-of-cost.

Answer 3

I posted the original question 2 years ago. In the end I did not pick a a database and wound up coding the market data ticker processing functionality from scratch in .Net. This is because I could not find a database that worked on my budget (close to $0) and on the hardware I had available (one multicore intel server). In addition to expensive licensing, many high-performance options required server clusters or other elaborate setups.

While doing the research to find a database that met the needs in the original question, I found several databases that were worthy of mentioning to help others facing the same problem. Keep in mind that my research was done two years ago, so my list should be considered a starting point.

• Kdb+ (not in-process but has a .net client and notifications)
• ExtremeDb Financial Ed. (can be in-process within .Net and has notifications)
• RavenDb (can be in-process within .Net and has notifications)