We are working on a project of type (intensive insertion) should able to handle millions of insertion per day.
The Current situation
- we have two table one of them includes 2 billion record (Id & meta data) and another one includes 30 million record (Id & meta data).
- the total storage is 600 GB.
- we are using PostgreSQL on one node (32G RAM, 24 Core, 2TB Storage).
The Issue now is PostgreSQL is not able to handle all this insertion so we tried to shard the tables to 1000-sub-tables and handle the insertion from the backend-service. but this solution is not stable and hard to maintain.
Now we are looking for simple key-value storage able to handle intensive insertion Also we are using Golang. so the database required to have Golang connector
what we did find :
- Redis : very fast database, but required the data to be able to fit in memory
- rocksdb & leveldb : both of them looks like they will work for the problem, but they don't have official Golang connector.