I'm doing a Monte Carlo study in which I am doing two things asynchronously:

  1. Generating virtual scenarios (and storing them)
  2. Running them through an algorithm (loading the virtual scenarios and then deleting them)


The data I'm working with has the following characteristics:

a. "Large" (In the order of a couple of terabytes)

b. Highly ordered (pretty much a 1-D array for each table)

c. Floating point decimal numbers

d. Highly sequential

The last point is the most important, I would like the data to be stored like the old magnetic tape storage devices did, completely sequential. When I want to write to it, I want it to be appended to the very last position. And when I want to read from it, I want it to start reading from the very first value of the table pull a chunk out, and after I'm done, delete it. These read/write processes will be happening simultaneously (or near simultaneous) so I would need it to handle multiple "users" querying it. And as mentioned in the title, this database would need to be able to be interfaced via Python.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.