We are looking for a simple library to help us with logging the state of a real-time numerical system. For now, we use a simple csv-like format, but are interested in logging more things, faster, and in a more flexible way:
- At each step (a few ms), we need to log about ~100 floating point numbers.
- As the state of the system evolves the number of columns (and their header) will change a few times.
Is there any product that would help us with this ? So basically, a column-oriented, flexible schema, binary format.
We are not fundamentally opposed to querying a distant endpoint, but we'd rather not have to send data over the network.
We thought about rolling our own, by inserting new headers in a single file whenever there's a change in the columns, but we would then need to handle by ourselves the reprocessing...
This library should also ideally have python bindings for later data exploration.