0

We are looking for a simple library to help us with logging the state of a real-time numerical system. For now, we use a simple csv-like format, but are interested in logging more things, faster, and in a more flexible way:

  • At each step (a few ms), we need to log about ~100 floating point numbers.
  • As the state of the system evolves the number of columns (and their header) will change a few times.

Is there any product that would help us with this ? So basically, a column-oriented, flexible schema, binary format.

We are not fundamentally opposed to querying a distant endpoint, but we'd rather not have to send data over the network.

We thought about rolling our own, by inserting new headers in a single file whenever there's a change in the columns, but we would then need to handle by ourselves the reprocessing...

This library should also ideally have python bindings for later data exploration.

Improve this question
1
  • Since you want to exploit the data in python, you might look at picklingtools which seems to be a C++ library compatible with python pickle (version 2, binary). That page also links to a pdf describing the resulting binary format.
    – meuh
    Feb 2 '17 at 18:35
1

I believe what you're asking for is a DBMS (possibly with streaming functionality, or schema-less) - and definitely not a logging library (such as these).

Logging libraries don't generally involve columns (except with fields such as the timestamp, thread id, log level etc. - that is, log output formatting); it is not intended for you to log large amounts of data; log formats should not evolve; etc.

Improve this answer
1
  • That's an interesting thought, I'll try to tinker with a few products. I found that InfluxDB looks pretty good, but not sure how to get the data back out.
    – Urukann
    Feb 3 '17 at 2:17
1

You don't want an DBMS; it'll be too slow. You want a time-series database. Start with InfluxDB. https://www.influxdata.com/

Improve this answer
2
  • Thanks for the suggestion, but Influx can not be used as a library as far as I can tell...
    – Urukann
    Feb 6 '17 at 6:45
  • Don't use it as a library. Just spawn it or require it as a dependency.
    – Brannon
    Feb 6 '17 at 16:25
0

In the end, we rolled out our (hopefully) temporary solution:

  • Serialize all our objects using flatbuffers
  • Write them to disk in a separate thread
  • A small tool to bring the data back out as a CSV / DBMS file

We looked at:

  • InfluxDB: great performance, has streaming to disk, but not usable as a library unfortunately.
  • WhiteDB: performance is not so great, no streaming to disk and needs to manually select the amount of available memory.
  • Akumuli: performance looks great, is usable as a library! Unfortunately, it is still very unstable (I couldn't get it to build a new volume without segfaulting).
  • Rosbag: performance is poor, does not have built-in support for streaming to disk (but it can be done by threading), has a lot of good ROS-enabled tooling.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.