I'm writing some remote monitoring software in Python 3. I currently gzip CSV data collected over about 8 hours for transmission to the server but I'm looking for something with lower bandwidth requirements as some of the devices have fairly extreme bandwidth restrictions. I'm looking for a Python 3 library that can serialise numeric record data with the following properties:
- The file must have a header row (ie. a single record with the column titles)
- Rows are fixed length (ie same number of fields in each row)
- All data (except the column titles) are 8-byte floating-point
- It's relatively common to have whole columns that are all zero (and could therefore be omitted) but for any particular column, this can't be known for certain until the whole file is written (though it's possible to know with 99% certainty after a few records and I can live with the 1% loss this implies)
- It must be possible to re-open a file and append new records
- Files are typically around 80 columns wide and 80,000 rows. Rows are generally added in batches of a hundred or so.
- Python 3 library support - I don't want to roll my own
- Data integrity checking is a bonus
HDF5 is a possibility but it seems that its size performance is not that great at this size file. A major upside to it though would be that it would be relatively easy to remove all-zero columns (I think) which would help file size considerably.
I'm presently gzipping the CSV files. XZ could do somewhat better (about 30%, informal tests suggest) but at the cost of a LOT of CPU cycles.
Can anyone recommend something?
