I am going to set up an experiment with plenty of sensors, an Raspberry-PI (most probably) with Linux for a long duration experiment with chances of data corruption and/or reboot due to ionization radiation.
In this configuration a continuous steam of data with low data rate (<50kBit/s) is generated. To prevent loss of experiment data the data shall be stored locally on a SD-Card and/or USB-Flash device, streamed to a server and to a client for on-line diagnostic. Each of these data stream shall be independent and loss of network connection shall not influence the local storage nor shall problems with the local storage result in failure of network transfer.
The network part should also retry connection in case of loss and/or reboot of the system automatically. It would be perfect when loss of network connection and reconnection could synchronize the missed data as well. (But without loading the system too much to prevent problems in the data acquisition task) Also a reboot shall not lead to data corruption or overwriting of already logged data. (A journalling file system with disabled write cache shall be OK)
We implemented such systems already on our own in combination of python, c++, pipes and shell but it is very hard to do this all right and testing of such systems is a nightmare.
Is there is some already tested piece of code/library to handle such or similar requirements? Or do you have some thoughts how to solve this in a maintainable and flexible way?