I'm working on a project where it is assumed that a few thousand distributed custom devices need to be monitored. So there are pretty simple questions which should be answered like:
- What's the load/memory/disk space of device X now?
- What was the disk usage lately?
- What was the load at the first time the device was online, maybe years back in time?
And also more complicated questions like: Which devices did run a custom task, and how long did it take? This one would be used for accounting purposes.
One option to solve this is to build a custom monitoring and reporting tool. But this won't be a simple one day hack, so I wanted to ask if anyone knows some open source software which is able to
- add new metrices in a very flexible way like Ganglia's gmetric tool
- keep the data forever in some kind of storage to query for simple data points
- use a JSON API to query datapoints and to be able to generate complex queries via some kind of API
- build graphs like everyone already knows from Ganglia or such
I've tried Ganglia already, but gmetric seems to dump the collected data as soon as the device is offline. But the flexiblity of gmetric is one of the key features since we don't know what might be relevant to monitor in the future.