The running of an industrial process I'm in charge of produces time series data (approximately several thousand parameters, each 1/8th of a second, during a month).

My goal is to look at two month-long executions and compare the data; something like, for each parameter or group of parameters, max diff, mean diff, Generalized Least Squares regression, and other standard statistical metrics.

I've succeeded in prototyping the ingestion of data and producing a basic visualization using the ELK stack.

But as I'm moving forward, I'm wondering: sure ELK can store my data and offer some nice visualization, but is it really a tool to produce such time series analysis/comparison/statistics? What would be the best software/stack to turn to here?


I would suggest taking a look at pandas going forwards as it excels at this sort of task. It is free, gratis & Open Source plus cross-platform. It is a tool within the Python ecosystem so you will require a recent copy of Python on your system, it also free, etc., this can also be installed from the Anaconda distribution of Python.

The excellent Python Data Science Handbook by Jake VanderPlas has a section on time series which gives the example of comparing closing stock market prices for a company (Google in this case) over time. It includes various visualisation techniques and the possibility of time shifting data so that you could, for example, take the values grouped by year with the data time shifted to align the first Monday of each year.

| improve this answer | |
  • 1
    Could you please give an example of a scenario and say what parts pandas would be involved in? Thanks! – Nicolas Raoul May 19 at 1:34
  • 1
    You might want to mention that pandas is part of the Python ecosystem so you need to install Python first (unless you are using Anaconda). – Eric Shain Jun 21 at 19:31
  • @EricShain - Done – Steve Barnes Jun 22 at 5:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.