we have statistics from multiple sources and would need to display them at a central place.
At the moment, we have some custom charts using PHP and ExtJS for things stored in a MySQL database, then we have Kibana for information out of ElasticSearch, and we have some external sources such as HappyApps which we can at the moment only view through their respective admin panels, although there would be REST APIs available as well to get their data.
In the moment, we have the following problems:
- We have some data spread across different platforms.
- Not only that some reports have to be generated in the custom MySQL statistics system and some reports have to be generated in Kibana or somewhere else, there is also the problem that there are reports which would require connecting information from multiple platforms, e.g. a report that puts the number of rows in the MySQL database into a relation to the number of documents in ElasticSearch and shows a percentage (with filtering of course).
- The client's experience is poor because they need to fiddle around with multiple systems, enter "cryptic" search queries in Kibana and do some calculations on top in order to understand the full picture.
- Modifying reports takes too long, because when the client needs a new kind of report for MySQL, we need to have somebody who understands the custom system update it.
- In Kibana specifically, we are lacking certain features which would be important for our clients.
- Most importantly a "dummy" way to filter data by various fields, using something like an "add filter" button where you can select the field, and sensible values depending on the field to filter for (e.g. freetext, number range, enumeration of predefined values, etc.).
- We would also need a way to annotate data based on information stored elsewhere, such as in a MySQL database. For example, we currently see only IDs in some of the reports, but we would need to automatically show the names of the corresponding objects which are stored in a MySQL database.
- Additionally, we would need to perform post-processing in ways Kibana doesn't offer, such as calculating percentages in a more flexible way (e.g. number of events type X makes 100%, and in relation to this, we would like to know the number of events type Y and Z as percentage, e.g. for all events X, there were only 60% corresponding Y and 20% corresponding Z events), or automatic calculation of relations such as we have events of type "impression" and of type "click" and we would need to see the CTR value in % (= number of clicks / number of impressions) in a histogram.
- Plus, we would sometimes need to "join" data such as that we have user IDs, and events of type X which contain more information such as the user's country, which are not available in events of type Y and Z. But we have the user ID in all, and we would need to allow filtering the data of type Y and Z by the user's country, which is only known by checking the corresponding type X events' values.
We are therefore looking for a system which helps as combine it all, but still keeps us fast-moving with client requests. The idea is that we would ideally have some kind of tool which has the following capabilities:
- Merge, analyze and display data from multiple sources, including MySQL, ElasticSearch and custom (can be REST API, can be some kind of JSON file we can fill, or whatever - that's in order to integrate with external sources such as HappyApps).
- Allow creation of visualizations and dashboards similar to how it works with Kibana, but without the issues and lacking features mentioned above.
- Have an intuitive user interface for the charts and dashboards, with easy filter tools where we (the admins) can specify which fields to allow filtering for and with what kind of values, without requiring our customers to type filter queries and remember syntax/grammar of the query language. (The filtering is supposed to be not only client-side as some JS chart libraries offer, because there will be very large amounts of data which should be filtered as part of the corresponding DB queries and can't all be loaded into the browser's memory.)
- Is easy to use for admins as well, for managing data sources and building visualizations. Ideally there is a graphical way to build basic visualizations and for customizations such as post-processing of values, allows some kind of scripting language or similar on top.
- Is not too expensive and ideally can run on our own servers. (Open source would of course be even better, in case we would need to modify a part of it in response to a client request we can't fulfill with the existing features.)
- Takes a reasonably short time to get started with. We prefer a solution we can start with quickly and then spend some time refining and improving over a solution which takes us a week to get the first chart to display only to find out that it doesn't work out for us.
- Is a modern web application (not a Windows application or something like that).
I hope it's not too much to read, and I really appreciate that you took the time reading it - but I guessed it's better to specify exactly what we are looking for, because just asking for a "dashboard for analytics" wouldn't really get us anywhere.
I hope you guys have some ideas, since I already spent hours googling and reading product websites but didn't really find a good result.
What I've seen so far is:
- DashBuilder, which looks interesting but the latest version which has feature we need isn't finished and doesn't have proper filtering tools yet, the way we need them
- Re:dash, which also appears not to offer the intuitive filtering we are looking for
- KPI Watchdog, which isn't self-hosted and doesn't support ElasticSearch
- Cloud9Charts, which appears to get very expensive quickly as soon as we want to build a larger system with it
- Sisense, which appears to be a Windows application and a huge system to start with
- ...and some other smaller things I don't remember
Thanks a lot in advance, David