2

We are looking for a business level logging solution that has some of the features of both logging to a database and regular log files.

1) Regular log files are only readable by developers. Customers want to access some logs too. Like events of a purchase they made.

2) Regular log files are not queryable. We don't need complex queries, joins etc. but finding logs about a particular thing in the system between two dates and such things are desired.

3) Databases are not "storable", like I can't just move old logs to my machine without disturbing the real running system.

4) Databases require maintenance. Application's own DB maintenance is a workload and I don't want to add a second DB's maintenance to it. Log files are still readable after a serious system crash etc. and don't require much maintenance.

5) For both: business logs are important, they are not like website access statistics where losing some does not matter.

Pretty much like this. Isn't there a solution that has a bit of best of both worlds?

  • What OS would be relevant, Windows would offer quite different solutions from something Linux based. – thorsten müller Aug 27 '14 at 10:51
  • Development machines are Windows and real application runs on CentOS. – uylmz Aug 27 '14 at 11:07
2

Well, let's give it a try in case nobody else with better knowledge comes in. So it's not really an answer, but more like pointing towards a few options that may help you.

First it would be interesting what amount of 'logs' you expect. Point one (purchase events) seems to be not very large. Also about point 3, you can always dump some SQL to a file which should not disturb the system very much.

It seems that what you want would be on the line of a NoSQL database. (Though you say you don't want any additional DB maintenance workload). But some types of key value stores (eg Redis) would be very easy to use and others would be easy enough and offer exactly the kind of search you want to do.

As written in my comment on Programmers.SE the ELK stack could be one way to go. Elasticsearch for search (with Kibana as a comfortable enough frontend) and Logstash as one of the industry standards for logs (with Redis for storage as one optin). Though this is meant for high volume logs like Apache logs, sb logs and similar system based logging.

You could easily use this as a starting point. If you don't have a very high volume, you could still use Elasticsearch together with Kibana and add some API in front for authentication.

Let's maybe also try to go through all points one by one:

1) Most systems of this kind (also most NoSQL DBs) do not offer much on the line of user authentication. So your only option in this case would be to write a frontend for the users that handles this. (Say devs would use Kibana with full access and the frontend would still be able to run queries with the rather simple syntax of Elasticsearch)

2) Exactly what ELK offers (and other NoSQL solutions).

3) Unless you are talking about very high data volumes that's simply not true.

4) Well, either you want easy querying or simple storage as files. If you install a 'system' you will have to maintain it to some degree. I wouldn't know of anything that handles data, guarantees persistence and doesn't require maintenance.

5) Most NoSQL solutions allow to do what you want. Which exactly is the best would depend, but Elasticsearch or MongoDB would allow to be configured with 100% persistence. The tradeoffs to be made (as in 2 out of 3 CAP theorem are only relevant with super high data volumes over many clusters).

Maybe have a look at the Elasticsearch site, they have a few video tutorials there that demonstrate ELK and wouldn't cost you more than one hour of your time to watch. If nothing else it will give you some interesting insights in what is possible there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.