Stop logging to files. But what system to choose?

Question

Logging to files works since the unix epoch 1970.

I think it is time for a change.

My reasons:

• rotating files is a pain: Server needs to get reloaded after rotating ...
• logging structured data means this workflow:
  • nice high level data structures gets serialized to a byte string
  • later the byte string needs to be constructed to a nice high level data structures again. Can break if two processes log to the same file.
• Yes, grep is nice for most cases. But complex queries (spanning several lines) are hard.

But what to choose?

My environment:

• Linux severs
• Mostly Python software
• Satellite systems:
  • Our software runs in intranets of customers.
  • The satellite runs on its own, but gets managed from our central office
  • Network connection between satellite and central office is sometimes slow or down.

Requirements

• Network outage between satellite and central office can happen daily. Some sort of buffering is needed
• grep-replacement: We need simple way to "grep" the logs from the shell (on satellite and central office)
• structure data support: We need to be able to log structured data (json or yaml)
• Support for Python virtualenvs: We run several virtualenvs on one linux server. Logs need to be kept separated from each virtualenv

Answer 1

You might be able to use: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

Logstash forwards the logs to the central logging server.

---

Also, rsyslog can be configured to forward to a central store, and cache logs locally with the following config setting:

$WorkDirectory /rsyslog/work # default location for work (spool) files Through extended configuration, it can even be configured to log to a MySQL server. Configuration can generate complex entries.

Answer 2

You might want to log to SQLite databases instead of plain text files for the following reasons:

• SQLite is part of the python standard runtime library. Thus, databases can be created, filled and maintained very easily
• binary distribution consists of a single (scriptable) commandline executable, the sqlite shell, which can be used to easily query the logs
• sqlite provides a locking mechanism that prevents files getting clobbered by a concurrent write access
• JSON and YAML data could either be serialized and stored as text (SQLite allows text field size of 2³¹-1 bytes) or if structure is limited in size and consistent across log entries, the database tables could be structured accordingly
• querying the databases by means of using SQL can be much faster, and more convenient than using grep on text files. SQLite also implements indexed full-text search to speed up the search for strings
• Largely decreased the need for log rotation: theoretical maximum number of rows in SQLite databases (which corresponds to a number of possible log entries) is 2⁶⁴. Even without log rotation, recent entries can quickly be queried if time stamp is explicitly integrated as field in the tables

The recommendation would be to use a separate local SQLite database for any virtualenv. The logs could then be collected by the satellite. As soon as the network is available, satellite database can be queried from central office. Alternatively satellite and central office could as well use any other RDBMS for integrating data (in general database management systems also include a command line client for administering or quickly querying - or as you named it - "grepping" the database)

Admittedly there are some limitations:

• writing performance: can be an issue, if entries are generated very fast and each one should go into the DB in a separate transaction. In this case, one only can write a few dozen entries per second. Otherwise, if one can combine multiple entries in one transaction, this is a far smaller problem (please have a look at https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite)
• maximum sqlite database size can be a limitation: if generated data volume is very high, the database size limit of ~ 140 TB is not that far away. In that case, one would prefer a system with more capacity if the rotation is to be avoided completely. For a comparison of different RDBMS see https://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems#Limits