Monitoring software for production software?

Question

We've some servers in linux and these servers get hung (stuck) but not stopped. So, how can I deal with those servers? It's not clear what the cause of this stuckness is. Any guidance will be appreciated.

The problems:

1. The server hangs from time to time. It doesn't stop. It just hangs. Theoretically it's still up but practically it has stopped working. The one way to trace it is to monitor the logs, you'd see logs not being printed anymore.

Cause: Unknown

2. The server goes down time to time, too frequently on some servers.

Cause: Huge log size

Solution: logrotate

3. The server goes down time to time, too frequently on some servers.

Cause: Unknown

Solution: Script that auto-restarts the service in timely manner. I've less hopes that it will work though.

4. The clients want to be able to monitor these services by themselves and do things like restarting by themselves. What's the best monitoring tool that allows to restart the service as well(i.e something that runs scripts as I like)?

Are nagios, zabbix, monit used for this purpose? what's the best tool for this purpose?

We're using centos 7 (Yes it's reaching end of life). The servers are on virtual machine. We only have remote access. The applications are:

• java servers

• glassfish servers

• tomcat servers

Answer 1

Yes, nagios etc. are used for this purpose. You can ask nagios, for example, to regularly try a HTTP request to some specific health-check URL, and it will then alert you (via mail, SMS etc.) if it cannot connect. It works by regularly polling some ports or data from servers and also lower level components, and has concepts like a server being behind some network, so if the network goes away then the server will not be flagged "red" (which would make no sense if the server is not reachable to nagios).

As you mention that you do not know the cause of some of your problems, let me suggest other software:

Prometheus for gathering metrics. "Metrics" are things like "this web server served 100 pages in the last hour, with an average response time of 100ms". It acts as a central database for this and many more metrics - for web servers, Linux hosts, etc. etc.; including custom metrics which your developers can program themselves.

Grafana as a visualization tool for the metrics kept my Prometheus. You can configure all kinds of realtime dashboards to your hearts content. You can easily check historic data in the same dashboards. This can help loads if you have unknown causes - maybe your Linux server just ran out of RAM, or it showed 100% CPU for an hour each time before it crashed, and so on and forth.

These two give you Monitoring.

Then you also maybe want to invest in a Logging stack. A popular one is ELK stack (ElasticSearch - Logstash - Kibana). In this case, ElasticSearch has the role of data management / database, logstash is responsible for shoveling the data in, and Kibana is for queries and visualization. There are alternative, e.g. you could use fluentd instead of logstash if it suits your tastes more.

Speaking of which: as you are writing about a rather small server environment, any of those should be more than capable of serving your needs. You can indeed go somewhat by feel/taste; maybe you want something strictly open source or free; maybe you prefer something with a paid license (and commercial support to go along with it), etc.

The logging stack gives you a) a central storage for all the logs of all your components. You do not need to have gigabytes of storage on each individual web server anymore; the logs are usually transferred to the stack in near real-time. Especially handy if a server crashes and you somehow manage to lose the log files... And b) it gives you a central GUI to go through all your logs. This makes it very easy to get all logs from different components in a given time span, and makes it much easier to see if there may be some connections between different components that caused the problem.