Secure "push" monitoring solution for linux 'servers'

Question

I have a bunch of devices (20-50 range), different architectures (x86_32/64 pc,arm - raspberrypi, beaglebone) running mostly recent Ubuntu and Debian that are deployed in various networks, that are mostly unreachale from outside (mobile networks, firewalls), but all of them have web access (allowed outbound connections to ports 80 and 443).

I would like to monitor system parameters (for starters, cpu load, ram usage, disk usage, basic network stats.. custom 'plugins' are a plus), so I need a solution that would push the data to a server, and visualizations will be done there (alarms are a plus, but we cak hack them together). First idea was to use something 'mainsteam' but most of them use pull model, which is unusable, since you cannot connect to those devices from outside. The next idea was to hack a script that sends those five parametes in a http post to a server, then a script to put it into mysql/postgres, + grafana for visuals.. but it seemed like reinveinting the wheel.

Security (ssl or whatever except clear text) is a "should" (by rfc standards of what 'should' means). But for starters we can even do without that.

Any recommendations?

Answer 1

Icinga2 passive checks with nsca server/client. I wrote an article about it once, I will paste it for you here. All package names are Debian. If you just want to look at my example code go to gitlab.

Package Requirements

Be careful nsca and nsca-ng are NOT compatible.

On the master server:

apt install icinga2
apt install nsca-ng-server

On the nodes:

apt install nsca-ng-client

It you may also want to install nagios-plugins on the nodes, as it contains many helpful monitoring scripts you can use as a starting point in /usr/lib/nagios.

Master Server

NSCA-Server

In /etc/nsca/nsca.cfg specify the named pipeline that is used to write incoming messages to icinga and the user as which to run. The command file should already be specified in /etc/icinga2/icinga2.conf by default.

command_file = "/run/icinga2/cmd/icinga2.cmd"
user = "nagios"

Then add the host(s) from which to receive status messages. You can restrict for which services a host can submit results within the { } separated by commata, but generally that is not necessary for a simple use-case, so we just write * to allow all. This does not mean that the remote node can submit a service for another host.

authorize "example" {
    password = "PASSWORD"
    host     = "example.com"
    services = {
        "*",
    }
}

Icinga

You have to add each host in the /etc/icinga2/ host.conf. You should have a group or a variable that identifies the host as a node, in case you ever want to have non-node servers (aka non-passive-checks).

object Host "example"{
    import 
    name = "example.com"
    address = "example.com"

    # mark as node
    var.is_node = "true"

    # default check intervals
    max_check_attempts = 7
    retry_interval = 1m     

    # add this if you want mail notifications
    vars.notification["mail"] = {
        groups = [ "icingaadmins" ]
    }
}

Then we create a template (passive check) service in services.conf and the checks themselves. Obviously you don't need a template but it makes things shorter. The services themselves can override the check-/retry intervals if necessary. Other than that the services themselves must have the same names as the checks in the monitoring.conf on the remote server.

Template:

template Service "remote_passive" {
    import "generic-service"
    check_interval = 10m
    retry_interval = 1m
    check_command = passive
}

Service:

apply Service "serivce-name" {
    import "remote_passive"
    assign where host.var.is_node
}

It should be mentioned that you do not need to use different files for the your configuration. Every configuration file is simply included by the main icinga configuration /etc/icinga2/icinga2.conf. For (mail) notifications you need a working mail setup which is outside the scope of this article. A starting point for that should be the official icinga documentation.

Node Server

NSCA

On the node server you first need to configure the nsca-ng-client in /etc/send_nsca.cfg:

server   = "example.com"
identity = "example"
password = "PASSWORD"

Checks/Python-script/monitoring.conf

Then you may simply use my python script with a configuration file that looks like this:

user<TAB>service-name<TAB>command-to-execute-with-args

For example:

nobody  test-apt /usr/lib/nagios/plugins/check_apt -o

And start it like this:

/etc/monitoring/monitoring-report.py -c /PATH/TO/monitoring.conf

Finally add this line to your crontab and you are done. Happy becoming sad because you now notice how often your shit breaks.

---

Copied from myself, Cheers!