Exporting Jamulus stats as prometheus metrics

In the last post, I introduced the grafana and prometheus combination and described two ways, how to monitor a Jamulus installation. The only thing I wanted to monitor is, how many clients are connected. The two ways however are not very reliable. Both use the tool mtail, that can parse log files and make the logs available as prometheus metrics. The first method uses the direct log entries of Jamulus, the second method uses a status page created by Jamulus. But since mtail is about log files, it really only reads new lines added to the files, which it is tailing (like tail -f). Jamulus doesn’t log disconnects, only connects, so the log file doesn’t even provide all information. The status file lists all connected clients, but it is not a log file, it would be need to be read at a whole.

I’ve looked at grok_exporter, another tool similar to mtail. While it has a configuration property “readall”, which sounded like it would be useful for the status file, it still is tailing the file and only processes new lines. The readall property only has an effect, when grok_exporter is started: then it will read all lines of the log files before tailing it. But in the end, it also tails it and updates to the status files are not processed correctly (same as with mtail).

global:
  config_version: 3
input:
  type: file
  path: /var/run/jamulus/jamulus*-status.html
  readall: true
  fail_on_missing_logfile: false
metrics:
  - type: gauge
    name: jamulus_clients_grok
    help: number of connected jamulus clients
    match: '  <li>.*</li>'
    value: '1'
    cumulative: true
    labels:
      logfile: ''
server:
  protocol: http
  host: localhost
  port: 9144
  path: /metrics

You can start it with grok_exporter -config /etc/grok_exporter_config.yml -disable-exporter-metrics and test it with curl http://localhost:9144/metrics.

So, grok_exporter also doesn’t work.

Solution

Then, let’s create a prometheus metrics exporter for Jamulus on our own. How hard can it be?

The status file “/run/jamulus/jamulus-status.html” (option --htmlstatus for Jamulus) might look like this:

<ul>
  <li>Andreas Dangel</li>
</ul>

It is intended to be included in some webspace content.

If there are no clients connected, the file looks like this:

  No client connected

I’ll use bash scripting and nc (netcat) for providing the server part. We just need to count the lines, that contain “<li>” in them, e.g.

grep '<li>' /run/jamulus/jamulus-status.html|wc -l

We’ll need to convert this result into the prometheus metrics format and wrap around some minimal HTTP headers. That is the server script, which we’ll place at /usr/local/bin/Jamulus_stats:

#!/usr/bin/env bash

logfile=/run/jamulus/jamulus-status.html

response="# TYPE jamulus_stats gauge
jamulus_stats{logfile=\"$(basename $logfile)\"} $(grep '<li>' $logfile|wc -l)
"

header="HTTP/1.0 200 OK
Content-Type: text/plain
Content-Length: ${#response}"

echo -en "$header\n\n$response"

Make sure to make it executable (chmod 755 /usr/local/bin/Jamulus_stats).

Next we’ll need a service script, that will open a tcp port on localhost and expose this information. This service script is /usr/local/bin/Jamulus_statsd:

#!/usr/bin/env bash

set -e

while true; do
    nc -l -p 9144 -e /usr/local/bin/Jamulus_stats
done

Also make this script executable (chmod 755 /usr/local/bin/Jamulus_statsd).

Then let’s add this to systemd: Create a new file /etc/systemd/system/jamulus-statsd.service:

[Unit]
Description=Jamulus-Stats-Server
After=jamulus.target

[Service]
Type=simple
User=jamulus
Group=nogroup
NoNewPrivileges=true
ProtectSystem=true
ProtectHome=true

ExecStart=/usr/local/bin/Jamulus_statsd

Restart=on-failure
RestartSec=30
StandardOutput=journal
StandardError=inherit
SyslogIdentifier=jamulus-statsd

[Install]
WantedBy=multi-user.target

Enable and start this service:

sudo chmod 644 /etc/systemd/system/jamulus-statsd.service
sudo systemctl daemon-reload
sudo systemctl enable jamulus-statsd
sudo systemctl start jamulus-statsd

Now, you can test this metrics endpoint via curl:

$ curl localhost:9144/metrics
# TYPE jamulus_stats gauge
jamulus_stats{logfile="jamulus-status.html"} 0

Time to add this to prometheus as an additional scrape config in /etc/prometheus/prometheus.yml:

- job_name: jamulus_statsd
  static_configs:
    - targets: ['localhost:9144']

Reload prometheus: sudo systemctl reload prometheus

Now you can use the query jamulus_stats{logfile="jamulus-status.html"} to get the current number of connected clients and to draw nice charts in grafana.