Build a monitoring dashboard with resin.io

Johannes 'fish' Ziemke is Platforms Engineer at Jodel. Jodel is a Berlin startup building a location based social network. Join them if you want to work with a great team on a technically challenging product!

Introduction

We're running Prometheus for monitoring both OS level metrics like CPU usage as well as application level metrics like error counters or latency of requests to our data stores. To make those metrics easily accessible, we're using Grafana which allows you to create dashboards from various data sources.

But the best dashboards are useless if nobody is looking at them, so we decided to put up a big screen in our office showing our "top level dashboard".

Now the remaining questions was: How to actually display the dashboard there?

Implementation

jodel-dashboard

We bought a Raspberry Pi, sshed into it, installed software here and there and tried things out. It felt like system engineering in the 90s, with nobody really knowing what is installed and no way to reliably reproduce the image. Maybe okay for an office dashboard, but nothing that meets our engineering standards in general.

I've ran into resin.io before but haven't had a use case for it back then. This project though sounded like the perfect use case. I love that you can just boot up the Raspberry Pi and have it available for deployments straight away. No need to get up and reflash the SD card after some breakage, no manual tampering with the system until it works and all the other advantages of immutable deployments.

So I've created a new project and flashed the images to my Raspberry Pi as described in the getting started documentation. After that, I've created a Dockerfile which installs Xorg, OpenVPN and a browser. Figuring out a browser which works was actually the hardest part.

Browser

The epiphany browser is highly recommended for it's low resource footprint but it failed to load grafana properly. Chromium on the other hand crashes due to a two year old bug in V8 so I went for good old Firefox, or to be precise it's Debian fork iceweasel which works surprisingly well.

Since Firefox, like most browsers, is intended to be used by a human we need to customize it to work well for displaying our dashboards.

First of all, it has no way to start it in fullscreen. Instead of relying on the xautomation tools like xte, we installed R-kiosk which, beside other things, forces firefox to run in fullscreen.

Firefox also has a feature to prompt for killing unresponsive scripts. Since the Raspberry Pi is pretty slow, we ran into this several times when displaying grafana dashboards. To disable this, we need to modify the Firefox preferences.

Since we can't install extensions and modify Firefox settings as part of the build process, we needed to include a existing profile. For that we started Firefox on a dev machine, changed the settings and installed the extensions, then stripped down the profile to only include the extension and preferences.

The stripped down profile is part of the dashboard-viewer's repository and gets included in the Docker build. That way, when Firefox starts for the first time it already has the extension enabled and 'Unresponsive
script' dialog disabled.

VPN

Since our dashboard isn't available in the public internet, I've included openvpn. It reads a set of environment variables for configuration, then starts openvpn in client mode and waits for the specified dashboard URL to come up before it starts the browser.

Thanks to OpenVPN, implementing this was straight forward. The only caveat is that the resin.io environment variables can't include newlines, so those need to get escaped as \n.

Use it

We made all this available on https://github.com/Jodel/dashboard-viewer. If you want to use our dashboard viewer, you can simply create a new resin.io, check out the repository and push it to your resin.io remote:

$ git clone https://github.com/Jodel/dashboard-viewer.git
$ cd dashboard-viewer
$ git remote add resin your-user@git.resin.io:your-user/your-repo.git
$ git puysh resin master

After that, you should set the following environment variables:

  • URL: Dashboard URL to open
  • PASS: The dashboard runs dropbear, this var will be the root password. If unset, defaults to 'root'

To use openvpn, set the following options:

  • REMOTE_HOST: Address of openvpn server
  • REMOTE_PORT: Port the openvpn server is running on
  • VPN_USER: VPN Username
  • VPN_PASS: VPN Password
  • CERT: Client certificate, newlines replaced by \n

If you have any questions you can chat to us gitter.

comments powered by Disqus
Terms of Service | Privacy Statement | Master agreement | Copyright 2019 Balena | All Rights Reserved