What is balenaOS?
BalenaOS is an operating system optimised for running Docker containers on embedded devices, with an emphasis on reliability over long periods of operation, as well as a productive developer workflow inspired by the lessons learned while building balena.
The core insight behind balenaOS is that Linux Containers offer, for the first time, a practical path to using virtualisation on embedded devices. VMs and hypervisors have lead to huge leaps in productivity and automation for cloud deployments, but their abstraction of hardware as well as their resource overhead and lack of hardware support means that they are out of the question for embedded scenarios. With OS-level virtualisation as implemented for Linux Containers, both those objections are lifted for Linux devices, of which there are many in the Internet of Things.
BalenaOS is an operating system built for easy portability to multiple device types (via the Yocto framework) and optimised for Linux Containers, and Docker in particular. There are many decisions, large and small, we have made to enable that vision, which are present throughout our architecture.
The first version of balenaOS was developed as part of the balena platform, and has run on thousands of embedded devices on balena, deployed in many different contexts for several years. balenaOS v2 represents the combination of the learnings we extracted over those years, as well as our determination to make balenaOS a first-class open source project, able to run as an independent operating system, for any context where embedded devices and containers intersect.
We look forward to working with the community to grow and mature balenaOS into an operating system with even broader device support, a broader operating envelope, and as always, taking advantage of the most modern developments in security and reliability.
Variants of balenaOS
BalenaOS currently comes in 3 different variants all built from the same source but with slightly differing features enabled or disabled. Each version of balenaOS produces the following variants:
|Version Name||Variant Type||Description|
|2.0.0+rev2||production||This is the production version of the balena managed OS. This is the OS you should use for any production fleet deployments|
|2.0.0+rev2-dev||development||This version the development version of the above and should be used when you are developing a new application and want to use the fast local mode workflow. This variant should never be used in production.|
|2.0.0+rev2.dev||standalone||This version is an unmanaged version of balenaOS, it doesn't have the balena supervisor and does not connect to the management console. Use this variant if you want a stable linux OS to run Docker on.|
Dev vs. Prod images
The Development images are recommended while getting started with balenaOS and building a system. The dev images enable a number of useful features while developing, namely:
- Passwordless SSH into balenaOS on port
22222as the root user. So one can do
ssh root@<DEVICE_IP> -p22222and poke around to see how the system runs.
- Docker socket exposed on port
2375, which allows
deployto do remote Docker builds on the target device (see Deploy to your Fleet).
- Getty console attached to tty1 and serial.
- Capable of entering local mode for rapid development of application containers locally.
Note: Raspberry Pi devices don’t have Getty attached to serial.
The production images have all of the above functionality disabled by default. In both forms of the OS we write logs to an 8 MB journald RAM
buffer in order to avoid wear on the flash storage used by most of the supported boards. However, persistent logging can be enabled by setting
"persistentLogging": true key in the
config.json file in the boot partition of the device. The logs can be accessed via the host OS at
Both Prod and Dev variants will also allow the setting of a custom hostname via the
config.json, just add
"hostname": "my-new-hostname". Your device will then broadcast (via Avahi) on the network as
my-new-hostname.local. If you don't set a custom hostname, the device will default to
On the production variant, nothing is written to tty1, on boot up you should only see the balena logo on the HDMI screen and this will persist until your application code takes over the framebuffer. If you would like to replace the balena logo with your own custom splash logo, then you will need to replace splash/resin-logo.png file that you will find in the first partition of our images (boot partition or
resin-boot) with your own image. NOTE: As it currently stands plymouth expects the image to be named
BalenaOS also comes in a Standalone or "Unmanaged" variant. This variant is exactly the same as the
-dev variant but does not have the balena supervisor agent and has the balena VPN service disabled, so it will never connect to the balena management system.
The standalone version of balenaOS is meant as an excellent way to get started with Docker containers on embedded systems and you can read more about this over at balena.io/os.
The balenaOS userspace tries to package only the bare essentials for running containers while still offering a lot of flexibility. The philosophy is that software and services always default to being in a container, unless they are generically useful to all containers or they absolutely can’t live in a container. The userspace consists of many open source components, but in this section we will just highlight some of the most important services.
The balena supervisor is a lightweight container which runs on your device, manages your applications and communicates with our servers - downloading new application containers and updates to existing containers as you push them, sending logs to your dashboard. It also provides a useful HTTP supervisor API interface, which allows you to query update status and perform certain actions on the device.
We use systemd as the init system for balenaOS and it is responsible for launching and managing all the other services. We leverage many of the great features of systemd, such as adjusting OOM scores for critical services and running services in separate mount namespaces. systemd also allows us to easily manage service dependencies.
The Docker engine is a lightweight container runtime that allows us to build and run linux containers on balenaOS. BalenaOS has been optimized to run Docker containers and has been set up to use the journald log driver and DNSmasq for container DNS resolution. We use AUFS as the underlying storage driver since it is arguably the most production tested storage driver in the Docker ecosystem. It also allows us to more easily support devices with older kernel versions and additionally gives us the ability to run on devices with Unmanaged NAND flash.
NetworkManager and ModemManager
BalenaOS uses NetworkManager accompanied by ModemManager, to deliver a stable and reliable connection to the internet, be it via ethenet, wifi or cellular modem. Additionally to make headless configuration of the device’s network easy, we have added a
system-connections folder in the boot partition which is copied into
/etc/NetworkManager/system-connections. So any valid NetworkManager connection file can just be dropped into the boot partition before device commissioning.
DNSmasq is here to manage the nameservers that NetworkManager provides for balenaOS.
NetworkManager will discover the nameservers that can be used and a binary called
resolvconf will write them to a tmpfs location, from where DNSmasq will take over and manage these nameservers to give the user the fastest most responsive DNS resolution.
In order to improve the development experience of balenaOS, there is an Avahi daemon that starts advertising the device as
<hostname>.local on boot if the image is a development image.
BalenaOS will provide the user with an OpenVPN server that they might use. It is worth noting that this server will be disabled by default and manual interaction from the user is needed to activate and configure this server to their needs.
Stateless and Read-Only rootFS
BalenaOS comes with a read-only root filesystem, so we can ensure our host OS is stateless, but we still need some data to be persistent over system reboots. We achieve this with a very simple mechanism, i.e. bind mounts. BalenaOS contains a partition named resin-conf that is meant to hold all this persistent data, inside we populate a Linux filesystem hierarchy standard with the rootfs paths that we require to be persistent. After this partition is populated we are ready to bind mount the respective rootfs paths to this read-write location, thus allowing different components (e.g. journald) to be able to write data to disk. A mechanism to purge this partition is provided, thus allowing users to rollback to an unconfigured balenaOS image.
A diagram of our read-only rootfs can be seen below:
Image Partition Layout
The first partition,
resin-boot, holds important boot files according to each board (e.g. kernel image, bootloader image). It also holds a very important file that you will find mentioned elsewhere in this document (i.e.
config.json file is the central point of configuring balenaOS and defining its behaviour, for example you can set your hostname, allow persistent logging, etc.
resin-rootA is the partition that holds our read-only root filesystem; it holds almost everything that balenaOS is.
resin-rootB is an empty partition that is only used when the rootfs is to be updated. We follow the A-B update strategy for the resin HostOS upgrades. Essentially we have one active partition that is the OS’s current rootfs and one dormant one that is empty, we download the new rootfs to the dormant partition and try to switch them, if the switch is successful the dormant partition becomes the new rootfs, if not, we go back to the old active partition.
resin-state is the partition that holds persistent data as explained in the Stateless and Read-only rootfs.
resin-data is the partition that holds downloaded Docker images. Generally any container data will be found here. If you want to read a bit more about the partition layout, have a look at the balenaOS github repo.
OS Yocto Composition
The OS is composed of multiple Yocto layers. The Yocto Project build system uses these layers to compile balenaOS for the various supported platforms. This document will not go into detailed explanation about how the Yocto Project works, but will require from the reader a basic understanding of its internals and release versioning/codename.
|Codename||Yocto Project Version||Release Date||Current Version||Support Level||Poky Version||BitBake branch|
We will start looking into balenaOS’s composition from the core of the Yocto Project, i.e. poky. Poky has released a whole bunch of versions and supporting all of them is not in the scope of our OS, but we do try to support its latest versions. This might sound unexpected as we do not currently support poky’s last version (i.e. 2.1/Krogoth), but this is only because we did not need this version yet. We tend to support versions of poky based on what our supported boards require and also do a yearly update to the latest poky version for all the boards that can run that version. Currently we support three poky versions: 2.0/Jethro, 1.8/Fido and 1.6/Daisy.
On top of poky we add the collection of packages from meta-openembedded.
Now that we are done with setting up the build system let’s add Board Support Packages (BSP), these layers are here to provide board-specific configuration and packages (e.g. bootloader, kernel), thus enabling building physical hardware (not emulators). These types of layers are the ones one should be looking for if one wants to add support for a board; if you already have this layer your job should be fairly straightforward, if you do not have it you might be looking into a very cumbersome job.
At this point we have all the bits and pieces in place to build an OS.
The core code of balenaOS resides in meta-balena. This layer handles a lot of functionality but the main thing that one should remember now is that here one will find the
resin-image.bb recipe. This layer also needs a poky version-specific layer to combine with (e.g. meta-balena-jethro), these two layers will give you the necessary framework for the abstract balenaOS generation.
Now for the final piece of the puzzle, the board-specific meta-balena configuration layer. This layer goes hand in hand with a BSP layer, for example for the Raspberry Pi family (i.e. rpi0, rpi1, rpi2, rpi3) that is supported by the meta-raspberrypi BSP, we provide a meta-balena-raspberrypi layer that configures meta-balena to the raspberrypi's needs.
Below is a representative example from the Raspberry Pi family, which helps explain meta-balena-raspberrypi/conf/samples/bblayers.conf.sample.
|meta-balena||https://github.com/balena-os/meta-balena||This repository enables building balenaOS for various devices|
|meta-balena-jethro||https://github.com/balena-os/meta-balena||This layer enables building balenaOS for jethro supported BSPs|
|meta-balena-raspberrypi||https://github.com/balena-os/balena-raspberrypi||Enables building balenaOS for chosen meta-raspberrypi machines.|
|meta-raspberrypi||https://github.com/agherzan/meta-raspberrypi||This is the general hardware specific BSP overlay for the Raspberry Pi device family.|
|meta-openembedded||http://git.openembedded.org/meta-openembedded||Collection of OpenEmbedded layers|
|meta-openembedded/meta-python||https://github.com/openembedded/meta-openembedded/tree/master/meta-python||The home of python modules for OpenEmbedded.|
|meta-openembedded/meta-networking||https://github.com/openembedded/meta-openembedded/tree/master/meta-networking||Central point for networking-relatedpackages and configuration.|
|oe-meta-go||https://github.com/mem/oe-meta-go||OpenEmbedded layer for the Go programming language|
|poky/meta||https://git.yoctoproject.org/cgit/cgit.cgi/poky/||Core functionality and configuration of Yocto Project|