Architecture Overview

Introduction

BalenaOS is an operating system optimised for running Docker containers on embedded devices, with an emphasis on reliability over long periods of operation, as well as a productive developer workflow inspired by the lessons learned while building balena.

The core insight behind balenaOS is that Linux Containers offer, for the first time, a practical path to using virtualisation on embedded devices. VMs and hypervisors have lead to huge leaps in productivity and automation for cloud deployments, but their abstraction of hardware as well as their resource overhead and lack of hardware support means that they are out of the question for embedded scenarios. With OS-level virtualisation as implemented for Linux Containers, both those objections are lifted for Linux devices, of which there are many in the Internet of Things.

BalenaOS is an operating system built for easy portability to multiple device types (via the Yocto framework) and optimised for Linux Containers, and Docker in particular. There are many decisions, large and small, we have made to enable that vision, which are present throughout our architecture.

The first version of balenaOS was developed as part of balena, and has run on thousands of embedded devices, deployed in many different contexts for several years. BalenaOS v2 represents the combination of the learnings we extracted over those years, as well as our determination to make balenaOS a first-class open source project, able to run as an independent operating system, for any context where embedded devices and containers intersect.

We look forward to working with the community to grow and mature balenaOS into an operating system with even broader device support, a broader operating envelope, and as always, taking advantage of the most modern developments in security and reliability.

OS composition

The OS is composed of multiple yocto layers. The Yocto Project build system uses these layers to compile balenaOS for the various supported platforms. This document will not go into detailed explanation about how the Yocto Project works, but will require from the reader a basic understanding of its internals and release versioning/codename.

Codename Yocto Project Version Release Date Current Version Support Level Poky Version BitBake branch
Pyro 2.3 Apr 2017 Development
Morty 2.2 Oct 2016 2.2.1 Stable 16.0 1.32
Krogoth 2.1 Apr 2016 2.1.2 Stable 15.0 1.30
Jethro 2.0 Nov 2015 2.0.3 Community 14.0 1.28
Fido 1.8 Apr 2015 1.8.2 Community 13.0 1.26
Dizzy 1.7 Oct 2014 1.7.3 Community 12.0 1.24
Daisy 1.6 Apr 2014 1.6.3 Community 11.0 1.22
Dora 1.5 Oct 2013 1.5.4 Community 10.0 1.20

We will start looking into balenaOS’s composition from the core of the Yocto Project, i.e. poky. Poky has released a whole bunch of versions and supporting all of them is not in the scope of our OS, but we do try to support its latest versions. This might sound ironic as we do not currently support poky’s last version (i.e. 2.1/Krogoth), but this is only because we did not need this version yet. We tend to support versions of poky based on what our supported boards require and also do a yearly update to the latest poky version for all the boards that can run that version. Currently we support three poky versions: 2.0/Jethro, 1.8/Fido and 1.6/Daisy.

On top of poky we add the collection of packages from meta-openembedded. Now that we are done with setting up the build system let’s add Board Support Packages (BSP), these layers are here to provide board-specific configuration and packages (e.g. bootloader, kernel), thus enabling building physical hardware (not emulators). These types of layers are the ones one should be looking for if one wants to add support for a board; if you already have this layer your job should be fairly straightforward, if you do not have it you might be looking into a very cumbersome job. At this point we have all the bits and pieces in place to build an OS. The core code of balenaOS resides in meta-balena. This layer handles a lot of functionality but the main thing that one should remember now is that here one will find the balena-image.bb recipe. This layer also needs a poky version-specific layer to combine with (e.g. meta-balena-jethro), these two layers will give you the necessary framework for the abstract balenaOS generation. Now for the final piece of the puzzle, the board-specific meta-balena configuration layer. This layer goes hand in hand with a BSP layer, for example for the Raspberry Pi family (i.e. rpi0, rpi1, rpi2, rpi3) that is supported by the meta-raspberrypi BSP, we provide a meta-balena-raspberrypi layer that configures meta-balena to the raspberrypi's needs.

Below is a representative example from the Raspberry Pi family, which helps explain meta-balena-raspberrypi/conf/samples/bblayers.conf.sample.

Layer Name Repository Description
meta-balena https://github.com/balena-os/meta-balena This repository enables building balenaOS for various devices
meta-balena-jethro https://github.com/balena-os/meta-balena This layer enables building balenaOS for jethro supported BSPs
meta-balena-raspberrypi https://github.com/balena-os/balena-raspberrypi Enables building balenaOS for chosen meta-raspberrypi machines.
meta-raspberrypi https://github.com/agherzan/meta-raspberrypi This is the general hardware specific BSP overlay for the Raspberry Pi device family.
meta-openembedded http://git.openembedded.org/meta-openembedded Collection of OpenEmbedded layers
meta-openembedded/meta-oe https://github.com/openembedded/meta-openembedded/tree/master/meta-oe
meta-openembedded/meta-python https://github.com/openembedded/meta-openembedded/tree/master/meta-python The home of python modules for OpenEmbedded.
meta-openembedded/meta-networking https://github.com/openembedded/meta-openembedded/tree/master/meta-networking Central point for networking-relatedpackages and configuration.
oe-meta-go https://github.com/mem/oe-meta-go OpenEmbedded layer for the Go programming language
poky/meta-yocto https://git.yoctoproject.org/git/meta-yocto
poky/meta https://git.yoctoproject.org/git/poky Core functionality and configuration of Yocto Project

Userspace Components

The balenaOS userspace tries to package only the bare essentials for running containers while still offering a lot of flexibility. The philosophy is that software and services always default to being in a container, unless they are generically useful to all containers or they absolutely can’t live in a container. The userspace consists of many open source components, but in this section we will just highlight some of the important ones.

Systemd

We use systemd as the init system for balenaOS and it is responsible for launching and managing all the other services. We leverage many of the great features of systemd, such as adjusting OOM scores for critical services and running services in separate mount namespaces. Systemd also allows us to easily manage service dependencies.

Docker

The Docker engine is a lightweight container runtime that allows us to build and run linux containers on balenaOS. BalenaOS has been optimized to run Docker containers and has been set up to use the journald log driver and DNSmasq for container DNS resolution. We use AUFS as the underlying storage driver since it is arguably the most production tested storage driver in the Docker ecosystem. It also allows us to more easily support devices with older kernel versions and additionally gives us the ability to run on devices with Unmanaged NAND flash.

NetworkManager and ModemManager

BalenaOS uses NetworkManager accompanied by ModemManager, to deliver a stable and reliable connection to the internet, be it via ethenet, wifi or cellular modem. Additionally to make headless configuration of the device’s network easy, we have added a system-connections folder in the boot partition which is copied into /etc/NetworkManager/system-connections. So any valid NetworkManager connection file can just be dropped into the boot partition before device commissioning.

DNSmasq

DNSmasq is here to manage the nameservers that NetworkManager provides for balenaOS. NetworkManager will discover the nameservers that can be used and a binary called resolvconf will write them to a tmpfs location, from where DNSmasq will take over and manage these nameservers to give the user the fastest most responsive DNS resolution.

Avahi

In order to improve the development experience of balenaOS, there is an Avahi daemon that starts advertising the device as balena.local or <hostname>.local on boot if the image is a development image.

OpenVPN

BalenaOS will provide the user with an OpenVPN server that they might use. It is worth noting that this server will be disabled by default and manual interaction from the user is needed to activate and configure this server to their needs.

Image Partition Layout

The first partition, resin-boot, is meant to hold boot important bits according to each board (e.g. kernel image, bootloader image). It also holds a very important file that you will find mentioned elsewhere in this document (i.e. config.json). The config.json file is the central point of configuring balenaOS and defining its behaviour, for example you can set your hostname, allow persistent logging, etc. Resin-rootA is the partition that holds our read-only root filesystem; it holds almost everything that balenaOS is. Resin-rootB is an empty partition that is only used when the rootfs is to be updated. We follow the Blue-Green deployment strategy. Essentially we have one active partition that is the OS’s current rootfs and one dormant one that is empty, we download the new rootfs to the dormant partition and try to switch them, if the switch is successful the dormant partition becomes the new rootfs, if not, we rollback to the old active partition. Resin-state is the partition that holds persistent data as explained in the Stateless and Read-only rootfs. Resin-data is the partition that holds downloaded Docker images. Generally any container data will be found here.

Stateless and Read-Only rootFS

BalenaOS comes with a read-only root filesystem, so we can ensure our hostOS is stateless, but we still need some data to be persistent over system reboots. We achieve this with a very simple mechanism, i.e. bind mounts. BalenaOS contains a partition named resin-state that is meant to hold all this persistent data, inside we populate a Linux filesystem hierarchy standard with the rootfs paths that we require to be persistent. After this partition is populated we are ready to bind mount the respective rootfs paths to this read-write location, thus allowing different components (e.g. journald) to be able to write data to disk. A mechanism to purge this partition is provided, thus allowing users to rollback to an unconfigured balenaOS image.

A diagram of our read-only rootfs can be seen below:

Dev vs. Prod images

BalenaOS comes in two flavours, namely Development (dev) and Production (prod). The Development images are recommended while getting started with balenaOS and building a system. The dev images enable a number of useful features while developing, namely:

  • Passwordless SSH to balenaOS
  • The device broadcasts as balena.local or <hostname>.local on the network for easy access.
  • Docker socket exposed on via port 2377
  • Getty console attached to tty1 and serial

Note: Raspberry Pi devices don’t have Getty attached to serial.

The production images have all of the above functionality disabled by default. In both forms of the OS we write logs to an 8 MB journald RAM buffer in order to avoid wear on the flash storage used by most of the supported boards. However, persistent logging can be enabled by setting the "persistentLogging": true key in the config.json file in the boot partition of the device.

OS Tools

Base Images

To help getting started with containers on embedded systems, balenaOS comes with a full complement of over 500 Docker base images. We currently have base images for Debian, Fedora and Alpine Linux distributions, as well as Nodejs, Python, Go and Java language base images. For a more in-depth look into all the available base images head over to the balena base images wiki or the Docker Hub repository.

Balena Command Line Tool

The balena CLI is a set of useful tools that help with setting up and developing containers with a balenaOS device. The goal of the CLI is to provide a simple and intuitive developer experience. Currently, local mode part of the CLI are evolving fast and is considered 'beta', so as always, we love it when you report bugs, you can report them here: github.com/balena-io/balena-cli

Installation

Currently the CLI is a node.js based command line tool which requires that our system has the following dependencies installed and in our path:

Once we have those setup we can install the CLI using npm:

$ npm install --global --production --unsafe-perm balena-cli

Usage

Configure

balena local configure allows you configure or reconfigure a balenaOS system image or SD card. Currently, this allows for configuration of wifi settings, hostname and enablement of persistent journald logs.

$ balena help local configure
Usage: local configure <target>

Use this command to configure or reconfigure a balenaOS drive or image.

Examples:

	$ balena local configure /dev/sdc
	$ balena local configure path/to/image.img
Flash

balena local flash command helps to easily and safely flash the balenaOS system image on an SD card or USB drive.

Note: Currently, balena local flash doesn't work with the Intel Edison board.

$ balena help local flash
Usage: local flash <image>

Use this command to flash a balenaOS image to a drive.

Examples:

	$ balena local flash path/to/balenaos.img
	$ balena local flash path/to/balenaos.img --drive /dev/disk2
	$ balena local flash path/to/balenaos.img --drive /dev/disk2 --yes

Options:

    --yes, -y                           confirm non-interactively          
    --drive, -d <drive>                 drive     
Push

The balena local push command enables you to quickly build and deploy a Docker container to a target device. It also allows you to easily sync code between your laptop directory and your running container on the target. Once you have a balenaOS host device running and advertising on the network, balena local push will allow you to iterate code on a container service. push has a lot of advanced functionality, which all gets encoded into the .balena-sync.yml file in your project directory. To better understand the .yml file, let's look at an example:

  app-name: myapp
  build-triggers:
    - Dockerfile: 9396730559c4107a0d82f95373c45d8e0634556342fb4ee276a7bc1c84c51570
    - package.json: 1e40740d7cf5018cf887124ee75917eb139d49473c51464580feda247b3a4641
destination: /usr/src/app
ignore:
  - .git
  - node_modules/

Let's look a bit into some of these settings keys.

  • app-name: represents the name of the Docker image and corresponding running container on the target device.
  • build-triggers: is a list of file names and their content hashes. Any changes in the files contained in this list will result in a docker build . on the target device.
  • destination: this sets the destination sync directory in the container running on the device. This allows you to synchronise a --source directory on your laptop to a running container on the device. This is very useful when developing with interpreted languages like Python or Node.js, as you can sync just your source code across, without rebuilding the entire container.
  • ignore: is a list of files or directories that you would like to be ignored during the directory sync. This is useful in the case where you have node_modules in your source directory that is compiled to run on your x86 laptop, but you are pushing your code over to an ARM based embedded device.

The push has quite a bit of functionality and a few useful flags. Check out the balena help local push:

$ balena help local push
Usage: local push [deviceIp]

Warning: 'balena local push' requires an openssh-compatible client and 'rsync' to
be correctly installed in your shell environment. For more information (including
Windows support) please check the README here: https://github.com/balena-io/balena-cli

Use this command to push your local changes to a container on a LAN-accessible balenaOS device on the fly.

If `Dockerfile` or any file in the 'build-triggers' list is changed,
a new container will be built and run on your device.
If not, changes will simply be synced with `rsync` into the application container.

After every 'balena local push' the updated settings will be saved in
'<source>/.balena-sync.yml' and will be used in later invocations. You can
also change any option by editing '.balena-sync.yml' directly.

Here is an example '.balena-sync.yml' :

	$ cat $PWD/.balena-sync.yml
	destination: '/usr/src/app'
	before: 'echo Hello'
	after: 'echo Done'
	ignore:
		- .git
		- node_modules/

Command line options have precedence over the ones saved in '.balena-sync.yml'.

If '.gitignore' is found in the source directory then all explicitly listed files will be
excluded when using rsync to update the container. You can choose to change this default behavior with the
'--skip-gitignore' option.

Examples:

	$ balena local push
	$ balena local push --app-name test-server --build-triggers package.json,requirements.txt
	$ balena local push --force-build
	$ balena local push --force-build --skip-logs
	$ balena local push --ignore lib/
	$ balena local push --verbose false
	$ balena local push 192.168.2.10 --source . --destination /usr/src/app
	$ balena local push 192.168.2.10 -s /home/user/myBalenaProject -d /usr/src/app --before 'echo Hello' --after 'echo Done'

Options:

    --source, -s <path>                 root of project directory to push                                                                                       
    --destination, -d <path>            destination path on device container                                                                                    
    --ignore, -i <paths>                comma delimited paths to ignore when syncing with 'rsync'                                                               
    --skip-gitignore                    do not parse excluded/included files from .gitignore                                                                    
    --before, -b <command>              execute a command before pushing                                                                                        
    --after, -a <command>               execute a command after pushing                                                                                         
    --progress, -p                      show progress                                                                                                           
    --skip-logs                         do not stream logs after push                                                                                           
    --verbose, -v                       increase verbosity                                                                                                      
    --app-name, -n <name>               application name - may contain lowercase characters, digits and one or more dashes. It may not start or end with a dash.
    --build-triggers, -r <files>        comma delimited file list that will trigger a container rebuild if changed                                              
    --force-build, -f                   force a container build and run                                                                                         
    --env, -e <env>                     environment variable (e.g. --env 'ENV=value'). Multiple --env parameters are supported.              
SSH

balena local ssh discovers balenaOS devices on the local network and allows you to drop a SSH session into any of the containers running on the device. It also enables to you to drop in to the underlying host OS by doing balena local ssh --host, however you can of course always just do ssh root@balena.local -p22222.

$ balena help local ssh
Usage: local ssh [deviceIp]

Warning: 'balena local ssh' requires an openssh-compatible client to be correctly
installed in your shell environment. For more information (including Windows
support) please check the README here: https://github.com/balena-io/balena-cli

Use this command to get a shell into the running application container of
your device.

The '--host' option will get you a shell into the Host OS of the balenaOS device.
No option will return a list of containers to enter or you can explicitly select
one by passing its name to the --container option

Examples:

	$ balena local ssh
	$ balena local ssh --host
	$ balena local ssh --container chaotic_water
	$ balena local ssh --container chaotic_water --port 22222
	$ balena local ssh --verbose

Options:

    --verbose, -v                       increase verbosity                 
    --host, -s                          get a shell into the host OS       
    --container, -c <container>         name of container to access        
    --port, -p <port>                   ssh port number (default: 22222)  
Logs

balena local logs allows the fetching of logs from any of the running containers on the device.

$ balena help local logs
Usage: local logs [deviceIp]


Examples:

	$ balena local logs
	$ balena local logs -f
	$ balena local logs 192.168.1.10
	$ balena local logs 192.168.1.10 -f
	$ balena local logs 192.168.1.10 -f --app-name myapp

Options:

    --follow, -f                        follow log                         
    --app-name, -a <name>               name of container to get logs from
Balena
An open source project byBalena
Terms of Service|Privacy Statement|Master Agreement|
Copyright 2018 Balena
|
All Rights Reserved