Improve this doc

SSH access

To help you debug and develop your applications, we've provided a browser-based terminal and a command line tool for easy SSH access to your devices. With these tools, you have console access to any of your running containers, as well as to the host OS, letting you test out small snippets of code and check system logs on your device. You can also access your device via a standalone SSH client.

Note: Host OS SSH access is available for devices running balenaOS version 2.7.5 and above.

Using the dashboard web terminal

To use this feature, navigate to your application and select the device you want to access. You will see a Terminal window below the Logs window:

SSH Terminal

If your device is online, select a target as either the host OS or a running service, and click the blue >_ Start Terminal session button. In order to start a terminal session for a service, you need to ensure that the service container is running. If the container code crashes or ends quickly, it is not possible to attach a console to it.

A terminal session should be initiated for you in a second or two. If you would like a bigger window for the terminal, you can click the Expand button in the upper-right corner.

Note: To copy and paste in the terminal window, you cannot use the normal Ctrl + C and Ctrl + V shortcuts. You can either select Copy and Paste from a menu, or use Ctrl + Insert for copy and Shift + Insert for Paste. For MacOS users, ⌘ + C and ⌘ + V work as expected.

Using balena ssh from the CLI

Note: balena ssh to application services/containers requires CLI version 11 or above.

If you prefer to work from the command line, you can use balena ssh to connect to your application containers and the host OS. First, you will need to install the balena Command Line Interface (CLI). Once that is set up, run the following in your development machine's terminal:

$ balena ssh <device-uuid>

<device-uuid> is the unique identifier for the device you want to access, which can be found via the dashboard or in the output of the balena devices CLI command. By default, SSH access is routed into the host OS shell. However, you can SSH into a service by specifying its name as part of the command:

$ balena ssh <device-uuid> main

This also works in multicontainer applications, simply pass the name of the appropriate service (as defined in docker-compose.yml) instead of main.

Note: To run a command in a non-interactive way, you can pipe commands to the CLI's stdin. For example, echo "uptime; exit;" | balena ssh <device-uuid>.

When an application name or device UUID is used as above, balena ssh uses the balena VPN to create a secure tunnel to the device and then forward SSH traffic between the device and your development machine. For this method, you must have a SSH key configured on your development machine and added to the balenaCloud dashboard.

If an IP address or a .local hostname is used (instead of an application name or device UUID), balena ssh establishes a direct connection that does not rely on the balena VPN:

$ balena ssh 192.168.1.23
$ balena ssh <device-uuid>.local

This should work without further configuration in the case of devices running a development balenaOS image, which allows passwordless root SSH access (and for this reason, should never be directly exposed to the public internet). In the case of a production balenaOS image, an SSH key needs to be present in the config.json file of the device. More details about configuring SSH keys in config.json may be found here.

Add an SSH key to balenaCloud

To add an SSH key, go to the Preferences page of balenaCloud and select the SSH Keys tab.

SSH key preferences

You may either import an existing SSH key from GitHub or manually enter the public SSH key of an existing SSH key on your development machine.

If you do not have an existing key, you can follow GitHub's documentation, skipping the step about adding the key to your GitHub account, and instead adding the key to your balenaCloud account.

Using a standalone SSH client

If you prefer to use a standalone SSH client to connect to the device, the SSH server on a device listens on TCP port 22222. While development images have passwordless root access enabled, production images require an SSH key to be added to the config.json file.

$ ssh -p 22222 root@<device_ip_address>

balena tunnel

The SSH server of the host OS will always be accessible on TCP port 22222 unless this has been blocked at the network level. To get around this, you can use the balena tunnel command of the balena CLI, which may be used to map local ports to listening ports on the device. For example, the following command maps the local port 4321 to port 22222 on the device:

$ balena tunnel <device-uuid> -p 22222:4321

The device can then be accessed on port 4321 via the balena ssh command of the balena CLI or standalone SSH client:

$ balena ssh 127.0.0.1 -p 4321
$ ssh -p 4321 root@127.0.0.1

Troubleshooting with host OS access

Note: For an in-depth guide to debugging balena devices see the device debugging masterclass.

Host OS SSH access gives you a handful of tools that can help you gather more information about potential issues on your device.

Warning: Making changes to running services and network configurations carries the risk of losing access to your device. Before making changes to the host OS of a remote device, it is best to test locally. Changes made to the host OS will not be maintained when the OS is updated, and some changes could break the updating process. When in doubt, reach out to us for guidance.

BalenaOS services

BalenaOS uses systemd as its init system, and as such, almost all the fundamental components in balenaOS run as systemd services. In general, some core services need to execute for a device to come online, connect to the balenaCloud VPN, download applications, and then run them:

  • chronyd.service - Responsible for NTP duties and syncing 'real' network time to the device.
  • dnsmasq.service - The local DNS service which is used for all host OS lookups.
  • NetworkManager.service - The underlying Network Manager service, ensuring that configured connections are used for networking.
  • os-config.service - Retrieves settings and configs from the API endpoint, including certificates, authorized keys, the VPN config, etc.
  • openvpn.service - The VPN service itself, which connects to the balenaCloud VPN, allowing a device to come online.
  • balena.service - The balenaEngine service, the modified Docker daemon fork that allows the management and running of application service images, containers, volumes, and networking.
  • resin-supervisor.service - The balena Supervisor service, responsible for the management of applications, including downloading updates for and self-healing (via monitoring) of those applications, variables (application/device/fleet), and exposure of these services to applications via an endpoint.
  • dbus.service - The DBus daemon socket which can be used by applications by applying the io.balena.features.dbus label, which exposes it in-container. This allows applications to control several host OS features, including the Network Manager.

Additionally, there are a couple of utility services that, while not required for a barebones operation, are also useful:

  • ModemManager.service - Deals with non-Ethernet or Wifi devices, such as LTE/GSM modems.
  • avahi-daemon.service - Used to broadcast the device’s local hostname.

You may see all enabled services on the host OS with the following command:

$ systemctl list-unit-files | grep enabled

To check the status of a service, use the systemctl status <serviceName> command. The output includes whether the service is currently loaded and active, together with detail about the process, including the latest entries from the journal log. For example, to obtain the status of the OpenVPN service use the following command:

$ systemctl status openvpn.service

Checking logs

journalctl

Information from a variety of services can be found using the journalctl utility. The output from journalctl can be very large, and you can filter the output using the --unit (or the short version -u) option to only output logs from a single service.

A typical example of using journalctl might be following a service to see what’s occurring in real-time by using the --follow (-f) option. For example, to follow the latest supervisor logs on the device:

$ journalctl --follow --unit resin-supervisor

To limit the output to the last x messages, use the -n option. The following example lists the last 10 messages from the chronyd service:

$ journalctl -n 10 -u chronyd

The --all (-a) option may be used to show all entries, even if long or with unprintable characters. This is especially useful for displaying the service container logs from applications when applied to balena.service.

$ journalctl --all -n 100 -u balena

dmesg

For displaying messages from the kernel, you can use dmesg. Similar to journalctl, the output from dmesg will be very large without additional options. The following example limits the output to the last 100 lines:

$ dmesg | tail -n 100

Monitor balenaEngine

beginning with version 2.9.0, balenaOS includes the lightweight container engine balenaEngine to manage Docker containers. If you think the supervisor or application container may be having problems, you’ll want to use balena for debugging.

From the host OS this command will show the status of all containers:

$ balena ps -a

You can also check the journalctl logs for messages related to the balenaEngine service:

$ journalctl --follow -n 100 -u balena

Note: For devices with balenaOS versions earlier than 2.9.0, you can replace balena in these commands with docker.

Inspect network settings

NetworkManager

NetworkManager includes a CLI that can be useful for debugging your ethernet and WiFi connections. The nmcli command, on its own, will show all interfaces and the connections they have. nmcli c provides a connection summary, showing all known connection files with the connected ones highlighted. nmcli d displays all network interfaces (devices).

Another useful place to look for NetworkManager information is in the journalctl logs:

$ journalctl -f -n 100 -u NetworkManager

ModemManager

Similar to NetworkManager, ModemManager includes a CLI, mmcli, to manage cellular connections. mmcli -L provides a list of available modems.

Look up version information

Knowing what version of a specific service is being run on your device can help you troubleshoot compatibility issues, known bugs, and supported features. Many services provide a direct option for displaying their version:

$ udevadm --version
$ systemctl --version
$ openssl version

Understand the file system

In some cases, you may need to examine the contents of certain directories or files directly. One location that is useful for troubleshooting purposes is the /data directory, which contains your device's Docker images, persistent application data, and host OS update logs. The boot directory includes configuration files, such as config.json, config.txt and NetworkManager connections.

Note that the filesystem layout may look slightly different from what you’d expect—for example, the two locations mentioned above are found at /mnt/data and /mnt/boot respectively.