04 July 2022 / Last updated: 04 Jul 2022

Sunsetting vpn.balena-cloud.com and Introducing Cloudlink

We have introduced Cloudlink (cloudlink.balena-cloud.com) as a replacement for balena VPN (vpn.balena-cloud.com) on May 07, 2022. This is because balena VPN will no longer be usable on December 10, 2023 due to its CA Certificate expiring and additionally because we have found that there may be confusion with the name and its function.
Sunsetting vpn.balena-cloud.com and introducing Cloudlink
Read on to learn more.

How does this affect fleets and devices?

This shouldn’t change anything on the device, just the connection instance that the device is connected to. This means that OpenVPN is still used as an underlying technology to Cloudlink and components such as open-balena-vpn will still be around. Additionally, the majority of the users should not have experienced any problems with the deployment as devices have a mechanism to automatically transfer to the new connection (os-config).
However, as of our current investigation, there could be 2 causes that a device may have problems connecting to Cloudlink – (1) Devices with old balenaOS versions (<= 2.14.0) may not have os-config and (2) your firewall may be blocking cloudlink.balena-cloud.com and your devices can’t connect to this endpoint, resulting in a Heartbeat only state. If your devices are in this situation, please follow the guidelines below in detail.
On May 07, 2022, we introduced Cloudlink as a replacement to the balena VPN connection. This is the connection that provides features like SSH access, the public device URL, and the web terminal in balenaCloud. Although balena VPN (vpn.balena-cloud.com) is still up and running, we will be sunsetting this in the near future and will be using Cloudlink (cloudlink.balena-cloud.com) from then on. This is because of 2 main reasons:

1. Expiring CA Certificate

balena VPN(vpn.balena-cloud.com) will become unusable on December 10, 2023 because that will be when the CA Certificate will expire. The CA Certificate is required for any of our connection instances to validate the devices in our platform. When this expires, any device that is still connecting to that subdomain on the date of expiry will effectively be unable to use features like SSH access, Public device URLs and the web terminal in balenaCloud.
The primary purpose of this connection isn't to act as a VPN, but as a mechanism for connecting to the cloud. We still use OpenVPN as an underlying technology for Cloudlink to provide features like SSH access, Device Public URLs, and the web terminal. Components and services such as the open-balena-vpn or the openvpn.service are still present and unchanged on the device. Nevertheless, we have faced instances in the past where there may be confusion about expected functions that a VPN usually has because of the name "balena VPN" and domain vpn.balena-cloud.com. Hence, we have abstracted it from the name of the technology and we have chosen a name that reflects its main function. This also allows us the flexibility to change the underlying stack to provide us the same or even allow us more features in the future.
The change over from vpn.balena-cloud.com to cloudlink.balena-cloud.com isn't triggered by a host update but rather from a tool that runs on the device every 24hrs – os-config. If you want to check if your devices are on Cloudlink, you can look at the addresses of cloudlink.balena-cloud.com and check it against the connected address that your device’s openVPN service is on.
root@<uuid>:~# nslookup cloudlink.balena-cloud.com
Server:    127.0.0.2
Address 1: 127.0.0.2 a5fa2e0

Name:      cloudlink.balena-cloud.com
Address 1: 34.226.166.12 ec2-34-226-166-12.compute-1.amazonaws.com
Address 2: 52.7.228.224 ec2-52-7-228-224.compute-1.amazonaws.com
Address 3: 3.225.166.106 ec2-3-225-166-106.compute-1.amazonaws.com
Address 4: 2600:1f18:6600:7f01:dc24:54f2:d95f:abc0
Address 5: 2600:1f18:6600:7f02:bda3:7af0:e21:425b
Address 6: 2600:1f18:6600:7f00:b18f:40:ea2:57ca

root@<uuid>:~# journalctl --unit openvpn
Jun 23 14:39:02 a5fa2e0 openvpn[1833]: Thu Jun 23 14:39:02 2022 [vpn.balena-cloud.com] Peer Connection Initiated with [AF_INET]3.225.166.106:443

Who could be affected and what should be done?

The majority of our existing users shouldn't have experienced the change directly as we have mechanisms (os-config) that allow the smooth transition between connection instances. If your fleet has not experienced any issue with connectivity for a long period of time since May 07, 2022, this means that the mechanism has worked correctly and your devices are now connected to Cloudlink.

Old balenaOS versions

Some devices may still be using the old instance because of old balenaOS versions. This can be due to a number of factors but the most obvious one is that the device is on the balenaOS version <= 2.14.0. These versions won't have os-config which unfortunately means with a balenaOS version this old (or older), the update to the new connection portal (cloudlink) won't work out of the box. Your devices should not have experienced any downtime related to this because we are keeping our vpn.balena-cloud.com subdomain alive until the certificate expires December 10th, 2023. To prevent unusable connectivity in the future, your best recourse is to update your balenaOS version on the devices that are still with balenaOS version <= 2.14.0. If you notice that your device may be on the old instances and are above the version 2.14.0, we will be in contact with you to investigate and help with the situation. We have identified these rare cases and are looking to solve them before the certificate expires.

Experiencing “Heartbeat only” status on device

Additionally, if your devices are experiencing "Heartbeat only" statuses for extended periods of time, you should check if the local network is blocking or using a static access list (whitelist) that doesn't have cloudlink.balena-cloud.com. As suggested in our network requirements, a wildcard that includes all subdomains (*.balena-cloud.com) would solve it if this is the case you are experiencing.

Doublecheck network access lists

There may be vendors or local networks with problems with a wildcarded access list, so we would like to reiterate that balenaCloud is a multi-tenant SaaS built on top of other multi-tenant SaaS products, like AWS and DockerHub to name a few. Due to the nature of these types of multi-tenant cloud environments, and because we regularly add new features which require new endpoints to the platform, we are not able to provide an unchanging FQDN list for access listing purposes.
It would be possible today to perform a network capture of regular traffic in your device's networks, which would produce a list of the endpoints that your devices are using today, but we do not guarantee that this will remain static as your device's consumption of features and the platform itself shift and change over time. For this reason, we provide a list of a few specific domains, and the wildcard *.balena-cloud.com. Some customers solve these problems by placing balena devices in a DMZ, exempting their DHCP range from certain types of firewalling, or installing the root certificate for a MITM proxy within balenaOS to allow for traffic sniffing before it leaves the network, instead of FQDN access listing. BalenaOS also supports RedSocks for proxy configuration.

Feedback is welcome

Please sound off in the comments below (our Forums) if you have any questions or concerns.
by Jao MaloyProduct Builder and Fleet Reliability Engineer at balena