12 August 2015 / Last updated: 05 Nov 2019

Support-driven Development

Execution time:
Update November 2019: We've grown a lot since this post was initially published in 2015, and as such our support tools and processes have changed and grown too. However, the idea of support driven development described below still holds true and continues to be practised at balena to this day.

Traditionally, support is treated as an afterthought, a thing you outsource, or perhaps something you deal with to get back to real work. Who hasn't been stuck waiting on the phone to speak to a representative of their favourite ISP? Even silicon valley companies such as Google almost take pride in doing as little of it as they can get away with. On the other hand, Steve Jobs responded to customers personally even as a supremely busy CEO. We take pride in joining other startups in realising there is gold in them hills. In the spirit of "customer development", not speaking to customers who are coming to you is basically insane. Having these conversations doesn't need you to even get out of the building! We like to think we've taken that approach even further, to its logical conclusion.
If you've used the magic questionmark (provided by the lovely intercom.io) at the bottom right of every resin.io dashboard page, or you've used our forum, you've more than likely spoken to a senior member of the team or a founder, likely within 10 minutes of asking your question. A lot of our support load is currently lifted by Lorenzo, a full stack developer if there ever was one, who's been at resin.io since day one and has also worked on things as diverse as the linux kernel, the chromium project, and built our original AngularJS UI. The fact that our team is distributed across the globe also means you'll likely get an initial answer fast, no matter the time.
The first thing we do is to focus on getting that user unblocked and back to working on their project as soon as possible. Support requests are linked in our internal chat, and whoever the relevant team member is gets pinged to help with the investigation. This helps team members connect with real people out there using their work, and also gets the users top-notch responses. Some times we're able to fix things on the fly, especially when it applies to our open source tools or providing some example code. At this point we've not done much more than a good job at supporting our users, but in many ways the fun has just begun.

Gold in them hills

We treat each support inquiry as a failure that should have been caught in several points prior to becoming a support issue, but also an opportunity for improvement. The word "failure" here should not be treated as a put-down or cause for concern. We believe that the best way to build a company that fails as little as possible is to have extremely well-trained failure-response mechanisms as part of its daily routine, but also digging deep for micro-failures before they become large. We know that users are unlikely to request support, so we consider each user that does ask most probably representative of several other users with the same question, who left with the wrong impression. So our priority at that stage is to make sure the next user with this issue doesn't reach the support portal at all.
To this effect, we ask ourselves the following questions:
  • Could the product itself have prevented the issue? This is the case where an outright bug or awkward abstraction has led to a very reasonable question. In this case we file an issue in JIRA, our development management tool.
  • Is this a matter of the correct way not being obvious? Could we have had UI/UX hints to point to the appropriate way to accomplish what the user wanted? This also warrants a JIRA ticket.
  • If not product or UX, could we have covered this in the documentation? How can we document this so the user would have been able to find this information? In case a documentation issue exists, we work to improve it. Our docs are open source on github, so we issue a pull request like any other user could. Having our docs as an open source project allows us to short-circuit this process, with users occasionally just fixing stuff directly.
  • Is this an interesting question? Some times the issue is not necessarily a failure, but is interesting in other ways. Could this become a blog post? Some times the user has a genuinely interesting issue, e.g. trying to connect a new sensor, deploying an interesting new library, etc. The answer to these issues is a useful draft for a potential blog post, and we aspire (but not always succeed) to produce blog posts every week coming from support conversations with users. The thing to be careful of here of course is that as the product changes, the "correct" way also changes, so these blog posts may have to be maintained in the future like we do all other documentation, but at least the fact that blogs are dated makes this issue a bit less severe.
  • Is this user asking for a new feature? Is this request common? Finally, users will often suggest a new feature that could be built into resin.io. If the feature is being built or is planned, which is very common, we add the user to the notification list and get back to them when the feature is deployed. Many users have extremely strong, positive surprise reactions when, on completion of a new feature, we let them know it's there. Some features are common requests, and oftentimes, the work needed to build them isn't much. Users have an unbiased way of looking at the product, which allows them to identify low hanging fruit that would make a big difference. These requests, especially if they make sense within our vision and roadmap, also get discussed internally and recorded in JIRA as to-do items. A closely related variation is where a user will ask a question or display a misunderstanding that could have been avoided with a new product feature. We think of these as "implicit feature requests".
  • Is this a common question that will inevitably end up getting asked again? While we try to figure out how to solve users' problems before they get to support, that process can take time. Interesting questions go into our internal support scratchpad, an informal wiki-reference to the support team. Adding things to the scratchpad is not necessarily the end of the process, but it helps to have a low-friction place to store knowledge before it becomes stale. It may end up triggering one of the above actions, but for now we have a simple answer while we wait for development to complete, or we need to wait for more users to come forward in order to better understand the solution that would make sense.

Will it scale?

A process this intense depends on having highly talented people executing it. The need for end-to-end product knowledge, technical chops, but also people skills and insight, means that this position is not something we take lightly. The way we've chosen to scale this in the future, is to have our senior engineers each be the point-person on support for a week at a time. With Lorenzo being there to train the person, help them out of tricky situations, and process the conversations after they're done. We're just starting the scaling process, and so far one thing we've seen is that it certainly helps build empathy for the support position. It turns out, it's easy to underestimate how much work goes into support, even for our own engineers.

Taking it Further

So a question that comes from users, especially one that comes often, is treated like a bug report on one or more of these layers. However, adding tickets to a long list may not be a convincing way forward to you (and it wasn't to us). We recently decided to go even further. The engineers doing support are now empowered to set one task per week to each of our development teams (devices, infrastructure, experience). In this way, tasks that are painful to users, but not necessarily "sexy" get prioritised and resolved, with a real justification behind them, in terms of the users that have already hit the problem. The requests get chosen by balancing the support burden that would be alleviated, the difficulty of the request, and, of course, the amount of user pain that would get cured. This process has already been extremely rewarding, in our case indicating that a lot of the user desired functionality is particularly related to a specific component (our device agent) that while operational, was clearly in need of more attention. As such, we greatly strengthened the team working on our next-generation agent, and used the list of user-requested features as one of the inputs in designing the new agent.
While we have our guiding vision for resin.io, devoting a portion of our development to incremental improvements driven by repeated user feedback seems obvious in retrospect. In a sense our vision can be seen as making long-range jumps to new kinds of experiences, with support offering the much needed hill-climbing improvement pathway. However it's not unheard of that a support conversation will provide fertile ground for new thoughts on our vision for resin.io itself, so even the grand vision is not immune from being altered through a humble support conversation.
So keep talking to us, and never feel embarassed in starting a support conversation. you not only help yourself, but other users that will follow, and you may even impact the future direction of the product. We work hard to make sure you have a great experience using resin.io, and it's all worth it when we get users unblocked and happy:
@resin_io have the best support I've ever witnessed. And they just rebuilt a tool just for me. Keep up the awesome work guys!
— Max (@schedin) May 13, 2015
Have questions, or just want to say hi? Find us on our community chat. Find us in the forums at https://forums.balena.io
by Alexandros MarinosFounder/CEO, balena

Share this post