#FlattenTheCurve – Introducing Our Remote Setup
Even Tux helps to flatten the curve
The Corona crisis is a challenge that has hit many people as well as most companies quite unexpectedly. The entire team of Pengutronix wants to thank those that currently ensure our essential supplies, health system and civil infrastructure!
But, there will also be a time after the crisis and people and economy will recover. Thus stopping our work was never an option, but we need to ensure that we operate in responsibility for our employees and our fellow people.
Two weeks ago, we introduced our strategy for protecting our staff, doing our part to flatten the COVID-19 curve and slow down the pandemic by switching to remote work.
Despite already having a few full-time remote workers at our company, most of our employees work at the main office. In normal times we are used to meeting at our coffee bar to chat, discuss recent topics and share ideas. So, being isolated is a new situation to most of us, too. However, extending our distributed infrastructure to serve all colleagues (and not only a handful remote workers) was managed in half a day and all colleagues were able to seamlessly continue working.
Luckily, we are able to automate most of the things that need to happen for embedded development, but for the few tasks that cannot be done remotely (like receiving parcels, switching cables on local hardware, etc.), there is always one colleague present at our local office.
Due to our normal workflow most of the required infrastructure was already in place, but we also had to extend it here and there to face the additional challenges of being all remote now.
We like to share recent experiences with our remote environment, that helps us to keep physical distance without social distancing:
Working Remotely on Software
For doing our everyday work, we have an extended server infrastructure to manage all our developing projects in Git repositories. Compiling code and building Board Support Packages needs lots of CPU power, so we have our ICECC compiling cluster and remote server access via SSH. We have been using this infrastructure for decades, but in this case it helped our colleagues to set up their working places at home in little to no time and continue their work as usual.
Remote Conversation
Chatting like always – IRC
IRC and e-mail still are our main communication tools.
In times of all kinds of new instant messengers flying around, IRC seems to be a bit dusty – you might think. But it perfectly fits our communication use case, with the advantage of minimal disturbances: You are not cluttered with unrelated pictures and this “messenger” is really running on our own server infrastructure. Also, most of our developers already follow various OSS projects' IRC channels, so IRC puts itself forward for this job.
Our communication via mailing lists is still unrestricted and working as usual.
Still Talking to Each Other – Mumble
While we have been using Mumble, an open source VoIP chat server, for a long time for talking to our remote-working colleagues, this voice chat has now become a coffee bar replacement. It also allows us to continue our biweekly stand-up meeting, as well as discussing topics in small groups by joining a conference channel.
The most important advice for your personal setup - no matter which VoIP-tool you are using: Configure your setting to usage of PushToTalk. Your colleagues will thank you.
Show and Tell – Jitsi
Sometimes talking to each other is not enough and it's actually helpful to share a screen, look at a piece of hardware together or just talk face to face. For these cases our admins set up a Jitsi server for video conferencing in the browser. From first experiences it works quite well, even in long-lasting conferences with a handful of clients and on a small bandwidth. We are planning to do some performance test in the next few days.
The fact that Jitsi doesn't need any clients suits our “no corporate desktop” policy well.
Working Remotely on Hardware
The main difference between Pengutronix and most other software companies is that we are working directly on hardware (e.g. for developing and testing kernel drivers). This means that our colleagues need access to custom boards.
What we can benefit from during this crisis is that fully working remotely actually is not that much different from what we usually do in our daily business already. After initial bring up, most of the customers' hardware we work on is installed in one of our several remote testing and development labs. This way, our developers can work with the hardware from their office desk and are able to automatically switch power, get a serial console, have network access, etc.
Our Remote Setup
We've worked a lot on both hardware and software tools that enable working remotely on hardware during the past years; and we'd like to give a short insight into our setup in order to possibly also inspire others:
Power Switch at a Remotelabs
Each remote lab consists of the following core components:
- a network controllable (Gude or netio) power switch
- a network controllable serial server (accessible over the RFC 2217 protocol)
- a network switch
- a small server
Some of them are also equipped with
- CAN infrastructure for IO switching
- USB hubs for controlling USB devices
Our open source hardware automation tool labgrid then brings these physical resources to the developer's desk.
Each remote lab then runs a labgrid exporter instance that automatically provides labgrid “resources” for the available remote hardware. A resource can be for example a power switch, a serial console, a USB device, or a CAN remote IO.
These resources can be grouped to places, which allows grouping resources with a similar purpose, and controlling them by a simple abstract place name like “my-devboard”.
Then each developer has a local instance of the labgrid client running that allows him to lock places, switch power, get a serial console, etc.
Finally, a central labgrid coordinator instance runs in a VM, coordinating access between the labgrid exporters and the clients.
This setup works both on local office desks as well as from home connected through SSH.
Helpful Tools
USB-SD-Mux
But, there are still things left that one can find hard to automate, like populating SD cards with a disk image and plugging it back into the device. For this purpose, our partner company Linux Automation GmbH brought up the USB-SD-Mux. This device consists of a small PCB with one end formed like a µSD card that you can put into your board's µSD slot instead of the physical µSD card. The physical µSD card is then put into the USB-SD-Mux instead. Connected over USB to a host computer it then allows us to switch the SD card between the host and the device under test, and re flash the SD card from the host like a standard USB mass storage.
We also have similar hardware in the pipeline or in the final test phase for Ethernet switching (like actually plugging cables in and out) and for USB switching (e.g. to simulate plugging of an USB stick). Stay tuned to find out about those :-)
We hope, you'll find our experiences helpful. Even though our remote setup works very well, we are looking forward to an end of this crisis and meeting each other in person again.
Further Readings
Managing Complexity with Open Source
A few days ago, something exciting happened: I revisited my very first embedded system - a 34 year old stepper motor controller, driving the telescope mount of the Public Observatory Rothwesten, which was built by me back when I was in class 12 in highschool. Comparing those embedded systems from back in the days with the recent industrial systems, it is impressive to see that the latter ones are not manageable any more without the use of open source software.
Pengutronix at FrOSCon 2024
On August 17th and 18th, 2024, it's that time again: FrOSCon will take place at the Bonn-Rhein-Sieg University of Applied Sciences in Sankt Augustin - and Pengutronix will be there again as a Partner.
umpf - Git on a New Level
Modern software development is commonly accompanied by a version control system like Git in order to have changes to the source code being documented in a traceable manner. Furthermore, any version state should be easily reproducible at any time. However, for work on complex projects such as the BSP ("Board Support Package") of an embedded system with its multiple development aspects, the simple stacking of the individual changes on top of each other does not scale.
Yes we CAN... add new features
Have you ever experienced an otherwise fine product that is missing just the one feature you need for your application?
DSA in Barebox
The v2022.05.0 Release of barebox introduced initial support for the Distributed Switch Architecture (DSA) Framework. DSA is originally a subsystem from the Linux Kernel, which exposes the individual ports of a network switch IC as virtual network interfaces.
First Steps using the candleLight
So you went and got yourself one of our fancy rocket-penguin branded CandleLight dongles or, being the die hard hacker you are, went and soldered one up in your toaster oven labeled "not food safe". What's next then? How do you use this thing? Let's answer these question by grabbing a Raspberry Pi and exploring some of the possibilities.
Fixing ICECC, our Workhorse for Building BSPs
While developing operating system infrastructure for industrial devices for our customers, we build lots of embedded Linux board support packages, kernels, bootloaders etc. at Pengutronix. Although ICECC should be a good tool to distribute the computing power to a cluster of machines, it recently turned out that things are not that simple.