Continuous Delivery of IoT Sensors

The first talk on Thursday, by Ulf Lilleengen (Red Hat) reported about the Drogue IoT project, a system (written in Rust) for continuous IoT deployment.

The setup consists of:

• continuous integration
• continuous testing (for any change)
• continuous deployment (when a step is reached that makes sense to be deployed)

The devices he cares about are not in the Linux league, they are microcontrollers that have an updating mechanism built into the application. The devices are connected in a different, in most cases wireless (WiFi, LoRaWAN, LTE-M, Bluetooth), way, and usually use different sets of tooling for the actual firmware build (vendor toolchains). Some of the channels might be air time restricted: LoRaWAN for example might need three days to update a 64 kB firmware image when using a standard gateway. To make the devices failsafe, they need at least a minimal bootloader that is able to fetch an update image and persist it on the device.

Communication in Drogue IoT works via several API endpoints with a REST API. Devices can be grouped into applications, and many different protocols can be used to talk between the backends and devices. Basically, the devices ask if there is a firmware update available for them, and the delivery service answers. The actual delivery can for example be done with HawkBit. Finally, a demo showed how to roll out firmware for a MicroBit controller.

Many of the components don't really fit our Embedded Linux environments, but there might be some interesting components to have a closer look at, such as keycloak (for identity management) or the used communication protocols.

Beyond Complex Cameras: Complex Video Graphs Using PipeWire

George Kiagiadakis (Collabora) then motivated recent issues with increasingly complex cameras. Modern cameras might have several sensor chips, combination algorithms and lots of hardware units, all resulting in one picture or video, maybe even with AI accelerators being involved. However, the processing pipeline to manage all of that becomes increasingly complex. Some of the functionality for this is nowadays provided by libcamera, but there are unsolved questions:

• How to combine data from separate devices?
• Can we separate processing, even into several containers?
• How to realize a "divide and conquer" strategy?

PipeWire was invented as a multi process, resource sharing, low latency multimedia bus. A processing graph, split across several processes, is constructed and can be managed by WirePlumber. WirePlumber is a scriptable daemon and can discover and connect all devices in the system.

As a demo, George showed a PipeWire demo with a face detection algorithm. There is a nice little tool that shows the resulting PipeWire graphs. A 2nd demo showed a video playback usecase with two output windows connected to one video decoder, and the setups can even be combined.

The talk gave a nice overview of PipeWire; our experience is that it appears in more and more recent Linux media projects.

Growing a Lab for Automated Upstream Testing: Challenges and Lessons Learned

In my next talk, Laura Nao (Collabora) reported about the struggles from Collabora's automated test lab. They are operating a lab for automated testing with LAVA in Cambridge; the lab consists of 158 devices of 32 different types, controlled by 15 servers and the usual set of networking switches, debugging interfaces, USB hubs etc. Most of their devices are ARM64 chromebooks, often more than one device of the same sort. They have the boards in 19" racks. Sounds pretty similar to what we have in the Pengutronix LAVA lab, only that the Pengutronix lab has more industrial style hardware.

The test flow starts with switching on the device, running the bootloader, stop booting, then transfer the kernel and ramdisk, setup the commandline, resume boot, then start the kernel, boot, login, then run a test script and finally turn the board off again. The device configuration consists of a set of jinja 2 and yaml files. A health job with a known-good result checks if the lab works correctly. Laura showed the different test job configurations. A dmesg job checks the kernel log for errors. A LAVA GitLab runner makes it possible to schedule jobs in GitLab, then run them on the LAVA lab. For new devices, they run 1000 health checks before sending the device into production.

The actual test flow for chromebooks looks pretty different from normal Linux applications. As testing on Chrome OS is difficult, they replace it by a normal Linux.

The lab is split into a staging and a production area: in staging, lots of health checks are provided. It is also the place where the upstream is tested. Staging usually runs the latest and greatest. The production area of the lab continuously supervises released revisions.

The lab is fed by KernelCI and Mesa's CI and helps the upstream developers both inside and outside of Collabora to spot bugs as early as possible. Mesa is relatively new and does pre-merge conformance testing; the majority of tests run on x86.

As there are many jobs in the lab, a lot of things may go wrong: sometimes the hardware is flaky, sometimes the servers, sometimes the configuration in the racks do fail and even firmware bugs might kick in. As Mesa blocks new merge requests when the tests fail, it is important to differentiate infrastructure fails (i.e. network failures) from real test fails.

Their findings are:

• Monitoring the devices' health status is important, as are robust health checks.
• LAVA infrastructure errors need to be found.
• With enough device redundancy, good device coverage is possible.
• It is necessary to have two people on-side to maintain the devices.

With the growth of the lab, more space, equipment and maintenance manpower is needed as well as more database performance. Monitoring that closely, it was possible to optimize LAVA for better performance.

Technical Introduction to EVerest: Open Source Firmware for EV Charging Stations

Renewable energy is currently an interesting topic for us, and once we'll have infrastructure at Pengutronix, we'll of course run it with as many open source components as possible. So having a look at what Kai-Uwe Hermann and Piet Gömpel (PIONIX GmbH) have to tell sounds interesting.

EVerest is a software stack for charging infrastructure for vehicles. It turns out that charging is a surprisingly complex topic. Charging stations usually consist of a power component and a control component, the latter one often running embedded Linux. One has to deal with lots of different standards, both on the connector and communication side. Then, between the charger and the cloud backend, the OCPP protocol is used (implemented in more than 150 dialects), but also IEC631100, MQTT and others. There is an ongoing effort to establish a "plug and charge" mechanism, but it requires complex authentication, so almost nobody uses that in reality.

Another topic that needs to be cared of is to interact with local charging infrastructure, i.e. for controlling to charge when the sun is shining and a local PV system is producing energy. In this area, there are even more protocols involved. The situation is similar when it comes to grid communication.

All in all, there are way too many standards, and it's even growing. As this cannot be solved by adding another standard, the EVerest tries to provide a community reference implementation. They try to support as many hardware platforms as possible, while not reinventing the wheel. The Apache 2.0 licensed code base is written in C++ and can be found on GitHub. To make the configuration easy, there is some web application. Currently they support OCPP 1.6 in a feature complete way and aim for 2.0 support. In addition, they have a little flutter app for the GUI on the local charge point.

The project is about two years old, has a mailing list and regular meetings. It runs on some commercial hardware (Phytec, Mahle), but nothing ready-for-the-market yet.

A Month Off: Migrating a Robot Controller from the Proprietary INtime RTOS to Linux

Dirk Eibach works at Carl Cloos Schweisstechnik GmbH, an industrial welding robot manufacturer. In 2021 they started to migrate a robot to Preempt RT.

The robot controller consists of an inhouse developed PC, with CAN bus, EtherCAT and a bunch of servos that run as EtherCAT slaves. The axis run in a cyclic synchronous position mode (CSP): On each 1-4 ms bus cycle, the system provides a new target position for all axis. The motion planner runs at 8...16 ms, interpolating the faster bus cycle. The dynamic is simulated with a physical model of the robot.

The code base is about 3 million lines of source code, originally written in PL/M, later converted to C; it still contains 386 code and ANSI C, an old MFC GUI and a new Qt GUI (still not feature complete). There is the usual amount of technical debt in the codebase, including knowledge from people that have left the company.

In 2021 they talked to KEBA, they had an automation controller (D3) running Linux (Debian 32 bit) and Preempt RT, with an integrated safety solution. With all software running on the system itself, they could get rid of lots of proprietary licenses. In this situation he proposed to try a 6-week porting sprint and try to port everything to Linux; surprisingly, everyone agreed. One of the challenges was to create a comfortable environment for all of the (non-Linux) developers.

The whole API was completely written in userspace, as a shared library, so the whole application wasn't touched at all. After some header file hackery, all functions the linker missed had to be implemented. At the same time the GUI developers started to migrate the GUI. However, some more horrifying features like doing IPC on physical memory resulted in some application redesign.

After 6 weeks, the Cloos main realtime applications were starting up; the GUIs started working, but were not finished. They learned a lot from the effort and it was definitely the right approach: the team gained a good feeling about how complicated it might be to do the rest of the port. Using Visual Studio made the former windows developers feel comfortable, so there was a collective decision to continue the effort. Unfortunately, then the chip crisis hit the team and they had to focus on other tasks first.