DevOps for Embedded Development: Bridging the Gap Between Software and Hardware

The world of embedded systems has traditionally been seen as distinct from mainstream software development, often characterized by manual processes, intricate hardware dependencies, and lengthy release cycles. However, as embedded systems become increasingly complex and connected, the principles of DevOps, which emphasize automation, collaboration, and continuous delivery, are proving to be not just beneficial but essential.

This article expands upon the concepts introduced in my online presentation for the Embedded Israel meetup, also available recorded.

Why DevOps for embedded?

The core promise of DevOps - faster feedback loops, automated pipelines, and increased reliability - is even more critical in embedded development. Unlike web or mobile applications, embedded systems directly interact with physical hardware, introducing unique complexities. Manual firmware flashing, plugging and unplugging cables, and delayed testing on physical boards can significantly slow down development.

By embracing DevOps, we can automate many of these historically manual steps. It means:

• Faster iteration: Build once, test often, and get feedback quickly, not just on code changes but actual device behavior.
• Early problem detection: Catch issues early in the development cycle, reducing the cost and effort of fixing them later.
• Increased reliability and scalability: Automating testing and deployment processes leads to more stable products and enables teams to handle larger, more complex projects.

Navigating the unique challenges

While the benefits are clear, embedded DevOps comes with its own set of challenges to address:

• Hardware-software integration: Code must work seamlessly with specific hardware, making simulation and automated testing more complex.
• Limited resources: Microcontrollers often have severe constraints on CPU, RAM, and storage, demanding optimized code and efficient toolchains.
• Real-time requirements: Many embedded systems have strict timing constraints, which continuous integration and deployment (CI/CD) setups must help verify.
• Safety and reliability: In critical industries like automotive or medical, rigorous testing and certification are paramount.
• Diverse hardware platforms: Teams often work with multiple hardware platforms, each with toolchains and debugging procedures.

Building the foundation: Toolchain and environment management

One of the first critical decisions in an embedded workflow is how to manage the build environment.

Native vs. virtualized builds

Native builds are straightforward for small projects, but can lead to "it works on my machine" problems due to environment drift. Virtualized builds, especially with Docker, provide a consistent and reproducible environment by locking in compiler versions, dependencies, and build scripts. This stability is crucial for CI/CD and collaborative development.

Cross-compilation and static linking

Since embedded devices often lack the resources for on-device compilation, cross-compilation is fundamental. We compile code on a more powerful host machine for the target embedded architecture. Many vendors provide official toolchains, such as Espressif's ESP-IDF for ESP32 or ARM compilers for STM32 via Buildroot or Yocto, making this process more streamlined.

Static linking is another common practice, where all necessary libraries are linked directly into the final binary. It simplifies deployment by eliminating missing library issues, but can result in larger binaries.

Emulation with QEMU

QEMU acts as a virtualizer, allowing developers to emulate embedded systems (like a Raspberry Pi or STM32) on their host machine. This is incredibly valuable for:

• Early development: Testing software before physical hardware is available.
• CI/CD integration: Running virtual test runs within the CI pipeline.

However, QEMU has limitations, especially regarding peripheral support (e.g., SPI, I2C sensors, custom GPIO). Despite this, it remains a powerful tool for accelerating early development and CI.

Docker: A game-changer for embedded development

Now, here’s my favourite part: Docker is a game-changer for all development these days, mainly because it solves some of the messiest problems we face — inconsistent toolchains, conflicting dependencies, and onboarding pain.

Let’s look at three ways it makes life easier:

• Reproducible toolchain environment: Docker images package the exact compiler, linker, and SDK versions, ensuring consistency across all development environments.
• Isolated cross-compilation targets: Teams can create isolated Docker images for different boards or architectures (e.g., STM32, ESP32), avoiding toolchain conflicts.
• CI/CD integration & faster onboarding: Docker images integrate seamlessly with CI/CD platforms like GitHub Actions, and new developers can get started quickly with a pre-configured environment.

CI/CD with GitHub Actions

GitHub Actions is an ideal choice for embedded projects due to its seamless integration with Git repositories and adaptability. Workflows, defined in YAML files within the .github/workflows directory, enable automated builds, cross-compilation, and testing. These workflows outline a series of automated steps, such as building, testing, or deploying code, triggered by events like pushes, pull requests, or scheduled intervals.

Key features for embedded workflows include:

• buildx: A Docker CLI plugin that supports building Docker images for multiple embedded architectures directly within the pipeline.
• QEMU runners: Enable GitHub Actions to emulate target CPUs, allowing for significant testing without physical hardware.

GitHub runners: Public vs. self-hosted

These workflows don’t run in isolation; they require runners, which are virtual machines or physical machines where the actual jobs execute.

There are two types of runners: GitHub hosted runners and self hosted runners.

Self-hosted runners are crucial when physical hardware interaction is required for testing. Scaling self-hosted runners involves setting up multiple instances to handle concurrent actions.

Scaling embedded CI: Labels & matrix builds

To enhance flexibility and scalability in CI/CD workflows, GitHub Actions provides advanced configuration features that allow teams to tailor their automation to complex hardware and software needs. Two handy features are:

• Runner labels: Self-hosted runners can be tagged with labels (e.g., arm64, lcd-test, usb-flash). Workflows can then target specific runners based on these labels, routing jobs to the appropriate hardware.

name: Flash Firmware

on:
  workflow_dispatch:

jobs:
  flash:
    runs-on: [self-hosted, arm64, usb-flash]
    steps:
      - uses: actions/checkout@v3

      - name: Flash firmware to device
        run: |
          ./scripts/flash.sh

• Matrix builds: This powerful feature allows for the parallel execution of multiple build variants. For example, a matrix can be defined to build firmware for different boards (e.g., STM32, ESP32) and modes (e.g., debug, release) simultaneously, accelerating testing cycles.

name: Build Firmware

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        board: [stm32, esp32]
        mode: [debug, release]
    steps:
      - uses: actions/checkout@v3

      - name: Build ${{ matrix.board }} firmware in ${{ matrix.mode }} mode
        run: |
          make BOARD=${{ matrix.board }} MODE=${{ matrix.mode }}

Optimizing your embedded CI pipeline

Optimizing your embedded CI pipeline improves build efficiency and reduces feedback time, enabling faster iteration and higher code quality. Focus on minimizing unnecessary steps, leveraging caching mechanisms, and parallelizing tasks where possible.

Additionally, ensure sensitive data like credentials and API keys are securely stored using environment variables or secrets management tools. Regularly audit your pipeline for vulnerabilities and apply security best practices to prevent potential breaches.

Caching

Embedded toolchains are often large. Caching build outputs and toolchains can significantly reduce build times by avoiding repeated downloads and compilations.

See below for some examples of caching using the action cache.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache ARM GCC Toolchain
        uses: actions/cache@v4
        with:
          path: ~/.toolchains/gcc-arm-none-eabi
          key: ${{ runner.os }}-arm-gcc-${{ hashFiles('toolchain-install-script.sh') }}
          restore-keys: |
            ${{ runner.os }}-arm-gcc-

      - name: Install Toolchain
        run: ./toolchain-install-script.sh

In this example, the script installs the ARM toolchain only if it's not found in the cache. The hashFiles ensures the cache invalidates if the script changes.

Dependency cleanup

Review pipelines for unnecessary dependencies and use flags like no-install-recommends during installation to keep containers smaller and build faster.

Secure pipelines

Treat sensitive information (signing keys, tokens, Wi-Fi passwords) as secrets. Use GitHub Actions secrets and permissions instead of hardcoding credentials. Implement code scanning actions to identify hidden keys, outdated libraries, and vulnerable GitHub Actions versions.

Example how to use secret key in your GitHub Action:

name: Secure Firmware Upload

on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  upload:
    runs-on: [self-hosted, usb-flash]

    steps:
      - uses: actions/checkout@v3

      - name: Decrypt firmware signing key
        run: |
          echo "${{ secrets.SIGNING_KEY }}" | gpg --batch --import
[...]

Finally, regarding security, it is advisable to implement GitHub Actions that scan your code for hidden keys, outdated libraries, and even outdated GitHub Actions versions that require updating.

Below is an example of how to use Codeql action for analyzing your code.

security-scan:
    name: Code & Dependency Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # GitHub's built-in code scanning with CodeQL
      - name: Initialize CodeQL
        uses: github/codeql-action/init@v3
        with:
          languages: cpp

      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@v3

Testing strategies: From unit to hardware

Testing embedded systems extends beyond simple code compilation. We typically categorize testing into three levels: unit, integration, and hardware-in-the-loop (HIL) tests.

Unit tests: 

These run on the host system, focusing on logic correctness. They are highly automatable and execute quickly.

Integration tests: 

These combine various modules and can be cross-compiled to run on emulators like QEMU or directly on target boards.

Hardware-in-the-Loop (HIL) tests:

This level involves running code on actual hardware. While HIL testing requires a more intricate setup, DevOps principles can automate key aspects such as flashing devices, executing tests, and capturing results. It often necessitates a physical connection between the target board and a self-hosted CI runner, typically via USB, UART, or GPIO. Additional components like relays, measurement tools, or even cameras (for display output testing) might be needed, depending on the test requirements.

Therefore, although DevOps tooling streamlines the testing workflow, a foundational infrastructure and cabling setup must always be established and maintained. Once in place, this groundwork enables powerful and repeatable testing on the very hardware your customers will utilize.

Infrastructure as Code (IaC)

Defining infrastructure (e.g., cloud resources, CI/CD environments) as code with tools like Terraform or Pulumi offers numerous advantages.

Using declarative tools ensures that infrastructure is consistently created similarly, eliminating human error inherent in manual cloud resource creation. This approach simplifies the setup process: instead of figuring out how to configure cloud resources, you simply declare them and provide necessary parameters such as names, address ranges, or sizes.

Setting up an environment with Terraform to describe your ideal infrastructure might take time, but subsequent environments will be created much faster. Additionally, implementing changes across all Infrastructure as Code (IaC) described environments will only necessitate a single module alteration and the application of Terraform code.

Summary

This article has covered much about DevOps for Embedded Development, yet we've only scratched the surface. It's a deep subject, encompassing software, hardware, infrastructure, testing, deployment, and security. We've laid the groundwork, demonstrating how tools like Docker, QEMU, GitHub Actions, and Terraform can introduce workflow structure and automation. It isn't just about accelerating development but about enhancing repeatability, scalability, and reliability, even when real hardware is involved.

Implementing these practices demands time and discipline, but the long-term benefits offer a significant advantage for your team, product quality, and customers. We've seen the potential of DevOps; now, it's time for action. You don't need to overhaul your entire process immediately. Start small - perhaps by Dockerizing your toolchain or setting up a simple GitHub workflow. Prioritize repeatability and feedback, and integrate hardware into your pipeline from the outset, rather than treating it as an afterthought.

Crucially, don't undertake this journey alone. Share your setups, build internal documentation, and contribute to tooling. The more accessible we make embedded DevOps, the stronger our teams and products will become. It is no longer a luxury; it's a competitive imperative.