how-tohardwareCI/CD

How to Build Local Hardware-in-the-Loop CI with Raspberry Pis for Fast Regression Testing

UUnknown

2026-02-18

11 min read

Build a fast, repeatable Raspberry Pi 5 hardware-in-the-loop CI cluster: networking, runners, orchestration, and teardown for reliable regression tests.

Stop slow, flaky regression cycles — wire Pi 5 hardware into CI for repeatable hardware‑in‑the‑loop (HIL) tests

If your team wastes hours waiting for slow or inconsistent hardware regression runs, or if production bugs slip past because staging doesn't exercise real peripherals, a local Raspberry Pi 5s based HIL CI cluster can change that. This tutorial walks you through building a resilient, fast, and secure Pi‑based CI fleet that integrates with common CI systems (GitHub Actions / GitLab CI), runs deterministic HIL jobs, and supports automated teardown to cut costs and avoid environment drift.

The value proposition in 2026 — why Pi 5 HIL CI matters now

By 2026, teams are moving more verification left into preprod and CI. Edge workloads, ARM-native builds, and hardware-accelerated ML at the edge are common; using cloud-only testbeds misses device-specific regressions. The Raspberry Pi 5 (improved CPU, I/O, and accessory ecosystem since 2023) hits the sweet spot: affordable, powerful enough for containerized runners, and widely available accessories (AI HATs, relay HATs, PoE hats) for realistic peripheral testing.

Key trends backing this approach:

Edge-first architectures and ARM-native CI pipelines are mainstream in enterprise dev teams (late‑2025 adopters report 2–3× faster feedback on device bugs).
Infrastructure automation matured — k3s, Ansible, and GitHub/GitLab self‑hosted runners are standard building blocks.
Cost and sustainability pressure — ephemeral test clusters that spin down deliver measurable cost reductions vs. always-on lab benches.

What you’ll build (high level)

A small cluster of Raspberry Pi 5 nodes acting as self-hosted CI runners.
Network isolation (VLAN + DHCP reservations) and service discovery for test orchestration.
Automated provisioning and teardown via Ansible + k3s for running isolated test workloads (Docker/OCI or Kubernetes pods) that can access hardware peripherals.
Power control and reset via networked relays or smart PDUs so jobs can reliably recover a hung device.

Prerequisites & bill of materials

3–8 x Raspberry Pi 5 boards (fewer for initial proof-of-concept)
Network switch that supports VLANs and PoE (optional PoE hat for Pis)
USB‑C PD power supplies or a high‑quality powered USB hub
MicroSD or NVMe boot media (recommended: write images via Raspberry Pi Imager)
Optional: relay HATs or networked power strip for remote power cycling
A laptop or admin PC for provisioning (Ansible control node)
GitHub or GitLab repository where CI jobs will run

Network layout and considerations

Keep your Pi HIL fleet segregated from production and developer VLANs:

Assign a dedicated VLAN (e.g., vlan 50) with its own DHCP scope and DNS entries.
Use DHCP reservations so each Pi has a stable hostname and IP.
Enable mDNS/avahi for easy discovery during development; rely on static DNS in CI production runs.
Restrict ingress: allow SSH from admin nets and CI controller IPs only.

Why this matters: hardware tests often require predictable device mapping (USB serial paths, I2C addresses). Unpredictable IPs or conflicting network services create non-deterministic failures.

Step 1 — Image the Pis and baseline OS hardening

Use the Raspberry Pi OS Lite or a minimal 64‑bit image. For reproducibility create a base image and version it in your infra repo.

Essential post‑flash steps (can be automated with Ansible):

Create a /etc/hosts entry for the controller and set a predictable hostname.
Disable unused services (bluetooth, GUI) to reduce jitter.
Set up a non-root user, disable password auth, and drop in an SSH authorized_keys file.
Install jq, docker or containerd, k3s (optional), and system monitoring agents.

# Example: Ansible task fragment to create pi user and harden SSH
- name: Ensure pi ci user
  user:
    name: cicd
    groups: docker
    shell: /bin/bash

- name: Install SSH key
  authorized_key:
    user: cicd
    key: "{{ lookup('file', 'files/id_rsa_ci.pub') }}"

- name: Disable password authentication
  lineinfile:
    path: /etc/ssh/sshd_config
    regexp: '^PasswordAuthentication'
    line: 'PasswordAuthentication no'

Step 2 — Deploy the runner (GitHub Actions example)

Decide whether to run the runner as a system service (container) or via Kubernetes (k3s). For small fleets, Dockerized self‑hosted runners are simplest.

Runner as a systemd unit (container runner)

Create a docker-compose or container run that mounts device nodes and required artifacts.

# systemd unit to run GitHub Actions runner container
[Unit]
Description=GitHub Actions Runner
After=docker.service

[Service]
Restart=always
User=cicd
WorkingDirectory=/home/cicd/actions-runner
ExecStart=/usr/bin/docker run --rm \
  --name gh-runner \
  -e RUNNER_NAME=%H \
  -e RUNNER_REPO_URL=https://github.com/org/repo \
  -e RUNNER_TOKEN={{ runner_token }} \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /dev:/dev \
  ghcr.io/myorg/runner-image:latest

[Install]
WantedBy=multi-user.target

Key points: mounting /dev exposes hardware (serial ports, USB devices) to jobs; mount the docker socket for nested container builds if needed. Use short-lived runner tokens and rotate them with automation.

Runner as Kubernetes pods

For scale and isolation, join the Pis into a small k3s cluster and run the runner as a DaemonSet or Deployment using the Kubernetes executor.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gitlab-runner
spec:
  selector:
    matchLabels:
      app: gitlab-runner
  template:
    metadata:
      labels:
        app: gitlab-runner
    spec:
      containers:
      - name: runner
        image: gitlab/gitlab-runner:latest
        volumeMounts:
        - mountPath: /dev
          name: dev
      volumes:
      - name: dev
        hostPath:
          path: /dev

Set nodeSelector or tolerations to pin hardware test pods to Pi nodes.

Step 3 — Expose hardware safely to CI jobs

Expose only the devices the test needs. Common patterns:

USB serial: bind specific /dev/ttyUSB* nodes into the container.
I2C/SPI/GPIO: use privileged containers with specific device mappings or a helper sidecar app that performs hardware operations and exposes a small RPC surface.
Network devices: isolate with separate bridge networks and firewall rules.

Example: Docker run for a test job that needs a USB serial device:

docker run --rm \
  --device=/dev/ttyUSB0:/dev/ttyUSB0 \
  -v $(pwd):/workspace \
  myorg/hil-test-image:latest /workspace/run_tests.sh

Prefer narrow device permissions and non-root runtime where possible. Consider using udev rules to create predictable device symlinks like /dev/my-sensor-01 for test scripts.

Step 4 — Orchestration and coordination

For HIL tests you need more than a runner: orchestration to claim devices, coordinate multi‑node tests, and handle retries. Options:

Use a simple allocator service — a small HTTP service that tracks device leases in Redis.
Use Kubernetes with CustomResources that represent devices (Device CRDs) and a controller to bind workloads.
Use GitHub/GitLab job labels and runner tags to schedule jobs to specific Pis.

Simple device allocator (pattern)

# Pseudocode: /acquire?device=serial-1
POST /acquire { device: "serial-1", ttl: 300 }
# returns claim id; job holds claim and periodically renews

This prevents parallel jobs from stepping on the same hardware. Implement exponential backoff and a finalizer that triggers a power-cycle if the device doesn't respond after a few attempts.

Step 5 — CI job examples and wiring

Here’s a minimal GitHub Actions workflow that targets a self-hosted runner labelled "pi-hil" and performs a hardware test against /dev/ttyUSB0.

name: HIL regression
on: [push]

jobs:
  hil-test:
    runs-on: [self-hosted, pi-hil]
    steps:
    - uses: actions/checkout@v4
    - name: Run hardware test
      run: |
        docker run --rm \
          --device=/dev/ttyUSB0:/dev/ttyUSB0 \
          -v ${{ github.workspace }}:/workspace \
          myorg/hil-test-image:latest /workspace/run_hw_tests.sh

Make the test script idempotent, verify device state, and store test artifacts (logs, traces) back to the CI artifacts store. If device access fails, surface clear diagnostics: serial dumps, last reboot timestamp, unit test logs.

Step 6 — Power control and reliable teardown

You need deterministic teardown for both reliability and cost control.

Automated teardown scenarios:
- Normal completion: runner marks device free; Ansible/allocator removes ephemeral files and optionally powers down node.
- Failed/locked device: trigger a remote power-cycle via relay HAT or smart PDU and re-run hardware health checks.
Ephemeral images: rebuild a fresh OS image on the Pi as part of a long-running cleanup (e.g., nightly) to eliminate drift.
Cost optimization: power down Pis between heavy test batches or at night; schedule wake windows with smart PDUs.

Example: Ansible task to power cycle a Pi via a networked PDU API:

- name: Power cycle pi via PDU
  uri:
    url: "http://pdu.local/api/outlets/3/power"
    method: POST
    body: '{"state": "cycle"}'
    body_format: json

If you use HAT relays attached to the Pi, be careful: you cannot reliably power cycle the board from itself when it hangs. Use an external networked relay or a PDU for robust resets.

Step 7 — Observability, debugging, and flaky test handling

Visibility wins HIL CI. Instrument everything.

Prometheus node_exporter on Pis for CPU, temp, load, and disk metrics.
Collect serial logs (ring buffer) per device and push artifacts on CI failures.
Tag runs with HW revision, HAT firmware version, and image SHA so regressions are traceable.
Introduce a flaky-test policy: automatic retries with exponential backoff, but record flaky incidents for triage.

Security and compliance

HIL setups are often outside the main security perimeter — treat them like production but tighter in some ways:

Use ephemeral runner tokens and rotate them automatically.
Restrict network egress and block unapproved outbound connections from test jobs.
Run non-root where possible; minimize device access privileges.
Record a chain of custody for test units when tests exercise sensitive workloads (encryption keys, credentials).

Advanced strategies and 2026 predictions

As device CI matures, expect these practices to become common:

Edge-simulated CI farm orchestration: Declarative device CRDs and controllers will be used to manage thousands of tiny boards, similar to how Kubernetes manages node pools.
ARM-native cross-compiles integrated into CI: With more ARM targets in production, CI pipelines will default to native testing on physical ARM devices like Pi 5 rather than emulation.
Synthetic peripherals and virtualization of sensors: Hybrid tests that combine physical and simulated sensors to increase coverage while reducing hardware bottlenecks.

Teams that invest in reproducible HIL CI gain faster regression feedback and fewer escapes to production. In enterprises, expect this pattern to be part of any robust preprod gating policy by 2027.

Troubleshooting common issues

Runner times out claiming a device: check allocator TTL, and verify DNS/mDNS resolution between runner and allocator.
Serial device paths change after reconnect: add udev rules to create persistent symlinks based on vendor/product IDs.
Job fails intermittently due to thermal throttling: monitor temperature and add forced cooldown or fewer concurrent tests per Pi.
Power-cycle doesn't recover hung Pi: ensure the reset is done by an external device; on‑board HATs may be insufficient.

Real-world example: 5‑node Pi 5 cluster for IoT product tests

Summary of a typical deployment we’ve seen:

Hardware: 5x Pi 5, 1x managed switch, external PDU, relay HAT for per‑sensor control.
Orchestration: Ansible for initial provisioning, k3s for service orchestration, and a small allocator web service backed by Redis.
CI: GitHub Actions self‑hosted runners on each Pi; runners tagged by model and HAT revision (pi5, hat-v2).
Result: regression feedback time dropped from 12 hours to under 45 minutes for complete hardware test-suite runs; cloud costs reduced by 60% by shutting down lab devices outside working hours.

"The biggest win wasn't raw speed — it was confidence. Running the exact peripheral stack caught a timing bug that emulation never showed."

Checklist: get to a production-grade Pi HIL CI fleet

Define device identity: hostnames, hardware labels, and device naming conventions.
Automate base image creation and version it in Git.
Deploy self‑hosted runners (container or k3s) and secure tokens.
Implement a device allocator and device-level health checks.
Add power control (external PDU), automated teardown, and nightly image resets.
Instrument metrics and logs; archive artifacts on CI failures.

Conclusion — start small, iterate fast

Raspberry Pi 5-based HIL CI is affordable and practical in 2026. Start with a 3-node PoC, automate everything (provisioning, runner lifecycle, and teardown), instrument the fleet, and iterate. The payoff is faster regression cycles, fewer production escapes, and measurable cost savings.

Actionable next steps:

Clone a starter repo with Ansible playbooks and runner containers (create one if you don't have it).
Provision 1–3 Pis and wire a single sensor/peripheral into a GitHub Actions job.
Implement an allocator and a power-cycle path before you scale.

Call to action

Ready to cut regression time and stop chasing environment drift? Start your Pi 5 HIL CI PoC this week: provision 3 Pis, automate their runners with the patterns above, and run a full hardware test in under an hour. For a jumpstart, clone our reference repo (Ansible + k3s manifests + allocator) at https://github.com/preprod-cloud/pi-hil-reference and share your results — we’ll help you optimize for scale and security.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.