Building Governance-First Physical AI

Most AI safety work targets the model. Physical AI asks for more: invariants that hold at the protocol layer and at the end of a wire, enforced before the actuator moves. This demonstrator makes four claims, and each one is checked by reading code or wiring rather than a statement of intent.

The full source is available at github.com/thierrysays/governed-edge-ai: code under Apache 2.0, documentation under CC BY 4.0. Hardware design files will be published under CERN OHL-P v2 when there are any; none exist yet.

docs/architecture.md names these four constraints invariants A to D and says what enforces each: the actuation co-processor enforces A by rejecting any frame without an audit reference, the bistable relay enforces B and is the only one of the four that holds with every processor switched off, and the arbiter enforces C by retaining digests off-host and D by exposing no inbound message that releases its latch. All four are structural, not procedural.

＃Five boards, one job each

The organising rule fits in one sentence: no board both decides and enforces. That is a property you check by looking at the wiring, not by reading a policy.

The field of view sets the safety envelope

Two Arducam IMX219 8 MP, 62.2° horizontal field of view each. The decided arrangement puts 52° between the optical axes, 26° either side of the centreline: about 114° of coverage and about 10° of overlap in the middle. Some earlier material rounds this to "roughly 120°"; 114 is the figure the decision was actually taken on.

The overlap serves a precise function. A mounting tolerance of a couple of degrees per camera would turn a seam specified as exactly closed into a real gap directly ahead of the robot, the worst possible place for one. It also pays for itself as a free self-test: an object in the shared wedge should be seen by both channels, so a camera that has failed, been knocked or lost focus shows up as a disagreement rather than as silence. The splay angle is therefore a calibrated, documented parameter, because the safety envelope is defined by it: a mount that shifts is a control that drifts silently.

A consequence worth writing down plainly: about 246° of blind sector remains, behind and to the sides, which the audit log cannot distinguish from an empty one. That belongs in the threat model, in those words.

A naming note: the co-processor simulator is still called MockSTM32H5, inherited from the VENTUNO Q STM32H5 the demonstrator started on. The rt-control/ directory remains an empty specification: arbitration was assigned permanently to the UNO R4, a separate board with its own power, rather than to a core sharing a circuit board with the host it is meant to arbitrate over.

ALog before act

The audit reference is a SQLite row id obtained before any command frame is transmitted. The structure prevents the alternative: the send sits inside the block that follows the log call, and the robot's own firmware rejects any command that does not carry a reference.

            # Governance invariant: log-before-act.
            # audit_ref must be non-zero; log_event() guarantees this by
            # returning the SQLite rowid (>= 1).
            audit_ref = self._logger.log_event(AuditEvent(
                session_id=self._session_id,
                actor="ai",
                detection_type=detection.detection_type,
                detection_label=detection.label,
                confidence=detection.confidence,
                command=action_type.name,
                command_sent=should_send,
                stm32_ack=None,
                notes=override_note,
            ))

            # Witness-before-act: the oversight node holds a digest covering
            # this row before the command frame is written.
            self._witness(audit_ref, command_sent=should_send)

            if should_send:
                command_sent_this_frame = True
                ack = self._send_command(detection, action_type, action_param, audit_ref)
                if ack is not None:
                    self._logger.update_stm32_ack(audit_ref, ack)

The interesting case is the failed write, which the nominal path leaves in the dark. The row already claims command_sent=1, and the journal is append-only: the record cannot be withdrawn. The filter flags the event for human review instead, which is the only honest way to record that a frame was composed and never reached the wire.

The same rule, reapplied at the actuation boundary

The governed body executes a motor command only after four gates, in this exact order. That is what separates the demonstrator from a software policy: a patch to the governance board that bypassed the journal would produce frames with no audit reference, which the body would refuse to execute.

def _handle(msg, alvik, stdout, kill_switch_active: bool) -> None:
    # Governance gate 1: audit_ref must be non-zero (log-before-act)
    if msg.audit_ref == 0:
        stdout.write(encode_reject(msg.audit_ref, REJ_AUDIT_REF_ZERO))
        stdout.flush()
        return

    # Governance gate 2: kill-switch overrides all commands
    if kill_switch_active:
        stdout.write(encode_reject(msg.audit_ref, REJ_KILL_SWITCH_ACTIVE))
        stdout.flush()
        return

    # Governance gate 3: float32 confidence threshold (defence-in-depth)
    if msg.confidence < CONFIDENCE_THRESHOLD:
        stdout.write(encode_reject(msg.audit_ref, REJ_CONFIDENCE_BELOW_THRESHOLD))
        stdout.flush()
        return

    # Governance gate 4: unknown action type
    action_fn = MOTOR_MAP.get(msg.action_type)
    if action_fn is None:
        stdout.write(encode_reject(msg.audit_ref, REJ_UNKNOWN_ACTION))
        stdout.flush()
        return

The firmware qualifies gate 2 itself: KILL_PIN is a local test input and is not a governance control. It was briefly driven by the arbiter, an arrangement wrong on two counts, and that mistake is what produced invariant B.

Default command map · conservative by construction: any unmapped label triggers HALT

CWitness before act

A SHA-256 chain head reaches a board the decision host does not control before the command frame is written. That ordering is what makes the retained digests evidence rather than a log of a log.

    digest_0 = GENESIS                                  (32 zero bytes)
    digest_n = SHA-256( digest_{n-1} || canonical(row_n) )

The row is read back from SQLite rather than reconstructed from the in-memory object: the chain must commit to what the database holds, not to what the process believes it wrote. The UNO R4 retains 64 digests in its own memory, 2.5 KB of the RA4M1's 32 KB SRAM. Recomputing the chain from the database and comparing it against the retained digests detects any retroactive edit, deletion or reordering, because the first altered row changes every digest after it.

The direction of authority is written into the protocol

Everything the VENTUNO Q sends on this link is a report. Everything the UNO R4 sends back is an instruction. The governance tier cannot tell its supervisor to stand down, and no message type would let it: the protocol contains no OVERRIDE_DENY.

A limit worth stating rather than hiding: the filter does not block on the ATTEST_ACK before transmitting. A chain-break verdict stops the next command, not the one in flight. Blocking on the ack would put the oversight link inside the actuation latency budget, which is the wrong trade for a link whose job is to observe.

BEnforcement outlives its enforcer

The stop is a bistable relay contact in the motor supply. It holds position with no current at all, so cutting power to the board that opened it does not restore motor power. And it needs no cooperation from the robot, because the robot has no pin to read. It is the only one of the four invariants that holds with every processor in the system switched off.

This is the newest claim and the only one with a story attached, and the story is worth telling: the project's credibility rests on it more than on the feature list.

What 611 passing tests did not see

The physical stop used to be a signal wire from the arbiter into a pin on the robot. It had two faults. It would release when the arbiter lost power: a driven line goes low when its board dies, so a power failure at the supervisor would have un-isolated the motors. A safety control that stops enforcing the moment its own board dies protects nothing. And it needed the governed component's cooperation: it worked only because the robot's firmware chose to read that pin. Reflash the robot and the control evaporates. That was a governance module bolted onto the thing it is meant to govern, which breaks the project's own design rule.

Both faults were found by reasoning about the design rather than by watching it fail: no board has ever been energised. That distinction matters, and the other version would make the better story.

Neither appeared in 611 passing tests. The test doubles modelled a state machine: there was no power to lose. Coverage does not find a fault whose failure mode the model has no vocabulary for.

    def test_open_contact_survives_a_power_cycle(self, latch, sim):
        """The regression that motivated the whole redesign.

        The GPIO kill line released when its board lost power, so a power cut
        at the oversight node un-isolated the motors. A bistable contact holds
        its position with no coil current at all.
        """
        latch.enforce_halt()
        assert sim.contact is LatchState.OPEN
        sim.power_cycle()
        assert sim.contact is LatchState.OPEN
        assert latch.poll().observed is LatchState.OPEN

A second fault, found while writing the deployment instructions

The arbiter read the contact back on a single sense line, and the question "what does that line read when its wire is cut" had the answer "open", which the arbiter reports as the motors are isolated. One physical position became indistinguishable from a fault, and it was precisely the position this module exists to refuse to assert lightly.

The observation is now antivalent: two opto-isolated channels wired to disagree with each other. Any fault reads as "cannot see", and nothing rounds that up to "isolated". A fault in the observation costs availability and never safety.

class LatchState(IntEnum):
    """Contact position, named for what it does to the motors.

    The contact is normally open and wired in series with the motor supply,
    so OPEN is the safe state and the power-on state of an unlatched relay.
    """

    OPEN = 0      # contact open, motor supply cut, HALT enforced
    CLOSED = 1    # contact closed, motor supply available
    UNKNOWN = 2   # no reading yet, or the sense line is unreadable


    @property
    def enforcing(self) -> bool:
        """True when the contact is observed open: motor supply actually cut.

        Deliberately false when the observation is UNKNOWN. An unreadable
        sense line is not evidence that the motors are isolated.
        """
        return self.observed is LatchState.OPEN

DOversight its subject cannot revoke

The override latches at the arbiter, and no message on any link releases it: the protocol contains none. Releasing an override is a physical act at that board. The governance tier may ask for the contact to be opened, which is always honoured; it may ask for it to be closed, which is refused outright while an override stands.

A claim like this one is worth only the attempt made to break it. The test suite throws the entire outbound vocabulary at a latched node, including the R4's own messages replayed against it, and checks that neither the override nor the motor power cut moves.

    def test_no_message_type_clears_an_override(self, node, channel):
        """The protocol has no OVERRIDE_DENY, and nothing the governance tier
        can send acts as one. Throw the whole outbound vocabulary at it."""
        node.press_button()
        node.release_button()
        assert node.override_active

        for frame in (
            encode(SupervisorHeartbeat(1, SystemState.ARMED, 1, 0)),
            encode(AttestDigest(1, b"\x01" * 32)),
            encode(OverrideClear(timestamp_us=1)),      # the R4's own message,
            encode(OverrideAssert(1, OverrideReason.REMOTE_CONSOLE)),  # replayed
        ):
            channel.write(frame)
        time.sleep(0.3)

        assert node.override_active is True
        assert node.motor_power_cut is True

The limit is real and documented: this link is worth what the cable is worth. An attacker who can write to it forges an OVERRIDE_CLEAR and releases the soft veto, and a test establishes exactly that. The relay contact is reachable by no message at all. That is why there are two paths rather than one, and a control whose limits are undocumented is a control nobody can rely on.

≡Eleven steps shipped, six designed

The first eleven steps are in the repository and tested. The next six are specified and ordered by dependency, but not written. Telling them apart is part of the demonstration.

✓What is measured

737 tests across two modules, 100% line coverage on both, with the build gate set at 98%. No hardware is needed to run the suite.

The doubles are real implementations of the state machines, driven over pseudo-terminals, where a stub would return canned values. The path exercised in CI is the one that would run on a rig. That is a stronger claim than mocking allows, and a weaker one than having run it.

Adversarial security tests. They attack the design from four positions: a compromised governance host, an attacker on the oversight cable, a compromised host with database write access, and hostile input on either link. They assert what holds and what does not.

Firmware parity tests. The R4's C++ state machine is compiled for the host with -Wall -Wextra -Werror and checked against the Python reference model: byte-identical frames, identical verdict sequences, identical state transitions, identical constants. Two implementations of one state machine drift unless something checks them.

What the security audit actually caught

A pass deeper than the routine quality chain: bandit at every severity rather than medium and above, a secret scan, a sweep for dangerous calls, a review of the SQL and network surfaces, and a bounds check on the C++ added at step 11. Three defects, none of them in the parts of the system that get written about.

A failed motor call was acknowledged as a success. The governed board caught any exception from the motor driver and replied CommandAck anyway, so the journal recorded stm32_ack = 1, which means accepted and executed, for motion that never happened. It is the exact shape of the error this project exists to refuse: the component asked to act reporting that it acted. It now answers SYSTEM_FAULT, and the attempt is still on record because the row was written before the command was sent.

    try:
        action_fn(alvik, msg.action_param)
    except Exception:
        # Caught rather than propagated: a motor driver fault must not take
        # the governance loop down with it. The board stays responsive and
        # says what happened.
        stdout.write(encode_reject(msg.audit_ref, REJ_SYSTEM_FAULT))
        stdout.flush()
        return

    stdout.write(encode_ack(msg.audit_ref))

Why it survived. alvik-firmware/main.py had no tests at all: an unconditional import of the Arduino library made it unimportable under CPython. The four governance gates on the governed board were therefore the least tested code in the repository while being among the most safety-relevant. The import is now optional and the file has thirteen tests.

The TCP listener had no length cap. It binds 0.0.0.0:9100 on the board holding the audit journal, and read a four-byte length prefix from an unauthenticated peer before anything else. A peer sending 0xFFFFFFFF made it accumulate four gibibytes. A malformed message also escaped the accept loop and ended the listener. This is the third instance of the same bug class in this repository. The rule is worth writing down rather than fixing a fourth time: a length that arrives before its data is a promise, and a promise from an unauthenticated peer needs a bound.

Two protocol messages were in the specification and not in the port. LATCH_REQUEST and LATCH_REPORT existed in the Python model and not in the firmware, so the arbiter could neither report the contact position nor accept a request to move it. The harness gap is the more instructive part: the parity suite compared only the messages both sides implemented, so a message the firmware never got was invisible and the suite stayed green. It now enumerates the model's types and fails on any the firmware lacks. A harness that only checks what exists cannot find what is missing.

What the audit did not find, recorded so the next pass knows what has been looked at: no secrets in the tree, no eval, exec, pickle, os.system, shell=True or yaml.load, no SQL built by interpolation, no bandit findings at any severity after the fix, and no known CVEs in dependencies. One observation rather than a finding: requirements specify a minimum version rather than pinning it, so a future release of a dependency can change behaviour without a commit here.

The full quality chain runs in CI with no hardware: ruff, mypy in strict mode, bandit and pip-audit, all clean. The .github/workflows/qa.yml workflow runs the repository's own make qa target on every pull request and every push to main, so the rule no longer depends on what a contributor remembers to run locally.

!What this project does not solve

These limits belong in any serious write-up of the demonstrator. Omitting them would contradict the argument the project makes.

▸Running it yourself

With no hardware at all, in about twenty minutes.

git clone https://github.com/thierrysays/governed-edge-ai.git
cd governed-edge-ai
python3 -m venv .venv && source .venv/bin/activate
pip install -r linux-stack/requirements.txt -r audit-service/requirements.txt
make qa

Then the whole stack against the mock peers:

cd linux-stack
PYTHONPATH=../audit-service python3 -m governance.ventuno_q_service \
    --alvik mock --supervisor mock --db /tmp/audit.db

Hardware-free mode is an explicit flag, never a silent fallback: --source synthetic, --alvik mock, --supervisor mock. The distinction matters, because a system that switched silently to synthetic data would keep producing plausible detections while real perception was down.

With the boards on the bench, docs/deployment-guide.md takes it from an unconfigured machine to a verified rig: bill of materials, flashing both firmwares, wiring the relay into the motor supply, systemd units, and a nine-test procedure for checking that the governance controls actually work. It assumes no prior embedded experience.