Commissioning Bench

Overview

An independent gate that decides whether an industrial AI system is fit to approve, and notices when that stops being true.

Details

Project type: Assurance system for industrial AI
My role: Defined what the product commissions and how it decides, set the design system, and implemented the bench and its test library.
Category: Systems
Status: Prototype2026

What I did

Framed the unit under test as the whole configuration, not the model
Designed the commissioning cycle, from approval to detected change to reapproval
Wrote the test library of 56 cases across ten failure families
Set the industrial design system and implemented the interface, the backend and the evidence trail

Built with

Python: Backend, retrieval, validators and the worker
FastAPI: The API the interface and the tooling both talk to
PostgreSQL with pgvector: Documents, runs, findings and the retrieval index
TypeScript: Interface code
Next.js: The bilingual interface
Playwright: End-to-end tests over the whole demonstration flow

Next project

Virksom

The Commissioning Bench start screen: a Simulated badge and a synthetic-data banner above the system under test, a button to run twelve tests, and a red result reading that six of twelve tests failed with four critical

The need

An industrial company wants to put an AI assistant next to work that can hurt someone: isolation procedures, inspection deadlines, permits. Before anyone signs that off, they have to answer a question nobody has a good process for. Is this thing fit to be trusted here, and how would we know if that changed?

Ordinary evaluation does not answer it. It scores the model. But the model is only one part of what is actually deployed, and the parts around it move constantly: the instructions, the documents it reads, the retrieval settings, the tools it can use, its permissions and its guardrails. A configuration that passed last month may be a different system this month, and nothing tells anyone.

Decision

Commission the configuration, not the model

Context: Approving a model is approving the wrong thing. The behaviour that reaches a technician on a platform comes from the model plus its instructions, its corpus, its retrieval, its tools, its permissions, its guardrails, its environment and its application version.
What I decided: The whole configuration is taken as one controlled unit and reduced to a fingerprint: a deterministic hash over a canonical snapshot of all of it. An approval belongs to one fingerprint and to nothing else. When the running configuration drifts from the approved one, the change is classified as minor, major or critical, the approval is invalidated, and the suites it affects are named for rerun.
Trade-off: It is stricter and slower than approving a model once. Small operational changes can invalidate an approval and force a rerun. That is the cost of an approval that means something rather than an approval that has quietly expired.

Failure, described

A score alone tells an engineer nothing they can act on. So a failure is not a number here: it is a finding that states what the system did, what it should have done, and why the difference matters, with the passage it drew on quoted underneath and a fix proposed.

The library behind those findings is 56 cases across ten families of industrial failure: citing a superseded revision, inventing a deadline no source states, silently resolving a contradiction between two approved documents, answering where it should say it does not know, guessing which pump an ambiguous tag refers to, treating an unapproved vendor draft as authority, or following an instruction hidden inside a supplier PDF.

The Commissioning Bench findings list: each finding states the observed behaviour, quotes the source it came from and proposes a fix, labelled critical, above a note that the result is decision support and does not certify the agent

The test that matters is the second one

Two versions of the same maintenance assistant sit in the demonstration, and they differ only in configuration. Under ordinary questioning version 0.8 looks useful. Put through twelve industrial failure conditions it scores 47.7 and is blocked, with four critical findings. Version 1.0 scores 100 and reaches ready for human review.

The comparison then sets the two runs against each other case by case: six fixed, no regressions, the score up 52.3, four fewer critical findings. That view is the point of the whole product. A single pass is a moment of confidence. A comparison against the last approved version is the only way to see that a change which fixed one thing broke another.

The Commissioning Bench comparison view: version 0.8 blocked at a score of 47.7 with four critical findings beside version 1.0 ready for human review at 100, with six tests fixed, no regressions and a case-by-case table below

What it refuses to be

The product does not certify safety and does not approve operational use. It says so on the screen, in both languages, and a check run before any demonstration fails outright if the report has drifted into claiming otherwise.

That limit is the product's credibility. A bench that quietly implied it had certified something would be exactly the unaccountable thing it exists to guard against. It produces evidence; a named person makes the decision and has to write down why.

How it works

The configuration fingerprint is a deterministic SHA-256 over a canonical snapshot. The snapshot is kept alongside the hash, because a hash can tell you something changed but not what.
Every finding is traceable to a document version, a revision and a hash, so a failure can be reproduced rather than argued about.
The evidence package is a tamper-evident bundle: what was tested, against which configuration, the results, the findings, the decision and who made it. The assurance case is derived from the results rather than written by hand.
Three execution modes, never silently swapped. Simulated composes answers locally from the retrieved sources and still runs them through the real validators, so the bench works on a laptop with no API key and no spend.
A verification command run before any demonstration checks the dataset, the injection fixture, the superseded revisions, both agent versions, the replay snapshots, the audit chain, both language files, and that the report does not claim a certification. It exits non-zero if anything is missing.
The demonstration dataset is entirely synthetic, built around a fictional asset, so nothing proprietary is in the repository at all.
The palette is fixed and industrial, and status is never carried by colour alone: every state has an icon and a word beside it, which is an accessibility requirement and, in a product about accountability, a credibility one.

Where it stands

It runs end to end as a demonstration: the full cycle from test to approval, through a configuration change that is detected and classified critical, an invalidated approval, a targeted rerun, and a second approved baseline once the change is undone.

It is a working prototype on synthetic data, not a system in production use. What it establishes is a position rather than a feature: that the thing worth approving is the whole deployed configuration, that a failure has to be described before it can be fixed, and that the second run matters more than the first.