All guides

How-to guides · 5 min

Read a compatibility report

A compatibility report is the result of one test run of a profile version. It tells you whether the version passed, what exactly was tested, what blocked the result and what evidence it rests on.

Where to find the report

On the profile page, the “Validation and deterministic tests” panel lists the runs. Open the one you need. A validated version also has a “Report that validated this version” button at the top.

While a run is QUEUED or RUNNING there is no report yet. The report appears once the state is PASSED, FAILED, ERROR or CANCELLED. After that it never changes.

How to read it, in order

  1. Verdict

    At the top are the run state, “Lab readiness” (VALIDATED or NOT_READY) and the profile with its version. VALIDATED means the version can be bound to a source in the Lab.

  2. What was actually tested

    “Firmware actually tested” is the simulated reference firmware 1.0.0 and the fixture: CANONICAL_REFERENCE for the simulator, IN_PROCESS_MOCK for MQTT and Home Assistant. “Declared range” is what you declared in the profile.

  3. Stages

    SCHEMA → SEMANTIC → SAFETY → EXECUTION. Each stage has its own result. The first one that failed usually explains the rest.

  4. Blockers

    The list of codes that stopped the version from passing. Fix them in a new or edited draft and start a new test.

  5. Compatibility matrix

    Expand “Compatibility matrix: PASS / FAIL / UNKNOWN”. The first row is the result for firmware 1.0.0. The second is your declared range, always UNKNOWN: other firmware and physical devices were not tested.

  6. Individual tests

    In “Test results and evidence” each test has a name, a state and a failure reason. Expand “Inputs, expected and observed result” and “Test evidence timeline”. The filter shows only PASS, FAIL, ERROR or “Not applicable”.

Test states

StateWhat it means
PASSExpected and observed matched.
FAILThe simulated device did not behave as the profile or safety policy requires.
ERRORThe test could not run, for example BOUNDED_FIXTURE_EXECUTION_FAILED. This is not a PASS.
NOT_APPLICABLEThe check does not apply to this type, for example SOC tests for a load. Neither PASS nor failure; the reason is in the evidence.

Hashes and export

The Definition, Report, Runner and Simulator SHA256 let anyone check that the report belongs to exactly this definition and this version of the test code. Binding a profile needs a PASS report with the current runner and simulator hashes.

The “Export JSON” and “Export Markdown” buttons download the report for your team. Export is available only to workspace members and is audited.

A report is simulated Lab evidence. It grants no production eligibility, market participation or physical device qualification.

Common blockers

BlockerWhat to do
restore_after_ttlThe safety policy turns off restoration after TTL. Turn it on; without it the restoration test fails too.
UNSAFE_RECOVERY_POLICYCheck restore_power_kw, replay_completed and the stale/offline policies (they must be REJECT).
SAFETY_TELEMETRY_MISSINGRequired telemetry fields are missing. Add them to telemetry.
REFERENCE_FIRMWARE_OUTSIDE_DECLARED_RANGEThe declared range does not include 1.0.0, so the reference test proves nothing. Widen the range.
ASSET_TYPE_PACK_NOT_QUALIFIEDThere is no test pack yet for EV_CHARGER and GENERIC_GENERATOR.
REQUESTER_AUTHORITY_REVOKEDThe person who started the test no longer has an Owner, Admin or Developer role. Start the test again as a member with one.
PROFILE_OR_RUNNER_PROVENANCE_CHANGEDThe definition or test code changed during the run. Start a new test.
RUNNER_RESTART_INCOMPLETE_EVIDENCEThe worker restarted mid-run. The evidence is incomplete; start a new test.

A test name in the blocker list means that test did not pass. Find it in the results and compare expected with observed.