Reliability evals
Deterministic regression cases that verify Relay chooses the expected safe outcome under each failure mode.
Latest suite: 12/12 passed
These failure semantics are regression-tested: retry when absence is proven, do not retry when the effect already exists, replan on stale state, stop when the outcome cannot be proven, require review for high-risk actions, and prevent duplicate effects.
12 deterministic casesFailure modes run through the coordinator.
Expected outcome checkedEach case asserts the safe recovery behavior.
Persisted evidenceEval results are saved for inspection.
Started 9/7/2026, 5:07:11 AM · readiness READY_FOR_DEMO
| Case | Result | Checks passed | Failure |
|---|---|---|---|
| E01_HAPPY_PATH | PASS | 5/5 | None |
| E02_TIMEOUT_BEFORE_WRITE | PASS | 4/4 | None |
| E03_TIMEOUT_AFTER_WRITE | PASS | 3/3 | None |
| E04_PARTIAL_COMPLETION | PASS | 4/4 | None |
| E05_STALE_VERSION | PASS | 3/3 | None |
| E06_DUPLICATE_SEMANTIC_REQUEST | PASS | 3/3 | None |
| E07_HIGH_RISK_REVIEW | PASS | 4/4 | None |
| E08_REVIEW_INVALIDATED | PASS | 3/3 | None |
| E09_MALFORMED_RESPONSE_AFTER_WRITE | PASS | 3/3 | None |
| E10_AMBIGUOUS_WRITE_AND_READBACK_FAILURE | PASS | 3/3 | None |
| E11_RETRY_EXHAUSTION | PASS | 3/3 | None |
| E12_ILLEGAL_TRANSITION | PASS | 3/3 | None |