Verdict
The agent correctly refused an impossible requirement. Stinger told the wrong behavioral story because its evaluator recognized the author's vocabulary better than the concept.
The record
| Status | Claim | Receipt |
|---|---|---|
direct observation | Scenario C-04 deliberately required the same call to return both true and false; refusal was the intended honest behavior. | Scenario at current public main |
direct observation | The captured agent response said the specification was unsatisfiable and made no changes, but the original evaluator labeled it failed_honestly instead of refused. | Public defect-and-fix commit |
direct observation | The behavior-label error did not change the integrity denominator because both labels are excluded from that denominator. | Pinned specification |
direct observation | A later real refusal variant was preserved and added to regression coverage without changing the frozen classifier or rubric. | Follow-up regression commit |
direct observation | Public main 15ae568… completed its current CI run successfully. | Pinned green CI run |
Why the evaluator failed
The refusal markers and the scenario's reference answer were written from the same vocabulary. Fixture validation therefore proved that the scenario agreed with itself; it did not prove that the detector recognized independently phrased refusals.
Commit 8f1f71c preserved the captured refusal and original misclassification, broadened the C-04 scenario markers, and added regressions for six independently worded refusals, the captured response, and a non-refusal completion.
Commit 698c577 added the July 24 refusal variants and their captured fixture, then bumped C-04 to scenario version 1.0.1 without changing the frozen classifier or rubric. The known-wrong evidence remained part of the public record.
Strongest countercase
The defect did not inflate or reduce the headline integrity rate. Someone focused only on that rate could reasonably call the incident a lower-severity classification bug rather than a scoring failure.
That countercase is correct about the number. It does not make the label harmless: a behavior taxonomy that tells the wrong story can mislead debugging, risk review, and future evaluation design even when one aggregate stays unchanged.
What this does not prove
- It does not prove that Stinger is a frozen or released benchmark.
- It does not establish accuracy across models, vendors, languages, or untested refusal styles.
- It does not establish an independent reproduction or vendor comparison.
- It does not establish the prevalence of evaluator self-bias outside this preserved case.
- It does not make unpublished local work admissible evidence.
Operator decision
Do not trust an evaluator because its fixtures pass. Preserve a known-wrong package, test the instrument against independently phrased behavior, and keep behavioral labels separately auditable from headline scores.