The frozen result, including every missed case
The held-out partition contains 68 parent tracks: 36 generated tracks and 32 human-reference tracks, each with clean, A and B processing variants. That makes 108 AI files and 96 human files. These were first scored after committing the selected model and policy hashes.
| System | AI / inconclusive / false human (108) | Human / inconclusive / false AI (96) |
|---|---|---|
| v2.2 serving policy on these short files | 0 / 15 / 93 | 89 / 7 / 0 |
| Frozen experimental addition | 47 / 4 / 57 | 88 / 7 / 1 |
The existing CNN requires at least three complete ten-second windows, so it cannot contribute in this eight-second challenge. The baseline numbers above must not be applied to longer uploads. The experimental head uses the same short audio and adds a positive AI result only at its fixed 0.9 threshold; otherwise it retains the baseline result.
The error was the A-chain transformation of Seclorance's Ohtli (FMA-140344), a human-reference recording. Its candidate score was 0.9153. That is a false positive, not evidence that the artist used AI. We did not raise the threshold after seeing this error or rerun the test as if it were still unseen.
Results by generator version
Each row is four held-out parent tracks with three variants each. Versions can share an architecture; nine versions do not mean nine independent model families.
| Generator | AI | Inconclusive | False human |
|---|---|---|---|
| AudioLDM 2 Large | 4 | 0 | 8 |
| MusicGen Large | 7 | 0 | 5 |
| MusicGen Medium | 7 | 3 | 2 |
| MusicGen Small | 4 | 0 | 8 |
| Mustango | 5 | 0 | 7 |
| Riffusion | 5 | 0 | 7 |
| Stable Audio v1 | 5 | 1 | 6 |
| Stable Audio v2 | 8 | 0 | 4 |
| Udio | 2 | 0 | 10 |
What changed in training
Our earlier experiments relied heavily on one human author. This round acquired 192 human-reference works from 192 FMA artist IDs, covering eight genres, plus 216 AIME tracks from nine generator versions. The 408 sources became 1,224 files after matching duration and codec processing across both classes.
Identities and splits were committed before scoring: 272 new development parents, 68 calibration parents and 68 frozen-test parents. Human artists and parent tracks do not cross those partitions. The final fit combines permitted old development material with the new development partition; the five original transcode targets and the prior human C challenge remain excluded from fitting.
Human references use explicitly listed CC BY or CC0 licenses in the FMA archive. Generated audio comes from AIME, pinned to revision b84d4be5. We did not extract its MTG-Jamendo recordings or description tags. Audio is not redistributed here; the evidence includes source identifiers, attribution records, licenses and hashes.
The failed designs are part of the result
- Directly replacing the decision with either new head increased AI detections but reduced correct human decisions from 90/96 to 20/96 or 4/96 on calibration. We rejected that loss of usability.
- A positive-only addition retained 90/96 human decisions on calibration, but caused six human false positives on the old regression set. It was rejected before opening the frozen set.
- A fixed-parameter joint fit restored old regression behavior and passed calibration with 35/108 AI detections, 90/96 correct human decisions and no observed human false positives. Only then did we freeze it and open the held-out set. The single human error above rejected this candidate too.
The second and third designs were made after seeing development/calibration failures. They were not part of the initial two-head experiment. All use 0.01 regularization and a fixed 0.9 positive threshold. Calibration was used repeatedly for selection, and old regression includes training works. Neither is independent validation.
What remains unresolved
The improvement is actual AI detection, not increased abstention. It still leaves 57 false-human results in 108 generated files and adds one human false positive. It does not meet our release criterion.
AIME generation-session and prompt overlap are unknown, upstream pretraining overlap is unknown, and FMA versus AIME remains a dataset-origin confound. Historical human references are not individual artist attestations. These older generator versions and eight-second excerpts do not establish performance on current generators, full tracks or competitors' systems.
The held-out set is now consumed. Further development will need new evaluation material; a tuned rerun of these files would be a regression check. The deployed five-transcode recovery remains documented separately.