Evaluation Design
Status: evaluation is ongoing. External input preparation is in progress, and no performance figures are published yet. This page states the design that results will be held to.
Questions
- Does the learned technical score place more externally supported candidates in the Top-K than simple baselines on the same candidates?
- Is the ranking stable across reasonable choices of K and across data subsets?
- Does every report field trace back to a recorded source?
Labels
Reference labels come from external evidence sources. They are evaluation labels, not cardiac ground truth. A candidate without external support is counted as unconfirmed, never as an artifact.
Metrics at fixed K
The protocol will specify K values before comparison and report them together, rather than choose a favorable K after seeing results.
Confirmed yield@K = (confirmed candidates in Top-K) / K
Confirmed-positive recall@K = (confirmed candidates in Top-K) / (all eligible confirmed positives in the evaluation pool)
Confirmed yield is not true precision when the remaining candidates have unknown status. External evaluations use their documented candidate pool; they do not automatically represent the full cardiac caller union. All methods in each comparison share the same pool, labels and review budget. Missing inputs and failed scores remain accounted for; if fewer than K candidates can be returned, the shortfall is reported.
Repeated observations of the same event must stay in the same evaluation group. A single-donor study cannot establish generalization across donors. Confirmation-agreement metrics, where justified by the assay records, are distinguished from somatic-origin or artifact-detection accuracy.
Baselines
Baselines are included only where they can be run on exactly the same candidates and labels:
- ordering by caller concordance (number of agreeing callers);
- ordering by a single technical feature, such as alt-read support or allele fraction;
- a simple rule-based or linear combination of technical features.
Comparisons that would require different input data are not reported as head-to-head results.
Not claimed
- No end-to-end performance gain is claimed until it is independently confirmed.
- No diagnostic accuracy, clinical benefit or user-study outcome is claimed.
- Language-model output is assessed for faithfulness to evidence, not used as a performance metric for ranking.