SPEC-CXR, read for report-generation evaluation

01Why score findings one by one

The CT posts in this series kept hitting the same wall: BLEU, RadGraph-F1 and the like give one number per report, which can’t say which findings a generator misses. For Merlin, the finding-level score had to come from a later paper: a recall of 0.10 over 30 findings. Chest X-ray has long had a per-finding score, the CheXbert F1 over 14 observations. Its newer metrics, GREEN and FineRadScore, are LLM judges that list the errors of each report and still return one score per report. SPEC-CXR, from Lunit, joins the two ideas: an LLM judge with a fixed vocabulary of 122 entities, and three attributes on top of presence.

02How a report pair is scored

Three radiologists drew up the vocabulary, meant to be mutually exclusive: 89 findings in 11 categories, from 23 lung and pleura entries to 2 for the abdomen, and 33 differential diagnoses. Rib fracture and tracheostomy tube have their own entries, and every anatomical region has an “other” entry for the rest.

The scoring is then arithmetic. Our reconstruction from Section 2.2 and Figure 2, not the authors’ code:

# a record: entity, presence, location, severity, comparison

def report_errors(records):       # report level: lower is better
    return sum((r.presence in ("FP", "FN"))
               + (r.location == "not-aligned")
               + (r.severity == "not-aligned")
               + (r.comparison != "other") for r in records)

def entity_f1(pairs, entity):     # entity level, over a test set
    n = Counter(r.presence for records in pairs
                for r in records if r.entity == entity)
    return 2 * n["TP"] / (2 * n["TP"] + n["FP"] + n["FN"])

Attributes are scored on matched findings only, so a miss isn’t penalised twice. Two consequences of the design our observation: a finding that neither report mentions leaves no record, so there is an F1 but no specificity; and matching ignores attributes, so in the example a left pneumothorax matches a right one and costs one location error, not a miss plus an invention.

03Is the scorer right?

The report-level score was tested on ReXVal: 200 MIMIC-CXR report pairs on which six radiologists counted errors in six categories, the same six SPEC-CXR counts. The measure is Kendall’s tau-b between a metric and the radiologists’ mean total error count. The candidates aren’t model outputs: each is another patient’s report, retrieved because it scores well on BLEU, BERTScore, CheXbert or RadGraph, and only impressions were compared.

Three things to read off it:

1. The judge matters more than the vocabulary. The same metric moves from 0.597 with GPT-4o-mini to 0.777 with Claude 3.5 Sonnet. With one judge, o3-mini, going from CheXpert’s 14 labels to the 122 entities adds 0.06 (the paper’s Table 2).
2. One comparison holds the judge fixed. With o3-mini, SPEC-CXR reaches 0.734 and FineRadScore 0.692.
3. The ± isn’t a confidence interval. It is the spread of five runs of one judge. The 95% intervals other papers print on these 200 pairs span about 0.1 (0.680 to 0.788 for FineRadScore’s 0.738), enough to hold SPEC-CXR’s best value.

The entity-level scores, the new part, got a much smaller check: a manual assessment of 50 randomly sampled pairs. The scorer’s labels agreed with it 91.8% of the time on presence, 87.2% on location, 84.5% on severity and 75.3% on comparison. Fifty pairs can’t test 122 entities one by one, and the paper gives no per-entity agreement.

04What it shows about three generators

The paper then scores three generators on the 2,347 studies of ReXrank’s MIMIC-CXR test split: Microsoft’s MAIRA-2 and CXRReportGen, and Google’s Med-Gemini. The results are two radar charts over 15 groups of entities, with no table, so we read the values off the image our measurement.

Three things stand out:

1. Devices are found, most other things aren’t. Tubes, lines and implanted devices score 0.53, 0.53 and 0.63; no other group passes 0.5. Med-Gemini, the best model, is below 0.2 in 9 of the 15 groups, as the paper states; the ninth, the abdomen, sits at 0.196.
2. The weak groups aren’t only the rare ones. Differential diagnoses were extracted about 1,200 times, third among the groups, and all three models stay near 0.1 or below. The paper doesn’t say why. One candidate: diagnoses are usually written in the impression, and MAIRA-2 generates only findings our observation.
3. Matched findings are described well, with two cautions. The legends give 0.86 to 0.92 for location, 0.81 to 0.91 for severity and 0.94 to 0.95 for comparison. But on lung opacities, the most frequent group, Med-Gemini’s location and severity accuracies are 0.77 and 0.78 our measurement. And on comparison the scorer disagrees with the manual check 24.7% of the time, while it reports a 5 to 6% error rate for the generators derived.

05What the paper and the release leave out

• The validated pipeline. The released code (January 2026) extracts entities from each report separately, derives presence by comparing the two lists, and asks the LLM only for the attributes. The repository’s first README names the single-prompt version as the paper’s method, so the agreement figures above belong to a pipeline the release no longer contains. The default judge is GPT-4o (0.656 in the paper’s Table 1); the fine-tuned open judges, at 0.72, aren’t included; the licence is non-commercial.
• Entity-level detail. No per-entity support and no confidence intervals; nor who annotated the 50 pairs, which judge produced Figure 4, which report sections were compared, or how the generators were run.
• Coverage. By the authors’ LLM-based test, the 122 entities fully express 81.9% of test reports, against 56.8% for CheXpert’s labels. About one report in five holds a statement the set can’t represent derived.
• Scope. One reference dataset and one 200-pair benchmark, both from MIMIC-CXR. Reviewers asked for wider validation and a bias analysis of the LLM; the authors agreed to state the first as a limitation.

06Takeaways for evaluating a report generator

1. Put a per-finding table next to every report-level score. No report-level number shows that the best of three leading generators is below 0.2 in 9 of 15 finding groups.
2. Name the LLM judge and keep it fixed. One metric, eight judges: 0.597 to 0.777. The release’s default, GPT-4o, is at 0.656 in the paper.
3. Extract the reference once. The released code re-extracts it on every call. Cache it, or two models can be scored against different reference entities.
4. Publish the averaging rule and the support. CXRReportGen’s 0.189 becomes 0.203 on the 13 groups behind MAIRA-2’s 0.205.
5. Don’t rank LLM error counts on ReXVal alone. The best five sit between 0.76 and 0.79, under an inter-expert bound of 0.81, and 200 pairs don’t separate them.

ASources

1. Lee, J. O., Cho, J., et al. SPEC-CXR. MICCAI 2025, LNCS 15966, pp. 594–604: paper page with the reviews and the rebuttal · PDF · DOI.
2. SPEC-CXR code, read at commit 0cb653f (14 Jan 2026); its first README, of 15 Jul 2025, is in the repository’s history.
3. Ostmeier, S., et al. GREEN: arXiv:2405.03595 (2024), Table 6. Huang, A., et al. FineRadScore: arXiv:2405.20613 (2024), Tables 10 and 11.
4. Yu, F., et al. RadCliQ: Patterns 4(9), 2023; ReXVal on PhysioNet. Zhao, W., et al. RaTEScore: arXiv:2406.16845 (2024).
5. Zhang, X., et al. ReXrank: arXiv:2411.15122 (2024). Bannur, S., et al. MAIRA-2: arXiv:2406.04449 (2024) and its model card.
6. Smit, A., et al. CheXbert: arXiv:2004.09167 (2020). Delbrouck, J.-B., et al. RadGraph-F1: arXiv:2210.12186 (2022).