Phrase-grounded fact-checking, read for report-generation evaluation

* Our arithmetic on the paper’s Tables 3 and 5.

01Why check a report against the image

The blog’s first post on evaluation read SPEC-CXR, which scores a generated chest X-ray report against a reference report, finding by finding. This paper checks the report against the image instead. That matters at the point of use, where no reference exists.

It comes from Rensselaer Polytechnic Institute, IBM Research, Stanford and Massachusetts General Hospital. Some of the authors built a first checker in 2023: a frozen CLIP encoder with an SVM that labelled whole sentences real or fake. The new one, the FC model, reads normalised finding phrases, trains its encoders and also says where the finding is.

So the verdict is about presence. The first preprint says the location words are dropped from the model’s input. A misplaced finding shows only as a low overlap between two boxes.

02The training data: fakes made by rule

With no large set of labelled generator errors to learn from, the authors make their own. Each ground-truth report yields real pairs: a finding phrase, and a box around the anatomical regions the finding is attached to. Each real pair then spawns fakes of three kinds. Omissions and wrong severities, the other error types the paper names, are left out.

Look at the two relocated fakes. They keep the image and the phrase of the real pair, yes|edema; only the target changes. Since the model’s input is the image and the phrase, nothing it receives tells the three apart our observation, and the paper doesn’t say what it should learn from them.

Table 1 — Five public datasets, three of them cut from MIMIC-CXR. The silver set supplies the bulk; the paper sets the gold one aside for the generator test. Source: Table 3 (silver counts in full from the first preprint); last column derived.

03Test 1: held-out synthetic pairs

The first test asks the model to do on held-out pairs what it was trained to do: label each pair real or fake, and regress its box.

Four things to read off it:

1. An accuracy without its mix. The paper gives no confusion matrix and no balance of the test partitions. By Table 3 the pairs are 83 to 92% fake on three of the four sets derived. If the test partitions keep that mix, answering “fake” every time scores as much: the model is 0.02 to 0.09 above it, the 2023 classifier below it on two or three.
2. The only spread given is wider than the gaps. Over a ten-fold re-split the paper reports 0.92 ± 0.12, of unstated meaning. The ablations sit within 0.05 of the full model in accuracy.
3. The box baselines may not be on home ground. MAIRA-2 is 0.05 to 0.15 behind and Med-RPG 0.11 to 0.31, and the paper doesn’t say how either was run. Med-RPG’s own paper reports 0.59 on its MS-CXR split, against 0.32 here, and 0.39 for a version that sees only the phrase.
4. A mean IoU needs a floor. In our sample of the released pairs, half of the real boxes cover more than 37% of the image, and one fixed box per phrase, set without looking at the image, overlaps them at a mean IoU of 0.50 our count. That is the silver set, not the table’s test sets, but it is the kind of yardstick the 0.49 to 0.57 lacks.

04Test 2: seven real report generators

The second test gives the abstract its only number. Seven generators write reports for the same images, 439 of them on the gold set: RGRG, XrayGPT, R2GenGPT, CvT2DistilGPT2, CheXRepair, MAIRA-2 and an in-house GPT-4 tool. Each report gets the error score of Figure 1 twice: once from the checker’s verdicts and boxes, once with the dataset’s annotations in the checker’s place. The second is a metric from a companion paper by the same group. The two agree with a concordance correlation coefficient of 0.997, which the authors read as support for the checker as a “surrogate for ground truth”.

Lin’s coefficient on the 28 printed pairs is 0.997 derived, the paper’s number to the third decimal. The paper doesn’t say what its coefficient was computed on and gives no per-report value. The rebuttal counts its evidence the same way: six metrics, four datasets and six generators make “144 measurements”.

At that level the agreement holds: the two means never differ by more than 0.022, and within each dataset the seven generators come out in the same order, bar one swap between two that the reference separates by 0.007. Two things temper it. First, 82% of the variance in the reference score lies between datasets derived: both scores are near 0.6 on the gold set and VinDr-CXR and near 0.4 on the other two. Second, the two sides share a front end our observation: the same extractor and region boxes turn each report into findings and locations before either comparison, so their errors move both scores together.

05What the paper leaves out

The three reviewers started at 2, 3 and 6 out of 6 and all ended at accept.

• Verdicts on real errors: no per-finding accuracy on generator output, no radiologist, no weighting of errors. Asked whether synthetic errors transfer, the authors pointed to Table 5. And by the first preprint the reference side, too, keeps only the findings a report mentions.
• The promised comparison: a reviewer asked how the verdicts line up with GREEN, RadGraph-F1, CheXbert-F1 and RadCliQ. The rebuttal quotes concordances of 0.998, 0.999 and 0.998 for the first three and promises the full results in the arXiv version, which has the camera-ready text without them.
• Uncertainty and bookkeeping: no interval beyond the ±0.12, and no word on which pairs the mean IoU averages over. The gold set has 439 images in the text and 461 in Table 3.
• Code and model: the paper page lists neither. Without the extractor and the region detector, a report can’t be turned into the phrases and boxes the check needs.

06Takeaways for evaluating without a reference

1. Validate a checker per finding, on real generator output. Agreement between averages, which gives the 0.997 here, can’t tell a checker that flags the right findings from one that flags the right number of them.
2. Print the floor next to every score. An accuracy of 0.92 reads differently once 83 to 85% of the dataset’s pairs are fake, and a mean IoU of 0.54 once a fixed box reaches 0.50 on the released silver pairs.
3. Make rule-made errors as hard as a generator’s. When seven fakes in ten carry a phrase that is almost never real, as in the released sample, accuracy says little about the hard case: a common finding that is or isn’t there.
4. Use an image-based check beside a reference-based score, not in its place. It needs no second report, but it can’t see what a report leaves out; a score like SPEC-CXR can.

ASources

1. Mahmood, R., Machado-Reyes, D., Wu, J., Kaviani, P., Wong, K. C. L., D’Souza, N., Kalra, M., Wang, G., Yan, P., Syeda-Mahmood, T. Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports. MICCAI 2025, LNCS 15966, pp. 441–452: paper page with the reviews and the rebuttal · PDF · DOI · arXiv:2509.21356 (the camera-ready text).
2. The first preprint: Mahmood, R., et al. Anatomically-Grounded Fact Checking of Automated Chest X-ray Reports. arXiv:2412.02177 (Dec 2024). The same model and results, with a report-correction step the MICCAI version drops.
3. The companion metric: Mahmood, R., et al. Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings. arXiv:2412.01031. The 2023 classifier: Mahmood, R., et al. Fact-Checking of AI-Generated Reports. arXiv:2307.14634.
4. The released pairs: razi-mahmood/RadCheck on Hugging Face, read through its dataset viewer on 5 Oct 2026: exact counts for the validation and test splits, and 330 evenly spaced pages of 100 validation rows as the sample. The fixed box of section 03 is the median box per phrase, fitted on half of the sampled images and scored on the other half.
5. Chen, Z., et al. Med-RPG: arXiv:2303.07618 (2023), Tables I and II. Bannur, S., et al. MAIRA-2: arXiv:2406.04449 (2024).
6. Datasets: Chest ImaGenome, MS-CXR and VinDr-CXR on PhysioNet; ChestX-ray8.
7. Lin, L. I.-K. A concordance correlation coefficient to evaluate reproducibility. Biometrics 45, 255–268 (1989).
8. Our post on SPEC-CXR.