CT-Bench, a four-choice lesion exam where GPT-4V scores 28%

01What a report must get right, lesion by lesion

Much of a CT report is a list of lesions: where each one is, what it is, how big, what it looks like. The public data behind CT report generators mostly doesn’t work at that grain. Merlin and CT-RATE pair a whole volume with a whole report, so nothing in them ties a sentence to a place in the image. DeepLesion, NIH’s 2018 release, marks lesions on key slices with a box and two diameters, and has no words.

CT-Bench, from NIH and with DeepLesion’s senior author among its authors, adds the words. It appeared in MELBA’s special issue on MICCAI Open Data, so the paper is mostly the description of a resource in two parts: a set of lesions with text, and an exam built on them.

02The lesion set

Table 1 — The lesion set, split by patient. No patient is in two splits. Source: Table 1.

DeepLesion’s public release holds 32,735 lesions, so CT-Bench keeps about 62% of them derived: those with a valid box and a report sentence that could be tied to the lesion without ambiguity.

Each lesion has a description of one sentence (where it is, what it is, sometimes what it suggests), a size as text, the box, and DeepLesion’s metadata: age, sex, pixel and slice spacing, display window. A second file maps each description to standard radiology terms: a lesion type and a body part for over 98% of lesions, but an imaging property such as “enhancing” or “calcified” for only 4,841 lesions (23.8%). Scanner and contrast phase aren’t available.

The sentences were not written from scratch. A GPT-4 model, fine-tuned on 500 curated cases, drafted each description and size from the radiologist’s original sentence; people then reviewed every draft: two annotators and a medically trained expert for validation and test lesions, one annotator and a medical professional for training lesions. The paper shows why the medical check matters: in the first round, annotators with a PhD but no medical training were right 56% to 73% of the time, the one with an MD 93%. It gives no error rate for the released descriptions.

03The exam

Table 2 — Seven tasks, four choices each. Question counts are without + with a box drawn around the lesion; “find the box” exists only with boxes. Sources: Table 3; Appendix B.3.

Wrong answers drawn at random were too easy, the authors report, so they are picked to be hard: BiomedCLIP, an image-text model, retrieves lesions that look similar, and a doctor chooses three plausible wrong answers among them. The same questions come in two formats, one for models that score an image against each text (CLIP-style) and one for chat models.

04What the baselines score

Three things to read off it:

1. General chat models were close to guessing. GPT-4V scores 28.4%; on 1,900 questions, pure guessing lands in 25 ± 1.9 derived. On the four “describe the lesion” settings it gets 27.5, 26.5, 24.0 and 22.0, all inside the 25 ± 6 that the arXiv version prints for guessing on 200 questions. Gemini 1.5 Pro reaches 33.8%. Both are models of 2023–2024 and no later one was run: the numbers date the baseline, they don’t describe current models.
2. Far below 25% means not answering. RadFM and LLaVA-Med total 15.3% and 14.5%, with zeros or near-zeros on the image-retrieval and size tasks. A guesser practically can’t score 0 on 100 or 200 four-choice questions, so these rows most likely measure whether a model returns a usable letter, not whether it reads CT our observation. RadFM fine-tuned on the captions scores zero on every task. The journal version prints those rows without comment; the arXiv version reads them as catastrophic forgetting.
3. Fine-tuning helps mostly with the box. BiomedCLIP leads the models as released at 38.1%. Trained on the lesion set with boxes drawn, it goes from 41.0% to 62.0% on with-box questions, 21 points. A second copy trained without boxes goes from 34.9% to 41.4% on questions without them, about 7 points. The box-trained model scores 37.1% on questions without a box. The “fine-tuned” row of the paper’s summary table is the box-trained model.

05How solid is the human yardstick?

The exam’s reference is three readers: two senior radiologists and a junior doctor with radiology experience, on 100 cases in total. The arXiv version gives the layout: 10 cases for each setting of the description, retrieval and box tasks, 5 for each setting of the attribute and size tasks. One case is worth 10 or 20 points in its cell.

With boxes, the fine-tuned model’s 62.0% falls inside the junior doctor’s interval, 55% to 79%, and below both senior readers’, whose printed totals are 79.2% and 81.5%.

One caution on the readers’ averages derived. Counting correct cases from the task cells gives 80, 82 and 56 of 100, close to the printed totals. But the printed with-box and no-box averages don’t follow from the cells: those give the seniors 47 and 49 of 55 with boxes (85% and 89%, printed 90.0 and 90.5) and 33 of 45 each without (73%, printed 66.0 and 70.5). Neither pooling cases nor averaging tasks reproduces them, while the models’ with-box and no-box averages follow from their cells exactly. The headline survives either way: seniors at about 8 in 10, the junior doctor under 6 in 10.

06Writing the description, not picking it

The lesion set also supports the task closest to report generation: write the sentence for a lesion. The paper scores it with word-overlap and embedding metrics only.

Fine-tuned RadFM reaches a BLEU-1 of 0.24, against 0.17 for the best model as released. BERTScore sits between 0.84 and 0.89 for every system, so it separates little. None of the five metrics checks whether the organ, the side or the lesion type is right, and the paper’s one worked example shows what that leaves out: for a nodule in the left lung, 4 of the 6 printed outputs put their main finding in the right lung. The structured labels in the release (body part, lesion type) are the material for a score that could catch this; the paper doesn’t build one.

07What the paper leaves out

• A validation table to trust. 12 of the 91 model cells in the validation results can’t come from the 100 or 50 questions the paper gives each validation setting (24.5% of 100, 49% of 50), and each of the 12 equals the test-set value for the same model and task. Compare against the test table.
• Intervals, and the make-up of the questions. The journal version prints no interval (the arXiv version has them per task: about ±6 points on 200 questions). Neither gives the answer mix per task, the number of distinct lesions behind the 2,850 questions, or the split the readers’ 100 cases came from.
• Some of the promised files, going by name. The paper mentions preprocessing code, benchmark construction scripts and a data dictionary. The Kaggle listing shows five scripts (captioning, caption scoring, the two question runners, fine-tuning), a README and a requirements file. We read the listing, not the files.

08Takeaways for testing a CT model on lesions

1. Report with-box and no-box scores separately. The best model is at 62.0% with boxes and 37.1% without; one total hides whether a system finds the lesion or only describes one it is shown.
2. Count non-answers apart from wrong answers. Two medical models total 15.3% and 14.5%, below guessing, because of zeros.
3. Treat the human reference as “about 80%”. It is 100 cases and two readers; with boxes, each senior’s interval starts below 80%.
4. For generated text, score the facts. Body part and lesion type are in the release; in the paper’s example, 4 of the 6 outputs had the wrong lung.

09What you can use today

• The data. CT_Bench on Kaggle (DOI 10.34740/kaggle/ds/7422536): 15.14 GB, about 80,200 files. Neither the paper nor the Kaggle page links a code repository; the scripts ship inside the dataset.
• The licence. CC BY-NC-SA 4.0, on the Kaggle page and in the paper: use and adapt it for non-commercial work, credit the authors, share what you derive under the same licence. Commercial use isn’t allowed. The paper adds: research only, not for clinical decisions, no attempt at re-identification.
• The files, by name. lesion_nobox/ and lesion_bbox/: key slices as 8-bit PNG, without and with the box drawn. Metadata.csv: lesion id, description, size, split. CT-Bench-Final-Merged-with-DeepLesion-Metadata .csv (the name has a space before the extension): the same lesions with box coordinates, diameters, spacing, window, age and sex. CT-Bench Lesion Ontology Mapping.csv: lesion type, body part and property terms. qa_clip.json and qa_llm.json, with the image folders clip/ and llm/: the questions in the two formats, split into test and validation. CT-Bench-code/: five scripts.
• Filter by the metadata. Kaggle lists 32,735 files in each lesion folder: DeepLesion’s lesion count, not CT-Bench’s 20,335 our observation. We read the listing, not the images; take the list of lesions from Metadata.csv.
• What you need from DeepLesion. Nothing for single-slice work. The released slices are already windowed to 8 bits; for raw Hounsfield values, or neighbouring slices for the training lesions, go back to DeepLesion (16-bit PNG; subtract 32,768) likely, not confirmed: the listing shows nine-slice inputs only for the exam questions, and the paper says identifiers were replaced, so check that the merged table lets you match files.
• How to start. There is no loader package. Read Metadata.csv for description, size and split; train on the 15,380 training lesions; score on the 1,900 test questions with qa_llm.json (a prompt, an image, choices A to D) or qa_clip.json (image-text similarity).
• Numbers to compare with (test questions, total): guessing 25%; GPT-4V 28.4%; Gemini 1.5 Pro 33.8%; BiomedCLIP 38.1% as released and 50.3% fine-tuned (62.0% with boxes, 37.1% without). Readers, on 100 cases: senior radiologists 80.4%, junior doctor 58.2%.

ASources

1. Zhu, Q., Jin, Q., et al. CT-Bench. Machine Learning for Biomedical Imaging 2026:036, pp. 739–757 (published 21 Sep 2026): paper page · PDF · DOI. The reference for every number here.
2. The earlier version, “CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomography”: arXiv:2602.14879 v2 (19 Feb 2026). Same result tables; used here for the per-task intervals, the reader study’s layout and the remark on fine-tuned RadFM.
3. The Kaggle dataset page, read on 12 Oct 2026: licence, size, file listing and the README it displays. No data file was opened.
4. Yan, K., et al. DeepLesion: arXiv:1710.01766 (2017) for the preprint’s counts; the NIH release and the Hugging Face mirror the paper links, whose card gives 32,735 lesions on 32,120 slices.
5. Zhang, S., et al. BiomedCLIP: arXiv:2303.00915 (2023).