nnFoundation, read for report generation

* Our arithmetic from the stated patch size and merge factor, for a 192³ input.

01Why this experiment matters

Most 3D report-generation papers change several things at once: the encoder, the connector, the language model and the training data. That makes it hard to tell what helped. The report-generation experiment in nnFoundation does the opposite. Everything after the image encoder is fixed: Qwen2.5-VL-3B without its own 2D vision tower, one 3D adapter design, one two-stage training schedule and the same two datasets. Only the frozen encoder changes, along with a per-encoder sweep of learning rate and field of view.

Because the encoder weights never move, differences in BLEU, F1 and CRG show what each pretrained representation already contains. If you are choosing a vision backbone for CT report generation, that is more useful than yet another end-to-end system. The experiment covers just two of the paper’s 108 tasks, and most of its numbers sit in Supplementary Tables 9, 10 and 15.

The line-up covers every major pretraining approach:

• Masked autoencoding: nnFoundationViT and nnFoundationCNN
• DINOv2 self-distillation on 2D slices: CURIA
• Contrastive learning: CT-FM and VoCo
• Vision–language pretraining with reports: Merlin
• Supervised segmentation: VISTA3D
• Older self-supervised objectives: SwinUNETR (inpainting + contrastive + rotation) and MIS-FM (volume fusion)

02The data: 2.16 million volumes, zero reports

The pretraining corpus is the hardest part to copy. The team gathered 3,271,454 imaging series from 125 sources and kept 2,158,570 volumes after curation. The corpus has three streams:

Clinical partnerships dominate. The DKFZ radiology research PACS alone contributes 821,692 volumes, followed by Bonn with 264,803 and Thoraxklinik Heidelberg with 228,732. Together that is 1,315,227 volumes, or 60.9% of the corpus. Large research cohorts add 34.2%, for example the Osteoarthritis Initiative’s 240,867 MRIs, NLST’s 129,592 chest CTs, OpenNeuro and I-SPY 2. The 91 small public datasets that most papers are built on, mostly from TCIA, make up only 4.9%. The scale comes from institutional access within the Human Radiome Project, not from a clever scrape of public data.

Curation was aggressive and mostly automatic. A third of the aggregated series were dropped. The largest cuts were volumes derived from 4D series (458,754), volumes with fewer than 10 slices along an axis (229,693), and series that were not CT, MR or PET, or not images at all (180,051). Two steps matter directly for report generation:

• An out-of-distribution detector whose flags were reviewed by hand. This removed 25,456 volumes, 9,110 of them non-human scans.
• Hash-based de-duplication against every downstream dataset, which removed 88,477 volumes. It keeps CT-RATE and MERLIN scans out of pretraining.

Pretraining uses no labels and no text. Volumes are not resampled to a common spacing, so the model sees native resolutions, and every image is z-score normalised.

03Stage 0: pretraining an encoder that never reads

The report-generation encoder is nnFoundationViT, a plain 3D Vision Transformer built on the Primus design, DKFZ’s ViT for 3D medical segmentation. A CT crop is cut into 8×8×8-voxel patches. Each patch becomes a 1,056-dimensional token, and 40 transformer layers process the sequence.

It is trained by masked autoencoding. For every 192³ crop, 80% of the patch tokens are hidden at random. The encoder sees the remaining 20%, and an 8-layer decoder reconstructs the missing voxels. The decoder is discarded after pretraining. No text, label or contrastive pair is used at any point in this stage.

Architecture

Primus-style 3D ViT · depth 40 · width 1,056 · 674M parameters

Tokens

8×8×8-voxel patches → 24³ = 13,824 tokens per 192³ crop derived

Objective

MAE, random patch masking, ratio 0.8 (≈2,765 visible tokens per crop, derived)

Decoder

8 transformer layers, used only during pretraining

Input

Random 192³ crops at native spacing, z-score intensities

Optimiser

AdamW, weight decay 0.05, peak LR 2×10⁻³, 50k linear warm-up steps, polynomial decay, gradient clipping 1, mixed precision

Schedule

Global batch 96 × 625,000 steps = 60M crops, ≈28 passes over the corpus (derived)

Compute

≈12,500 A100 GPU-hours (130 h on 96 GPUs)

Why MAE, and why a large ViT

The choice came from a controlled comparison run before scaling up. On a subset of ~630k public volumes, separate teams tuned MAE, DINOv2 and SimCLR-style contrastive learning, each for both a CNN (nnU-Net ResEnc-L) and a Primus ViT. Each team had a development budget of about 25k A100-hours, and each pretraining run was capped at 400 GPU-hours. MAE came out on top for both architectures.

Scaling then went differently for the two. Going from 205M to 674M parameters clearly helped the ViT but did not consistently help the CNN, so the final pair is a 102M CNN and a 674M ViT.

One detail from the development runs is relevant to report generation. With a shallower (16-layer) encoder, the depth of the MAE decoder shifted what the encoder learned: shallow decoders favoured segmentation, deeper ones favoured classification. The authors read this as decoder capacity trading fine local detail against more abstract features. The final model pairs its 40-layer encoder with the deepest decoder they tried (8 layers). Our reading is that this fits the ViT’s later strength on global, semantic tasks such as writing reports.

04Stages 1–2: from frozen features to a report

Report generation reuses Qwen2.5-VL-3B but discards its 2D vision tower. The frozen 3D encoder takes its place, and a small adapter connects the two. At 674M parameters, the 3D encoder is about the same size as the 2D tower it replaces (roughly 0.65B, by our count from the Qwen2.5-VL config).

The adapter: Qwen’s patch merger, in 3D

Qwen2.5-VL shortens its sequence of vision tokens by concatenating each 2×2 group of neighbouring tokens and projecting the result into the language model’s embedding space. The nnFoundation adapter does the same over 2×2×2 groups. Eight neighbouring encoder tokens (8 × 1,056 = 8,448 features) are concatenated and projected to one LLM token. This keeps local 3D context together and makes the sequence 8× shorter.

The paper describes the adapter in one paragraph. Here is a minimal sketch of it, reconstructed from that description (not the authors’ code):

class PatchMerger3D(nn.Module):
    """Merge each 2x2x2 block of frozen-encoder tokens into one LLM token."""
    def __init__(self, d_vis=1056, d_llm=2048, m=2):
        super().__init__()
        self.m, d_cat = m, d_vis * m ** 3          # 8 x 1056 = 8448
        self.norm = nn.LayerNorm(d_vis)            # Qwen2.5-VL uses RMSNorm
        self.mlp = nn.Sequential(
            nn.Linear(d_cat, d_cat), nn.GELU(), nn.Linear(d_cat, d_llm))

    def forward(self, x):                          # x: (B, 24, 24, 24, 1056)
        B, D, H, W, C = x.shape
        m = self.m
        x = self.norm(x).view(B, D // m, m, H // m, m, W // m, m, C)
        x = x.permute(0, 1, 3, 5, 2, 4, 6, 7)      # group each 2x2x2 block
        x = x.reshape(B, -1, m ** 3 * C)           # (B, 1728, 8448)
        return self.mlp(x)                         # (B, 1728, 2048)

If the adapter copies Qwen’s two-layer MLP exactly, it has about 89M trainable parameters (8,448² + 8,448 × 2,048). A single linear projection would have about 17M. The paper does not say which it uses.

Training: two stages, one loss

Training is plain supervised next-token prediction on the reference reports, in two stages. Stage 1 trains only the adapter, for 1.2 epochs. This places the visual tokens somewhere the LLM can use them before the LLM itself is allowed to change; the authors describe it as a guard against feature collapse. Stage 2 adds LoRA to the language model for another 10.8 epochs while the adapter keeps training. The encoder stays frozen throughout.

Every encoder got its own sweep over learning rate and field of view on internal validation splits, and each reported number is that encoder’s best configuration. The two ViTs, nnFoundationViT and CURIA, both preferred a learning rate of 1e-3. All the other encoders settled at 4e-4 (Table 1 below).

05CT preprocessing and field of view

Preprocessing is standardised per encoder. Each scan is resampled to the encoder’s native spacing, which for nnFoundation (no fixed spacing) means 1 mm isotropic. It is then normalised with the encoder’s own intensity scheme: z-score for most, [0, 1] scaling for VISTA3D and Merlin.

Each scan is then cropped to the anatomy. On CT-RATE, the region of interest is the union of TotalSegmentator’s lung labels. On MERLIN, it is the union of all TotalSegmentator foreground labels. Connected components under 400 voxels are discarded as outliers, and the bounding box is enlarged by 4%. Three field-of-view modes were then compared:

• resize: trilinearly resize the ROI crop to the encoder’s native input size, which keeps all the anatomy at lower resolution;
• native crop: centre-crop the ROI to the native input size, which keeps full resolution but may cut off anatomy;
• 192-crop: centre-crop to 192³.

On MERLIN the abdominal ROI is larger than every encoder’s input, so only the resize mode was run. Two encoders needed special handling:

• CURIA is a 2D model, so its per-slice tokens are mean-pooled into 8, 12 or 16 axial bins (part of the sweep) to form a volume grid.
• Merlin, on its own dataset, keeps its original 224×224×160 pad-and-crop pipeline with no ROI crop, so it is not disadvantaged by a shift away from its pretraining setup.

Table 1 — Report-generation settings chosen by each encoder’s sweep. “Native patch” is the pretraining input size listed in the paper. Source: Table 1a/b and Methods 4.6.

06Results on CT-RATE (chest CT)

CT-RATE pairs 25,692 non-contrast chest CT scans from 21,304 patients with their radiology reports. The reports were written in Turkish, machine-translated into English and reviewed by bilingual medical students. It is the only benchmark for which the paper uses structured abnormality labels, so it is the only one scored with clinical metrics.

Table 2 — CT-RATE results with 95% CIs. Sorted by F1. “Pretrain vol.” is as listed in the paper’s Table 1; CURIA’s 1M counts 2D slices as volume equivalents. Source: Supplementary Table 15 and Table 1a.

Five things stand out:

1. Two leaders, then a large drop. nnFoundationViT (F1 0.437) and CURIA (0.428) are far ahead; the third encoder, CT-FM, reaches 0.276. The two leaders are the only plain ViTs in the line-up and, together with nnFoundationCNN, have the three largest pretraining sets. Their objectives are very different: MAE on 3D crops versus DINOv2 on 2D slices.
2. Same data, different architecture, three times the F1. nnFoundationCNN has the same corpus and the same MAE objective as the ViT, yet reaches only F1 0.146. Architecture is not the only difference, though: the CNN has 102M parameters to the ViT’s 674M, and about 500 A100-hours of pretraining to its 12,500. The CNN is the paper’s pick for segmentation. For report generation from frozen features, it is roughly level with VISTA3D and VoCo.
3. Report supervision on a small dataset does not carry over to chest CT. Merlin’s encoder was pretrained with reports on ~15k abdominal CTs. On chest CT-RATE it reaches F1 0.227, well behind both self-supervised ViTs. On its own abdominal data it ties nnFoundationViT (section 08), so part of this gap is domain, not supervision.
4. Some encoders give the LLM nothing to work with. MIS-FM scores F1 0.000 and a CRG of exactly 1/3, with CIs of zero width. That is chance level: its reports essentially never flag an abnormality. SwinUNETR is barely higher (0.022 and 0.336). With these features the LLM appears to write much the same normal-looking report for every scan, and that still earns BLEU 0.241–0.247.
5. Absolute scores are modest. The best CRG, 0.444, corresponds to a pooled true-positive rate minus false-positive rate of about 0.37 (section 09). That leaves a lot of room to improve.

Where does 0.437 sit?

Other CT-RATE report generators, scored with CT-RATE’s RadBERT labeler (probably the one used here), give some context. Test sets and averaging differ between papers, so treat this as a rough guide, not a leaderboard.

Table 3 — CT-RATE context. A frozen, report-free encoder behind a 3B LLM clears the 2024-era systems by a wide margin. A closely related DKFZ paper, which tunes how visual tokens are reduced, reaches 0.495. Sources: arXiv:2505.17167 and arXiv:2608.08713.

07Is the lead real?

The paper uses a strict rule: nnFoundation “outperforms” a comparator only if the 95% paired-bootstrap interval for the difference lies entirely above zero. By that rule, the win over CURIA holds for CRG only (+0.018, 95% CI +0.014 to +0.023). The F1 lead (+0.009, CI −0.005 to +0.022) and the BLEU lead (+0.001) are within noise. Against every other encoder, nnFoundationViT is clearly ahead on all three metrics.

The paper’s ranking summary tells the same story. On report generation, nnFoundation’s mean rank is 1.00 (95% CI 1.00–2.50) and CURIA’s is 2.00 (1.50–3.00). A fair headline is therefore “best or joint best”. The more interesting result is the tie itself: a 2D slice encoder, with slice features mean-pooled into a coarse volume grid, nearly matches a native 3D ViT at writing chest CT reports.

08Results on MERLIN (abdominal CT)

MERLIN is Stanford’s abdominal CT dataset: 25,528 scans in the paper’s count, with findings organised by organ system. The Merlin encoder was pretrained on its roughly 15k-scan training split. Only BLEU is reported for this benchmark, so it says nothing about clinical correctness.

Four encoders are tied at the top: nnFoundationViT at 0.172, and CURIA, CT-FM and Merlin at 0.170, with overlapping intervals. Merlin is the in-domain reference here. It was pretrained on this very dataset and evaluated with its original preprocessing, so matching it with an encoder that never saw these scans is a good sign for transfer. Below the top four, scores roughly halve (0.079–0.100), and MIS-FM drops to 0.019.

09Reading the metrics

BLEU

The paper uses corpus-level BLEU: n-gram overlap between generated and reference reports, pooled over the test set. It doesn’t state the n-gram order; the common CT-RATE evaluation code (pycocoevalcap, as in the VLM3D challenge) reports BLEU-1 to BLEU-4 and their mean. BLEU rewards fluent, templated wording, which is why an encoder with F1 = 0 still scores 0.241 on CT-RATE. Its range across the nine encoders on CT-RATE is 0.076, against 0.437 for F1.

Abnormality-label F1

F1 is computed only where structured labels exist, which here means CT-RATE. Each CT-RATE scan carries 18 multi-label abnormality annotations:

• Medical material
• Arterial wall calcification
• Cardiomegaly
• Pericardial effusion
• Coronary artery wall calcification
• Hiatal hernia
• Lymphadenopathy
• Emphysema
• Atelectasis
• Lung nodule
• Lung opacity
• Pulmonary fibrotic sequela
• Pleural effusion
• Mosaic attenuation pattern
• Peribronchial thickening
• Consolidation
• Bronchiectasis
• Interlobular septal thickening

To score a generated report, its abnormalities first have to be read out of the text. The paper doesn’t name its extractor. The standard tool is the RadBERT classifier released with CT-RATE, fine-tuned on 1,000 manually labelled reports and thresholded at 0.5. The same classifier produced CT-RATE’s ground-truth labels.

CRG score

CRG (Hamamci et al., 2025) counts true positives, false negatives and false positives over every label slot in the test set. It weights them so that an empty report and an exhaustive one score the same:

A little algebra shows that CRG is a rescaling of Youden’s J (sensitivity + specificity − 1) over the pooled label slots:

So 1/3 is chance level (J = 0). That is also exactly what a model scores if it never reports an abnormality, or always reports all of them. The scale runs from 0.2 (J = −1) to 1. MIS-FM’s exact 0.333, alongside F1 = 0, means its reports essentially never flag an abnormality. Converted back to J, the results are easier to read:

10What the paper leaves out

The protocol is described in two short paragraphs. Several details you would need to reproduce the numbers, or to compare them with other CT-RATE work, are missing:

• Language-model training: LoRA rank, alpha and target modules; the report-generation optimiser and schedule; batch size; context length; the prompt. The DKFZ companion paper with a similar recipe uses LoRA r = 32, α = 64 and dropout 0.1, which may carry over.
• Targets: which report sections are generated (findings, impression or both) and any text cleaning.
• Scoring: the label extractor behind F1 and CRG, and whether F1 is micro- or macro-averaged. Most likely it is the RadBERT classifier with macro F1 (section 09).
• Evaluation set: test-split size, and how scans with several reconstructions were handled.
• Decoding: greedy or sampled, and the maximum length.
• Cost: compute for the report-generation runs. The whole evaluation suite used more than 50k A100-hours.
• Deeper analysis: per-abnormality results, radiologist review, or comparison with published CT-RATE report generators.
• Ablations: on the adapter design or the two-stage schedule, and a baseline with the encoder unfrozen.
• Code: the paper says fine-tuning pipelines for vision–language tasks are provided. The only nnFoundation code repository it links is nnssl, which covers pretraining, and the model card lists report generation as “TBA”. The encoder weights themselves are public under CC-BY-SA-4.0.

11Takeaways for building a CT report generator

1. Treat the encoder as the main lever, and benchmark it frozen. With the LLM, adapter and recipe held fixed, swapping encoders moved F1 from 0.00 to 0.44. A frozen-encoder comparison like this one is cheap and gives clear answers.
2. For report generation, choose a ViT pretrained at scale. Both leaders are plain ViTs. Every CNN and Swin encoder lags far behind, including one trained on the same 2.1M volumes. For segmentation the order flips, and the paper’s CNN wins.
3. You may not need reports to pretrain the encoder. Image-only self-supervision at scale tied a report-supervised encoder (Merlin) on Merlin’s own abdominal data and beat it on chest CT, where Merlin is off-domain. Reports are needed to train the adapter and LoRA weights, and a CT-RATE-sized dataset was enough (its official training split has ~20k patients; the paper doesn’t say which split it used). The encoder weights are public, so you can reuse the 12,500 GPU-hours of pretraining.
4. Test a strong 2D encoder before committing to 3D. CURIA, with slices mean-pooled into axial bins, is statistically tied on F1 and BLEU.
5. Don’t rely on BLEU alone. Track a clinical metric such as label F1 or CRG, and watch for template collapse: F1 near 0, or CRG stuck at chance (1/3).
6. Keep the two-stage schedule and sweep per encoder. Warming up the adapter alone before adding LoRA is cheap insurance. Sweep ROI cropping and field of view for each encoder: the settings that won differ from one encoder to the next (Table 1).
7. There is plenty of headroom. Each constraint here has an obvious upgrade: the encoder is frozen, the LLM has only 3B parameters, and the LLM is tuned only through LoRA. Candidates are partial unfreezing, a larger LLM, tiling the full lung volume, and reinforcement learning against a clinical reward such as CRG.

ARecipe sheet & sources

Everything the paper specifies for report generation

Datasets

CT-RATE (chest, 25,692 volumes) · MERLIN (abdomen, 25,528 volumes)

Language model

Qwen2.5-VL-3B, 2D vision tower replaced by the frozen 3D encoder

Adapter

Spatial-to-channel merge of 2³ neighbouring tokens, projected to the LLM hidden size

Loss

Supervised next-token prediction on reports

Stage 1

1.2 epochs · adapter only

Stage 2

10.8 epochs · adapter + LoRA on the LLM

Encoder

Frozen throughout, for every method

Sweeps

Learning rate × field of view per encoder, internal validation split, best configuration reported

nnFoundationViT

LR 1e-3 · native crop (192³ at 1 mm) · lung ROI on CT-RATE, resize on MERLIN

Metrics

Corpus BLEU; abnormality F1 and CRG on CT-RATE; bootstrap 95% CIs (1,000 replicates); paired-bootstrap significance

Not reported

LoRA rank/alpha/targets, batch size, optimiser, prompt, decoding, report sections, label extractor, F1 averaging, test-split size, compute

Likely, not confirmed

RadBERT labeler, macro F1, pooled CRG. LoRA r = 32, α = 64, dropout 0.1, as in the DKFZ companion paper

References

1. Ulrich Harsy, C., Wald, T., et al. nnFoundation: 3D Foundation Models for Radiology. arXiv:2609.26924 (2026).
2. MIC-DKFZ. nnssl: self-supervised learning framework for 3D medical images (linked from the paper).
3. Hamamci, I. E., et al. CT-RATE / CT-CLIP. arXiv:2403.17834 · dataset.
4. Hamamci, I. E., et al. CRG Score: a distribution-aware clinical metric for radiology report generation (2025).
5. Suprijadi, J., Stock, R., et al. Resolution Meets Reduction: efficient visual context for 3D radiology report generation (2026). DKFZ companion paper.
6. VLM3D challenge report-generation evaluation code: forithmus/VLM3D-Dockers.
7. MIC-DKFZ. nnFoundationViT model card (CC-BY-SA-4.0).
8. Blankemeier, L., et al. Merlin: a computed tomography vision–language foundation model and dataset. Nature 652, 1318–1328 (2026).
9. Qwen Team. Qwen2.5-VL technical report (2025).
10. Wasserthal, J., et al. TotalSegmentator.