µ²LLM, read for report generation

* From the released code and configuration. The paper’s text says 1,024 queries.

01Why this paper matters for report generation

Two choices shape every system in this series: how much of the scan reaches the language model, and what the model is trained on. µ²LLM makes an unusual one of each. It reads 256 slices but hands the LLM only as many tokens as one 32-slice block yields, chosen with the question in view. Then it trains on its judge: the model drafts reports, GREEN ranks them, and direct preference optimisation (DPO) teaches it to prefer the winners.

So GREEN is trainer and examiner at once. The design-space paper read earlier raised GREEN by 0.078 with canned sentences that its other metrics did not reward; does training on GREEN move anything else?

Two of these facts come from the repository, not the text:

• 256 tokens, not 1,024. The text mentions 1,024 learnable queries, but its notation gives the tokenizer one block’s worth of output tokens, and the released configuration sets 256. That is the design-space baseline’s budget, drawn from eight times as many slices; Merlin passes 490 tokens and NV-Reason-CT 13,824.
• The training scripts freeze nothing. The paper does not say what trains. The released stage-1 scripts for Llama-3.2-1B update the ViT, the projector, the tokenizer and the whole LLM, one model per dataset (4 epochs at a learning rate of 4×10⁻⁶ in the scripts of July 2025). “1B” counts the LLM alone; as released, the tokenizer adds about 0.36B parameters derived.

02Three datasets, and what the baselines were given

The abstract announces four CT report datasets; the introduction says three, and three are described and scored. The paper gives sizes but no split and no test-set size; Table 1 adds what the evaluation scripts in the repository’s history show.

Table 1 — The three datasets, all public; one has machine-written references. The last column is from scripts kept in the repository from March 2025 to February 2026, which may not be the exact runs behind the paper. Sources: Section 3, the challenge page, the CT-RATE and RadGPT papers.

Three things stand out:

1. In the scripts, the baselines are public checkpoints, used as they are. The paper does not say how they were run. The evaluation scripts load LaMed, RadFM and CT-CHAT as published, LaMed and RadFM with generic prompts.
2. One row matches another paper’s table. RadGPT-N’s scores (ROUGE-1 0.247, METEOR 0.112) match the 24.7 and 11.2 of the RadGPT preprint’s Table 4, which were measured on a private set from another hospital.
3. Trained on the data, the same baseline is another model. LaMed-Phi-3-4B scores 0.009 GREEN on AMOS-MM here. In the design-space paper, the M3D stack with that Phi-3, trained on AMOS-MM’s 1,287 scans, scores 0.365, with the same judge checkpoint in both releases (400 validation cases, mean over regions; prompts and decoding differ).

So, by these scripts, the claim that a 1B model outperforms models of 7B to 14B compares fine-tuned with as-released.

03The tokenizer: what the ablation shows

The µ²Tokenizer starts from LinVT, a module that lets image LLMs read video: score the frame tokens, keep the top k, pool them at several scales, and let learned queries, which also read the text, aggregate them. The paper treats the eight blocks as frames and changes three steps: a learned bias per relative position in attention (RPE); 1,024 soft selections, each a weighted sum of all tokens, in place of the hard top-k (DTS); and learned weights on the pooled scales (DMTP).

1. Only GREEN and METEOR put the full tokenizer first. RPE alone matches it on ROUGE-1 (0.421) and on BLEU (0.281 against 0.279); DTS alone has the top BERTScore (0.888 against 0.881). A reviewer noted the uneven pattern.
2. The text and the table disagree on DTS. The text credits it with “up to 0.2 points” of GREEN; the table shows 0.095. The rebuttal said the sentence belonged to a second, subtractive ablation and promised to print both. The final paper keeps the sentence and one table.
3. The RPE row may measure more than positions. In the repository’s code until a commit of January 2026, the settings without RPE called PyTorch’s attention with the batch axis where it expects the sequence, so a query could not attend to the other 255. Whether Table 2 ran on that code is not stated our observation.
4. No row removes the tokenizer, drops the question or uses one block instead of eight.

The reviewer who scored the paper lowest doubted two claims: that a question can steer feature extraction when report prompts hardly vary, and that a model fed one input shape is multi-scale. The rebuttal placed the scales in the tokenizer’s pooling; the reviewer was not convinced. The released code gives both doubts some ground our observation. The pooling averages 1, 2 and 4 neighbouring entries of a list of soft selections that has no spatial order, so a scale is not a size in the scan. And the evaluation scripts ask one of three region prompts on AMOS-MM, and one fixed sentence on CT-RATE.

04DPO on GREEN: what moves, and what can’t be told

Stage 2 needs no reward model and no reinforcement learning, only a better and a worse report for the same input. Condensed from the released code (drafting and ranking, pairing):

# 1. draft, judge, rank: for each example of the training split
drafts = []
while len(drafts) < 8:
    d = model.generate(scan, question, do_sample=True,
                       temperature=1.0, top_p=0.9)
    if usable(d):            # 20+ characters, none Chinese
        drafts.append(d)
scores = {d: green(reference, d) for d in drafts}
ranked = sorted(scores, key=scores.get, reverse=True)

# 2. one preference pair per example, unless every draft scores 0
if scores[ranked[0]] != 0:
    pairs.append({"image": scan, "question": question,
                  "chosen": ranked[0], "rejected": ranked[-1]})

Each pair is the model’s own best and worst draw (0.63 and 0.1 in the paper’s schematic). DPO then raises the likelihood of the chosen report over the rejected one, relative to a frozen copy of the stage-1 model.

Which GREEN? The abstract names GREEN-RedLlama (sic), presumably the 7B RadLlama2 checkpoint, as DPO’s guide; the checkpoint that scores the results is not named. In the released scripts, a checkpoint from a folder named GREEN-RadPhi2 (2.8B) ranks the drafts and GREEN-RadLlama2-7b (6.7B) scores the results. Either way, one family of judges, fitted mostly on chest X-ray reports, trains the model and examines it.

GREEN gains 12 to 23% of its starting value and ROUGE-1 4 to 9% derived. This is not the pattern of the design-space paper. There, canned normal sentences added 0.078 GREEN while RaTEScore moved 0.002 and BLEU fell 0.057: a gain only the judge saw. Here the model’s own text changes, and no score the paper prints goes down.

05What the paper leaves out

• Test sets and uncertainty: no split, no count, no decoding setting, no interval or repeat run, and one example report without its reference. The committed AMOS-MM script for µ²LLM stops after 100 validation cases. The lead over CT-CHAT is still called significant, though a reviewer asked for a test or another word.
• The recipe: what trains, for how long, how many preference pairs, and which β between 0.1 and 0.5 (0.1 in the scripts). The released training transform also rotates and flips volumes while the reports keep their lefts and rights our observation.
• The release: the tokenizer, both training stages and the pair-building scripts are public. The weights are three Qwen3-based successors, not the paper’s Llama-3.2-1B checkpoints, and the data are CT-RATE derivatives, not the preference pairs.

06Takeaways for building a CT report generator

1. DPO is a cheap way to put a judge in the loop. The paper reports 0.045 to 0.065 more GREEN with no reward model; the released recipe is eight samples per training example and one scoring pass.
2. If you train on a metric, add a ruler it doesn’t share. ROUGE-1 and METEOR followed GREEN, but all three compare text with the same reference; label F1 on CT-RATE’s 18 findings compares labels.
3. Decode greedily before and after. When drafts and test reports are both sampled at temperature 1.0, fewer bad draws and better reports look the same.
4. Train your baselines on your data. The same stack scores 0.009 GREEN here, run as published going by the scripts, and 0.365 in a study that trained it on AMOS-MM.

ASources

1. Li, S., Qin, P., Wu, H., Nie, D., Thirunavukarasu, A. J., Yu, J., Zhang, L. µ² Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation. MICCAI 2025, LNCS 15964, pp. 3–12. The page holds the reviews and the rebuttal. PDF · doi · arXiv:2507.00316: v1 still has the typos the reviewers flagged; v2 has the final text plus a section and an appendix on synthetic question–answer data. The table values are the same in all three.
2. Code: Siyou-Li/u2Tokenizer (MIT), read on 5 Oct 2026 at commit 71ed693; the per-dataset training and evaluation scripts are in the history, at commit 7d1ecb7 (2 Jul 2025). Weights and data: AlpachinoNLP on Hugging Face.
3. Ostmeier, S., et al. GREEN: arXiv:2405.03595, Findings of EMNLP 2024; checkpoints GREEN-RadLlama2-7b and GREEN-RadPhi2.
4. Rafailov, R., et al. Direct Preference Optimization: arXiv:2305.18290, NeurIPS 2023. Gao, L., et al. LinVT: arXiv:2412.05185 (2024). Bai, F., et al. M3D: arXiv:2404.00578 (2024).
5. Datasets: AMOS-MM challenge (Codabench); Hamamci, I. E., et al. CT-RATE and CT-CHAT: arXiv:2403.17834; Bassi, P. R. A. S., et al. RadGPT and AbdomenAtlas 3.0: arXiv:2501.04678.
6. Our posts on the 3D MLLM design space, Merlin and NV-Reason-CT.