Artificial Intelligence in Radiology: nnU-Net Segmentation

Why nnU-Net is still the strongest reference

Published in 2018 by the Heidelberg team (Fabian Isensee), nnU-Net is a "self-configuring" U-Net type framework. It analyzes the spacing, voxel intensities, anatomical structure, and sample count of the dataset, and automatically designs the most appropriate architecture configuration. The user does not perform manual hyperparameter tuning.

From 2018 to 2026, nnU-Net has consistently ranked in the top three in large competitions such as the Medical Segmentation Decathlon, KiTS, BraTS, AMOS, and FLARE. The 2024 nnU-Net v2 release came with a PyTorch native design, a ResEncoder L (large residual encoder model) option, and faster data loading. As of 2026, it remains the reference baseline for medical segmentation.

In this article I explain where nnU-Net is strong and where it is weak, how it compares with modern alternatives (MONAI, SwinUNETR, TotalSegmentator), and how to design a TRIPOD-AI compliant pipeline for clinical validation.

The four configurations of nnU-Net

nnU-Net v2 has four basic configurations:

2d: Slice-by-slice 2D U-Net. Fast, low GPU. But the 3D context is lost. It may miss thin structures (for example, vessels, nerves).

3d_fullres: Full resolution 3D U-Net. The original voxel spacing is used. Highest accuracy, but GPU memory is expensive. Typically 12-24 GB VRAM is required.

3d_lowres: Low resolution 3D U-Net. For roughly localizing large structures. It is not used on its own; it is the first stage of the cascade.

3d_cascade_fullres: Two stages. First 3d_lowres finds the region, then 3d_fullres performs detailed segmentation. The most accurate result on large volumes (for example, whole-body CT). Training time is doubled.

Practical recommendation: First set up a baseline with 2d (you will see results in 1-2 hours), then move to 3d_fullres. Use the cascade only if the data is large (>10 GB per voxel) and 3d_fullres exceeds the memory budget.

Data preparation: the most frequently skipped step

Running nnU-Net is simple, but if the data preparation is wrong, the model learns in vain. Three critical steps:

1. Format conversion. nnU-Net expects NIfTI (.nii.gz). dcm2niix is used to convert from DICOM. During conversion, orientation (LPS / RAS / RAI) must be preserved. Wrong orientation causes the model to confuse left and right.

2. Label numbering. Labels must start from 0 and be consecutive (0=background, 1=organ1, 2=organ2). There must be no empty label number. Wrong: 0, 1, 2, 5. Correct: 0, 1, 2, 3.

3. dataset.json. nnU-Net requires a JSON for the dataset metadata. Channel IDs, label names, and the number of training cases are written in it. Training cannot start until this file is written correctly.

Typical mistakes:

• Inconsistent voxel spacing (mixed 1mm vs 0.5mm): nnU-Net resamples, but information is lost
• Not cropping (when only the pelvis is needed from a whole-body CT)
• Label imbalance (for example, a tumor segmentation with a 0.1% ratio): the loss function needs to be adjusted

2D vs 3D decision matrix

Which configuration and when?

The 2D fallacy: "2D is fast, so I will do 2D too." If the anatomical structure is connected in 3D (for example, a vessel tree, an organ contour), using 2D breaks the continuity of the segmentation. Consistency between slices is lost.

The 3D fallacy: "3D is always better." If your GPU budget is tight (8 GB VRAM) or the data is a 2D modality (for example, X-ray, fundus retina photo), 3D is an excessive cost.

Transfer learning and foundation models

Since 2024, the transfer learning paradigm for medical imaging has changed. There are three layers:

Old approach: ImageNet pre-trained. ResNet, EfficientNet trained on natural images. It transfers to medical images, but the domain gap is large.

Modern: medical foundation model. MedSAM, RadImageNet, MONAI Bundles. Pre-trained with millions of examples on medical images. It gives good results with little data (50-200 cases).

State-of-the-art: SAM-based interactive segmentation. SAM (Segment Anything Model) and SAM-Med2D / SAM-Med3D. Semi-automatic segmentation with a click or a bounding box. It speeds up the ground truth creation process.

As of 2024, nnU-Net added the ResEnc-L (residual encoder large) variant. There is also a pre-training option (MAE based). On small datasets (n=50-100), these variants can give a Dice score 2-5% better than standard nnU-Net.

MONAI and alternative frameworks

The alternative to nnU-Net is MONAI (Medical Open Network for AI). A partnership between NVIDIA and King's College. Its advantages:

• Modular (you add your own loss function, metric, transform)
• MONAI Bundle: pre-trained model packages
• MONAI Label: active annotation tool (3D Slicer integration)
• MONAI Deploy SDK for deployment

Comparison:

Practical recommendation: nnU-Net for research and baseline. MONAI for production deployment and custom pipelines. The two can also be used together (train with nnU-Net, deploy with MONAI).

TotalSegmentator (Wasserthal 2023), on the other hand, is a ready-made model that recognizes 117 anatomical structures for CT. For general anatomy segmentation there is no need for retraining; it is used directly.

Detection and classification

For tasks other than segmentation:

Detection (lesion detection):

• nnDetection (Baumgartner 2021): the detection version of nnU-Net, RetinaNet based
• MONAI Detection: 3D RetinaNet, deformable DETR
• Yolo3D variants: fast but medical accuracy is low

Classification (for example, malignant vs benign):

• 3D ResNet, 3D DenseNet
• MedMNIST pre-trained models
• ViT (Vision Transformer) variants
• SwinUNETR-Cls

The most frequent mistake in classification: using a 3D ViT with a small dataset (below n=500). ViTs are data-hungry; without 1000+ examples they cannot surpass ResNet.

TRIPOD-AI compliant validation

You developed a model, Dice 0.92. Is it publishable? The answer: without validation, no.

TRIPOD-AI (Collins 2024, BMJ) is a reporting standard for clinical prediction models. 27 items for an AI model paper:

• Separation of development cohort + validation cohort
• Internal validation (cross-validation or bootstrap)
• Temporal validation (data collected from a date after training)
• External validation (different center, different scanner, different population)
• Calibration analysis (is the model's probability estimate reliable)
• Subgroup analysis (performance by age, sex, ethnicity)
• Failure mode analysis (where the model makes mistakes, and why)

Typical nnU-Net papers report only internal cross-validation. TRIPOD-AI finds this insufficient. The minimum for clinical use:

1. Discovery cohort: train + 5-fold cross-validation
2. Temporal validation: same center, data from 6 months later
3. External validation: at least 1 different center, a different scanner

If external validation is skipped, the model falls into the "internal success + external failure" trap. The Aetherly 2024 systematic review showed that 43% of medical segmentation models dropped 0.15+ in Dice on external validation. To see how these validation steps are brought together as regulatory evidence, you can also review how the SaMD clinical-evidence study is structured.

DECIDE-AI: an additional standard for clinical deployment

TRIPOD-AI standardizes the reporting. DECIDE-AI (Vasey 2022, BMJ), on the other hand, proposes 27 items for clinical deployment evaluation. Testing early-stage AI on real physicians in the clinic.

• How often does the physician accept / change the AI recommendation
• Decision-making time without AI vs with AI
• When the AI makes a mistake, does the physician catch it
• Was the final clinical decision influenced by the AI or the physician

An nnU-Net model may come out "good," but if the physician does not use it or if it leads to wrong decisions, there is no clinical benefit. DECIDE-AI fills this gap.

Dataset and KVKK

In Turkey, the curation of medical artificial intelligence datasets is two-sided:

Technical: DICOM anonymization (PHI removal, defacing), patient ID hashing, study UID re-generation. The pydicom + dicom-anonymizer libraries are standard.

Legal: Patient explicit consent or ethics committee approval. Under KVKK, health data is classified as "special category personal data." A DPA (data processing agreement) is mandatory between the data controller (hospital) and the processor (the team developing the model).

Federated learning alternative: The data is not moved, the model trains locally at each center, and only the weights are combined. The NVIDIA Clara and OpenFL frameworks exist. A KVKK-friendly option for multi-center studies.

Practical core checklist

For a medical artificial intelligence project to proceed soundly:

• Was orientation preserved in the DICOM to NIfTI conversion
• Is the label numbering consecutive starting from 0
• Is the train/val/test split at the case level (not the slice level)
• Was 5-fold cross-validation done
• Is there temporal validation
• External validation at least 1 center
• Was calibration analysis done
• Subgroup analysis (age, sex)
• Was the TRIPOD-AI reporting checklist applied
• Was the DICOM anonymization protocol verified
• Was a KVKK-compliant DPA signed
• Was a model card prepared (intended use, limitations)

Three commonly made mistakes

Mistake 1: Data leak. If different slices of the same patient fall into train and val, the model gives a high score because "I know this patient," and external validation crashes. The split must always be done at the patient level.

Mistake 2: Excessive augmentation. The nnU-Net default augmentations (rotation, scaling, mirror) are sufficient for most domains. Adding more augmentation often produces over-regularization.

Mistake 3: Focusing on a single metric. Reporting only the Dice score is not enough. Hausdorff Distance (boundary accuracy), Surface Dice (clinically meaningful boundary), and sensitivity / specificity should be reported separately. In tumor segmentation the Dice may be 0.88, but if the boundary shift is 5mm it cannot be used clinically.

Serteser Danismanlik support for medical artificial intelligence

Developing a medical artificial intelligence model rests on two wings, the technical and the methodological. Serteser Danismanlik offers support on both wings:

• nnU-Net pipeline setup, hyperparameter guidance
• MONAI Bundle integration
• DICOM anonymization protocol and dataset curation
• TRIPOD-AI and DECIDE-AI compliant validation design
• External multi-center validation study management
• Methodology section for ethics committee applications
• AI methodology for TUBITAK 1001/1002 applications

With a research infrastructure that jointly establishes the technical + methodological disciplines, backed by international peer-reviewed clinical research experience published in an international peer-reviewed journal and active systematic review management, we provide end-to-end support in medical artificial intelligence projects.

If your organization has a similar artificial intelligence or data engineering need, we can evaluate it together within the scope of professional consulting.