Slice-by-slice segmentation: 3D demo

The lesion is one elongated synthetic shape cut into 2 mm slices, as in a CT or MRI series. A 2D model sees each slice on its own. Where it finds the lesion it follows the outline closely, with a small inconsistency from slice to slice that the slider controls. In a few slices it finds nothing, which in practice happens where the lesion is faint, blurred by motion or unlike the training examples.

The 3D view stacks the slices: blue where the model found the lesion and amber where it missed it. Dice on the slices it segments is the figure a slice-level evaluation reports. The whole-volume Dice counts the missed slices too. The other two readouts are what a clinician would take from the mask: how many separate lesions there are, counted as connected pieces, and the longest diameter of the largest piece.

A gap of one slice is enough to split a lesion in two. Counts, longest-diameter measurements and volumes are then wrong even though most voxels are right, and those are the numbers that go into response criteria and follow-up. That is why 3D segmentation should be evaluated in 3D: volume overlap, the number of connected lesions against the reference, and the error of each derived measurement. See medical imaging AI.

Picture a team that trains a 2D network to outline lesions on CT, one axial slice at a time. They evaluate it the way it was trained: Dice per slice, averaged over the slices that contain a prediction. The figure is excellent, and the model goes on to produce lesion counts and diameters for a study.

The demo’s default settings show one such lesion: 30 mm long, in 15 slices of 2 mm, with 5% inconsistency between slices and 2 slices where the model finds nothing. Dice on the slices it segments reads 0.98. Dice on the whole volume reads 0.88, which still sounds acceptable.

The other two readouts do not. Lesions reported reads 3 for 1: the two missed slices cut the lesion into three pieces, and any count of lesions is now wrong. Longest diameter reads 20 of 30 mm, because the largest piece is much shorter than the lesion. The line under the readouts adds that the measured volume is 4.3 cm³ against a true 5.1 cm³, an error of −16%.

Set the missed slices to 0 and the problem disappears: volume Dice is 0.98, one lesion is reported and the diameter is 30 of 30 mm. Set it to 1 and the lesion is already split in two, with a volume Dice of 0.93. With 3 missed slices there are 4 pieces and the volume Dice is 0.82. Throughout, Dice on the slices the model segments stays at 0.98.

The correct practice is to evaluate a 3D task in 3D. Report overlap on the whole volume, compare the number of connected lesions with the reference, and report the error of every measurement the mask is used for, such as longest diameter and volume. Where a 2D model is used, add a step that enforces consistency between slices, and test on the cases where slices are hardest: thin ends, motion and low contrast.

• Is overlap reported on the whole volume, not only averaged over slices?
• Are slices with no prediction counted, or silently left out of the average?
• Is the number of connected lesions compared with the reference for each case?
• Are the derived measurements, such as longest diameter and volume, evaluated directly?
• Does the pipeline enforce consistency between neighbouring slices?
• Are results broken down by lesion size and by scan quality?

Related: our medical imaging AI service, the wiki page on medical AI validation, and the demos on Dice score vs clinical error and nodule volume and slice thickness.

Why does my 2D segmentation model give fragmented 3D lesions?

A 2D model decides each slice on its own, so nothing prevents it from finding the lesion in one slice and missing it in the next. A single missed slice splits one lesion into two connected pieces. In this demo 2 missed slices out of 15 turn one lesion into 3, while the Dice on the slices it does segment stays at 0.98.

Should segmentation be evaluated per slice or per volume?

Per volume, and per lesion, for anything that will be used in 3D. A per-slice average can leave out the slices where the model found nothing and cannot show whether the pieces connect. In the demo the per-slice figure is 0.98, the whole-volume Dice is 0.88, and the lesion count and longest diameter are both wrong.

How do I check a 3D segmentation for lesion count and diameter errors?

Label the connected components of the predicted and reference masks, match them, and compare the counts. Then compute each clinical measurement from both masks and report the difference. In the demo the longest diameter is 20 mm from the prediction against 30 mm from the reference, and the volume is 16% too small.

Is a 3D network always better than a 2D network for segmentation?

Not always. 3D networks use context between slices, which helps consistency, but they need more memory and more training data, and thick or uneven slice spacing can limit their benefit. A 2D or 2.5D model with a consistency step afterwards can work well. Whichever is used, the evaluation should be volumetric.

What is the difference between slice-level Dice and volume Dice?

Slice-level Dice is computed on each 2D slice and then averaged, often only over slices that contain a mask. Volume Dice is computed once over all voxels of the 3D mask. They differ most when whole slices are missed. In the demo the first is 0.98 and the second is 0.88.