Image Segmentation¶
Introduction¶
This section contains the process of partitioning an MR image into multiple segments(sets of pixels, i.e. Label Map). In this process a label is assigned to each pixel in an MR image so that pixels with the same label represent a specific structure (e.g. bone, ligament, cartilage, etc). These segments will be used to create the volumetric reconstruction of the desired tissue or bone.
To familiarize yourself with the segmentation software and procedure, first review Loading Data into Slicer, Segmentation Procedure, Label Map Smoothing and Generate Triangulated Surface (STL) from Label Map) sections then move on to Tissue Specific Procedure section for guidline on suggested steps for segmentation of bones, cartilage and soft tissues.
Prerequisite¶
3D Slicer: 3D Slicer is an open source software platform for medical image informatics, image processing, and three-dimensional visualization https://www.slicer.org/ .
Inputs¶
OpenKnees Data¶
Sets of MRI in NifTI format (.nii). Download the desired image from the desired specimen (click here for the download links). Selection of desired MRI set depends on the tissue you want to segment. Each of these MRI sets are suitable for visualization of specific anatomical structures of the knee.
- For segmentation of bones and cartilage, it is recommended to use Cartilage MRI (MRC) which is a 3D T1-weighted with fat suppression, anisotropic voxel size, gradient echo, with sagittal plane resolution of approximately 0.35 mm, and an out-of-plane resolution of approximately 0.7 mm. Start off the segmentation at the sagittal plane and refer to the coronal and axial planes to get better visualization of boundaries.
- For segmentation of ligaments, it is better to use Connective Tissue MRI (MRL) for insertion sites definition. Connective Tissue MRI consists of three image sets each corresponding to one of the anatomical planes (i.e. MRL-s:sagittal, MRL-c:coronal and MRL-a:axial). They are all Multi Echo Spin Echo (MESE) sequence with an in-plane resolution of approximately 0.35 mm, and an out-of-plane resolution of approximately 2.8 mm. Based on the desired ligament, you may start off the segmentation at sagittal plane or coronal plane window.
- General Purpose MRI (MRG) will be used for segmentation of registration markers. This MRI set is a 3D T1-weighted without fat suppression, isotropic voxel size with with 0.5 mm x 0.5 mm x 0.5 mm resolution.
Specimen/Sample Labeling protocols
Labeling protocols will be followed for each structure, which allows use of pre- and post-processing procedures to be generically applied after segmentation.
- oks003: Open Knee Specimen 3
- Bone:
- FMB: Femur
- TBB: Tibia
- FBB: Fibula
- PTB: Patella
- Cartilage:
- FMC: Femoral Cartilage
- TBC: Tibial Cartilage
- PTC: Patellar Cartilage
- Ligaments:
- amAcl: Anteromedial Bundle of Anterior Cruciate Ligamnet
- plAcl: Posterolateral Bundle of Anterior Cruciate Ligamnet
- alPcl: Anterolateral Bundle of Posterior Cruciate Ligament
- pmPcl: Posteromedial Bundle of Posterior Cruciate Ligament
- sMclProx: Proximal Fibers in Superficial Bundle of Medial Collateral Ligament
- sMclDist: Distal Fibers in Superficial Bundle of Medial Collateral Ligament
- dMcl: Deep Bundle of Medial Collateral Ligament
- lcl: Lateral Collateral Ligament
- all: Anterolateral Ligament
- pfl: Popliteofibular Ligament
- oplMid: Middle Portion of Oblique Popliteal Ligament
- pMFL: Posterior Meniscofemoral Ligament (Ligament of Wrisberg)
- aMFL: Anterior Meniscofemoral Ligament (Ligament of Humphrey)
- Mpfl: Medial Patellofemoral Ligament
- Lpfl: Lateral Patellofemoral Ligament
Loading Data into Slicer¶
Detailed procedures for loading the above described MRI data sets are given to help reduce the uncertainty for reproducibility across groups. Note the segmentation procedure, which applies to each tissue (Tissue Specific Procedure), is described below. In Slicer, to load MRI data in NifTI format use the following steps (for loading OpenKnees data):
- Click
data
button from the top toolbar section or go toFile > Add Data
. - Click
Choose File(s) to Add
, select desired MRI file (.nii) and clickOpen
. 3 volume slices (axial, sagittal and coronal) will appear in the slice viewers (red, yellow and green windows). you can choose between a number of predefined layouts of these viewer windows from the layout drop-down menu.- A slider is provided at the top of each viewer windows for manually going through slices and a slice number entry widget is also provided at the top right of each viewer for manually indexing a particular slice.
- You can change the brightness and contrast of the images by clicking the default mouse cursor and dragging up/down or left/right in any 2D viewer windows.
- Hover you mouse over the pin icon on the top left corner of the red slice viewer to display the slice viewer controller. Click on the chain icon to link the slice viewers. By doing so, the configuration options and interactions will apply across all slice viewers otherwise options apply only to the viewer being adjusted.
Load segmented label maps:
- Click
data
button from the top toolbar section or go toFile > Add Data
. - Click
Choose File(s) to Add
, select desired label map file (.nrrd or .nii). - Check the
Show Options
box on top right of the window. A couple of checkboxes will appear in front of the name of the file you have selected. For label maps always check theLabelMap
box. - Click
Open
.
Notes¶
Following points will be useful throughout the segmentation procedure.
1. Display the slice view controller window by moving your mouse over the pin icon on the top left corner of any of the slice viewer you want. The panel may be opened persistently by clicking on the pin icon.
- you will see 3 layers (
L,F,B
) specific options (If you can not see layers option click on the double arrow icon for showing all the options). - Layer labeled as “
L
” is the current Label Map Volume that you are editing. You can switch between available Label Maps using the drop-down menu. The visibility of the Label Map can be changed using the spinbox widget, from 0.0 (transparent) to 1.0 (opaque). You can toggle between showing the Label Map with regions outlined or filled using the square icon next to the spinbox. - Layer labeled as “
B
” is the Background Layer. In theEditor
module the background gray-scale is known as theMaster Volume
since it is used as the input forEditor
module tools (e.g.PaintEffect, DrawEffect
,etc). Note that you can load multiple MRI sets into Slicer (e.g. those that are cartilage specific or ligament specific). Any of these MR images can be used as the background layer, you can toggle between them using the background layer drop-down menu. For example one may wish to create the label map volume based on a high resolution MRI and then edit it while looking at a low resolution MRI. If you are doing so, make sure that you first create your label map based on theMaster Volume
you have already chose in theEditor
module. - Layer labeled as “
F
” is the Foreground Layer. Two label maps can be overlayed on the MRI in order to compare label map boundaries between two different structures. The label map you want to edit should be set as the label layer (L
). The other label map can be loaded as the foreground layer (F
). You can control the opacity of the Foreground label using the spinbox opacity widget next to it.
2. In the lower left corner of each slice viewer, three layers mentioned above and their assigned image/label are being displayed.
3. Crosshair
options are available for precise cross-correlation between slice viewers. Click on the Crosshair
drop-down menu in the toolbar. It provides a set of options for customizing the appearance
and behavior of the crosshair
across all slice viewer windows. Hold Shift
and move mouse in a slice view to position the Crosshair in all slice viewers.
4. It’s recommended to save your work frequently, to avoid repeating all the tedious work through program crashes. Click on Save
icon on the toolbar, a list of different files with different
names and formats will appear. By default, Slicer will select the scene and any modified data sets. You can change this selection using checkboxes in the leftmost column or through three shortcuts
icon on the top left of the panel. Make sure to change the directory of each file in the directory column. To save label map volumes you have to make sure to select the file with the same name as your
Merge Volume
in the Editor
module with .nrrd or .nii format.
Segmentation Procedure¶
Manual Segmentation¶
- After loading a volume (MR images) enter the
Editor
module from the toolbar.Editor
module is used for manipulating label map volumes. - If this is the first time entering
Editor
, you will be prompted to choose a color table. Select theGenericAnatomyColors
. If this is not the first time, the result of previous editing will be automatically selected when you enterEditor
module. - Under the
Create and Select Label Maps
tab, there is aMaster Volume
drop-down menu. The MRI set displaying in slice viewers is automatically selected as theMaster Volume
. You can change it using the drop-down menu. - If your background is named “x1”, the label will be automatically named “x1-label”. You can create a new one under the
Merge Volume
drop-down menu and then clickCreate a new LableMapVolume
. If you have loaded a previously segmented label map, make sure that the label map being edited in the slice viewers is the same as the Merge Volume. - Highlight the anatomy of interest in every slice, in each image orientation as appropriate (coronal, sagittal, axial).
- If you want to segment multiple structures from one MRI, it is better to create separate label maps for each of them. Before start highlighting the structure of interest, under
Create and Select Label Maps
tab and inPer-Structure Volumes
section click theAdd Structure
. You will be prompted to choose a color (label value) based on the Color Table selected when the Merge Volume was created. In such cases the Merge Volume will contain all the structures (label maps).
- If you want to segment multiple structures from one MRI, it is better to create separate label maps for each of them. Before start highlighting the structure of interest, under
- Different tools are available in Editor module to highlight anatomy of interest:
- Use the
Paint Effect
(underEdit Selected Label Map
tab, third icon from left, on the top row) with an appropriate/largest radius, to fill in the area using a brush. Select theSphere
checkbox to also applies the radius to slices above and below the current slice. In this case you have to be careful on your brush size since it will also paint in other slices and it may overlap with other structures. To avoid this issue try not to highlight areas very close to boundaries of anatomy of interest and after you are done highlighting, make sure to go through all slices and check for overlaps. In general, spherical brush is a handy tool for segmenting big structures (e.g. femur and tibia). It will help you saving a lot of time doing highlighting. - Use the eraser (under
Edit Selected Label Map
tab, second icon from left, on the top row) to change the behaviour of the tool, e.g. brush or pencil, to erase rather than draw. - Use the Draw Effect (under
Edit Selected Label Map
tab, forth icon from left, on the top row) to draw the boundaries of the area. By dragging mouse right click lay down a continuous line to outline the structure of interest (use “x” key to delete the last point added) and then left click to fill in the outline.
- Use the
Grow Cut Segmentation¶
- Perform first three steps of Manual Segmentation
- Choose the paint brush effect and highlight the anatomy of interest (L1) on roughly equally-spaced slices in each image plane orientation as appropriate (coronal, sagittal, axial)(Fig. 1)
- Under
Edit Selected Label Map
tab, change the label value (color) to highlight the area around the anatomy of interest (L2) on the slices you have also highlighted the anatomy itself(Fig. 1). - In an image plane orientation other than the one you are highlighting in, cap the area around the anatomy of interest to prevent L1 label leakage (Fig. 2).
- Click the
GrowCutEffect
(third icon from left on the bottom row). Make sure the L1 label (associated with the anatomy of interest) is selected under theLabel
drop-down menu and then hit theApply
button. Slicer window graphics will freeze until the grow cut algorithm is complete. - Remove the L2 label color (colors around the anatomy of interest) using
ChangeLabelEffect
. Choose the L2 label as the selected Label and also as theInput Color
and choose background label (black color) as theOutput Color
and click theApply
button.

Fig. 1 Grow Cut Segmentation.

Fig. 2 Grow Cut Segmentation, after performing label leakage control.
Notes¶
Some shortcuts in Editor
module:
- f , move to next slice
- b , move to previous slice
- z , undo
- y , redo
- Shift , scroll other slices to mouse location
- h , toggle the crosshair and put it into navigator mode
- e , toggles from the current color to 0 and back (use this while painting or drawing to quickly erase if you go over the line)
- c , show color picker
- Escape , cancel current effect
- p , select Paint effect
- d , select Draw effect
- w , select Wand effect
- r , select Rectangle effect
- Space , bring up tool selection window at current mouse location
- - or _ , shrink the current radius by 20%
- + or = , grow the current radius by 20%
Label Map Smoothing¶
Following smoothing step will be performed to remove the noises and outliers remaining from the grow cut segmentation step. This smoothing procedure is on the voxel level and will preserve the volume.
- From the
Modules
drop-down menu, selectSurface Models > Label Map Smoothing
. - Set Gaussian Smoothing Parameters, Sigma based on image resolution (e.g. 0.7 for cartilage MRI, resolution = 0.35 x 0.35 x 0.7 mm).
- From
Input Volume
drop-down menu, select desired label map. - From
Output Volume
drop-down menu, selectCreate new Volume as...
. - Enter new name as desired, click
OK
. - Click
Apply
.
Generate Triangulated Surface (STL) from Label Map¶
- From the
Modules
drop-down menu, selectSurface Models > Model Maker
. - Under
IO
tab, fromInput Volume
drop-down menu, select desired label map. - In
Models
drop-down menu, eitherCreate new ModelHierarchy as...
(e.g. default is ‘Models’), or select an existing Model Hierarchy. All models can be generated under the same or separate model hierarchies. - Under
Create Multiple
tab, specify a desiredModel Name
for the selected label map. - For creating raw STL models:
- Under
Model Maker Parameters
tab, setSmooth
to zero. - Under
Model Maker Parameters
tab, setDecimate
to zero.
- Under
- Click Apply button.
- Enter the
Editor
module and select theMakeModelEffect
(second icon from right on the bottom row) underEdit Selected Label Map
tab. - Set desired label map as the
Merge Volume
. If no smoothing is desired, deselectSmooth Model
. - Click Apply button.
- A 3D model of the label map will appear on the 3D view window. Roll your mouse over the pushpin icon on the blue bar across top of the 3D viewer window to open the configuration panel.
- Click on the
Save
icon on the toolbar. By default, 3D models will be saved with .vtk format. you can change it from theFile Format
drop-down menu in front of the file name. Make sure to choose the desired directory for saving the file.
Output¶
Output of the segmentation procedure is the triangulated surface representation of the tissue of the interest (.STL) in the raw and unfiltered format.
Tissue Specific Procedure¶
In the following sections segmentation procedure for bones, cartilage and soft tissues will be specifically described.
Bone & Cartilage Segmentation Procedure¶
Input: Cartilage MRI
Procedure:
- Perform Grow Cut Segmentation on desired bone.
- Perform Label Map Smoothing to remove Grow Cut boundary noise.
- Perform Manual Segmentation to more accurately define bone/cartilage boundary. It will be helpful to overlay the General Purpose MRI (as the Foreground label) on the Cartilage MRI in order to get a more accurate boundary definition.
- Iteratively repeat last two steps until boundary is as desired.
It is better to perform cartilage segmentation along with the bone segmentation. This way you can use bone boundaries to define cartilage boundaries, just remember to define a separate label map volume for the cartilage.
Soft Tissue Segmentation Procedure¶
Input: Connective Tissue MRI (MRL)
Procedure:
- Load sagittal ligament MRI
- Perform Manual Segmentation
- Perform Label Map Smoothing
- In order to have a high resolution segmentation volume from a low resolution MRI, one can open Cartilage MRI and Connective Tissue MRI in different viewer windows. When windows are linked Slicer
uses interpolation for coupled viewing and you can use Cartilage MRI as the
Master Volume
and segment the soft tissue on the interpolated MRI. - Alternatively, You can segment the tissue on the Connective Tissue MRI and then lay the label map over the Cartilage MRI (as the
Foreground Layer
, make sure to increase the opacity, the default is zero) Create a new label map as theMerge Volume
and start to segment the soft tissue. Use the Connective Tissue MRI label map as a guideline for the boundaries of the tissue.