Image Segmentation

Introduction

This document contains the process of partitioning an MR image into multiple segments(sets of pixels, i.e. Label Map). In this process a label is assigned to each pixel in an MR image so that pixels with the same label represent a specific structure (e.g. bone, ligament, cartilage, etc). These segments will be used to create the volumetric reconstruction of the desired tissue or bone.

To familiarize yourself with the segmentation software and procedure, first review Loading Data into Slicer, Segmentation Procedure, Label Map Smoothing and Generate Triangulated Surface (STL) from Label Map) sections then move on to Tissue Specific Procedure section for guidline on suggested steps for segmentation of bones, cartilage and soft tissues.

Prerequisite

3D Slicer: 3D Slicer is an open source software platform for medical image informatics, image processing, and three-dimensional visualization https://www.slicer.org/ .

Inputs

OpenKnees Data

Sets of MRI in NifTI format (.nii). Download the desired image from the desired specimen (click here for the download links). Selection of desired MRI set depends on the tissue you want to segment. Each of these MRI sets are suitable for visualization of specific anatomical structures of the knee.

  • For segmentation of bones and cartilage, it is recommended to use Cartilage MRI (MRC) which is a 3D T1-weighted with fat suppression, anisotropic voxel size, gradient echo, with sagittal plane resolution of approximately 0.35 mm, and an out-of-plane resolution of approximately 0.7 mm. Start off the segmentation at the sagittal plane and refer to the coronal and axial planes to get better visualization of boundaries.
  • For segmentation of ligaments, it is better to use Connective Tissue MRI (MRL) for insertion sites definition. Connective Tissue MRI consists of three image sets each corresponding to one of the anatomical planes (i.e. MRL-s:sagittal, MRL-c:coronal and MRL-a:axial). They are all Multi Echo Spin Echo (MESE) sequence with an in-plane resolution of approximately 0.35 mm, and an out-of-plane resolution of approximately 2.8 mm. Based on the desired ligament, you may start off the segmentation at sagittal plane or coronal plane window.
  • General Purpose MRI (MRG) will be used for segmentation of registration markers. This MRI set is a 3D T1-weighted without fat suppression, isotropic voxel size with with 0.5 mm x 0.5 mm x 0.5 mm resolution.

Denver Group Data

One set of DICOM file (.dcm files) including 112 slices of 3D T2-weighted MRI with fat suppression and sagittal plane resolution of 0.35 mm. Download the Denver group data from the following link: https://simtk.org/frs/?group_id=1061

Specimen/Sample Labeling protocols

Labeling protocols will be followed for each structure and will be consistent between the OpenKnees and the Denver data sets, which allows use of pre- and post-processing procedures to be generically applied after segmentation.

  • oks001: Open Knee Specimen 1
  • DU02: Denver Group Specimen 2

  • Bone:
    • FMB: Femur
    • TBB: Tibia
    • FBB: Fibula
    • PTB: Patella

  • Cartilage:
    • FMC: Femoral Cartilage
    • TBC: Tibial Cartilage
    • PTC: Patellar Cartilage

  • Ligaments:
    • amAcl: Anteromedial Bundle of Anterior Cruciate Ligamnet
    • plAcl: Posterolateral Bundle of Anterior Cruciate Ligamnet
    • alPcl: Anterolateral Bundle of Posterior Cruciate Ligament
    • pmPcl: Posteromedial Bundle of Posterior Cruciate Ligament
    • sMclProx: Proximal Fibers in Superficial Bundle of Medial Collateral Ligament
    • sMclDist: Distal Fibers in Superficial Bundle of Medial Collateral Ligament
    • dMcl: Deep Bundle of Medial Collateral Ligament
    • lcl: Lateral Collateral Ligament
    • all: Anterolateral Ligament
    • pfl: Popliteofibular Ligament
    • oplMid: Middle Portion of Oblique Popliteal Ligament
    • pMFL: Posterior Meniscofemoral Ligament (Ligament of Wrisberg)
    • aMFL: Anterior Meniscofemoral Ligament (Ligament of Humphrey)
    • Mpfl: Medial Patellofemoral Ligament
    • Lpfl: Lateral Patellofemoral Ligament

Loading Data into Slicer

Detailed procedures for loading the above described MRI data sets are given to help reduce the uncertainty for reproducibility across groups. Note the segmentation procedure, which applies to each tissue (Tissue Specific Procedure), is described below. In Slicer, to load MRI data in NifTI format use the following steps (for loading OpenKnees data):

  • Click data button from the top toolbar section or go to File > Add Data.
  • Click Choose File(s) to Add, select desired MRI file (.nii) and click Open. 3 volume slices (axial, sagittal and coronal) will appear in the slice viewers (red, yellow and green windows). you can choose between a number of predefined layouts of these viewer windows from the layout drop-down menu.
    • A slider is provided at the top of each viewer windows for manually going through slices and a slice number entry widget is also provided at the top right of each viewer for manually indexing a particular slice.
    • You can change the brightness and contrast of the images by clicking the default mouse cursor and dragging up/down or left/right in any 2D viewer windows.
    • Hover you mouse over the pin icon on the top left corner of the red slice viewer to display the slice viewer controller. Click on the chain icon to link the slice viewers. By doing so, the configuration options and interactions will apply across all slice viewers otherwise options apply only to the viewer being adjusted.

To load DICOM files (for loading Denver data):

  • Click File > DICOM.
  • On the top left of the window click on the Import and navigate to the desired dictory and hit Import. Yo can import multiple sets of DICOM data in to the slicer.
  • Select the desired DICOM file in the DICOM browser and click load.
  • To re-open the DICOM browser, from the Modules drop-down menu go to DICOM module and click on Show DICOM Browser.

Load segmented label maps:

  • Click data button from the top toolbar section or go to File > Add Data.
  • Click Choose File(s) to Add, select desired label map file (.nrrd or .nii).
  • Check the Show Options box on top right of the window. A couple of checkboxes will appear in front of the name of the file you have selected. For label maps always check the LabelMap box.
  • Click Open.

Notes

Following points will be useful throughout the segmentation procedure.

1. Display the slice view controller window by moving your mouse over the pin icon on the top left corner of any of the slice viewer you want. The panel may be opened persistently by clicking on the pin icon.

  • you will see 3 layers (L,F,B) specific options (If you can not see layers option click on the double arrow icon for showing all the options).
  • Layer labeled as “L” is the current Label Map Volume that you are editing. You can switch between available Label Maps using the drop-down menu. The visibility of the Label Map can be changed using the spinbox widget, from 0.0 (transparent) to 1.0 (opaque). You can toggle between showing the Label Map with regions outlined or filled using the square icon next to the spinbox.
  • Layer labeled as “B” is the Background Layer. In the Editor module the background gray-scale is known as the Master Volume since it is used as the input for Editor module tools (e.g. PaintEffect, DrawEffect,etc). Note that you can load multiple MRI sets into Slicer (e.g. those that are cartilage specific or ligament specific). Any of these MR images can be used as the background layer, you can toggle between them using the background layer drop-down menu. For example one may wish to create the label map volume based on a high resolution MRI and then edit it while looking at a low resolution MRI. If you are doing so, make sure that you first create your label map based on the Master Volume you have already chose in the Editor module.
  • Layer labeled as “F” is the Foreground Layer. Two label maps can be overlayed on the MRI in order to compare label map boundaries between two different structures. The label map you want to edit should be set as the label layer (L). The other label map can be loaded as the foreground layer (F). You can control the opacity of the Foreground label using the spinbox opacity widget next to it.

2. In the lower left corner of each slice viewer, three layers mentioned above and their assigned image/label are being displayed.

3. Crosshair options are available for precise cross-correlation between slice viewers. Click on the Crosshair drop-down menu in the toolbar. It provides a set of options for customizing the appearance and behavior of the crosshair across all slice viewer windows. Hold Shift and move mouse in a slice view to position the Crosshair in all slice viewers.

4. It’s recommended to save your work frequently, to avoid repeating all the tedious work through program crashes. Click on Save icon on the toolbar, a list of different files with different names and formats will appear. By default, Slicer will select the scene and any modified data sets. You can change this selection using checkboxes in the leftmost column or through three shortcuts icon on the top left of the panel. Make sure to change the directory of each file in the directory column. To save label map volumes you have to make sure to select the file with the same name as your Merge Volume in the Editor module with .nrrd or .nii format.

Segmentation Procedure

Manual Segmentation

  • After loading a volume (MR images) enter the Editor module from the toolbar. Editor module is used for manipulating label map volumes.
  • If this is the first time entering Editor, you will be prompted to choose a color table. Select the GenericAnatomyColors. If this is not the first time, the result of previous editing will be automatically selected when you enter Editor module.
  • Under the Create and Select Label Maps tab, there is a Master Volume drop-down menu. The MRI set displaying in slice viewers is automatically selected as the Master Volume. You can change it using the drop-down menu.
  • If your background is named “x1”, the label will be automatically named “x1-label”. You can create a new one under the Merge Volume drop-down menu and then click Create a new LableMapVolume. If you have loaded a previously segmented label map, make sure that the label map being edited in the slice viewers is the same as the Merge Volume.
  • Highlight the anatomy of interest in every slice, in each image orientation as appropriate (coronal, sagittal, axial).
    • If you want to segment multiple structures from one MRI, it is better to create separate label maps for each of them. Before start highlighting the structure of interest, under Create and Select Label Maps tab and in Per-Structure Volumes section click the Add Structure . You will be prompted to choose a color (label value) based on the Color Table selected when the Merge Volume was created. In such cases the Merge Volume will contain all the structures (label maps).
  • Different tools are available in Editor module to highlight anatomy of interest:
    • Use the Paint Effect (under Edit Selected Label Map tab, third icon from left, on the top row) with an appropriate/largest radius, to fill in the area using a brush. Select the Sphere checkbox to also applies the radius to slices above and below the current slice. In this case you have to be careful on your brush size since it will also paint in other slices and it may overlap with other structures. To avoid this issue try not to highlight areas very close to boundaries of anatomy of interest and after you are done highlighting, make sure to go through all slices and check for overlaps. In general, spherical brush is a handy tool for segmenting big structures (e.g. femur and tibia). It will help you saving a lot of time doing highlighting.
    • Use the eraser (under Edit Selected Label Map tab, second icon from left, on the top row) to change the behaviour of the tool, e.g. brush or pencil, to erase rather than draw.
    • Use the Draw Effect (under Edit Selected Label Map tab, forth icon from left, on the top row) to draw the boundaries of the area. By dragging mouse right click lay down a continuous line to outline the structure of interest (use “x” key to delete the last point added) and then left click to fill in the outline.

Grow Cut Segmentation

  • Perform first three steps of Manual Segmentation
  • Choose the paint brush effect and highlight the anatomy of interest (L1) on roughly equally-spaced slices in each image plane orientation as appropriate (coronal, sagittal, axial)(Fig. 1)
  • Under Edit Selected Label Map tab, change the label value (color) to highlight the area around the anatomy of interest (L2) on the slices you have also highlighted the anatomy itself(Fig. 1).
  • In an image plane orientation other than the one you are highlighting in, cap the area around the anatomy of interest to prevent L1 label leakage (Fig. 2).
  • Click the GrowCutEffect (third icon from left on the bottom row). Make sure the L1 label (associated with the anatomy of interest) is selected under the Label drop-down menu and then hit the Apply button. Slicer window graphics will freeze until the grow cut algorithm is complete.
  • Remove the L2 label color (colors around the anatomy of interest) using ChangeLabelEffect. Choose the L2 label as the selected Label and also as the Input Color and choose background label (black color) as the Output Color and click the Apply button.
alternate text

Fig. 1 Grow Cut Segmentation.

alternate text

Fig. 2 Grow Cut Segmentation, after performing label leakage control.

Notes

Some shortcuts in Editor module:

  • f , move to next slice
  • b , move to previous slice
  • z , undo
  • y , redo
  • Shift , scroll other slices to mouse location
  • h , toggle the crosshair and put it into navigator mode
  • e , toggles from the current color to 0 and back (use this while painting or drawing to quickly erase if you go over the line)
  • c , show color picker
  • Escape , cancel current effect
  • p , select Paint effect
  • d , select Draw effect
  • w , select Wand effect
  • r , select Rectangle effect
  • Space , bring up tool selection window at current mouse location
  • - or _ , shrink the current radius by 20%
  • + or = , grow the current radius by 20%

Label Map Smoothing

Following smoothing step will be performed to remove the noises and outliers remaining from the grow cut segmentation step. This smoothing procedure is on the voxel level and will preserve the volume.

  • From the Modules drop-down menu, select Surface Models > Label Map Smoothing.
  • Set Gaussian Smoothing Parameters, Sigma based on image resolution (e.g. 0.7 for cartilage MRI, resolution = 0.35 x 0.35 x 0.7 mm).
  • From Input Volume drop-down menu, select desired label map.
  • From Output Volume drop-down menu, select Create new Volume as....
  • Enter new name as desired, click OK.
  • Click Apply.

Generate Triangulated Surface (STL) from Label Map

  • From the Modules drop-down menu, select Surface Models > Model Maker.
  • Under IO tab, from Input Volume drop-down menu, select desired label map.
  • In Models drop-down menu, either Create new ModelHierarchy as... (e.g. default is ‘Models’), or select an existing Model Hierarchy. All models can be generated under the same or separate model hierarchies.
  • Under Create Multiple tab, specify a desired Model Name for the selected label map.
  • For creating raw STL models:
    • Under Model Maker Parameters tab, set Smooth to zero.
    • Under Model Maker Parameters tab, set Decimate to zero.
  • Click Apply button.
  • Enter the Editor module and select the MakeModelEffect (second icon from right on the bottom row) under Edit Selected Label Map tab.
  • Set desired label map as the Merge Volume. If no smoothing is desired, deselect Smooth Model.
  • Click Apply button.
  • A 3D model of the label map will appear on the 3D view window. Roll your mouse over the pushpin icon on the blue bar across top of the 3D viewer window to open the configuration panel.
  • Click on the Save icon on the toolbar. By default, 3D models will be saved with .vtk format. you can change it from the File Format drop-down menu in front of the file name. Make sure to choose the desired directory for saving the file.

Output

Output of the segmentation procedure is the triangulated surface representation of the tissue of the interest (.STL) in the raw and unfiltered format.

Tissue Specific Procedure

In the following sections segmentation procedure for bones, cartilage and soft tissues will be specifically described.

Bone & Cartilage Segmentation Procedure

Input: Cartilage MRI

Procedure:

  • Perform Grow Cut Segmentation on desired bone.
  • Perform Label Map Smoothing to remove Grow Cut boundary noise.
  • Perform Manual Segmentation to more accurately define bone/cartilage boundary. It will be helpful to overlay the General Purpose MRI (as the Foreground label) on the Cartilage MRI in order to get a more accurate boundary definition.
  • Iteratively repeat last two steps until boundary is as desired.

It is better to perform cartilage segmentation along with the bone segmentation. This way you can use bone boundaries to define cartilage boundaries, just remember to define a separate label map volume for the cartilage.

Soft Tissue Segmentation Procedure

Input: Connective Tissue MRI (MRL)

Procedure:

  • Load sagittal ligament MRI
  • Perform Manual Segmentation
  • Perform Label Map Smoothing
  • In order to have a high resolution segmentation volume from a low resolution MRI, one can open Cartilage MRI and Connective Tissue MRI in different viewer windows. When windows are linked Slicer uses interpolation for coupled viewing and you can use Cartilage MRI as the Master Volume and segment the soft tissue on the interpolated MRI.
  • Alternatively, You can segment the tissue on the Connective Tissue MRI and then lay the label map over the Cartilage MRI (as the Foreground Layer, make sure to increase the opacity, the default is zero) Create a new label map as the Merge Volume and start to segment the soft tissue. Use the Connective Tissue MRI label map as a guideline for the boundaries of the tissue.