Next: 9. Tissue Classification Up: 2 B. Anatomical Image Previous: 7. The Image Processing Contents
Subsections
- 8.1 Introduction
- 8.2 The Objectmap Editor Tools
- 8.3 The Surface Editor
- 8.3.1 Key Concepts
- 8.3.2 The Main Surface Editor Window
- 8.3.3 Objectmaps and the Objectmap Menu
- 8.3.4 The Spline/Objectmap Control
- 8.4 Delineating a surface: a step-by-step recipe.
8. The Interactive Segmentation Tools
8.1 Introduction
Image Analysis is the process of extracting quantitative information from images. Unlike image processing, this information is not an image per se, but rather a description of some aspect of the image. Examples of such information include the volume of gray matter in the brain, the volume of the left hippocampus, the volume of fat in the abdomen, the surface area of a given sulcus etc. Segmentation of extracting a structure or group of voxels from an image. It usually means one of two things:
Voxel Classification: The main applications of these techniques is the labeling of each voxel in the image according to tissue classes. Typical applications of these methods are the classification of brain tissue into gray matter, white matter and cerebrospinal fluid (CSF) in neuroimaging and fat/lean muscle classification in abdominal imaging. The output of this type of method is an image, or an objectmap, with labels (e.g. 0,1,2,3) each representing a different type of tissue.
Deformable Surface Segmentation: Deformable surfaces either using explicit parameterizations or levelset formulations are applicable to all parts of the body as a method for measuring the shape and volume of different organs and sub-cortical brain structures. In addition to automated methods, there are also applications of manual tracing and semi-automated methods in many practical applications both for complete segmentation or for the review/correction by an expert user of the results of automated segmentation such as in the BioImage Suite surface editor tool. The output of these types of methods are surfaces which bound structures of interest, e.g. the surface of the hippocampus.
There is a large degree of overlap between these two methods. For example, structures such as the hippocampus can be segmented either by painting all voxels belonging to the interior of the hippocampus with a certain color - this is the voxel classification approach, or by defining the outer surface that encompasses this structure. We can also often, with some loss of resolution, convert from one description (e.g. the surface) to another (e.g. the objectmap image).
8.1.0.0.1 Objectmaps:
In BioImage Suite we will use the word objectmap to mean an image which has a finite number of intensities (e.g. 1-10) where each intensity value represents a unique object or tissue class. In other words, Objectmaps are essentially images (and are stored in the same file format as regular images) whose intensities have special meaning.
8.1.0.0.2 Surfaces:
BioImage Suite stores surfaces using two major file formats. The most basic one is the polygonal surface (stored in files with a .vtk extension) which consists of a set of points and a set of connections between the points resulting, most commonly, in a set of connected triangular patches that form the surface. While this convention is very useful for visualization and post-processing, it is less convenient for editing. For this reason, the surface editor tools store the surfaces in .sur files where each surface consists of a set of planar curves, and each curve is represented by a b-spline. The nodes of the spline are essentially interpolated using piecewise cubic polynomials to yield a smooth curve.The editor applications provide powerful functionality for interactive segmentation of different anatomical structures. There are two distinct modes of operation. The surface tools allow the interactive outlining of surfaces which are represented as ``stacks'' of curves, one curve per image slice. The ``Editor'' menu is shown in figure 8.1.
BioImage Suite has four editor tools. The Mosaic Objectmap Editor and the Orthogonal Objectmap Editor allow only for the editing of objectmaps. The Surface Editor allows for both objectmap editing as well as surface editing. The 4D Surface Editor (under the cardiac menu) allows for only the editing of surfaces on temporal sequences of images.
8.2 The Objectmap Editor Tools
There are two tools here, one which allows for painting on orthogonal slices and the other on multiple parallel slices (this is slow!). We describe here the functionality of the Orthogonal Objectmap Editor (shown in Figure 8.2 - the use of the Mosaic Objectmap Editor is similar.
The Objectmap Menu (A): The objectmap (which is an image) can be loaded/saved/reinitialized using functionality under the Objectmap Menu. In particular note that the option ``VOI Volume/Stats Computation'' can be used to perform VOI analysis of the underlying image using the objectmap to define the VOIs! If Autosave is enabled, the objectmap is automatically saved periodically.
The Mask Slider (B): The objectmap is shown as a transparent overlay over the original image. The degree of the transparency is controlled using the mask slider (0=completely transparent, 100=opaque).
Objectmap Editor: The Objectmap Editor popup is invoked using the ``Show Paint Controls'' button (C). This has functionality for painting over the image.
D: Paint Control: Use the buttons 0-8 to select the color to paint with. If more colors are needed using the button marked ``More'' (E) to bring up an auxiliary control with 60 possible choices. The color definitions are controlled via the colormap which can be Loaded/Saved using functionality in the box marked as J. Individual colors can be edited using the ``Edit Color'' button (K).
The Painting mode is set via a number of controls. the Mode drop menu (G 
L) which selects the brush size (Off disables painting). If the
Mode menu is set to ``Off' then the left mouse button is use to navigate,
otherwise it is used to both paint and navigate.
The four checkbuttons in the row marked as H control for (i) whether the brush is two or three dimensional (3D), whether the paintbrush will overwrite any other existing non-zero label (No-Overwrite) and whether the painting will respect thresholding and connectivity constraints. If the ''Use Thrshld'' check button is enabled only voxels in the range specified in the threshold entries (marked as G) will be painted over. The ``Connect'' checkbox can be used to restrict the painting to voxels which are connected with the center voxel.
The Undo/Redo buttons I can be used to correct for mistakes during the painting. The unit of undo/redo is each mouse paint operation (from mouse press to mouse release).
Note: The objectmaps are not displayed in Oblique or Volume Rendering modes.
8.2.0.0.1 ObjectMap Regularization
The objectmap regularization/smoothing in BioImage Suite can be manually performed using the functionality in the objectmap editor. Figure 8.3 shows a screenshot of the objectmap editor and the parameters required to smooth an objectmap. The ``Factor'' and ``Conv'' settings are parameters that control the smoothing kernel. On completion, a popup dialog box informs the user about the percent of voxel that were changed in the smoothing process.
  |
8.3 The Surface Editor
The Surface Editor is the original BioImage Suite application. It was originally developed beginning in 1995 specifically for the segmentation of the bounding surfaces of the left ventricle of the heart (endocardium and epicardium) using then Silicon Graphics exclusive software libraries (Open Inventor). It has since been used for a number of applications including segmentation of neuroanatomical structure, region of interest definition of functional neuroimaging studies, cardiac segmentation, abdominal structure segmentation - including fat quantification - etc. The rest of this handout describes the core functionality of the surface editor tools. There are two related tools: the surface editor tool which includes both surface and objectmap tools for 3D images, and the 4D surfaceeditor tool which has only the surface editing tools but includes additional functionality for cardiac image analysis. Most of this handout is concerned with the 3D version of the tools.
8.3.1 Key Concepts
The SurfaceEditor tools operate and edit in 2D display in 3D paradigm. Editing is performed in a slice-by-slice mode using the slice editor tool and the 3D overall picture (i.e. 3D surface/3D objectmap) is displayed in the main Surface Editor window. The main window is updated either manually using the ``Update'' button in the editor or automatically when the slice is changed if the ``Auto Upd'' checkbutton is in the ``on'' state. The Surface Editor window consists of the same core BioImage Suite orthogonal viewer described in the viewer pages, but has additional functionality related to the interactive segmentation tasks that will be described next.Note that editing in the slice editor is performed on the XY slice of the underlying image, i.e. the axial slice if the image was acquired in an axial orientation, or the coronal slice if it was acquired in a coronal orientation. If a different 2D slice orientation is preferred, the image will need to be re-oriented or resliced (if oblique slices are required) using the Image Processing tool. The same applies to painting.
The Surface Editor has the ability to create surfaces from binary images. In this way segmentations generated using other tools such as itk-SNAP, which are typically stored as binary images, can be imported into the Surface Editor for more detailed manipulation. See the ``Creating Surfaces from Binary Images'' for more details.
8.3.2 The Main Surface Editor Window
The main Surface Editor window is shown to left in Figure 8.4. There are 5 parts to the user interface.
A: The Menu Bar: This contains the usual options of loading/saving the underlying images, image processing for smoothing, reslicing/cropping images, etc. In addition, the objectmap menu has options specific to the surface editor. Please note that the File Menu only contains operations for loading/saving the underlying image and not the surfaces/objectmaps. Separate functionality is provided to do this.
B. The Tab Selection: This selects which of three sets of controls appears on the right most pane of the window. These are the Image tab which has the usual orthogonal viewer options for manipulating how the image is displayed (e.g. single slice, orthogonal slices, volume rendering etc.), the Surface tab which contains most core surface operation and the Surface+ tab contains additional surface operations. The ``Surface'' tab (shown) is divided into two parts, namely:
C: Current-stack properties: Options for manipulating the current surface (how to display this e.g. open surface and fast surface) and the all-important definition of surface extent, i.e. on which slices does the structure appear. This is the first thing that needs to be set when creating a new surface.
D: Current-Stack Options: These are options for resampling the current stack, flipping the x-axis, changing the color and loading and saving. On the bottom of the window
E: Editor Slice: contains functionality for selecting the current slice on which the spline/objectmap editor operates and a ``button'' for opening the Editor.
The ``Image'' ``Surface'' and ``Surface+'' tabs in more detail
Much of the functionality of the surface editor is bound within the notebook tabs in the rightmost pane. The Image tab is identical to the standard orthogonal viewer.
The Surface tab contains most of the functionality needed for manipulating surfaces in a global sense. The surface editor can be used to outline a number of surfaces at a time (typically 4; but can be set to as high as 50 using the user preferences editor). These can all be shown or hid using the Show All/Hide All buttons in the ``Global Properties'' frame at the top.
The ``Spline Stack Properties'' frames has the following options. First the ``Current Stack'' option menu enables the selection of the current surface (e.g. 1 to 4) for editing/manipulation. All subsequent operations operate on this surface. The ``Draw Mode'' option has mostly cosmetic effect. Typically the surfaces are cylinder-like with open tops and bottoms, i.e. ``Open Surface''. The other options in this drop-menu enable the visual closing of either the top or the bottom or both ends of the surface. This adds computational cost to the surface updating and is best used sparingly. The ``Display Mode'' option has three choices ``Fast Surface'' which produces a fast but relatively low quality surface rendering, ``Full Surface'' which produces a slower but higher quality rendering and ``Hide'' which completely hides the surface. The ``Bot Trim:'' and ``Top Trim:'' controls enable the user to set the critical bottom slice and top slice extends of the surface. If the structure of interest appears only in e.g. slices 24-32, then bottom trim should be set to 24 and top trim to 32. Failure to do this will result in either additional or missing curves in the surface with erroneous results.
The ``Operations on Current Stack Frame'' contains options for global manipulation of the surface. The first line consists of a row of buttons that resample the surface. For example, ``x0.5'' resamples all curves to have half the number of control points (as discussed later each curve is parameterized as a b-spline), and similarly x0.8, x.1.25 and x2. The ``s2'' and ``s5'' buttons re-parameterize the surface to have different degrees of smoothness and result in non-uniform b-spline. Finally the ``FlipX'' button performs a left-right mirroring of the surface which is useful in some applications.� The ``Color'' button brings up a dialog box for selecting the color of the surface. Clicking the ``Cylinder'' button resets the surface to a cylinder of default location and size. The ``Volume'' button computes the volume of the surface by adding up the areas of the individual curves and multiplying by the image slice thickness. The VOI Prop tab performs VOI (volume-of-interest) analysis of the area inside the surface. The two thresholds can be set to restrict the analysis to only voxels in that range. The ``Load'' and ``Save'' buttons enable the loading and saving of surfaces in the custom ``.sur'' format. Finally the Copy and Paste options enable the copying of the surface to the clipboards and its pasting (presumably to another surface).
The Surface+ tab contains some additional functionality. The ``Current Stack'' drop menu mirrors the one in the ``Spline Stack Properties'' frame and can be used to set the current surface. The Load Set and Save Set options can be used to load/save all the surfaces in the editor in a composite ``.multisur'' format file. (Note that the .multisur file simply contains pointers to an additional number of individual surface files. E.g. if one saved a group of 4 surfaces to mysurfaces.multisur, the following files will be created: mysurfaces.multisur, mysurfaces.multisur_1.sur, mysurfaces.multisur_2.sur, mysurfaces.multisur_3.sur and mysurfaces.multisur_4.sur). The Autosave and Backup checkbuttons if enabled turn on the autosave features of the software (leave them on!). The Shift and Scale button brings up a legacy control for re-scaling surfaces generated with older versions of Surface Editor which did not store proper slice thickness information. The Cylinders button brings up a control for creating a custom cylinder by specifying both radii, slice extents, curve centroids and number of control points. The Export buttons brings up a control for exporting surfaces either in the custom ``.tstack'' format or in the more common ``.vtk'' format (as defined by vtkPolyDataReader/Writer). The user can specify the sampling of the spline surfaces in mm as well as the format and a number of smoothing iterations to be applied prior to exporting.
 |
Creating Surfaces from Binary Images A recent addition is the Extract From Binary Image button which allows for the creating of a surface (.sur) from a binary pre-segmented image. In this way segmentations generated using other tools such as itk-SNAP, which are typically stored as binary images, can be imported into the Surface Editor for more detailed manipulation. Pressing the Extract From Binary Image button, brings up a helper dialog which allows the user to specify four options: (i) Surface Smoothing: this determines how many control points will be used to parameterize the extracted surface, (ii) Image smoothing: which is the standard deviation of the kernel that is used to smooth the binary image prior to surface extraction and (iii) and (iv) bottom and top slice to limit the extraction range. The extraction routine automatically performs connectivity analysis on the underlying image to eliminate small holes etc. and will return only those consecutive slices for which the surface extraction has succeeded.
8.3.3 Objectmaps and the Objectmap Menu
Most of the objectmap editing functionality is identical to that shown in Figure 8.2 and will not be repeated here. The only limitation is that all editing is done with a 2D brush, the ``3D'' checkbox is ignored.
Objectmaps can be created either from the current displayed image in the viewer using the ``Create From Data/Grab from Current Displayed Image'' button or from the surfaces (i.e. convert surfaces to objectmaps) using the ``Surface Objectmap Tool'' control shown above that is accessed using the ``Create from Data/Create from Surfaces'' option. The default setup for this control (using the ``Create!'') button enables the user to specify which surfaces to use, and value inside each surface as well as the background. (The Distance Map function creates a ``2D Distance Map'' based on the surfaces which is useful in some applications.) The ``Mask'' button creates a mask where only the parts of the image inside the selected surface are left, the rest are blanked. The opposite (i.e. only parts of the image outside the selected surfaces) is accomplished using the ``Inverse'' button.
The (Objectmap | Fat Objectmap Operations) menu choice brings up two different customized options for application specific needs not described here.
8.3.4 The Spline/Objectmap Control
 |
The local editing of surfaces/objectmaps is done within the Spline/Objectmap tool shown at right in Figure 8.6. This is accessed using the ``Objectmap/Spline Editor'' button in the bottom left corner of the main SurfaceEditor window. This extremely powerful control enables the manipulation of individual curves by moving b-spline control points, the drawing of new curves by simply clicking points which are then converted to b-splines for further editing, as well as the manipulation of objectmaps using paint-brush like operations.
Editing Splines
In the viewer portion (i.e. the left part of the image in Figure 8.6), we show a single spline. If the viewer ``edit'' mode is turned on, manual mode off and the mode in the Fill Control is set to ``Spline'' (these are all the default settings), then the spline can be manipulated by moving individual control points by clicking and dragging them using the left mouse button. Holding down the shift-key while and left mouse button allows you to move the entire spline. Clicking on the curve with the left mouse button while holding down the ``Control'' key adds a new control point half way between the closest two control points. In this way, a fully customizable spline curve can be generated on each slice of an image, thus creating a user-defined surface.
 |
The Bottom Row - selecting slice and objectmap transparency.
Given the absence of a menu in this control, most core functionality is present in the bottom row ``buttonbar'' shown above. The ``Slice'' control determines which slice is currently being edited. This is mirrored in the slice control in the bottom row of the main Surface Editor window. The ``Mask'' control sets the relative opacity of the objectmap with respect to the image. If set to 0 the objectmap is completely transparent (effectively hidden) whereas if set to 100 the objectmap becomes completely opaque and hides the underlying image. The next label ``200(56,83,62)'' shows information about the last clicked voxel in the image with, in this case, image coordinates (56,83,62) and intensity=200.00. The ``SaveTiff'' button enables the saving of the contents of the viewer (i.e. a screen grab) as a tiff/jpeg file (depending on the suffix).The ``Zoom'' arrow buttons enabling zooming in and out (this is also accomplished by dragging with the middle mouse button). The ``ColorMap'' button brings up the standard colormap editor (as described in the viewers documentation) and the ``Reset'' button resets the viewer to default zoom/location levels. The ``Interp'' check button turns on/off standard anti-aliasing interpolation in the image display.
The Spline Controls - manipulating splines
This control has functionality for manipulating individual curves. We again note that in the SurfaceEditor each surface is made up of a single curve on each XY slice (within the bounds set by the bottom trim and top trim controls). These curves are manipulated using the spline control shown here. Most of the basic functionality of the spline control is described in Figure 8.8.The ``Global Properties'' frame has options for turning ``Edit Mode'' on and off, enabling and disabling ``Manual Mode'' described in detail below - showing and hiding the actual control points and showing and hiding all splines.
The (-) button is used to flip the relative depth positioning of the splines and the image. It is primarily needed when tracing on Sagittal images. If the splines are not visible a single click of the (-) button should fix the problem,
Individual spline properties are set using the options in the ``Spline Properties'' frame. The ``Current Spline:'' drop menu selects the current spline to edit/manipulate. Spline 1 is the spline corresponding to ``Surface 1'' in the main Surface Editor window, ``Spline 2'' is part of ``Surface 2'' etc. The spline can be hidden by unchecking the display button. The ``Size:'' optionmenu controls the size of the control points.
The ``Operations On Current Spline'' frame mirrors functionality in the ``Operations on Current Stack'' frame in the ``Surface'' tab of the main surface editor. The ``Snake'' and ``Update Snake'' options enable the use of interactive snake-like segmentation methods to be described in more detail in upcoming lectures.
The Copy/Paste options are the real workhorse options of this program. Given the general smoothness of most anatomical structures, the shape of a surface on slice 51 is very similar to that in slice 50. Hence the curve can be copied from slice 50, the slice number incremented to 51 and then the curve pasted giving an excellent initial position for further editing. The ``Undo'' button reverts to the version of the curve that is currently stored in the surface editor (i.e. prior to editing). Past flip performs an x-flip prior to pasting.
The Update and ``Auto Upd'' buttons control the communication between the spline/objectmap editor and the surface editor. On pressing update the versions of the curves and the objectmap slice (see below) as edited in the spline/objectmap editor are copied over to the main SurfaceEditor, and the 3D display in the Surface Editor is updated to reflect this. The ``Auto Upd'' checkbutton enables automatic updating upon slice change. The ``Update'' button automatically copies the current spline and places it in the clipboard so that it can be pasted directly.
8.3.4.0.1 Manual Mode:
. In certain cases the initial shape to be traced is very complex and cannot easily be captured by moving the control points of an elliptical spline to fit this. The use of ``Manual mode'' enables the use of the more traditional (and less flexible for subsequent editing purposes) method of clicking individual points on the boundary to create a polygonal curve which is then converted to a b-spline for editing. To use manual mode first enable this by selecting the ``Manual Mode'' checkbutton towards the top right corner of the ``Global Properties'' frame.
 |
Next, click individual points using the left mouse button to form a polygon (there is no need to close this, the fitting process automatically does this for you - fact the polygon is best left open). Once the polygon is mostly closed, turn off ``Manual Mode'' by clicking on the ``Manual mode'' checkbutton as before. Once this is done a pop-up dialog asks whether you want to replace the current spline with one fitted to the current set of manually clicked points. If you answer ``no'', the current manually clicked points are deleted. If you answer ``yes'' a spline is fitted to the polygonal points which can then be further edited.
The Paint Controls - manipulating objectmaps
The ``Paint control'' - see Figure 8.9 - enables manipulation of object-maps by appropriately coloring in image voxels (by using the left mouse button). The ``Fill In'' and ``Fill Out'' buttons use the threshold (but not connect) settings together with the current spline to paint the image. Fill In will fill the inside of a spline with a certain value, whereas ``Fill Out'' will fill the region outside the spline with the selected value. If threshold settings are set, these are also used.
8.4 Delineating a surface: a step-by-step recipe.
- Load the Image using the File/Load Option in the main Surface Editor window
- Adjust the orientation of the image to ensure that the slice orientation on which you want to draw curves on is the same as the orientation of the image. For example, if the structure you are interested in is best outlined on coronal slices, the image will need to be re-oriented in a coronal orientation (Use the tool under Image Processing/Reorient-Crop Image). Remember to make the change permanent using ``Copy Results to Image'' and to save the reoriented image for later use.
- Identify the extent of the structure of interest in the image, i.e. the bottom slice and the top slice that bound the surface. Set the bottom trim and the top trim in the main Surface Editor window, in the ``Surface'' tab to reflect this
- Save the surface - will enable the autosave functionality to be used. The autosave function automatically saves a surface each time it is updated in a file ``surfacename_auto.sur''. Should there be an unexpected (and these days rare) software crash, the autosave file can be used to recover most lost work.
- Select a slice roughly at the middle of the surface (i.e. half way between top trim and bottom trim), using the slice control in (E. Editor Slice ) on the main SurfaceEditor window
- Open the spline/objectmap editor
- Use the manual mode method to roughly outline the curve on this slice.
- Resample the spline as necessary to have the appropriate number of control points.
- Refine the curve by moving these control points.
- Update the 3D Window by clicking ``Update'' in the Spline Editor
- Change the slice.
- Paste the last traced curve. (The ``Update'' button automatically does the ``copy'' operation).
- Edit the curve
- Go back to step 9 until all slices have been traced.
Next: 9. Tissue Classification Up: 2 B. Anatomical Image Previous: 7. The Image Processing Contents