next up previous contents
Next: 3 C. Functional MRI Up: 2 B. Anatomical Image Previous: 11. Non Linear Registration   Contents

Subsections


12. Landmarks, Surfaces and Point-based Registration

12.1 Introduction

BioImage Suite can handle several types of objects. While images are by far the most important object type in BioImage Suite, point sets and surfaces also play significant roles in many applications. BioImage Suite has two specialized controls for manipulating these namely (i) The Landmark Control which is used to store and manipulate sets of points and (ii) The Surface Control which is used to manipulate surfaces. Surfaces and point-sets can be used as reference/target datasets for the Robust Point Matching registration algorithm - the Surface Objectmap Control can be used to define complex combinations of labeled surfaces. This session concludes with a look at the Point-based registration tools.

12.2 Acquiring Landmarks

Many operations in BioImage Suite require the use of landmark points. These include generating ``Talairach Transformations'' in neuroimaging as well as seeding of levelset-type segmentation algorithms etc. All landmark editing/storage etc. in BioImage Suite is handled by the Landmark Control. This is a multifunctional control that has capabilities for both landmark editing as well as curve tracing/extraction.

12.2.1 Invoking and Interaction with the Viewers

Figure 12.1: A viewer with the Features Menu.
Image LandmarkControl1
When supported, the landmark control lives under the ``Features'' menu as shown in Figure 12.1. There may be up to three option boxes in the top of this menu marked: If ``Shift/Click to Landmark'' is enabled (default), then to manipulate a landmark the user must hold the ``Shift'' key down and click with the left mouse button. This enables the use of the left mouse button alone for navigating in the images as usual. If desired, this option may be turned off, in which case navigation via the left mouse button is disabled and all left mouse button events are captured by the landmark control. (An exception to this is in some applications when the Electrode Editor is present; this is really a very specialized version of the landmark control and replaces its functionality). BioImage Suite also enables tracing on volume rendering or polygonal surfaces. These options are enabled using the ``Enable Volume Trace'' or ``Enable Geometry Trace'' menu selections. Enabling one of these disables the other. Volume tracing works by shooting a ray from the location of the virtual image camera and places a landmark at the first ``non-zero'' voxel in the image. It is most suitable for outlining cortical sulci from skulled-stripped brain images.

12.2.2 The Landmark Control

The Landmark Control is a complex tool for acquiring, storing, and manipulating ``point-sets'' which are simply collections of ordered points - the control can keep 9 different point sets in memory. Figure 12.2 shows a snapshot of the control, which is divided visually into four parts, namely: (i) the menu bar, (ii) the ``point set properties'' frame, (iii) the list of current points in the currently selected pointset and (iv) the ``global properties'' frame. All operations (either point acquisition or from the Landmark Control menu) are applied to the current pointset as selected using the option menu marked as (D) in Figure 12.2 (see also expanded version in Figure 12.3).

12.2.2.1 Handling Mouse Input

Mouse operations (typically shift/click with the left button) are interpreted depending on the current mode, which is selected using the left option menu in the ``General Properties'' frame (this is marked as (C) in Figure 12.2 and shown expanded in Figure 12.3). There are 5 different options:

12.2.2.2 PointSet Properties

In the pointset properties frame (see Figure 12.2) there are three items which control how a point-set is displayed.

The mode option menu ((A) in figure 12.2 and shown also in figure 12.3), lets you choose the display format for the set of points:

The point size menu ((B) in figure 12.2) controls the size of the points in mm, ((B) in figure 12.2). On the far right, a checkbox determines whether the points are displayed or not. At the bottom of the pointset properties frame, a status textbox displays information about the current pointset, namely its name and the number of points in the set.

Figure 12.2: The Landmark Control.
Image LandmarkControl2
Figure 12.3: Expanded views of the option menus A-D of the Landmark Control, as defined in Figure 12.2 above.
Image LandmarkControl3

The List of Current Points simply lists the coordinates of all points in the current point set in mm.

12.2.2.3 The Landmark Control Menu

The ``File'' Menu The File Menu contains options for loading and saving the pointsets. Pointsets can be loaded/saved to file in the default .land format. In addition they can be exported to standard vtk surface format (.vtk), as well as image objectmaps (where the voxel closed to each landmark is colored in).

The ``Edit'' Menu This contains self-explanatory options for copy/paste operations etc. The ``Edit Point'' option brings up a dialog box which enables direct manipulation of point coordinates by typing them into textboxes.

The ``Color'' Menu There are two options here: Landmark color, which enables setting the color of all the landmarks and Highlight color, which can be used to set the color of the ``current'' point - most commonly the last point.

The ``Setup'' Menu It is often desirable to guide the user to click a set of landmarks in a prescribed order. The Setup Tool has facilities to accomplish this. Pressing the  Edit Setup File option under the setup menu brings up the Label Editor. This can be used to edit/create a set of labels. The setup prescription can be loaded/saved using the Load Setup File/Save Setup File options under the Setup menu. One preset mode is the Talairach Mode - an example of using this can be found in the Coordinates page.

The ``PointSet'' Menu This is the equivalent of the ``Edit'' menu for whole point sets. ``Copy'' and ``Paste'' copy or paste the current landmark to/from the clipboard. ``Paste 2D'' enables pasting only the x and y coordinates. Create Circle creates a circle in the XY plane.

The ``Curve'' Menu This contains operations for manipulating the pointset as a curve. Options include area and length computation, use of the curve to define a region of interest to perform ROI analysis on the underlying image and options for smoothing and resampling the curve.

The ``Operations'' Menu There are two options here. (i) Angle returns the angle between the last two points and the x-axis. This is often useful in reslicing an image. (ii) Extract performs a local iso-contour extraction from the original image.

Figure 12.4: The Landmark Control Setup Tool. Landmark names can be added by typing them in the text box next to the update button and pressing the Add button.
Image landtool_setup

12.3 The Surface Control and the Surface Objectmap Control

Figure 12.5: The Surface Control Menu.
Image surfacecontrol1

Note: The Surface Control is a tool for manipulating and displaying surfaces parameterized as polygonal meshes. Tt can be used to create surfaces from images using the marching cubes algorithm - see the ``Extract Iso-Contour'' option below. However, for more detailed surface editing/creation see the Surface Editor webpage. Surfaces created using the Surface Editor can be exported to either .vtk or .tstack formats which can then be imported into the Surface Control. The Surface Control does not support direct loading of the native ``.sur'' format output of the Surface Editor as this captures the surface as a stack of parametric splines and NOT as a polyhedral surface.

The Surface Objectmap control is a specialized tool for creating surface objectmaps. These are used as inputs to the point-based registration tools. For more information see below. In the applications that they are available in, the surface and the surface objectmap controls appear under the Features Menu, as shown in Figure 1.

12.3.1 The Surface Control

12.3.1.1 The General Properties Frame

The Surface Control essentially consists of a menubar and the general properties frame below it. On the general properties frame there are three items: (i) The status label which provides information about the current surface. (ii) The Surface Selector which allows the user to select which surface to manipulate - all operations are performed on the current surface. (iii) The Display menu selects how the current surface is displayed. There are four options. ``Show as Surface'' results in an optimal rendering for 3D viewers. ``Show as Wireframe'' results in the surface being displayed as a wireframe - this is often useful for showing the intersection of the surface with image slices. ``Show as Points'' eliminates the lines joining the points and just shows the individual surface points. Finally the ``Do not show'' option is used to turn off the display of the individual surface.

The display of each surface can be further manipulated using options in the display menu, which is described below.

12.3.2 The Menus

12.3.2.0.1 The File & Edit Menus

There are four options under the File Menu:

The Edit Menu also has four options:

Figure 12.6: Surface Control Details. The Tools & Display Menus.
Image surfacecontrol2

12.3.2.0.2 The Tools Menu:

The Tools Menu contains a number of options for surface manipulation. They are briefly described below. Most of these tools invoke directly VTK classes which are listed in parenthesis in the description.

12.3.2.0.3 The Display Menu:

The display menu provides additional options for manipulating how the surface is displayed.

12.3.3 The Surface Objectmap Control

Figure 12.7: The Surface Objectmap Control.
Image surfaceobjectmap

The Surface Objectmap Control is designed to allow the combination of different surfaces and point sets into surface objectmaps which are used as inputs to the point-based registration tools. It can take as input up to 15 surfaces (in .vtk format) or point sets in (.land format, as saved by the Landmark Control.)

At the bottom of the surface objectmap control there are a number of buttons that can be used to Load/Save/Create the Objectmap. The Load Setup and Save Setup buttons enable loading/saving a list containing the filenames and attributes of the current set of surfaces.

The Resample All button causes all input surfaces to be resampled to have spacing equal to the product of the Target Spacing set in each surface (see below) multiplied by the scale factor set in the option menu next to the Resample All button.

The Create Combo Surface button appends all resampled surfaces into a single surface and places its output into the clipboard of the associated Surface Control, which can be invoked using the Show Surface Control button. The resulting surface objectmap can then be pasted into one of the surfaces in the Surface Control and saved for later use.

Image surfaceobjectmap2

Each surface or pointset can then be resampled to have approximate point spacing set by the ``Target Spacing'' field and assigned a unique label for inclusion into the surface objectmap. Each point set has its own set of controls shown above. The control is divided into three panes (left to right). The leftmost pane provides information about the surface/pointset (filename, type=Surface/Landmarks, Number of Points.). The middle pane provides options for resampling the surface (spacing) and for setting its label for the surface objectmap to be outputted from the control. The rightmost pane has one toggle button that enables the inclusion or exclusion of the surface from the final output (the ``Active'' toggle) as well as some information textboxes that display the actual number of points the surface is being resampled to.

12.4 Point-based Registration Tools

Point-based registration brings images into alignment based on feature points extracted from the images. The optimal transformation and correspondences must be determined.

12.4.1 Invoking and Interaction with the Viewers

Figure 12.8: Invoking the Point-Based Registration Tool.
Image pointbasedreg1

The point-based registration control is available in pxitclbrainregister and pxitclmouseregister. It is invoked using the point-based registration option under the Registration menu, as shown in Figure 12.8. The registration takes as inputs two sets of points saved as surfaces stored in the Surface Controls of the Reference and Transform viewers respectively. The output transformation is stored in the Transformation control of the Registration/Overlay tool.

In Robust Point Matching (RPM), the correspondence and transformation are determined together iteratively in a robust manner [19] accounting for outlier points which do not have a corresponding point in the other image. For more details see [30,76].

12.4.2 The Main Point-Based Registration Control

Figure 12.9: The Point-Based Registration Tool.
Image pointbasedreg2

The point based-registration control is divided into five parts:

  1. The ``Common'' controls frame (A in Figure 12.9) which defines the two surfaces to be registered, and some global parameters.
  2. The ``parameters'' controls frame (B1/B2 in Figure 12.9) which defines the specific parameters for the linear registration (B1 ) and the non-linear registration (B2 ).
  3. The ``top-frame'' controls frame (C in Figure 12.9) which has shortcuts to the two surface controls, and for closing the window.
  4. The ``Viewer'' frame containing a separate viewer for monitoring the progress of the registration (C in Figure 12.9).
  5. Finally, the update frame, at the bottom right (E in Figure 12.9) which contains settings for how often to update the display and how to display the surfaces.

An example: Computing a Linear (rigid registration)

The procedure to compute a linear registration between two surfaces is as follows:
  1. Load the reference surface in the surface control of the Reference Viewer. Note the index of the surface it is loaded in is 1 by default.
  2. Similarly, load the target surface in the surface control of the Target Viewer, and note its index.
  3. Open the point-based registration controls.
  4. In the ``Common Controls'' (A ), select the indices (most likely 1 and 1) for the reference and transform (target) surface.
  5. Press ``Show Surfaces'' in (C ) to show the surfaces and adjust the viewer to your liking. If no surfaces are visible, press the ``Va'' button in frame (D ) to reset the display.
  6. In Frame (B1 ), set the appropriate parameters. In particular, select the Linear tab and the ``Rigid'' transformation mode (other choices include Similarity and Affine). Set the desired number of points to use, in Max Landmarks . The only other parameters to touch are the ``Initial Temperature'' and ``Final Temperature'' which should be set to reflect (i) the maximum distance between the two surfaces prior to registration (initial temperature) and (ii) the point sampling distance of the surfaces, e.g. how closely sampled the points are (final temperature).
  7. Press the Start RPM button at the bottom of frame B1 to start the registration.

A second example: Computing a Non-Linear (nonrigid registration)

This is similar to the Linear case, with the following changes:

  1. A non-linear registration is often initialized by a linear registration. Run the linear registration first and verify that it is successful.
  2. In step 6 above, there are a few more parameters to adjust. In particular:
    • First, select the Nonlinear tab. Enable Use Initial Transformation to use the last computed linear transformation (or the currently selected transformation in the transformation control of the Registration/Overlay tool.
    • The Initial Temperature needs to be set to account for the distance at the end of the linear step.
    • The Initial Control Spacing and Final Control Spacing reflect the values for the spacing of the tensor b-spline grid transformation that is to be computed. A rule of thumb is to select the final value to give the desired accuracy/computational cost and multiply this by 2 to set the initial value.
    • The Initial Smoothness and Final Smoothness determine the value of the regularization weight at the start and end of the process. Most registrations will start with a relatively high smoothness to avoid local minima and progressively relax this to get improved accuracy.

  3. The computational cost is significantly higher!

12.5 Appendix: An Overview of Robust Point Matching

We present here a slightly modified form of the standard RPM methodology as can be found in Chui et al and Papademetris et al. This consists of two alternative steps: (i) the correspondence estimation step and (ii) the transformation estimation step. In the following discussion we will label the reference point set as $X$ and the transform point set as $Y$. The goal of the registration is to estimate the transformation $G:X\mapsto Y$. We will label $G^k$ the estimate of $G$ at the end of iteration $k$. $G^0$ is the starting transformation which can be the identity transformation.

12.5.1 Correspondence Estimation:

Given the point sets $X$ and $Y$ we estimate the match matrix $M$, where $M_{ij}$ is the distance metric between points $G^k(X_i)$ and $Y_j$. The standard distance metric is defined as:
\begin{displaymath} M_{ij} = \frac{1}{\sqrt{2\pi T^2}}e^{\frac{-\vert G^k(X_i) -... ... 1 \mbox{,\hspace{0.2 in}} \forall j \sum_i M_{ij} + R_j = 1 \end{displaymath} (12.1)

where $\vert X_i-Y_j\vert$ is the Euclidean distance between points $X_i$ and $Y_j$ and $T$ is the temperature that controls the fuzziness of the correspondence. If the correspondence problem is to be thought of as a linear assignment problem, the rows and columns of $M$ must sum to 1. The framework is further extended to handle outlier points by introducing an outlier column $C$ and an outlier row $R$. $C_i$ is a measure of the degree of `outlierness' of a point in the reference point set $X_i$ and $R_j$ is the same for a point in the transform point set $Y_j$. $C$ and $R$ are initialized with constant values. The ability to model outliers allows this method to robustly match features of high variability such as cortical sulci, and to incorporate robustly manually outlined structures where the ends of the structures are user defined and somewhat arbitrary (e.g. in the case of the kidney users typically trace on certain slices and the kidney may partially extend into one more slice either side). Once the normalization is completed we can compute the correspondence as follows. Let $V_i$ be the corresponding point to $X_i$ and $w_i$ the confidence in the match. Then $V_i$ is defined as a normalized weighted sum of the points $Y_j$ where the weights are the elements of the match matrix $M$.
\begin{displaymath} V_i = \frac{ \sum_j M_{ij} Y_j }{ \sum_j M_{ij}}, \hspace{0.2 in}\mbox{and }w_i = ( \sum_j M_{ij}) = 1 - C_i \end{displaymath} (12.2)

Note that a point that has a high value in the outlier column $C$ will have low confidence and vice-versa. We note that in our integrated method (Section 12.5.4) we simply use the correspondence piece of RPM.

12.5.2 Transformation Estimation

This is simply achieved by a regularized weighted least squares fit between $X_i$ and $V_i$ as follows:
\begin{displaymath} G^k = {\arg\min \atop g } \sum_i w_i( g(X_i)-V_i)^2 + f(T){\cal S}(g) \end{displaymath} (12.3)

where ${\cal S}(g)$ is a regularization functional (e.g. bending energy function) weighted by a function of the temperature $f(T)$. This last weighting term is used to decrease the regularization as we approach convergence.

12.5.3 Deterministic Annealing Framework

The alternating estimation of $M$ and $G$ is performed in a deterministic annealing framework. Starting with a high value of $T$ corresponding to a rough estimate of the maximum mis-alignment distance we first estimate $M$, and then $G$. Then $T$ is decreased by multiplying it with an annealing factor and the process is repeated until $T$ becomes sufficiently small.

12.5.4 Integrated Points + Intensity-based Registration

12.5.4.0.1 Intensity Module:

We use a slightly modified form of the intensity-based non rigid registration method first described by Rueckert et al. This method utilizes a free-form deformation transformation model based on tensor b-spline, and the normalized mutual information similarity metric. This metric can be expressed as:
\begin{displaymath} NMI(A,B)= H(A,B)/(H(A)+H(B)) \end{displaymath} (12.4)

where $A$ and $B$ are the two images, and $H()$ is the image intensity entropy. This similarity metric is combined with a regularizing term to yield an optimization functional which is maximized in a multi-resolution framework. Our own implementation of this method first estimates a linear affine registration and then uses this as an input to estimate a full non-linear FFD transformation in a multiresolution manner.


12.5.4.0.2 Integration Method:

We first estimate an initial registration using the RPM method alone, to ensure that landmarks are correctly aligned. We then proceed to refine the estimate of the transformation by minimizing the following optimization functional which is a trade-off between intensity similarity and adherence to point correspondence.


\begin{displaymath} {\hat g} = {\arg\max \atop g} \Bigl ( \underbrace{NMI(g(A),B... ...^2 \bigr)}_{\mbox{Adherence to Point Correspondences}} \Bigr ) \end{displaymath} (12.5)

where the first term is the intensity similarity distance (Equation 12.4), the second is a measure of adherence to the corresponding points as estimated by RPM (Equation 12.3) weighted by the constant $\lambda$ ($N$ is the number of points in the reference point-set). We note that during the optimization of this functional the correspondences maybe re-evaluated at each iteration at a constant temperature which is equal to the minimum temperature used in the initial registration. In practice, however, keeping the correspondences fixed produced marginally better results. The transformation estimated has the exact same parameterization as the FFD transformation estimated by the RPM algorithm.

This method is available only as part of the batch-mode command line tools.

next up previous contents
Next: 3 C. Functional MRI Up: 2 B. Anatomical Image Previous: 11. Non Linear Registration   Contents