3D Volume Reconstruction of a Mouse Brain

from Histological Sections using Warp

Filtering

Tao Ju ∗,1 Joe Warren 1 James Carson 2 Musodiq Bello 4

Ioannis Kakadiaris 4 Wah Chiu 2 Christina Thaller 2

Gregor Eichele 3

Abstract

Sectioning tissues for optical microscopy often introduces upon the resulting sec-tions distortions that make 3d reconstruction difficult. Here we present an automaticmethod for producing a smooth 3D volume from distorted 2D sections in the ab-sence of any undistorted references. The method is based on pairwise elastic imagewarps between successive tissue sections, which can be computed by 2D image reg-istration. Using a Gaussian filter, an average warp is computed for each sectionfrom the pairwise warps in a group of its neighboring sections. The average warpsdeform each section to match its neighboring sections, thus creating a smooth vol-ume where corresponding features on successive sections lie close to each other. Theproposed method can be used with any existing 2D image registration method for3D reconstruction. In particular, we present a novel image warping algorithm basedon dynamic programming that extends Dynamic Time Warping in 1D speech recog-nition to compute pairwise warps between high-resolution 2D images. The warpingalgorithm efficiently computes a restricted class of 2D local deformations that arecharacteristic between successive tissue sections. Finally, a validation frameworkis proposed and applied to evaluate the quality of reconstruction using both realsections and a synthetic volume.

∗ 6100 Main, MS-132, Houston, TX 770051 Rice University, Houston, USA2 Baylor College of Medicine, Houston, USA3 Max Planck Institute of Experimental Endocrinology, Hannover , Germany4 University of Houston, Houston, USA

Preprint submitted to Elsevier Science 27 July 2005

1 Introduction

In medical imaging, volumetric data generated by 3D imaging methods suchas MRI and CT have wide applications in the visualization and analysis oforgans. 2D imaging methods, such as optical microscopy, typically generateserial sections with much higher resolution than MRI or CT scans. Recon-struction of these 2D sections in 3D has therefore become an important toolfor understanding anatomical structures in 3D, and in particular, for buildinghigh-resolution templates (atlases) of organs (Timsari et al. (1999); Armstronget al. (1995); Rosen et al. (2000); Cannestra et al. (1997); MacKenzie-Grahamet al. (2004); Sidman (2005)), or of whole animals (Brune et al. (1999)). Ourwork is motivated by the need for automatic and efficient methods in re-constructing, from a stack of high-resolution Nissl-stained sections, a smoothrepresentative volume for building a high-quality 3D atlas of a mouse brain.

Unfortunately, direct 3D reconstruction of serial sections by stacking succes-sive sections will not produce a smooth volume. While 3D imaging methodscan be applied in vivo, planar imaging methods are applied ex vivo. The prepa-ration steps required by 2D imaging methods may introduce undesirable tis-sue distortions specific to the preparation procedures, which have been closelyexamined by several authors (Berthold et al. (1982); Deverell et al. (1989);Schormann et al. (1995); Brey et al. (2002)). The experimental data used inthis paper were acquired by cryo-sectioning a frozen mouse brain into 25 µmthin slices followed by histology staining (Carson et al. (2002)). Distortionson the resulted sections are mainly introduced in the sectioning stage, whichinclude global translation and rotation, local stretching and compacting, andvarious image artifacts (see a more detailed discussion in Section III A) . Sincethese deformations vary from section to section, as shown in the examples inFigure 1, distortion correction is required to reconstruct a smooth volumetricrepresentation.

Section 14 Section 15 Section 140 Section 141

Fig. 1. Four coronal (i.e., vertically cut from front to back) sections from a stack ofhistological sections of a mouse brain acquired by cryo-sectioning. Besides rotationaland translational differences, successive sections exhibit different nonlinear tissuedistortions, such as compacting, stretching, and even folds and tears, as well asartifacts, such as air bubbles and dust.

2

1.1 Related works

There has been tremendous amount of work on 3D reconstruction from dis-torted serial sections. Typically, reconstruction is achieved by deforming in-dividual 2D sections using image registration (warping) techniques. Here webriefly review related works on 2D image warping and 3D reconstruction.

1.1.1 Image warping

Image warping considers the task of finding the best deformation (warp) of asource image that matches a target image under a specific distance measuresuch as pixel intensities or mutual information (Maes et al. (1997)). Imagewarping methods have been intensively studied in the field of medical imag-ing, and we refer interested readers to survey articles by Maintz and Viergever(1998), by Glasbey (1998) and by Lester and Arridge (1999), as well as booksby Toga (1999), by Hajnal et. al. (2001), and by Modersitzki (2004) for excel-lent reviews. Software packages implementing state-of-the-art warping meth-ods are also available for medical applications, such as the Automated ImageRegistration (AIR) package from UCLA (Woods et al. (1998a)) and the In-sight Segmentation and Registration Toolkit (ITK) from NLM (Yoo et al.(2002)).

For the purpose of 3D reconstruction, we are interested in the classifica-tion of image warping methods by the nature of the resulting image defor-mations. A number of methods compute deformations represented as globalpolynormials, such as rigid-body transformations (Woods et al. (1993); Maeset al. (1997)), linear affine transformations (Rangarajan et al. (1997); Theve-naz et al. (1998)), and higher-degree polynomial deformations (Woods et al.(1998b)). Since these deformations are global in nature, they do not performwell in the case of local variations between images (Glasbey (1998)). To over-come this problem, transformation frameworks that allow more localized de-formations have been proposed, such as free-form B-spline based deformationsdefined on a regular grid of control points, which have been studied by numer-ous authors including Rueckert et. al. (1999) and Studholme et. al. (2000),and piecewise affine transformations considered by Pitiot et. al. (2003). Localwarping methods have more flexible mapping functions and are often regu-larized by some form of elastic energy, such as the thin-plate spline energy(Bookstein (1989)), to prevent excessive deformations. Common optimizationtechniques for computing a regularized solution include solving partial differ-ential equations (an excellent review of such methods can be found in the bookby Modersitzki (2004)), finite-element methods (Guest et al. (2001)), graphmethods such as max-flow (Roy (1999)) and min-cut (Boykov et al. (1999)),stochastic methods (Agazzi et al. (1993)), and dynamic programming.

Since its first successful application in speech recognition by Sakoe and Chiba

3

(1978), dynamic programming has been known in 1D as Dynamic Time Warp-ing (DTW) and has been extended in various ways for warping 2D images(Levin and Pieraccini (1992); Cox et al. (1996); Uchida and Sakoe (1998);Ronee et al. (2001)). Compared to other optimization methods in the con-tinuous domain, dynamic programming finds discrete minimizers in a robustmanner. However, due to the exponential time complexity of a fully 2D dy-namic programming task, existing DTW-based image warping methods canonly handle extremely low-resolution images.

1.1.2 3D reconstruction

Previous methods on warp-based 3D reconstruction from serial sections can beappreciated from two perspectives: the types of image deformations performed,and the subjects from which the deformations are computed.

Over the past two decades, many authors have reported how linear transforma-tions can be applied to automatically bring successive sections into alignment(Toga and Arnicar-Sulze (1987); Lamers et al. (1989); Rydmark et al. (1992);Schormann et al. (1995); Hess et al. (1998); Streicher et al. (2000); Ourselinet al. (2001a,b); Brey et al. (2002); Nikou et al. (2003)). The simplicity of lin-ear transformations not only reduces the complexity of computation but alsoallows convenient manual adjustment through software interfaces (Rosen et al.(2000); Chen et al. (2003); Karen et al. (2003); Andrey and Maurin (2005)).However, as shown in several studies (Deverell et al. (1989); Jones et al. (1994);Boyle et al. (1997)), the section distortions induced by the preparation pro-cess are local in nature. With the advance of image registration techniques,reconstruction methods were proposed to correct localized section distortionsby using local 2D deformations (Durr et al. (1989); Guest and Baldock (1995);Kim et al. (1997); Timsari et al. (1999); Ali and Cohen (1998)) as well as elas-tic 3D surface deformations (Thompson and Toga (1996); Mega et al. (1997);Gabrani (1998); Gefen et al. (2003)).

A large number of the above reconstruction methods compute the warp of eachsection so that the warped section matches a neighboring section. For example,Durr et. al. (1989) computes elastic deformation between pairs of distorted im-ages, while Karen et. al. (2003) uses software tools for user-assisted alignmentbetween every two consecutive sections by rigid-body transformations. How-ever, unlike image registration where exact matching is preferred, the goalof reconstruction is to form a smooth volume allowing natural progressionof features through successive images (Guest and Baldock (1995)). Recentmethods, and in particular elastic 3D surface deformation methods, focus onwarping distorted sections onto an undistorted reference, such as block-facephotos (Mega et al. (1997); Kim et al. (1997); Gefen et al. (2003)) or tissuemarkers (Streicher et al. (2000); Ourselin et al. (2001b)), 2D sections of a 3Din vivo image (Thompson and Toga (1996); Schormann et al. (1995); Ourselin

4

et al. (2001a)), or sections from an existing template (He et al. (1995); Ali andCohen (1998); Timsari et al. (1999)). The problem with this approach is thatin many applications an un-sectioned reference is not always available.

There have only been a few works so far (Montgomery and Ross (1994); Guestand Baldock (1995); Wirtz et al. (2004)) that address the problem of smooth3D reconstruction using elastic 2D image deformations in the absence of areference volume (note that existing methods using elastic 3D surface defor-mations (Thompson and Toga (1996); Mega et al. (1997); Gefen et al. (2003))can not be applied due to the lack of a reference). Without the knowledge ofthe original object before sectioning, these works based their reconstruction onthe following assumption: the shape of an anatomical structure varies slowlywith respect to section thickness. In other words, corresponding points on ad-jacent sections are likely to be located close together. Both Guest and Baldock(1995) and Wirtz et. al. (2004) consider all sections at the same time and aimat minimizing an overall energy functional consisting of distances between ev-ery two consecutive sections (e.g., spring forces between corresponding points(Guest and Baldock (1995)) or pixel-wise squared differences (Wirtz et al.(2004))) and the sum of elastic deformation potentials on every section. Al-though numerical solutions can be obtained by using finite element method(Guest and Baldock (1995)) or by approximating the differential equationsusing finite differences (Wirtz et al. (2004)), solving the global minimizationproblem involving all sections is non-trivial due to its massive size.

A completely different approach was taken by Montgomery and Ross (1994).The idea is to reposition a point on each section by applying a local Laplaciansmoothing operator on the position of that point and the positions of its corre-sponding points on the two adjacent sections. However, their method requiresmanual delineation of contours on each section to establish correspondence.Moreover, the Laplacian operator is only applied to points on the contours,hence the deformation is restricted to the overall shape of a contoured regionand has limited power in matching interior features with neighboring sections.

1.2 Contributions

In this paper, we introduce a new approach for reconstructing a smooth rep-resentative volume by elastically deforming serial sections, which requires nohuman intervention or the use of a reference volume. Since it is impossibleto “undo” the tissue distortions without the knowledge of the original objectbefore sectioning, we base our reconstruction on the same assumption fromprevious works (Guest and Baldock (1995)), that is, the section thickness (25µm) is small comparing to variations in the shape of anatomical structures. In-stead of considering all sections at the same time (Guest and Baldock (1995);Wirtz et al. (2004)) or only two neighbors for each section (Montgomery and

5

Ross (1994)), we consider an extended neighborhood to compute the deforma-tion of each section. Our goal is to produce a smooth volume where a pointon a section lies close to its corresponding points within its neighborhood. Toachieve this goal, we present:

(1) An automatic 3D reconstruction method, called warp filtering, based on2D image warps between each pair of adjacent sections. During the re-construction, each section is deformed using an average warp computedfrom the pairwise warps within a group of neighboring sections. The av-erage warp results in a smooth volume by effectively repositioning eachpoint on one section to the weighted-average location of the correspond-ing points on the neighboring sections. The algorithm works with any 2Dimage registration techniques for computing pairwise warps, and can beeasily parallelized for speed. We performed quantitative and qualitativevalidation of the reconstruction method on both real and synthetic data,and the results revealed the effectiveness of our method in building asmooth volume from distorted sections.

(2) A new image warping algorithm based on dynamic programming for com-puting regularized warps between adjacent serial sections. Due to thenature of sectioning distortions, we consider a class of 2D warps thatcan be decomposed into 1D piecewise linear deformations with elasticconstraints. The representation of such a decomposable 2D warp greatlyfacilitates the process of warp filtering. Moreover, the decomposition al-lows us to extend a well-known 1D discrete minimization method calledDynamic Time Warping (Sakoe and Chiba (1978)) to compute 2D elas-tic image warps even between images of high resolutions. Experimentalresults have shown that the proposed method achieves an efficiency com-parable to state-of-the-art warping methods, while the resulting defor-mations often match successive tissue sections with improved accuracy.

The rest of the paper is organized as follows. We present the reconstructionmethod in Section II and describe our image warping algorithm in Section III.In Section IV we present our validation framework involving quantitative mea-sures developed from the evaluation criteria suggested by Guest and Baldock(1995), and we report the experimental results on a MRI test volume and astack of 350 histology sections. Finally, we discuss limitations of the methodand future research in Section V.

2 Methods

2.1 Warp Filtering

To reconstruct a volume in which corresponding points on successive sectionsare located close to each other, warp filtering computes image warps thatmatch each section to a group of neighboring sections based on pairwise warps

6

between successive sections.

2.1.1 The algorithm

The input to the algorithm is a stack of N serial sections represented as2D images gk : R2 → R (k = 1, . . . , N) and pairwise warps represented asbivariate, bivalued functions φk,k+1 : R2 → R2 (k = 1, . . . , N − 1) betweenevery two successive images. The warps φk,k+1 can be computed using anyuser-specified 2D image warping method so that the warped image gk ◦ φk,k+15 best matches the neighboring image gk+1.

The algorithm computes new warps Φk : R2 → R2 (k = 1, . . . , N) that matcheach image gk to a group of its neighboring images {gk−d, ..., gk+d} (d > 0). Φk

is represented as the following weighted average,

Φk =k+d∑

i=k−d

γiφk,i (1)

where γi(i = k− d, ..., k + d) are binomially-distributed weights that approxi-mate a Gaussian filter,

γi =

2d

i− k + d

/ 22d,

and φk,i represent the warps from image gk to each image gi in the neighbor-hood i ∈ [k− d, k + d]. Given the pairwise warps {φk−d,k−d+1, . . ., φk+d−1,k+d},φk,i can be constructed inductively as 6

φk,i =

φk,i+1 ◦ φ−1i,i+1, k − d ≤ i < k

φk,k, i = k

φk,i−1 ◦ φi−1,i, k < i ≤ k + d

(2)

where φk,k denotes the identity.

In effect, under the warp Φk, each point on gk is repositioned not to exactlymatch its corresponding points on a single neighboring image, but to matchthe weighted-average location of its corresponding points on a group of im-ages {gk−d, ..., gk+d}. Consequently, in the warped stack of images gk ◦Φk (k =

5 Operator ◦ denotes function composition such that (f1 ◦ f2)(x) = f1(f2(x))6 If pairwise warps φk,k+1 are not invertible, reverse warps φk+1,k have also to becomputed using the chosen image warping method between successive image pairs.

7

(a) (b) (c)

(d) (e)

Fig. 2. Reconstruction of a smooth 2D image from deformed 1D columns usingwarp filtering. (a): the original grayscale image. (b): the same image with distortedcolumns. (c): the reconstructed image. (d,e): a subset of lines connecting corre-sponding points in successive columns before and after reconstruction.

1, . . . , N), high-frequency noise along lines of corresponding points throughsuccessive images, which are often induced by randomized sectioning distor-tions, are removed, and the distances between corresponding points on adja-cent images are reduced.

We illustrate warp filtering in a 2D example in Figure 2. Here we consider thesimplified problem of reconstructing a coherent 2D image from a sequence ofdeformed 1D columns of pixels gk : R → R. Figure 2 (a) shows a grayscaleimage with letters “TMI”, and Figure 2 (b) shows the same image but witheach column gk synthetically distorted using a randomized local function. Toreconstruct a smooth 2D image, pairwise warps φk,k+1 : R → R that establishcorrespondences between points on column gk to column gk+1 are computedbetween successive columns. In Figure 2 (d), a subset of lines connecting cor-responding points on successive columns is plotted based on these pairwisewarps. Observe that the lines are jagged due to the local distortions that varyfrom column to column. Our algorithm computes the filtered warps Φk frompairwise warps φk,k+1 and produces a stack of warped columns gk ◦ Φk thatconstitute a smooth 2D image shown in Figure 2 (c). Observe in Figure 2 (e)that, after warping, corresponding points on successive columns lie close toeach other and form smooth curves in space.

2.1.2 Comparison

The proposed algorithm is most closely related to the approach taken byMontgomery and Ross (1994). In comparison to their method, we present twomajor improvements. First, the correspondence between successive sections,

8

which was established manually using contour lines by Montgomery and Ross(1994), is now computed automatically as image warps. Second, the simpletwo-neighbor Laplacian operator on contour lines is replaced by a more generalde-noising Gaussian filter that operates on general warps between image pairsin an extended neighborhood.

Our method also offers several unique features when compared to global min-imization methods (Guest and Baldock (1995); Wirtz et al. (2004)):

(1) Simplicity: Without the need to setup and solve a 3D minimizationproblem, our local approach involves only 2D image warping and sim-ple weighted-averaging operations. In general, the algorithm can be usedin conjunction with any existing 2D image registration techniques forsmooth 3D reconstruction.

(2) Stability: In contrast with global minimization over all sections, com-puting pairwise warps between two individual sections is a minimizationproblem at a small scale and therefore less prone to errors due to vari-ations in the input data. Such stability is greatly desired due to thesystematic and random nature of section distortions.

(3) Efficiency: The decomposition of the reconstruction problem into warp-ing and filtering tasks makes substantial performance increase possiblethrough parallelization. In particular, since the pairwise warp betweeneach image pair as well as the filtered warp for each image are computedindependently of each other, linear reduction of computation time caneasily be achieved in a distributed computing environment.

2.1.3 Warp representation

Although the warp filtering algorithm can be used with any image warpingtechniques, efficient computation using formula (1) and (2) requires appropri-ate representation of pairwise image warps φk,k+1.

Regardless of how pairwise warps φk,k+1 are represented, the filtered warpΦk can always be implicitly constructed by evaluating the righthand side offormula (1) and (2) at every point (x, y). This approach, however, needs toevaluate {φk−d,k−d+1, . . ., φk+d−1,k+d} for every point in the image. For effi-ciency, we can construct Φk as an explicit function using formula (1) and (2),which can then be used to directly evaluate Φk(x, y) at every point (x, y). Thesecond approach requires the pairwise warps φk,k+1 to be represented for thefollowing operations on functions:

• Inversion: φ−1i,j , representing the warp from image gj to image gi.

• Composition: φi,j ◦ φj,k, representing the warp from image gi to image gk.• Addition: a φi,j + b φi,k, representing a weighted average of the two warps.

9

Although linear functions are feasible, warps represented as higher-degreeglobal polynomials or displacement fields in many local warping methods cannot be easily adapted to take advantage of these function operations. In thenext section, we describe a new image warping method for computing a local,piecewise linear warping function that is readily represented for efficient warpfiltering.

2.2 Computing image warps

It is well-known that the problem of computing the optimal warp is ill-posed(Modersitzki (2004)) and NP-complete (Keysers and Unger (2003)), henceregularization is required. For warping between successive serial cryo-sectionedimages, we consider a specific type of regularized image warps such that

(1) Each 2D warp can be decomposed into a single 1D deformation in thehorizontal direction and independent 1D deformations in the vertical di-rection for each column, and

(2) Each 1D deformation is represented as a monotonic piecewise linear func-tion with elasticity constraints.

As we shall see in the following, such regularized warps are capable of rep-resenting local deformations that are characteristic between adjacent serialsections. On the other hand, the regularization allows fast warp filtering bysupporting the three operations on warp functions (i.e., inversion, compositionand addition) and efficient warp computation using dynamic programming.

2.2.1 Decomposing 2D warps

In our experiment (Carson et al. (2002)), the frozen brain was cut coronallyinto 350 cryo-sections each 25µm thick. Tissue sections were placed on slidesand then a Nissl stain was applied. After staining, a coverslip was applied toeach slide. Assuming that the sections have been rigidly aligned to correct therotational and translational differences introduced when the tissue sections areplaced on the slides (see Section 3.2 for details), localized differences betweenadjacent sections include (see examples in Figure 1):

(1) Inherent anatomical variance between adjacent tissue sections.(2) Regional tissue deformations in the form of vertical stretching and com-

pacting induced both during cutting and when the section is placed ontothe slide. Extreme deformations may lead to tearing and folding.

(3) Image artifacts, such as dust (dark artifact) and air bubbles (black ring-shaped artifact), introduced during coverslipping.

10

While the anatomical variance and image artifacts typically do not have spe-cific orientation, regional distortions in the form of stretching and compactingmostly take place in the vertical (i.e., slicing) direction. To better representthe deformations caused by sectioning distortions while accommodating otherpossible local variances, we shall compute a restricted warp φ : R2 → R2 ofthe following form:

φ(x, y) = (φX(x), φY (x, y))

where φX , φY are single-valued functions. In other words, φX models an overall1D deformation in the horizontal direction (i.e., shifting of image columns) andφY models independent 1D deformations in the vertical direction for differentvalues of x (i.e., shifting of pixels in each column). A similar decompositionhas been considered previously by Agazzi et. al. (1993) for text recognition.Such decomposition greatly simplifies the task of warp computation, since 1D(vector) warps can be obtained much more efficiently and robustly than 2D(image) warps.

Previous authors, such as Cox et. al. (1996), have considered a more restrictedform of 2D warps. In their methods, each 2D warp consists solely of indepen-dent 1D warps on each column (i.e., φX is the identity function). To illustratethe difference, Figure 3 shows a simple example of warping between two im-ages containing rings of different radii, a typical type of deformation betweensuccessive tissue sections. Figure 3 (c) shows the source image deformed us-ing only 1D warps in the vertical direction, and Figure 3 (d) shows the samesource image deformed using the combination of 1D warps in both verticaland horizontal directions. Observe that the horizontal warp φX is necessaryto model the non-vertical variances between source and target images, whichis required to handle the anatomical differences and image artifacts in tissuesections.

2.2.2 Piecewise linear representation

For convenient representations that will facilitate the function operations inwarp filtering, we consider 1D warps {φX , φY } that are monotonic, piecewiselinear functions. In particular, given two images s and t with n + 1 columnsand m + 1 rows,

(1) The horizontal deformation φX is represented by a pair of piecewise linearfunctions σ, ψ : [0, K ∈ Z+] → [0, n], which match column σ(k) in images to column ψ(k) in image t for k = 0, . . . , K.

(2) The vertical deformation φY is represented by a sequence of piecewiselinear function pairs σk, ψk : [0, L ∈ Z+] → [0,m] for k = 0, . . . , K,which match the point (σ(k), σk(l)) in image s to point (ψ(k), ψk(l)) inimage t for k = 0, . . . , K and l = 0, . . . , L.

11

(a) (b)

(c) (d)

Fig. 3. Warping from a source image (a) to a target image (b) by only computingthe vertical deformation (c) and by computing deformations in both the horizontaland vertical direction (d).

We require functions σ, ψ and σΣ, ψΣ7 to be invertible, so that the warp φs,t

from image s to image t can be represented as:

φs,t(x, y) = (σ(x̂), σx̂(ψ−1x̂ (y)))

where x̂ = ψ−1(x) for x ∈ [0, n] and y ∈ [0, m] (linear interpolation is usedwhen subscript x̂ assumes a non-integer value). The symmetry of σ and ψallows the inverse warp φt,s from image t to image s to be represented in thesame way with symbols σ and ψ exchanged. Such convenience will later becomecrucial in performing function operations during warp filtering in Section IIID.

2.2.3 Elastic deformation

To prevent excessive image deformation, an image is typically modelled as anelastic material and image warps are constrained by some form of deformationenergy. Due to the piecewise-linear nature of our warp representation, weconsider a discrete form of deformation energy that consists of three terms:

(1) 1st and 2nd order deformation energy in the X direction:

EX(σ, ψ) = αX

K∑

k=0

δ(k)2 + βX

K−1∑

k=0

(δ(k + 1)− δ(k))2

7 σΣ is the short hand for writing the whole sequence σ0, . . . , σK .

12

where δ(k) = σ(k)− ψ(k) and αX , βX are constant weights.(2) Independent 1st and 2nd order deformation energy in the Y direction for

each vertical warp:

EY (σk, ψk) = αY

L∑

l=0

δk(l)2 + βY

L−1∑

l=0

(δk(l + 1)− δk(l))2

where δk(l) = σk(l)− ψk(l), and αY , βY are constant weights.(3) Coherence between neighboring vertical warp functions:

EC(σΣ, ψΣ) = γK−1∑

k=0

L∑

l=0

((σk+1(l)− σk(l))2 + (ψk+1(l)− ψk(l))

2).

where γ is a constant weight.

Note that the inclusion of the coherence term in 3) is necessary to ensure asmooth deformation field using a 2D warp that is decomposed into indepen-dent 1D warps.

Putting together, the warping problem that we consider becomes the mini-mization of the following error:

M(σ, ψ, σΣ, ψΣ) =K∑

k=0

MY (σ(k), ψ(k), σk, ψk) + EX(σ, ψ) + EC(σΣ, ψΣ)(3)

where MY is the warping error for each independent vertical warp under thel2-norm (e.g., sum of pixel-wise squared differences):

MY (σ(k), ψ(k), σk, ψk) =L∑

l=0

(s(σ(k), σk(l))− t(ψ(k), ψk(l)))2 + EY (σk, ψk)(4)

2.2.4 Dynamic programming

To compute a minimizer of (3), we first need an efficient method for minimizingthe 1D warping error in (4). Dynamic Time Warping (DTW) (Sakoe and Chiba(1978)) is such a method that robustly computes a discrete minimizer in 1Dmatching problems. Here we describe how 1D DTW can be extended to warp2D images by computing a discrete minimizer of (3).

2.2.4.1 1D Warping: First we consider the 1D problem of computing thevertical warp pair (σk, ψk) between column σ(k) in image s and column ψ(k)in image t, assuming the existence of σ and ψ. DTW discretizes (σk(l), ψk(l))(l = 0, ..., L) as a path on the integer grid from (0, 0) to (m,m). The original

13

DTW algorithm computes a Manhattan path from (0, 0) to (m,m), as shownin Figure 4 (a). However, the resulting piecewise linear functions (σk, ψk) wouldbecome non-invertible due to existence of zero derivatives on the linear seg-ments. To enforce invertibility, we adopt a similar technique to the methodof Kovar and Gleicher (2003) that restricts the slope of every segment of thepath to be 1/2, 1 or 2. An example path is shown in Figure 4 (b). UsingDTW, the path with bounded slope that minimizes the 1D warping errorMY (σ(k), ψ(k), σk, ψk) defined in (4) can be found in three steps:

(1) Construct a m + 1 by m + 1 error table ε, so that

εi,j = (s(σ(k), i)− t(ψ(k), j)2 + αY (i− j)2,∀ i, j ∈ [0,m]

(2) Construct a m + 1 by m + 1 auxiliary table e, in which ei,j is the errorof the minimal-error path with bounded slope from entry (0, 0) to entry(i, j) in the error table ε. We can compute ei,j inductively as follows:

ei,j =

Min

ei−2,j−1 + Left(i, j) + βY

ei−1,j−2 + Down(i, j) + βY

ei−3,j−3 + Diag(i, j)

, i > 0 or j > 0

εi,j, i = 0 and j = 0

∞, i < 0 or j < 0

(5)

where each function Left, Down, and Diag computes a weighted sumof entries in ε according to the direction and length of the previous pathsegment, as shown in Figure 5. Note that the sum of the weights is pro-portional to the length of the path segment in each direction.

(3) The minimal-error path that leads to em,m encodes the piecewise linearfunctions (σk, ψk) : [0, L] → [0,m] where L is the number of path seg-ments and eσk(l),ψk(l) are entries along the path for l = 0, . . . , L.

(0,0)

(m,m)

(0,0)

(m,m)

(a) (b)

Fig. 4. Comparison of a Manhattan path (a) and a restricted path (b) whose slopeis bounded between 1/2 and 2 and whose possible nodes (colored dark gray) lie ina skewed square along the diagonal.

14

1

2/3

2/3

1/3

1/3 1

2/3

1/3

2/3

1/3

ji,ε

1-j2,-iε

ji,ε

2-j1,-iε

1

1

1

1/2

1/21/2

1/21/2

1/2

ji,ε

3-j3,-iε

Fig. 5. Weights assigned to the entries in the error table ε for computing functionsLeft(i, j), Down(i, j) and Diag(i, j).

Both time and space complexity of the above algorithms is O(m2). However,we can dramatically improve the speed and the memory usage in practice.First, εi,j and ei,j only have to be computed for grid points i, j that lie withina skewed square of size (m+2)/3 along the diagonal, as shown in Figure 4 (b),which effectively reduces the computation by a factor of 9. Furthermore, wecan limit our computation to grid points {i, j} within a narrow band along thediagonal, whose half-width is the maximum deformation ‖σk(l)−ψk(l)‖. Whenwarping between adjacent tissue sections, we typically observe a maximumdeformation of less than 10% in both horizontal and vertical directions, hencebanding reduces the computation further by an approximate factor of 5.

2.2.4.2 2D Warping: The 2D warping task M(σ, ψ, σΣ, ψΣ) is achievedin two stages. First, observing a similar form on the right-hand side of the2D minimization goal in (3) (ignoring the coherence term EC) and the 1dminimization goal in (4), we consider functions (σ, ψ) such that (σ(k), ψ(k))(k = 0, ..., K) form a path on the integer grid from (0, 0) to (n, n) with boundedslope. Similarly, using DTW, such a path that minimizes the sum of verticalwarping error

∑Kk=0 MY (σ(k), ψ(k), σk, ψk) and the horizontal elastic energy

EX(σ, ψ) can be found in three steps:

(1) Construct a n + 1 by n + 1 error table ε, so that

εi,j = MY (i, j, σi, ψj) + αX(i− j)2,∀ i, j ∈ [0, n]

where (σi, ψj) is the minimal-error vertical warp between column i in sand column j in t, computed using the previous 1D algorithm.

(2) Construct a n + 1 by n + 1 auxiliary table e in the same way as in (5)except constant βY is replaced by βX .

(3) The minimal-error path that leads to en,n encodes the piecewise linearfunctions (σ, ψ) : [0, K] → [0, n] where K is the number of path segmentsand eσ(k),ψ(k) are entries along the path for k = 0, . . . , K.

Next, given (σ, ψ), the vertical warps (σΣ, ψΣ) can be then computed usingthe previous 1D algorithm. However, these minimal-error vertical warps yield2D warps that may vary significantly from column to column. For example,

15

Figure 6 (a) shows the result of applying a 2D warp computed from Section14 to Section 15 in Figure 1, without considering the coherence among verti-cal warps, to a test pattern (a sequence of uniformly spaced horizontal bars).Notice the high-frequency noise in the warped test pattern due to the largedisparity between vertical warps in neighboring columns. Typically, the true2D deformations induced during the sectioning process are much smoother. Toincorporate the coherence energy EC(σΣ, ψΣ), given (σ, ψ), instead of comput-ing a single best vertical warp (σk, ψk) for every k = 0, . . . , K, we compute asubset of the low-error warps (σkΣ

, ψkΣ) in step 3 of the 1D algorithm. Finally,

we add an extra dynamic programming pass to choose the best warp (σk, ψk)from each group (σkΣ

, ψkΣ) that minimizes the sum of vertical warping error∑K

k=0 MY (σ(k), ψ(k), σk, ψk) and the coherence energy EC(σΣ, ψΣ) (each ver-tical warp function must be first re-parameterized to have the same domain[0, L], see Section III E). Figure 6 (b) shows the coherent warp applied to thesame test pattern, which exhibits a much smoother deformation.

(a) (b)

Fig. 6. A test pattern warped by applying a non-coherent warp (a) and a coherentwarp (b) from Section 14 to Section 15 in Figure 1.

The dominant operation in 2D warping is the construction of error table ε instep 1, which has time complexity O(n2m2). However, the computation can beefficiently implemented by restricting the calculation to a subset of entries andusing appropriate banding, as described in 1D warping. Experiment resultshave shown that the speed of our algorithm is comparable to other state-of-the-art methods while producing higher quality warps.

2.2.4.3 Examples and comparison: Figure 7 shows two examples ofwarping adjacent Nissl-stained cryo-sections shown in Figure 1. In each exam-ple, the goal is to find the warp that deforms the source section s to matchthe target section t. Sections are represented as grayscale images of dimen-sions n = 850 and m = 670. We compare the result of applying our method 8

to the result of using two other popular warping methods: Automated ImageRegistration (AIR) (Woods et al. (1998a)), and NLM Insight Segmentationand Registration Toolkit (ITK) (Yoo et al. (2002)). For AIR, we used the182-parameter 12-degree global polynomial non-rigid transformation model

8 With elastic energy weights αX = αY = 0.00001, βX = βY = 0.02, γ = 0.005

16

9 . For ITK, we used the FEM-based local deformable registration method 10 .The l2-norm between the warped source image and the target image in eachmethod as well as the computation time are compared in Table 1. Note thatthe dynamic programming method used in this paper achieves a comparableefficiency to a global warping method, while the quality of the resulting 2Dwarp is often better than a conventional local warping method when deformingbetween successive tissue sections.

Example 1 Example 2

Time l2-norm Time l2-norm

Source Image (s) - 11717.1 - 17069.7

Polynomial transformation (AIR(s)) 36 sec 5907.5 252 sec 12419.5

Deformable registration (ITK(s)) 1325 sec 5005.9 1190 sec 14314.6

Dynamic programming (DP (s)) 165 sec 3951.9 173 sec 9207.4Table 1Performance comparison of three warping methods applied to the two examplesin figure 7. The l2-norm is computed between the source image (or warped sourceimage) and the target image.

Observe from Figure 7 that the proposed warping method also handles extremedeformations (e.g., sever folding in the second example) and image artifacts(e.g., the big air bubble in the first example) in a reasonable manner. Theserandom artifacts and distortions are examples of incompatible features be-tween the source and target images, which imply that there does not exist aperfect warp that exactly matches one image to the other. Such incompatiblefeatures may also assume the form of tissue tears, or appearing(disappearing)anatomical features (e.g., the dense circular feature in upper middle of thetarget image in the first example) that are typical through successive tissuesections. The proposed warping method matches compatible features betweenthe two images well while producing moderate modifications in places whereincompatibility arises without triggering excessive deformations.

9 In our test, align warp program was used with model menu number (m) 32,threshold (t1,t2) 2, and convergence threshold (c) 0.0000110 In our test, FEMRegistrationFilter was used with mean square metric, single-resolution, 8 pixels per element, 40 iterations with elasticity (E) and density (RhoC)set to 1.e5.

17

t t

s ‖s− t‖ s ‖s− t‖

AIR(s) ‖AIR(s)− t‖ AIR(s) ‖AIR(s)− t‖

ITK(s) ‖ITK(s)− t‖ ITK(s) ‖ITK(s)− t‖

DP (s) ‖DP (s)− t‖ DP (s) ‖DP (s)− t‖

Fig. 7. Two examples of warping from a source image s to a target image t usingglobal polynomial transformation in AIR package (AIR(s)), deformable registrationimplementation in ITK (ITK(s)), and the dynamic programming method presentedin this paper (DP (s)).

18

)x(*σ

)x(*ψ

(0,0)

(n,n)

x

(0,0)

(n,n)

(a) (b)

Fig. 8. (a): Re-parameterized functions σ∗ and ψ∗ of the restricted path (coloredin gray) along the diagonal from (0, 0) to (n, n). (b): Symmetric addition of twodiagonalized warps (colored in gray), the resulted warp is plotted in solid black.

2.2.5 Warp operations

As we mentioned before, the piecewise linear 2D warp representation facili-tates the three operations involved in warp filtering (i.e., inversion, additionand composition). First, we need to resolve a subtle problem that arises in ourdynamic programming approach. That is, horizontal warps (σ, ψ) between dif-ferent images may have different domain K, and even vertical warps (σΣ, ψΣ)in a same 2D warp may have different domain L. For convenience, we re-parameterize every horizontal warp σ, ψ : [0, K] → [0, n] onto a fixed domain[0, 2n] as follows:

σ∗(x) = σ(ω−1(x)), ψ∗(x) = ψ(ω−1(x)) (6)

where ω(x) : [0, K] → [0, 2n] is a monotonic, invertible mapping defined asω(x) = σ(x) + ψ(x). Intuitively, (6) re-parameterizes the two functions σ andψ along the diagonal from (0, 0) to (n, n), as shown in Figure 8 (a). Everyvertical warp (σk, ψk) can be diagonalized onto domain [0, 2m] in a similarfashion.

2.2.5.1 1D warp operations:

• Inversion: Given a pair of 1D warps {σ, ψ}, the inverted warp {σ, ψ}−1 iseasily obtained by exchanging the two functions as {ψ, σ}.

• Addition: Given two pairs of warps {σ, ψ} and {σ̂, ψ̂} with domain [0, 2n],the addition a{σ, ψ} + b{σ̂, ψ̂} is simply computed as the symmetric warp{aσ + bσ̂, aψ + bψ̂}. Note that the new warping functions preserves themonoticity, invertibility and the same domain. Figure 8 (b) illustrates thisoperation for a = b = 1/2.

• Composition: The composition of two pairs of warps {σ, ψ} and {σ̂, ψ̂} withdomain [0, 2n], represented as {σ, ψ} ◦ {σ̂, ψ̂}, is computed as a new sym-metric warp {σ(ψ−1), ψ̂(σ̂−1)}. Note that the new warping functions are

19

defined on the domain [0, n], so diagonalization (6) must be performed tore-parameterize σ(ψ−1) and ψ̂(σ̂−1) onto domain [0, 2n].

2.2.5.2 2D warp operations: Let {σ, ψ, σΣ, ψΣ} be a symmetric 2Dwarp between two images, the inverted warp is easily obtained by exchangingthe symbols σ and ψ.

Given two 2D warps {σ, ψ, σΣ, ψΣ} and {σ̂, ψ̂, σ̂Σ, ψ̂Σ}, the addition can becomputed as simply {aσ + bσ̂, aψ + bψ̂, aσΣ + bσ̂Σ, aψΣ + bψ̂Σ}. Compositionof the two warps can be performed as follows. First, the horizontal warp inthe composite 2D warp, denoted by {σ̄, ψ̄}, is computed as the compositionof horizontal warps {σ, ψ} ◦ {σ̂, ψ̂}. Then, the vertical warps in the composite2D warp, denoted by {σ̄Σ, ψ̄Σ}, are computed such that

{σ̄k, ψ̄k} = {σk1 , ψk1} ◦ {σ̂k2 , ψ̂k2}

where k1 = σ−1(σ̄(k)) and k2 = ψ̂−1(ψ̄(k)).

3 Results

Here we report the validation methods and the corresponding results for eval-uating the effectiveness of the proposed reconstruction method both qual-itatively (i.e., by visual examination) and quantitatively (i.e., by computingdistance and smoothness measures). These methods form a general frameworkfor validating any warp-based 3D reconstruction algorithm. The evaluation iscarried out on two sets of data: a synthetic volume with known distortions,and real serial sections with unknown distortions. All computations are per-formed on a commodity PC with 1.5GHz AMD Athlon processor and 2.5GBmemory.

3.1 Using a synthetic volume

We first apply our reconstruction method to a stack of synthetically distortedcross-sections from an existing 3D volume. In our experiment, we use a MRIvolume of the C57BL/6J mouse brain that has been generously provided by theLONI group at UCLA. The volume has dimension 2563 with a uniform spacingof 56µm. Figure 9 (a) shows a sagittal cross-section of the MRI volume. Wetake each of the 256 coronal cross-sections of the volume and apply a B-splinebased local image distortion that varies randomly from section to section.Figure 9 (b) and (c) shows one of the coronal section from the MRI volume

20

before and after the synthetic distortion. Notice that deformation mainly takesplace in the vertical direction so as to simulate the actual cutting distortionsin real cryo-sections. The sagittal cross-sectional view of the distorted volumeis shown in Figure 9 (d).

The reconstruction proceeds by first computing the pairwise warps betweensuccessive sections followed by warp filtering with width d = 5. The computa-tion took 742 seconds (12 minutes and 22 seconds) in total, with 661 secondsspent on warp computation and 81 seconds on warp filtering and final imagedeformation.

3.1.1 Visual validation

Figure 9 (e) shows the sagittal cross-section at the same location throughthe reconstructed stack. In comparison with Figure 9 (d), our reconstructionmethod removes the majority of high frequency noises caused by syntheticdistortions and is able to form smooth and easily-recognizable anatomicalstructures.

3.1.2 Quantitative validation

We further compute the l2-norm from each coronal cross-section in the originalMRI volume to the same section after synthetic distortion and after recon-struction, which are plotted in Figure 10. The sum of all section-wise l2-normare reduced by 46% after automatic reconstruction. The l2-norm quantitativelymeasures how close the reconstructed volume is to the real volume. However,one should realize that such measure is imperfect in that exact restoration ofthe un-distorted volume is not the goal of our reconstruction. Since there isno way to “undo” a distortion without the knowledge of the image before thedistortion, the goal of our reconstruction is to instead produce a smooth vol-ume representation. In the next experiment we present a better criteria thatmeasures the smoothness of the reconstructed volume.

21

(a)

(b) (c)

(d)

(e)

Fig. 9. Validation of warp filtering using an existing 3D volume. (a): A sagittalcross-section of a MRI volume of the mouse brain. (b): A coronal section from theMRI volume. (c): The same section in (b) after synthetic distortion. (d): Sagittalcross-section of the MRI volume after each coronal section has been randomly dis-torted. (e): Sagittal cross-section of the reconstructed volume from distorted sectionsusing warp filtering.

22

0

20

40

60

80

100

120

1 51 101 151 201 251

Coronal MRI Sections

L2

Dis

tan

ces

Fig. 10. The l2-norm from each of the 256 coronal sections of a MRI volume to thesame section after synthetic distortion (top curve) and to the reconstructed sectionafter warp filtering (bottom curve).

3.2 Using serial sections

We next test our method in reconstructing a 3D volumetric representation ofthe cell-density for the mouse brain from serial sections. The input is a stack of350 Nissl-stained images acquired by cyro-sectioning coronally a single frozenadult mouse brain (data preparation is detailed in Section III A). Each imageis size 850× 670 pixels at a resolution of 15µm per pixel.

3.2.1 Pre-processing

The images were first registered using rigid-body transformations that alignthe symmetry axis of each coronal section to the vertical midline of the im-age and vertically adjust the sections using reference sagittal histology sec-tions from Paxino’s Atlas (Paxinos and Franklin (2000)). The axis alignmentensures the vertical direction of sectioning distortion, to which our warpingmethod is tailored. The vertical adjustment avoids causing a straight cylin-der to be reconstructed as a sloping cylinder by warp filtering. The resultingstack exhibits a natural 3D profile with minimal translational and rotationaldifferences between adjacent sections. Figure 12 shows synthetic cross-sectioncuts in the sagittal (a1) and horizontal (b1) direction through the rigid-bodyaligned stack.

Since the distance between adjacent sections (25µm) far exceeds the size ofa cell (∼ 10µm), matching individual cells on two adjacent sections is notpractical. To avoid aligning cellular details while matching macro features

23

(e.g., the dark folds of the cerebellum) during warp computation, we firstapply a smoothing filter on the tissue sections before computing the pairwisewarps. Although a Gaussian filter could be used, an edge-preserving filteris more appropriate in that boundaries between anatomical structures areretained. Figure 11 shows the result of applying a bilateral filter (Tomasiand Manduchi (1998)) that we used in our experiment (other filters may alsobe used). Note that the filtered image exhibits clearer boundaries betweendifferent anatomical features. After performing warp filtering on the bilaterallyfiltered images, the final warps would then be applied to the original imagesfor accurate reconstruction.

(a) (b)

Fig. 11. A tissue section before (a) and after (b) applying the bilateral filter.

3.2.2 Performance

The computation of 349 pairwise warps between successive sections consumes27MB memory and a total of 976 minutes (16 hours and 16 minutes), averaging168 seconds for each single warp. The performance of the subsequent warpfiltering stage at different filtering width d is summarized in Table 2. We wantto point out that since every pairwise warp φk,k+1 is computed independentlyof each other (and so is every filtered warp Φk), both warp computation andfiltering can be greatly sped up by distributing the computation of differentwarps to different processors. Using this simple parallelization scheme on acluster of 16 processors, for example, all pairwise warps would be computedwithin an hour while warp filtering would finish in approximately ten minutesat d = 20.

3.2.3 Visual validation and comparison

Figure 12 shows cross-section cuts in the sagittal (a2) and horizontal (b2)direction through the smooth volume reconstructed using filtering width d =20 (while the filtered warps are computed from bilaterally filtered images, theyare applied to the original images). Note that anatomical features becomemuch more coherent in the reconstructed volume. The reconstruction alsorecovers the shape of some key structures, such as the folds in the cerebellum

24

Filter width d Time (min) Memory (MB) Mean Sk Max Sk

0 (Original stack) - - 85.43 1165.68

1 4.4 37 7.71 74.39

2 17.2 91 3.15 24.83

5 46.6 178 1.33 7.04

10 83.4 269 1.09 6.22

20 172.1 449 1.04 6.15Table 2Comparison of performance for different filtering width d and the correspondingmean and maximum smoothness measure Sk of the reconstructed volume.

to the left and the hippocampus region in the middle, whose profiles are barelyrecognizable or completely illegible in the original stack.

For qualitative validation, we compare the reconstructed volume to real his-tology sections from Paxino’s Atlas (Paxinos and Franklin (2000)) at similarsagittal (a4) and horizontal (b4) locations. We find that the shapes of theanatomical structures recovered in the reconstructed volume closely resemblethe shapes of the corresponding structures in real tissue sections.

For comparison, we show in Figure 12 (a3) and (b4) the results of applying theFEM-based reconstruction method of Guest et. al. (1995). The input to theirmethod consists of the same stack of coronal images after bilateral filtering asto our method, yet the deformations computed by their method were appliedto the filtered instead of the original images. Observe that in comparison, ourmethod reconstructs smoother anatomical shapes and tissue boundaries.

3.2.4 Quantitative validation

Objective validation with serial sections is difficult due to the lack of theknowledge of the original object before sectioning. One commonly used tech-nique is to measure the correlation between adjacent sections before and afterreconstruction, as done by Wirtz et. al. (2004). However, as pointed out byGuest and Baldock (1995), such criteria is inappropriate since the purposeof reconstruction is not to exactly match one section to the next section. In-spired by the smoothness measure proposed by Guest and Baldock (1995), wepropose the following two-step procedure to measure the smoothness of ourreconstruction:

25

Fig. 12. Comparison of sagittal (a1 to a4) and horizontal (b1 to b4) cross-sections:the original stack of 350 brain sections after rigid-body alignment (a1 and b1),the reconstructed volume using our method (a2 and b2), and reconstruction by themethod of Guest and Baldock (1995) (a3 and b3) on filtered images, and histologicalsections from the Paxino’s atlas (Paxinos and Franklin (2000)) at sagittal plate No.110 (a4) and horizontal plate No. 157 (b4).

26

3.2.4.1 Correspondence evaluation: We first evaluate the quality ofthe pairwise warps φk,k+1 that establish the correspondences between adjacentimages gk, gk+1 for k = 1, . . . , N . This is accomplished by computing the l2-norm between each warped image gk ◦φk,k+1 and target image gk+1, as plottedin Figure 13, which reveals the quality of matching. For comparison, we plotthe l2-norm between successive images in the original stack as well as in therigidly registered stack. Observe that the pairwise warps reduce the pixel-wise differences between neighboring sections substantially by 60% to 90% incomparison to the original stack. Correspondence evaluation is a necessarystep as the smoothness of corresponding features will make no sense if thecorrespondences are computed incorrectly.

0

5000

10000

15000

20000

25000

30000

35000

1 51 101 151 201 251 301

Coronal Sections

L2

Dis

tan

ce

Fig. 13. The l2-norm between successive images in the stack of 350 coronal sectionsbefore (top curve) and after (middle curve) rigid-body alignment. The elastic warp-ing error between successive sections (bottom curve) is computed as the l2-normbetween the warped image gk ◦ Φk and image gk+1 for k = 1, ..., 349.

3.2.4.2 Smoothness evaluation: We compute a smoothness measure Sk

on each section k as follows:

Sk =

∑ni=0

∑mj=0(B(i, j) + C(i, j)− 2A(i, j))2

nm(7)

where A(i, j) = Φ−1k (i, j) denotes the location of (i, j) in the warped image

gk ◦Φk, B(i, j) = Φ−1k−1(φk−1,k(i, j)) and C(i, j) = Φ−1

k+1(φ−1k,k+1(i, j)) denote the

locations of the corresponding points of (i, j) on the two neighboring sectionsgk−1 ◦Φk−1 and gk+1 ◦Φk+1 respectively after warping. As a reminder, φ refersto pairwise warps between adjacent sections, and Φ refers to the filtered warp.Sk is similar to the CAM measure proposed by Guest and Baldock (1995),which effectively measures how close a point lies to its corresponding pointson the neighboring sections. For comparison, the smoothness measure of each

27

section before reconstruction is computed by setting Φk−1, Φk and Φk+1 to beidentity functions.

Table 2 reports the mean and maximum smoothness measure Sk for all 350sections in the reconstructed volume for different filtering width d. Observethat Sk decreases by an order of magnitude just by warp filtering using asingle neighbor on each side of a section (i.e., d = 1). A reduction by twoorders of magnitude is observed in both the mean and maximum Sk for d ≥ 5.To interpret the numbers, we notice from (7) that Sk reflects the averagedistance from every point A(i, j) to the mid-point of its two correspondingpoints (B(i, j) + C(i, j))/2. For example, at d = 20, Sk = 1.04, hence theaverage distance ‖(B(i, j) + C(i, j))/2 − A(i, j)‖ = 0.51. In other words, inthe reconstructed volume, each point deviates from the middle of its two cor-responding points on the neighboring sections by only 0.51 pixels on average.These results appear to improve on the experimental results of a global recon-struction method reported by Guest and Baldock (1995) that uses a similarmeasurement of smoothness.

4 Discussion

4.1 Noise removal using majority filtering

Despite the use of elastic warping, image artifacts such as tissue folds andair bubbles remain after reconstruction. To clean up the image artifacts, aneffective solution we found is applying a majority filter to corresponding pointsthrough successive images based on the pairwise image warps. Specifically, foreach pixel on an image, if the intensity of the pixel is the highest or thelowest among its corresponding pixels in a group of nearby images, we let itinherit the average intensity of its neighbors (see Figure 14 top). The filterremoves outliers along matched pixels through different images, which areoften introduced by bubbles or tissue folding on a single image. The bottomof Figure 14 shows a coronal section in the reconstructed volume before andafter the majority filter is applied. Observe that the dark folds in the originalimage are much less apparent in the filtered image. It should be noted thatthe purpose of majority filtering is to ensure a smooth appearance of thereconstructed volume, which is ideal for overall visualization but may not besuitable for analysis of fine anatomical details. For efficient implementation,majority filtering of pixel intensities can be performed as the same time whenthe Gaussian filter is applied to the pairwise image warps.

Figure 15 (a) and (b) show the cross-section cuts of the reconstructed volumeafter incorporating the majority filter, which is applied to 3 neighbors on each

28

(a) (b)

(c) (d)

Fig. 14. (a): A dark pixel and its corresponding pixels on the neighboring sections.(b): Using majority filter, the dark pixel is replaced with the average pixel intensityof its neighbors. (c): A reconstructed coronal section. (d) The same coronal sectionafter the majority filter is applied to the stack.

side of a given pixel. In comparison with Figure 12 (a2) and (b2), the ma-jority filtering along lines of corresponding points through successive sectionsimprove the smoothness of the appearance.

(a) (b)

Fig. 15. Synthetic cross-section of the reconstructed brain volume after majorityfiltering in the sagittal (a) and horizontal (b) directions.

4.2 Smoothness of the volume

The smoothness of the reconstructed volume using our method is controlledby the filtering width d. Inappropriate choices of d, however, may cause themethod to either flatten the highly-curved or sharp features (when d is toobig) or fail to remove the distortion errors (when d is too small). This problemof obtaining the desired amount of smoothness in the reconstructed volumearises from all smoothness-based reconstruction methods (Guest and Baldock(1995); Wirtz et al. (2004)).

29

A good choice of d depends on the input data. In most cases, the scale ofrandomized tissue distortions is small in comparison to the size of anatomicalfeatures, and the value of d can be determined via experiments and visualevaluation. For example, Figure 16 shows the hippocampus region in a sagit-tal cross-section cut through the original stack of 350 images and the recon-structed stack with filter width d = 5, 10, 20. Note that although a choice ofd = 5 already achieves a good local smoothness measure in Table 2, the darkstripe at the top of hippocampus still exhibits low-frequency noise inherent tothe input data, which spans a larger neighborhood and requires a wider filter-ing width. At d = 20, the waviness is gone and the region becomes smooth.Still, the reconstructed volume needs to be examined to ensure that other keyanatomical features have not been overly smoothed.

(a) (b) (c) (d)

Fig. 16. Closeup looks at the hippocampus region in the synthetic sagittalcross-section of the rigid-body aligned stack (a) and the elastically aligned stackwith filter width d = 5 (b), d = 10 (c) and d = 20 (d). The region is indicated asred boxes in figure 12 (a1) and (a2).

Sometimes a reasonable choice of d fails to exist, for example, when the scaleof the largest distortion noise exceeds that of the smallest anatomical feature.To remove large distortions in one place while preserving small meaningfulfeatures in other locations, a more flexible method is to find independent filterwidths dk for each section gk or even independent widths dk(i, j) for eachindividual pixel gk(i, j). These local filter widths will adapt to the regionalnoise-signal ratio based on the correspondence information between adjacentsections. Such local choices of filter widths can be easily incorporated into thewarp filtering framework in (1) and (2). Moreover, edge-preserving filters, suchas a bilateral filter (Jiang et al. (2003)), can be used in place of the Gaussianfilter during warp filtering for preserving sharp features in the reconstructedvolume that are often flattened out using the Gaussian filter.

4.3 Limitation of 2D warps

Any restricted image deformation, such as the regularized 2D warp describedin Section IV, has its limitations. Specifically, our warp representation onlymodels those 2D deformations that can be characterized by non-uniform de-formations in the vertical direction and uniform deformation in the hori-zontal direction. Eligible deformations include translation, scaling, uniform

30

growth/shrinkage of anatomical structures in the horizontal direction, andsectioning distortions in the vertical direction. The deformations that are notwell modelled include rotation and non-uniform growth/shrinkage of anatom-ical structures in the horizontal direction. Although rotational differences be-tween adjacent sections can be eliminated using rigid-body registration, we doobserve, in a few cases, non-uniform variance of anatomical structures in thehorizontal direction that are not modelled accurately by our method. Evenfor these rare cases, our method behaves in a reasonable manner, as observedfrom the uniform appearance of the graph of warping error in Figure 13. Tobetter handle a wider range of 2D deformations, we are currently investigatingmore flexible ways of decomposing 2D warps into 1D warps, such as the ideaof allowing columns to bend and break during the horizontal deformation asdiscussed in (Levin and Pieraccini (1992); Uchida and Sakoe (1998); Roneeet al. (2001)).

While we will continue our research on more accurate and efficient warpingmethods, we notice that comparing the quality of 2D warps generated by var-ious methods is difficult due to the lack of standard benchmark data. In aneffort to promote the development of benchmarks for 2D image warping meth-ods as well as for 3D reconstruction methods, we have made the 350 histologi-cal sections in our experiment as well as our preliminary reconstruction resultavailable for download at http://www.geneatlas.org/gene/data/histology.zip.We encourage those interested in this problem to apply their methods to thisdata.

References

Agazzi, O., Kuo, S., Levin, E., Pieraccini, R., April 1993. Connected and de-graded text recognition using planar hidden markov models. In: IEEE Inter-national Conference on Acoustics, Speech, and Signal Processing (ICASSP-93). Vol. 5. pp. 113–116.

Ali, W., Cohen, F., 1998. Registering coronal histological 2-d sections of a ratbrain with coronal sections of a 3-d brain atlas using geometric curve in-variants and b-spline representation. IEEE Transactions on Medical Imaging17 (6), 957–966.

Andrey, P., Maurin, Y., 2005. Free-d: an integrated environment for three-dimensional reconstruction from serial sections. Journal of NeuroscienceMethods 145, 233–244.

Armstrong, J., Kaiser, K., Mller, A., Fischbach, K., Merchant, N., Strausfeld,N., 1995. Flybrain, an on-line atlas and database for the drosophila nervoussystem. Neuron 15 (1), 17–20.

Berthold, C., Rydmark, M., Corneliuson, O., 1982. Estimation of section-

31

ing compression and thickness of ultrathin sections through vestopal-w-embedded cat spinal roots. Journal of Ultrastructure Research 80, 42–52.

Bookstein, F., 1989. Principal warps: thin-plate splines and the decomposi-tion of deformations. IEEE Transactions on Pattern Analysis and MachineIntelligence 11, 567–585.

Boykov, Y., Veksler, O., Zabih, R., September 1999. Fast approximate energyminimization via graph cuts. In: Proceedings of International Conferenceon Computer Vision. Vol. 1. pp. 377–384.

Boyle, J., Jenkins, J., McKay, I., McPhaden, A., Lindop, G., 1997. An as-sessment of the distortion of arteries due to sectioning in endomyocardialbiopsies. Journal of Pathology 181 (2), 243–246.

Brey, E., King, T., Johnston, C., McIntire, L., Reece, G., Patrick, C., 2002.A technique for quantitative three-dimensional analysis of microvascularstructure. Microvascular Research 63, 279–294.

Brune, R., Bard, J., Dubreuil, C., Guest, E., Hill, W., Kaufman, M., Stark, M.,Davidson, D., Baldock, R., 1999. A three-dimensional model of the mouseat embryonic day 9. Developmental Biology 216 (2), 457–468.

Cannestra, A., Santori, E., Holmes, C., Toga, A., 1997. A three-dimensionalmultimodality brain map of the nemestrina monkey. Brain Research Bulletin43 (2), 141–148.

Carson, J., Thaller, C., Eichele, G., 2002. A transcriptome atlas of the mousebrain at cellular resolution. Current Opinion in Neurobiology 12 (5), 562–565.

Chen, X., Ai, Z., Rasmussen, M., Bajcsy, P., Auvil, L., Welge, M., Leach,L., Vangveeravong, S., Maniotis, A., Folberg, R., 2003. Three-dimensionalreconstruction of extravascular matrix patterns and blood vessels in humanuveal melanoma tissue: techniques and preliminary findings. InvestigativeOphthalmology and Visual Science 44 (7), 2834–2840.

Cox, I., Hingorani, S., Rao, S., Maggs, B., 1996. A maximum likelihood stereoalgorithm. Comput. Vis. Image Und. 63 (3), 542–567.

Deverell, M., Bailey, N., Whimster, W., 1989. Tissue distortion in three di-mensional reconstruction of wax or plastic embedded microscopic structures.Pathol Res Pract 185 (5), 598–601.

Durr, R., Peterhans, E., von der Heydt, R., 1989. Correction of distorted imagepairs with elastic models. European Journal of Cell Biology 48 (25), 85–88.

Gabrani, M., 1998. Multidimensional spline theory and surface-basedalignment of brains. Ph.D. dissertation, Elect. Comput. Eng., DrexelUniv.,Philadelphia, PA, 1998.

Gefen, S., Tretiak, O., Nissanov, J., 2003. Elastic 3-d alignment of rat brainhistological images. IEEE Transactions on Medical Imaging 22 (11), 1480–1489.

Glasbey, C., 1998. A review of image warping methods. Journal of AppliedStatistics 25, 155–171.

Guest, E., Baldock, R., 1995. Aautomatic reconstruction of serial sectionsusing the finite element method. BioImaging 3, 154–167.

32

Guest, E., Berry, E., Baldock, R., Fidrich, M., Smith, M., 2001. Robust pointcorrespondence applied to two- and three- dimensional image registration.IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 1–15.

He, L., Sarrafizadeh, R., Houk, J., 1995. Three-dimensional reconstructionof the rubrocerebellar premotor network of the turtle. Neuroimage 2 (1),21–33.

Hess, A., Lohmann, K., Gundelfinger, E. D., Scheich, H., 1998. A new methodfor reliable and efficient reconstruction of 3-dimensional images from autora-diographs of brain sections. Journal of Neuroscience Methods 84, 77–86.

Jiang, W., Baker, M., Wu, Q., Bajaj, C., Chiu, W., 2003. Applications of abilateral denoising filter in biological electron microscopy. Journal of Struc-tural Biology 144, 114–122.

Jones, A., Milthorpe, B., Howlett, C., 1994. Measurement of microtomy-induced section distortion and its correction for 3-dimensional histologicalreconstructions. Cytometry 15 (2), 95–105.

Karen, P., Jirkovska, M., Tomori, Z., Demjenova, E., Janacek, J., Kubinova,L., 2003. Three-dimensional computer reconstruction of large tissue volumesbased on composing series of high-resolution confocal images by gluemrc andlinkmrc software. Microscopy Research and Technique 62 (5), 415–422.

Keysers, D., Unger, W., 2003. Elastic image matching is np-complete. PatternRecogn. Lett. 24 (1-3), 445–453.

Kim, B., Boes, J., Frey, K., Meyer, C., 1997. Mutual information for automatedunwarping of rat brain autoradiographs. Neuroimage 5, 31–40.

Lamers, W., Laan, A., Huijsmans, D., Smith, J., Los, J., Strackee, J., 1989.Deformation-corrected computer-aided 3-d reconstruction of immunohisto-chemically stained sections of embryonic organs. European Journal of CellBiology 48, 103–106.

Levin, E., Pieraccini, R., 1992. Dynamic planar warping for optical characterrecognition. In: Proc ICASSP. Vol. III. pp. 149–152.

MacKenzie-Graham, A., Lee, E., Dinov, I., Bota, M., Shattuck, D., Ruffins,S., Yuan, H., Konstantinidis, F., Pitiot, A., Ding, Y., Hu, G., Jacobs, R.,Toga, A., 2004. A multimodal, multidimensional atlas of the c57bl/6j mousebrain. Journal of Anatomy 204 (2), 93–102.

Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P., 1997.Multimodality image registration by maximization of mutual information.IEEE Transactions on Medical Imaging 16, 187–198.

Mega, M., Chen, S., Thompson, P., Woods, R., Karaca, T., Tiwari, A., Vinters,H., Small, G., Toga, A., 1997. Mapping histology to metabolism: Coregistra-tion of stained whole-brain sections to premortem pet in alzheimers disease.Neuroimage 5, 147–153.

Modersitzki, J., 2004. Numerical Methods for Image Registration. Oxford Uni-versity Press.

Montgomery, K., Ross, M., 1994. Improvements in semiautomated serial-section reconstruction and visualization of neural tissue from tem images.Three-Dimensional Microscopy (Proc SPIE) 2184, 264–267.

33

Nikou, C., Heitz, F., Nehlig, A., Namer, I. J., Armspach, J.-P., 2003. A robuststatistics-based global energy function for the alignment of serially acquiredautoradiographic sections. Journal of Neuroscience Methods 124, 93–102.

Ourselin, S., Bardinet, E., Dormont, D., Malandain, G., Roche, A., Ayache, N.,Tande, D., Parain, K., Yelnik, J., 2001a. Fusion of histological sections andmr images: towards the construction of an atlas of the human basal ganglia.In: Proceedings of MICCAI 2001(4th International Conference, Utrecht,The Netherlands). pp. 743–751.

Ourselin, S., Roche, A., Subsol, G., Pennec, X., Ayache, N., January 2001b.Reconstructing a 3d structure from serial histological sections. Image andVision Computing 19 (1-2), 25–31.

Paxinos, G., Franklin, K., 2000. The Mouse Brain in Stereotaxic Coordinates,Second Edition. Academic Press.

Rangarajan, A., Chui, H., Bookstein, F. L., 1997. The softassign procrustesmatching algorithm. In: Information Proc. Med. Imaging. pp. 29–42.

Ronee, M., Uchida, S., Sakoe, H., September 2001. Handwritten characterrecognition using piecewise linear two-dimensional warping. In: Proceedingsof the Sixth International Conference on Document Analysis and Recogni-tion (ICDAR ’01). pp. 39–43.

Rosen, G., Williams, A., Capra, J., Connolly, M., Cruz, B., Lu, L., Airey, D.,Kulkarni, K., Williams, R., 2000. The mouse brain library @ www.mbl.org.In: International Mouse Genome Conference. Vol. 14. p. 166.

Roy, S., 1999. Stereo without epipolar lines: A maximum-flow formulation.International Journal of Computer Vision 34 (2), 147–161.

Rydmark, M., Jansson, T., Berthold, C., Gustavsson, T., 1992. Computer-assisted realignment of light micrograph images from consecutive sectionseries of cat cerebral cortex. Journal of Microscopy 165, 29–47.

Sakoe, H., Chiba, S., 1978. Dynamic programming algorithm optimizationfor spoken word recognition. IEEE Trans. Acoustics, Speech and SignalProcessing 26 (1), 43–49.

Schormann, T., Dabringhaus, A., Zilles, K., 1995. Statistics of deformationsin histology and application to improved alignment with mri. IEEE Trans-actions on Medical Imaging 14 (1), 25–35.

Sidman, R., 2005. High resolution brain atlas of c57bl/6j in 2d and 3d.Http://www.hms.harvard.edu/research/brain/atlas.html.

Streicher, J., Donat, M., Strauss, B., Sprle, R., Schughart, K., Mller, G., 2000.Computer-based three-dimensional visualization of developmental gene ex-pression. Nature Genetics 25 (2), 147–152.

Thevenaz, P., Ruttimann, U., Unser, M., 1998. A pyramid approach to sub-pixel registration based on intensity. IEEE Transactions on Medical Imaging7 (1), 27–41.

Thompson, P., Toga, A., 1996. A surface-based technique for warping threedimensional images of the brain. IEEE Transactions on Medical Imaging15 (4), 402–417.

Timsari, B., Tocco, G., Bouteiller, J., Baudry, M., Leahy, R., 1999. Accurate

34

registration of autoradiographic images of rat brain using a 3-d atlas. In:International Conference on Imaging Science, Systems, and Technology.

Toga, A., Arnicar-Sulze, T., 1987. Digital image reconstruction for the studyof brain structure and function. Journal of Neuroscience Methods 20 (1),7–21.

Tomasi, C., Manduchi, R., 1998. Bilateral filtering for gray and color images.In: Proceedings of the IEEE International Conference on Computer Vision.pp. 59–66.

Uchida, S., Sakoe, H., 1998. A monotonic and continuous two-dimensionalwarping based on dynamic programming. In: Proceedings of the 15th Inter-national Conference on Pattern Recognition. Vol. 1. pp. 521–524.

Wirtz, S., Fischer, B., Modersitzki, J., Schmitt, O., February 2004. Super-fast elastic registration of histologic images of a whole rat brain for three-dimensional reconstruction. In: Proceedings of the SPIE Medical Imaging2004. Vol. 5730.

Woods, R., Grafton, S., Holmes, C., Cherry, S., Mazziotta, J., 1998a. Auto-mated image registration: i. general methods and intrasubject, intramodal-ity validation. Journal of Computer Assisted Tomography 22, 139–152.

Woods, R., Grafton, S., Watson, J., Sicotte, N., Mazziotta, J., 1998b. Auto-mated image registration: ii. intersubject validation of linear and nonlinearmodels. Journal of Computer Assisted Tomography 22, 153–165.

Woods, R., Mazziotta, J., Cherry, S., 1993. Mri-pet registration with auto-mated algorithm. Journal of Computer Assisted Tomography 17 (4), 536–546.

Yoo, T., Ackerman, M., Lorensen, W., Schroeder, W., Chalana, V., Aylward,S., Metaxes, D., Whitaker, R., 2002. Engineering and algorithm design foran image processing api: A technical report on itk - the insight toolkit. In:Proc. of Medicine Meets Virtual Reality. IOS Press Amsterdam, pp. 586–592.

35