Application of Partial Differential Equation-BasedInpainting on Sensitivity Maps

Feng Huang,1* Yunmei Chen,2 George R. Duensing,1 James Akao,1 Andrew Rubin,1 andCharles Saylor1

Inpainting is an image interpolation method. Partial differentialequation (PDE)-based digital inpainting techniques are findingbroad applications. In this paper, PDE-based inpainting tech-niques are applied to the field of MR parallel imaging. A novelmodel and its corresponding numerical method are introduced.This model is then applied to sensitivity maps. Coil sensitivitymaps are important for parallel imaging, and they often requireextrapolation and hole filling (holes being dark regions of lowsignal in MR images). These problems can be solved simulta-neously by the application of inpainting techniques. Experimentsfor determining coil sensitivity maps for phantoms and cardiac MRimages demonstrate the accuracy of the proposed model. Imagesgenerated using sensitivity encoding (SENSE) that utilizes in-painted sensitivity maps, thin-plate spline (TPS) estimated sen-sitivity maps, and Gaussian kernel smoothed (GKS) sensitivitymaps are compared. From the experimental results, it can beseen that inpainted sensitivity maps produce better results thanGKS sensitivity maps. The TPS method generates results sim-ilar to those of the inpainting technique but is much moretime-consuming. Magn Reson Med 53:388–397, 2005. © 2005Wiley-Liss, Inc.

Key words: inpainting; sensitivity map; SENSE; partial differen-tial equation; total variation

The term “image inpainting” originally referred to theartistic restoration of a picture’s missing pieces. Computertechniques can significantly reduce the time and effortrequired to fix digital images—not only to fill in blankregions, but also to correct noise. Digital inpainting tech-niques (1–5) are finding broad applications, such as imagerestoration, disocclusion, perceptual image coding, zoom-ing and image super-resolution, and error concealment inwireless image transmission. Because of its broad range ofapplications, various methods for inpainting have beendeveloped, including nonlinear filtering methods, wave-lets and spectral methods, and statistical methods (espe-cially for textures). The most recent approach to nontex-ture inpainting uses partial differential equations (PDEs)and calculus of variations. According to Chan et al. (2),PDE-based total variation (TV) models and the Mumford-Shah model work very well for inpainting problems of amore local nature, such as hole filling. Hole filling (holesbeing regions of low signal in MR images) is one of themost important issues in the correction of MR sensitivitymaps, which are generally derived from MR images.

Sensitivity maps are meant to contain the sensitivityinformation of RF probe coils in MRI systems. Highlyaccurate knowledge of the spatial receiver sensitivity isrequired by both the simultaneous acquisition of spatialharmonics (SMASH) (6) and sensitivity encoding (SENSE)(7,8) techniques. Sensitivity maps are also crucial for cor-recting the inhomogeneities of MRI surface coils. It mustbe noted that accurate sensitivity information can only beobtained where signal is present. Some data sets have largeareas that contribute little or no signal. Such dark regions(i.e., holes) are common, for example, in pulmonary MRIusing fresh blood imaging (FBI) and cardiac imaging. Inthese cases, sensitivity map interpolation techniques arerequired to fix the holes. To deal with slightly varyingtissue configurations and motion, extrapolation over a lim-ited range is also necessary. We therefore need a techniquethat is capable of simultaneously interpolating and extrap-olating. There are some existing techniques for this pur-pose, such as the polynomial fit procedure (8), thin-platesplines (TPS) (9), and wavelets (10). These methods arebased on the assumption that the sensitivity map is suffi-ciently smooth (i.e., contains no sharp variations). Accord-ing to our experiments, this assumption is not always true.If the sensitivity map is piecewise smooth, which is usu-ally true for nonuniform loading, the existing methods arenot sufficient. The inpainting model proposed here is atechnique to handle interpolation and extrapolation ofpiecewise smooth maps simultaneously.

MATERIALS AND METHODS

In this section we first review the conventional TV inpaint-ing model. Then the modified model will be proposed forapplications in parallel imaging. The corresponding nu-merical method will also be given for the purpose of im-plementation.

It will be helpful to first explain some relevant terms.The process by which holes are fixed using inpainting isreferred to as diffusion. There are two basic types of dif-fusion: isotropic and anisotropic. In isotropic diffusion,pixels in hole regions are fixed by assigning them a valueequal to the isotropically weighted average value of theirsurrounding neighbors. Here “isotropically” means thatthe weights will be independent of direction from thepixel, but will decrease with distance. Anisotropic diffu-sion involves assigning these pixels values obtained byextending the intensity contours from non-hole regionsinto hole regions along the “intensity level set.” The in-tensity level set is simply the contour lines of equal inten-sity in an image, i.e., the set of equi-intensity curves.“Intensity” here refers only to the magnitude of the com-plex pixel values.

1Invivo Corporation, Gainesville, Florida.2Department of Mathematics, University of Florida, Gainesville, Florida.*Correspondence to: Feng Huang, Invivo Corporation, 3545 SW 47th Ave,Gainesville, FL 32603. E-mail: Feng.Huang@mridevices.comReceived 15 July 2003; revised 17 September 2004; accepted 19 September2004.DOI 10.1002/mrm.20346Published online in Wiley InterScience (www.interscience.wiley.com).

Magnetic Resonance in Medicine 53:388–397 (2005)

© 2005 Wiley-Liss, Inc. 388

Review of the TV Inpainting Model

If the inpainting problem is treated as an input-outputsystem, then the input is the image, u0, which requiresinpainting along with the specification of the hole regions(dark regions), D, in the image. The output is an image uwithout any holes.

The hole-free part of the image, where inpainting is notneeded, is not supposed to change much. Mathematically,we wish to keep u and u0 as close as possible on the partsof the image not contained in D. For this purpose we define� as the image domain, and �/D as the area in which theimage information is contained, which is referred to as“known locations.”

Let us define

�D�x� � �0, x � D1, x � �/D [1]

Then the desired similarity of u and u0 at “known loca-tions” can be achieved by minimizing the fidelity term,�

��D�x��u � u0�2dx.To reconstruct the missing parts without losing local

image texture near the holes, it is necessary for the imagemodel to smoothly extend the image into the holes andpreserve the edges of objects contained in the image. Thiscan be achieved by minimizing the smoothing term (3,11):

��

�ƒu�dx [2]

which is used to smooth and extend the image along theintensity level set.

The conventional TV model combines these two terms,and is given as

E�u�u0,D� �12 �

�

�D�x��u � u0�2dx � � ��

�ƒu�dx [3]

where � is a parameter used to balance the smoothing andfidelity terms. Given u0 and D, the u that minimizes theenergy functional will be the inpainting result.

Modification of the Exponent of �ƒu� in the TV Model

Let us consider a more general smoothing term�

����ƒu��dx, where �(�) is a functional. Minimizing theenergy of this smoothing term with respect to u can pro-duce a smoother u. The diffusion flow of �����ƒu��dx is

u�x,y,t�t

� div����ƒu��ƒu

�ƒu�� [4]

where div means divergence, and the prime symbol is thederivative operator.

If �(�) � (�)p, then

u�x,y,t�t

� �div� ƒu�ƒu�� , p � 1

�u, p � 2[5]

where � is the Laplacian operator. When p � 1 the term1/�ƒu� makes the diffusion be weighted differently in thetangential direction and normal directions to the level set;smoothing in the normal direction will be impeded when�ƒu� is large. In this case diffusion will not cross the edgesof the level set, thus the diffusion is anisotropic. If p � 2,the diffusion is isotropic because �u equally weighs eachneighboring pixel. Hence the exponent of �ƒu� determinesthe type of diffusion (i.e., isotropic or anisotropic). SeeRefs. 11 and 12 for additional details. This explains themain feature of the conventional TV model, i.e., its edge-preserving property. However, limitations of the TV modelarise when images in which the noise and edges are ex-tremely difficult to distinguish are reconstructed. This isoften true for sensitivity maps. If the conventional TVmodel is applied to a region where noise and edges aredifficult to distinguish, anisotropic diffusion is efficaciousin moving unwanted edges caused by noise into the holes.Because the exponent of �ƒu�p determines the type of dif-fusion, it is natural to consider models in which the expo-nent �ƒu� is a function (18), so that p � p(x). More specif-ically, one might want to consider p � p��ƒu0�� where,lim

y30p�y� � 2 and lim

y3 p�y� � 1.

Therefore, we wish to minimize

��

�ƒu�P�x�ƒu0�� dx [6]

where p can be either a constant (p � 1 or p � 2) or afunction. Here we choose this function to be

p�x,�ƒu0��

� �1 � g�x� � 1 �1

1 � ��ƒ�G�*u0��2,�ƒ�D� � 0

2, �ƒ�D� � 0[7]

which has the desired limiting properties. Here �,� areparameters, the operator “*” denotes convolution, and��D� � 0 denotes the boundary of hole regions. � is used toaccentuate the edges between regions of different intensi-ties; larger � will better preserve edges. � is for the Gauss-ian function, and G� � exp� � �x�2/�4�2��/� is used tosmooth the image by the standard method of weightedaverages. At areas of fairly uniform intensity, �ƒu0� issmall, causing p(x) to be close to 2. Hence the diffusion ismore isotropic. At edges between regions with very differ-ent intensities, �ƒu0� is large so that p(x) will be close to 1;anisotropic diffusion is then applied so that the edges willbe preserved. For the edges between D and �/D, �ƒ�D� islarger than 0, hence p(x) will be set to 2, and the image willbe isotropically extended. Therefore, the goal of choosingp(x) is to control the type of diffusion at specific locationsin the image, so that isotropic and anisotropic diffusion areeffectively combined (with little threshold sensitivity).

PDE-Based Inpainting on Sensitivity Maps 389

Specifically, TV diffusion is used to preserve edges andcreate varying combinations of isotropic and anisotropicdiffusions in regions that may or may not contain signifi-cant features.

Modified TV Model for Sensitivity Maps

To apply the model to sensitivity maps, the conventionalmodel may be modified based on the particular applica-tion. There will be a sensitivity map uj for each coil j. LetIj be the image generated by coil j, and let I be the homo-geneous image, then uj

0 is defined as Ij/I. To avoid singu-larities and consider every coil, ���Dj�x��uj � uj

0�2dxshould be replaced by

�j�

�

�Dj�x��ujI � Ij�2dx [8]

If the square root of the sum-of-squares of the Ij’s is used asthe homogeneous image I, then the sum-of-squares of ev-ery coil’s sensitivity map will be unity at each pixel.Therefore,

��

� ��j

uj2 � 1� 2

dx [9]

is to be minimized, where uj is the inpainted sensitivitymap for the jth probe coil. Then the modified TV model forthe sensitivity map is

E�uj�uj0,D� �

12 �

j�

�

�Dj�x��ujI � Ij�2dx

� ��j�

�

�ƒuj�p�x,�ƒuj0��dx �

�

2 ��

� ��j

uj2 � 1� 2

dx [10]

where �, � are used to balance terms in the energy func-tional. Increasing � in Eq. [10] will result in a smootherimage. Larger � makes the square root of the sum-of-squares of the sensitivity maps closer to unity.

Numerical Implementation Method

The Euler-Lagrange equation for Eq. [10] is

�Dj�x��ujI � Ij�I � �ƒ � ��ƒuj�P�2 Pƒuj�

� ��1 �1

��j

uj2uj � 0 [11]

with boundary condition uj/n� � 0, for the jth coil,where n� is the outward unit normal to the image boundary.

To derive the evolution equations corresponding to theEuler-Lagrange equation, the following difference scheme(13) can be applied to discretize Eq. [11]. Let u be an image,then we define

��x u�k,l� � u�k,l� � u�k � 1,l�,��

x u�k,l�

� u�k � 1,l� � u�k,l� [12]

��y u�k,l� � u�k,l� � u�k,l � 1�,��

y u�k,l�

� u�k,l � 1� � u�k,l�

and then

ƒu�k,l� � ���x u�k,l�,��

y u�k,l�� [13]

where k and l correspond to rows and columns of theimage, respectively.

Let v� � �v1, v2� be a vector field, and let h be the stepspace for this field (generally equal to unity), then accord-ing to (13)

div�v� � ���

x v1 � ��y v2

h, [14]

Applying the difference scheme above, the iteration for-mula for Eq. [11] is found to be

ujn�1

�

��Cjn,1uj

n�k � 1,l� � Cjn,2uj

n�k � 1,l� � Cjn,3uj

n�k,l � 1�� Cj

n,4ujn�k,l � 1� � �DjIjI

�DjI2 � ��Cj

n,1 � Cjn,2 � Cj

n,3 � Cjn,4� � ��1 �

1

��j

�ujn�2[15]

where

Cjn,1 �

p�k,l�aj

n�k,l�, Cj

n,2 �p�k � 1,l�aj

n�k � 1,l�,

Cjn,3 �

p�k,l�bj

n�k,l�, Cj

n,4 �p�k,l � 1�

bjn�k,l � 1�

ajn�k,l� � � ���

x ujn�k,l��2

� �ujn�k,l � 1� � uj

n�k,l � 1�

2 � 2� 2�p�k,l�/2

bjn�k,l� � � ���

x ujn�k,l��2

� �ujn�k � 1,l� � uj

n�k � 1,l�2 � 2� 2�p�k,l�/2

where j corresponds to the jth coil, k and l correspond torows and columns of the image respectively, and n corre-sponds to the nth iteration.

Pixels in the sum-of-squares image, I, with intensitybelow some particular threshold will be treated as holes,i.e., belonging to D. This threshold can be automatically

390 Huang et al.

determined in several ways. One simple approach is tofind the maximum pixel intensity in the entire image andset the threshold to be 0.05 times this value [i.e., 0.05 �(maximum intensity)].

A good initial guess for uj can reduce the number ofiterations involved in applying Eq. [15]. Raw sensitivitymaps for each coil, defined as

uj0�x� � � Ij�x�

I�x�, x�D

guess, x � D[16]

are used as the initial guess for uj. In our experiments, themagnitude of guess above is the average intensity of Ij, thephase of guess is taken to equal the phase of Ij. Anotherissue associated with sensitivity maps is the fact that theycontain complex data. Therefore, we can either inpaint thevector map directly or inpaint the magnitude and thephase part separately. However, there are some difficultiesin estimating the phase part (for details see the Discussionsection). In our experiments, only the magnitude was in-painted. The phase part will be the same as the phase ofguess.

RESULTS

The proposed inpainting model, Eq. [10], was applied tosensitivity map correction. The corrected sensitivity mapswere then utilized by SENSE for the reconstruction of MRimages. The accuracy and efficiency of the conventionalTV inpainting technique were demonstrated previously(2,3,5). To demonstrate the accuracy of the modified modelfor sensitivity maps, we first applied the proposed methodto a phantom without holes. Some holes were then ran-domly constructed and inpainted. The results were thencompared with the original sensitivity map that had noholes. Since the true sensitivity map is not known in realapplications, we were unable to directly determine theaccuracy of an inpainted sensitivity map. Therefore, weapplied the inpainted sensitivity maps to reconstruct anMR image by SENSE, and then compared the recon-structed image with the reference image, which was gen-erated using full k-space. The term “intensity difference”means the difference in magnitudes between the recon-structed and reference images at each pixel. We define the“relative error” as the L2 norm of the “intensity difference”divided by the L2 norm of the reference image. Note thatthe intensity of a reconstructed image by SENSE will belower than the reference image because of the reducedsignal in k-space. Following Parseval’s equation, we mul-tiplied the reconstructed images by their correspondingreduction factors before we calculated the intensity differ-ences in all of the experiments. This compensated for theloss of intensity in the SENSE images, and resulted inintensities comparable to those of the full k-space refer-ence images.

In 2001, a TPS-based method (14) was introduced for theestimation of coil sensitivities. The spline takes the form ofcombinations of log-based basis functions:

S�x,y� � �i�1

n

wiFi�x,y� [17]

where wi are weights, and Fi�x,y� � di2 log di, di

� ��x � xi�2 � �y � yi�

2. To approximate a cloud of datapoints, a least-squares fitting procedure was developed.The positioning of the basis functions is achieved auto-matically by restricting their locations to lie in regions ofhigh signal. The more basis functions used, the more ac-curate is the estimation (although it results in longer com-putation time). In our experiments, we chose 1% of allhigh signal points as basis function positions to balancethe computation time and accuracy.

Another way to fix the holes is to apply Gaussian kernelsmoothing (GKS). The basic idea is to first fill in the holeswith the average image intensity of the non-hole locationsand then smooth the hole-free image by Gaussian kernelconvolution. In our experiments, the width of the Gauss-ian kernel was 5. For the sake of comparison with ourmethod, sensitivity maps estimated by TPS and sensitivitymaps corrected by GKS were also applied to SENSE.

In some real applications, low-resolution images areused to produce sensitivity maps. According to Ref. 15, itis possible to extract the sensitivity map directly from afully sampled central band of k-space. Since the acquisi-tion of the sensitivity map data and the data to be recon-structed occur simultaneously, errors due to sensitivitymiscalibration are eliminated. An alternative method foracquiring the sensitivity map for dynamic MR, such as ina cardiac series, is to apply the time interleaved k-spacelines in sequential images. This method was introduced asUNFOLD (16) and TSENSE (17). In this manner, afterseveral time frames, one has acquired the entire k-space,thus yielding a set of full field-of-view (FOV) images. Sincethese full-FOV images are averaged from several timeframes, they are known to suffer from residual aliasingartifacts. Nevertheless, these images are of sufficient qual-ity for extracting coil sensitivity maps, thereby allowingintrinsic referencing. Both methods were tested in ourexperiments to show the flexibility of the inpainting tech-nique.

The choice of parameters may depend on the system andloading employed. Based on our experiments, the model isnot very sensitive to the choice of parameters. In ourexperiments, we set � � 1, � � 0.25, � � 10 and � � 1. Thenumber of iterations depends on the size of the holes(bigger holes may need more iterations). In our implemen-tation, we set two termination conditions: 1) when thesolution is convergent, and 2) when the number of itera-tions reaches it maximum. We set the maximum iterationnumber to be 50 in our experiments.

In all of our experiments, MATLAB codes were run on aCOMPAQ PC with a 1G Hz CPU and 1G RAM.

Phantom With No Intrinsic Holes

In this experiment, we sought to show that our modifiedinpainting technique can recover the missing sensitivityinformation. A phantom without holes was used to gener-ate the ground truth, i.e., the hole-free maps. Then several

PDE-Based Inpainting on Sensitivity Maps 391

holes were artificially produced. For comparison pur-poses, the inpainting technique, TPS method, and Gauss-ian smoothing technique were applied to the holes. Thephantom was constructed from an acrylic tube (197 mm indiameter, 180 mm long) filled with a solution of Cu2SO4

(2.0 g/L) and NaCl (4.5 g/L). Data were then collected by a1.5T Siemens system (FOV � 300 mm, matrix � 128 �128, TR � 300 ms, TE � 15 ms, flip angle � 90°, slicethickness � 5 mm, number of averages � 1) with aneight-channel tuned loop array of coils. The time requiredfor the inpainting of eight sensitivity maps was 4.35 s.

If we ignore the container, there are no holes at all in theoriginal sensitivity map. Figure 1a shows the magnitude ofthe original sensitivity map of coil 1. Five holes were thenrandomly made in the original sensitivity map (shown inFig. 1b). The local patterns are perfectly preserved in theinpainted result. Figure 1c shows the result of applying theproposed inpainting method to Fig. 1b. Figure 1d–f showthe difference maps of three methods for comparison.These difference maps use the same grayscale. It can beseen that the inpainted map has less difference than theother methods. Table 1 shows the relative error of eachchannel. The average relative error at hole regions of alleight channels using the modified TV method is 12.19%.The average relative error using TPS is 17.18%. The timeconsumed by TPS was 69.37 s for eight sensitivity maps.The average relative error using GKS was 43.03%, but thetime consumed was far less than 1 s.

Dynamic Images With Time-Interleaved k-Space

To show that there are big holes in the raw sensitivitymaps for dynamic imaging, and inpainted sensitivity mapscan generate reasonable results for SENSE, we show re-sults for cardiac images. Table 2 shows the results foroblique cardiac images collected by a 1.5T GE system(FOV � 280 mm, matrix � 160 � 120, TR � 4510 ms, TE �2204 ms, flip angle � 45°, slice thickness � 6 mm, numberof averages � 2) through fast imaging employing steady-state acquisition (FIESTA) with a GE four-channel cardiaccoil. Breath-holds ranged from 10 to 20 s, and 20 imagesper heartbeat were obtained. Full k-space data were ac-quired. Pseudo partial k-space data were utilized forSENSE reconstruction.

FIG. 1. Magnitudes of the various sensitivity maps for the phantom. The maps shown are for channel 1. a: The original hole-free map. b:The map with five randomly constructed holes. c: The map corrected by the proposed inpainting method. d: The difference map of theinpainting method. e: The difference map of the TPS method. f: The difference map of the GKS method.

Table 1The Relative Errors at Hole Regions

Coil Inpainting TPS GKS

1 8.46% 10.92% 35.24%2 5.76% 8.15% 40.97%3 21.81% 23.75% 61.64%4 10.60% 19.28% 45.37%5 8.29% 16.32% 41.22%6 9.03% 14.65% 31.77%7 26.89% 36.89% 44%8 6.64% 7.42% 44.05%

Average 12.19% 17.18% 43.03%

392 Huang et al.

We employed the time-interleaved k-space concept(16,17). Odd-numbered images used only the odd-num-bered phase-encode lines in their k-space, while even-numbered images used only the even-numbered phase-encode lines in their k-space. The phase-encode directionin this example was anterior–posterior. Each correspond-ing pair of odd and even images made a set that shared onespecific sensitivity map. This map was generated from theimage produced by combining their k-spaces (to form onefull k-space). SENSE was then applied to each image in thesequence. The reduction factor was 2 because one-half ofthe k-space was used in reconstruction. Reference imageswere reconstructed with full k-space.

Figures 2 and 3, and Table 2 show the results. Figure 2demonstrates the results of the third frame of 20 images.Figure 2a shows the magnitude of the raw sensitivity mapfor channel 1. The raw sensitivity maps were generated bydividing individual channel images by the sum-of-squaresimage and then applying a mask (see below). Figure 3ashows the magnitude of the 80th row of the sensitivitymap for channel 1. Figure 3b shows the intensity of the80th row of the sum-of-squares image for frame 3. Com-paring those two images, it can be seen that the sensitivitymap given by Fig. 3a has large jagged fluctuations at thelow signal (high noise) region (seen near right). To reducethe influence of noise (as well as define the hole regions),an intensity threshold is set and applied to the sum-of-squares image to define a mask. The mask is then appliedto the quotient of the individual channel images over thesum-of-squares image to generate the raw sensitivity maps.In this example, the 80th row of the raw map is shown asthe solid line in Fig. 3c. Because of low signal in someregions, we find huge holes in the raw maps. Figure 2bshows the magnitude of the sensitivity map corrected byTPS. Figure 2c shows the magnitude of the sensitivity mapcorrected by GKS. Because of the large size of the holes,

GKS was not entirely able to fix them, and the intensity inthe remainder of the holes was set to be the average inten-sity of the known region. Figure 2d shows the magnitudeof the sensitivity map corrected by inpainting. It can beseen that the intensity information extends smoothly intothe holes and preserves the original texture. Because of thefidelity term, we are also able to see more detail in Fig. 2dthan in Fig. 2c. In Fig. 2e, the cloudy region around theheart denotes a region of interest (ROI). This is the regionof importance in these images; hence the relative errorswill be calculated from this region. Figure 2f–h show theintensity differences between the reference image and theimages reconstructed with different sensitivity maps (TPS,GKS, and inpainted maps, respectively). For the purposeof comparison, they use the same intensity scale: [0 1]. Itcan be seen from these figures that the image reconstructedusing the inpainted sensitivity map (corresponding to Fig.2h) has less error over both of the ROIs, as well as theentire image. Table 2 shows the relative errors of the re-constructed images over the ROI for all 20 images in asingle heartbeat. The average relative errors of these recon-structed images (on the ROI) are 4.42% using the inpaintedsensitivity maps, 6.77% using the TPS maps, and 7.49%using the GKS maps. Over the entire image, the averagerelative errors of these reconstructed images are 4.63%using inpainted sensitivity maps, 7.3% using TPS esti-mated sensitivity maps, and 10.15% using GKS sensitivitymaps. Hence the inpainted sensitivity maps produce moreaccurate images. Note that it took 1599.4 s for the TPSmethod to fix the holes in 40 sensitivity maps. However, ittook only 23.47 s to inpaint those maps. From Fig. 2b andFig. 3c, it can been seen that the TPS method generatedresults similar to those produced by the inpaintingmethod. In the two experiments mentioned above, theinpainting technique outperformed the TPS method interms of accuracy. However, it is not always true that theinpainted map is more accurate than the TPS method. Inthe next example, it can be seen that the TPS method canbe better than the inpainting method. The accuracy of TPSdepends on the choice of basis functions and the smooth-ness of the sensitivity maps. Nevertheless, it can be seenthat inpainting requires much less time than the TPSmethod.

Figure 3 displays magnitudes of the 80th row of variousimages obtained from channel 1. From Fig. 3c, we can seethat the sensitivity changes (i.e., drops) quickly near theleft and right ends. This shows that sensitivity maps ob-tained from images may contain sharp changes. Figure 3ccan also help us see why the inpainting technique worksbetter than GKS. In general, when GKS is applied toknown regions, the Gaussian averaging also results in aloss of structure or texture. This can be seen in Fig. 3c fromthe dotted line between 20 and 40 on the horizontal axis.In contrast, the dashed line does a better job at preservingthe jagged structure of the raw map in this region. This canbe attributed to the fidelity term in the inpainting model.This can also be seen in Fig. 2c and d. A combination ofthe fidelity term and the smoothing term also results in amore accurate extension of texture into holes. This can beseen on the left of Fig. 3c, where GKS produces a humpover the hole due to the rapid drop in the sensitivity

Table 2The Relative Errors Calculated From Cardiac Images Over the ROI

Image Inpainting TPS GKS

1 4.47% 4.55% 9.92%2 5.34% 6.72% 9.16%3 5.10% 6.06% 8.20%4 5.08% 7.59% 7.75%5 4.55% 7.38% 8.42%6 4.74% 7.35% 7.10%7 3.91% 6.85% 7.29%8 3.73% 6.73% 6.60%9 3.69% 7.01% 7.12%

10 4.21% 7.06% 7.43%11 5.95% 9.37% 8.49%12 5.11% 9.47% 7.76%13 5.14% 7.29% 8.35%14 4.75% 6.86% 7.18%15 3.77% 6.21% 6.87%16 3.62% 5.76% 6.07%17 3.92% 5.90% 6.69%18 3.60% 6.29% 5.74%19 3.75% 6.14% 7.03%20 3.98% 4.79% 6.64%

Average 4.42% 6.77% 7.49%

PDE-Based Inpainting on Sensitivity Maps 393

causing the average over the known region to be largerthan the boundary of the right edge of the hole. In contrast,the dashed line drops smoothly from the edge of the holebecause of the isotropic diffusion applied at the bound-aries of the holes.

Images With Intrinsic Low-Resolution Sensitivity Maps

In our low-resolution experiments it is observed that in-painted sensitivity maps are still more accurate (althoughnot significantly so) than GKS-corrected sensitivity maps.If the sensitivity maps are sufficiently smooth, the TPSmethod can slightly outperform the inpainting technique,

but it takes 70 times longer. The same data were used inthis experiment as in the former one. Low-resolution sen-sitivity maps were generated with images reconstructedwith the 31 central rows of fully sampled data (15). Figure4 displays the results of the third frame of 20 images.Figure 4b–d all use the same intensity scale: [0 1]. Thereduction factor of k-space used here was only 1.6. Thenon-integer value is due to rows being skipped only out-side the 31-row central band. The ROI for relative errorcalculation is defined the same as in the previous example.Over the ROI, the average relative errors of the recon-structed images are 7.72% using the inpainted sensitivity

FIG. 2. Results for cardiac images. The third frame in the sequence is shown. a: The magnitude of the raw sensitivity map. b: The magnitudeof the TPS estimated map. c: The magnitude of the GKS-corrected map. d: The magnitude of the map corrected by inpainting. e: The cloudyregion around the heart shows the ROI. Parts f–h give intensity differences with respect to the reference image, and use the same grayscalemap. f: Obtained using the TPS-estimated map. g: Obtained using the GKS-corrected map. h: Obtained using the inpainted sensitivity map.

394 Huang et al.

maps, 7.24% using the TPS-estimated maps, and 9.40%using the GKS-generated maps. Figure 4e shows the 80throw of the three sensitivity maps for channel 1. The raw

sensitivity profile shows less detail than that in Fig. 3c.Hence the fixed holes in the inpainted sensitivity map alsoshow less detail than that of Fig. 3c.

This is to be expected, because low-resolution imagesutilize only the lower frequency central bands of k-spaceand thus constitute a form of smoothing (similar to theisotropic smoothing applied in GKS). This explains whythe images from inpainted low-resolution sensitivity mapsare similar to those resulting from GKS sensitivity maps.

DISCUSSION

A novel inpainting model has been proposed and appliedto sensitivity maps. From Figs. 1–4, it can be concludedthat several major issues involved in the correction ofsensitivity maps were effectively and simultaneously dealtwith: holes were fixed, and the original map was smoothlyextended to the entire image. From the experimental re-sults, it is further concluded that inpainted sensitivitymaps produce better results than GKS sensitivity maps.The inpainted sensitivity maps are far more capable offilling large holes compared to GKS. They are also superiorfor preserving detail and extending texture into hole re-gions. The TPS method generated results similar to thoseobtained with the inpainting technique, but consumedmuch more time (17–70 times more in our experiments).For low-resolution sensitivity maps, however, there is anintrinsic loss of detail. Thus if there are no large holes,GKS sensitivity maps may be sufficient (or better, since itis faster). Inpainting is more suitable for high-resolutionsensitivity maps with large holes—for example, the sensi-tivity maps generated with time-interleaved k-space forcardiac imaging. Using the sensitivity maps generatedfrom time-interleaved k-space, we found the relative errorsover the ROI with inpainted sensitivity maps ranging fromonly 3.62% to 5.95%. In comparison, the relative errorsobtained with TPS-estimated sensitivity maps ranged from4.55% to 9.47%, and those obtained with GKS sensitivitymaps ranged from 5.74% to 9.92%.

From our experiments, we found empirically that in-painting only the magnitude sufficiently improves theSENSE images. The inpainting phase of sensitivity mapsconsumes more time and may not generate better results.Because of the non-smoothness of the phase, it is ex-tremely difficult to estimate the missing data. Figure 5gives an example. Figure 5a is the phase map of channel 1from the third frame of the sequence. Note that the phasemap is not smooth, and changes sign suddenly at someplaces. Figure 5b is the phase map generated from time-interleaved k-space. Figure 5d shows the phase map at ahigh signal region generated by applying the mask for Fig.2a to Fig. 5b. To estimate the information at hole regions,we obtain the local information around the hole regions,and then smoothly extend the information to the holes.However, because of the non-smoothness, we can not re-cover Fig. 5a from Fig. 5d. The white arrows in Fig. 5ashow some regions that are not a smooth extension of theirneighbors and hence cannot be recovered from Fig. 5d. Onthe other hand, the similarity can be seen in Fig. 5a–c.Therefore, it is better to use the original noisy phase mapthan the inpainted phase map.

FIG. 3. The magnitude of the 80th row, from the third frame in thesequence, is shown. a: From the pointwise quotient of the image fromthe first channel, over the sum-of-squares image. b: From the sum-of-squares image. c: Each curve shown is derived from sensitivity profilestaken from the first channel. The solid line corresponds to the raw map,the dash-dot line to the TPS-estimated map, the dashed line to theinpainted map, and the dotted line to the GKS map.

PDE-Based Inpainting on Sensitivity Maps 395

FIG. 4. Results for cardiac im-ages with low-resolution sensitiv-ity maps, taken from the thirdframe in the sequence. a: Themagnitude of the inpainted sensi-tivity map from the first channel.Parts b–d give intensity differ-ences, all using the same gray-scale map. b: Obtained using theTPS-estimated map. c: Obtainedusing the GKS map. d: Obtainedusing the inpainted map. e: Eachcurve shown is derived from themagnitude of the 80th row of sen-sitivity profiles obtained from thefirst channel. The solid line corre-sponds to the raw map, the dash-dot line to the TPS-estimatedmap, the dashed line to the in-painted map, and the dotted lineto the GKS map.

FIG. 5. The phase map of the firstchannel, from the third frame inthe sequence. a–c: Phase mapsgenerated respectively from full k-space, time-interleaved k-space,and partial central k-space, re-spectively. d: Phase map at ahigh signal region generated byapplying the mask for Fig. 2a andb. e: Color bar for a–d.

396 Huang et al.

Inpainting is a useful tool for fixing holes, and it maypotentially be utilized for other MRI applications. Ourfuture work will be directed toward finding robust ways todelete the noise and fix the holes in the phase maps.

REFERENCES1. Ballester C, Bertalmio M, Caselles V, Sapiro G, Verdera J. Filling-in by

joint interpolation of vector fields and grey levels. IEEE Trans ImageProcess 2001;10:1200–1211.

2. Chan TF, Shen J, Vese L. Variational PDE models in image processing.Am Math Soc Notice 2003;50:14–26.

3. Chan TF, Shen J. Mathematical models for local non-texture inpaint-ings. SIAM J Appl Math 2001;62:1019–1043.

4. Chan TF, Shen J. Non-texture inpainting by curvature driven diffu-sions. J Vis Comm Image Rep 2001;12:436–449.

5. Caselles V, Morel J-M, Sbert C. An axiomatic approach to image inter-polation. IEEE Trans Image Processing 1998;7:376–386.

6. Sodickson DK, Manning WJ. Simultaneous acquisition of spatial har-monics (SMASH): ultra-fast imaging with radiofrequency coil arrays.Magn Reson Med 1997;38:591–603.

7. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. Coil sensitiv-ity encoding for fast MRI. In: Proceedings of the 6th Annual Meeting ofISMRM, Sydney, Australia, 1998. p 579.

8. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sen-sitivity encoding for fast MRI. Magn Reson Med 1999;42:952–962.

9. Angel M, Ballester G, Machida Y, Kassai Y, Hamamura Y, Sugimoto H.Robust estimation of coil sensitivities for RF subencoding acquisition

techniques. In: Proceedings of the 9th Annual Meeting of ISMRM,Glasgow, Scotland, 2001. p 799.

10. Lin FH, Chen YJ, Belliveau JW, Wald LL. Estimation of coil sensitivitymap and correction of surface coil magnetic resonance images usingwavelet decomposition. In: Proceedings of the 9th Annual Meeting ofISMRM, Glasgow, Scotland, 2001. p 801.

11. Rudin L, Osher S, Fatemi E. Nonlinear total variation based noiseremoval algorithms. Physica D 1992;60:259–268.

12. Aubert G, Kornprobst P. Mathematical problems in image processing:Partial differential equations and the calculus of variations. Springer;2002.

13. Aubert G, Vese L. A variational method in image recovery. SIAM JNumer Anal 1997;34:1948–1979.

14. Ballester MAG, Machida Y, Kassai Y, Hamamura Y, Sugimoto H. Robustestimation of coil sensitivities for RF subencoding acquisition techniques.In: Proceedings of the 9th Annual Meeting of ISMRM, Glasgow, Scotland,2001. p 799.

15. McKenzie CA, Yeh EN, Ohliger MA, Price MD, Sodickson DK. Self-calibrating parallel imaging with automatic coil sensitivity extraction.Magn Reson Med 2002;47:529–538.

16. Madore B, Glover GH, Pelc NJ. Unaliasing by Fourier encoding theoverlaps using the temporal dimension (UNFOLD), applied to cardiacimaging and fMRI. Magn Reson Med 1999;42:813–828.

17. Kellman P, Epstein FH, McVeigh ER. Adaptive sensitivity encodingincorporating temporal filtering (TSENSE). Magn Reson Med 2001;45:846–852.

18. Chen Y, Levine S, Rao M. Functionals with P(x) growth in imageprocessing. SIAM J. Appl Math 2004 (submitted).

PDE-Based Inpainting on Sensitivity Maps 397