Breast Conserving Surgery Outcome Prediction:A Patient-Specific, Integrated Multi-modalImaging and Mechano-Biological Modelling

Framework

Bjorn Eiben1(B), Rene Lacher1, Vasileios Vavourakis1, John H. Hipwell1,Danail Stoyanov1, Norman R. Williams2, Jorg Sabczynski3, Thomas Bulow3,

Dominik Kutra3, Kirsten Meetz3, Stewart Young3, Hans Barschdorf3,Helder P. Oliveira4, Jaime S. Cardoso4, Joao P. Monteiro4,Hooshiar Zolfagharnasab4, Ralph Sinkus5, Pedro Gouveia6,

Gerrit-Jan Liefers7, Barbara Molenkamp7, Cornelis J.H. van de Velde7,David J. Hawkes1, Maria Joao Cardoso6, and Mohammed Keshtgar8

1 Centre for Medical Image Computing, University College London, London, UKbjoern.eiben.10@ucl.ac.uk

2 Surgical and Interventional Trials Unit, University College London, London, UK3 Philips Technologie GmbH Innovative Technologies, Hamburg, Germany

4 INESC TEC, Porto, Portugal5 Imaging Sciences and Biomedical Engineering, King’s College London, London, UK

6 Champalimaud Foundation, Lisbon, Portugal7 Leiden University Medical Center, Leiden, Netherlands

8 Royal Free Hospital, London, UK

Abstract. Patient-specific surgical predictions of Breast ConservingTherapy, through mechano-biological simulations, could inform theshared decision making process between clinicians and patients byenabling the impact of different surgical options to be visualised. Wepresent an overview of our processing workflow that integrates MRimages and three dimensional optical surface scans into a personalisedmodel. Utilising an interactively generated surgical plan, a multi-scaleopen source finite element solver is employed to simulate breast defor-mity based on interrelated physiological and biomechanical processesthat occur post surgery. Our outcome predictions, based on the pre-surgical imaging, were validated by comparing the simulated outcomewith follow-up surface scans of four patients acquired 6 to 12 monthspost-surgery. A mean absolute surface distance of 3.3 mm between thefollow-up scan and the simulation was obtained.

Keywords: Breast imaging ·Oncoplastic breast surgery · Surgical plan-ning · Image registration · Surface reconstruction · Finite element ·Mathematical modelling

c© Springer International Publishing Switzerland 2016A. Tingberg et al. (Eds.): IWDM 2016, LNCS 9699, pp. 274–281, 2016.DOI: 10.1007/978-3-319-41546-8 35

Breast Conserving Surgery Outcome Prediction 275

1 Introduction

Breast cancer today is the most common cancer for women in the developedworld with 464,000 new cases reported in Europe in 2012 alone [1]. Early diag-nosis and improved treatment fortunately make breast cancer an increasinglytreatable disease and, as a result, the patients’ quality of life is a significantissue following the completion of primary cancer care. Breast Conserving Ther-apy (BCT) was shown to lead to an improved body image and a higher levelof satisfaction with the overall treatment result [2], however up to 30 % of BCTprocedures have been reported to produce cosmetically suboptimal outcomes [3].

Providing tools that enable predictions of the outcome of proposed BCTprocedures to be made, could inform a shared decision making process betweenpatient and clinician, and manage patient expectations regarding their expectedcosmetic outcome. Specifically, such tools could be used at various points in thepatient consultation process to develop an appropriate surgical option, given theconstraints of tissue excision and safe surgical margins etc., which is aestheticallyacceptable to the patient.

In this work we present an overview of a processing workflow that was devel-oped in the Picture project1. The workflow facilitates such patient-specific,surgical outcome predictions on the basis of clinical Magnetic Resonance Imag-ing (MRI) and optical surface scans. Development of surface reconstructionmethodologies that are designed to work with low-cost RGBD (abbreviationfor red, green blue, and depth) cameras, i.e. Microsoft Kinect, could make thesystem easier to adopt by removing the need for high-cost dedicated acquisitionsystems. To quantify the accuracy of the outcome predictions, we use follow-upsurface scans of four clinical cases. Figure 1 shows an overview of the implementedprocessing workflow, where each component is represented by a coloured box.

2 Input Data and Pre-processing

2.1 Surface Reconstruction

A 3D surface model of the patient’s torso is an important prerequisite to pro-duce a realistic visualisation of the personalised aesthetic outcome simulation.The patient’s upper body as well as the skin texture information is digitallyrepresented as a coloured, triangle-based mesh. This can be achieved by using adedicated scanning system provided by the company 3dMD that instantaneouslycaptures surface data and reconstructs three-dimensional models with sub-millimetre accuracy while utilising active multi-view stereo-photogrammetry. Asan alternative, low-cost source of surface data, we also make use of recent depth-sensing technology employing RGBD cameras of the Microsoft Kinect series.With both Kinect sensors working with light in the near-infrared range, the firstKinect device is a structured-light scanner while the second generation uses time-of-flight technology for depth estimation. In order to produce a single consistent

1 www.vph-picture.eu.

276 B. Eiben et al.

Surgical Outcome Prediction

MRI (prone)

OpticalSurface (pre)

OpticalSurface (post)

SurfaceReconstr.

SurfaceReconstr.

Biomech.Model

SupineSimulation

UprightSimulation

SurgicalPlan

SurgicalSimulation

ModelAlignment

OutcomePrediction

Validation

2.1

2.1

2.2

3.1

3.1

3.1 3.1

3.2 3.2 4

Fig. 1. Data processing and analysis work-flow. From the input MRI and surface data apersonalised mechano-biological model is generated that, together with a surgical plan,facilitates simulation of the surgical outcome. The prediction is validated using follow-up surface reconstructions. The numbers inside the boxes correspond to the sectionswhere more detail is provided (Color figure online)

dense surface profile, Kinect data acquisition follows a predefined protocol inwhich a sequence of images from a number of viewpoints, taken along a circularpath around the patient, is recorded. The data from these multiple, consecu-tive Kinect images needs to be fused into a unified model of the patient. Priorto this fusion, the subsurface visible in each individual image is aligned witha reference coordinate system (the SLAM problem). To that end, a processingpipeline based on the KinectFusion framework is used [4]. To improve results inour clinical data acquisition setting, the pipeline is extended via a pre-processingstep to discard unreliable data points and via global pose optimization step tomitigate pose errors [5]. Also, colour from the RGB images is integrated andmapped onto the surface. Having redundant systems for surface acquisition inplace, we aim to show that portable consumer-level depth sensors are capable ofcreating surface data of sufficient quality for this application, and hence have thepotential to reduce imaging costs and complexity in a clinical scenario. While thehighly-accurate 3dMD surface is generally superior to the Kinect-based surface,the latter provides a realistic geometry of good quality, able to resolve minordetails reliably. Figure 2 shows the estimated Kinect camera trajectory that wasrecovered during surface reconstruction alongside a side-by-side comparison ofthe surface reconstruction results of the three different systems.

2.2 Biomechanical Model Generation

A patient-specific biomechanical model is generated from routine clinical T1-or T2-weighted structural MRI scans. A binary region of interest mask is firstsegmented from these images, and discretised to create a three-dimensional meshusing a number of open-source libraries and tools, i.e. VTK, Gmsh and Netgen.Each tetrahedral finite element of the resulting mesh is then labelled as adipose

Breast Conserving Surgery Outcome Prediction 277

(a) (b) (c) (d)

Fig. 2. Surface reconstruction and side-by-side comparison for one exemplary patient(a) Reconstruction volume, estimated Kinect camera trajectory and overlaid recon-structed surface (b) 3dMD result (c) Kinect V2 coloured surface (d) Kinect V1 colouredsurface (Color figure online)

or fibroglandular tissue, according to segmentation of these classes from theinput MR volume, and assigned the appropriate constitutive material relationaccording to published population based statistics. The breast skin surface ismodelled using triangular membrane elements and is assumed traction-free inthe biomechanical model. Finally, the lateral planar boundaries are defined astraction-free, the superior and inferior planes are constrained to 2D in-planemotion, and the chest surface is fixed.

3 BCT Modelling Framework

3.1 Surgical Simulation Tool

We have developed a surgical simulation tool to model BCT interventions per-formed on breast cancer patients. The integrated tool is a three-dimensional,multiscale, finite element (FE) numerical framework, capable of simulatingbreast tissue deformations and the physiological tissue recovery process followingsurgery. The modelling framework encompasses breast tissue biology and solidmechanics in a multiscale manner. At the macroscopic level, tissue biomechanicsis described using a standard continuum mechanics approach, with breast tis-sues being modelled as a neo-Hookean material [6] and the skin as a hyperelasticmembrane [7]. At the microscopic level, cell (i.e. fibroblasts and epithelial cellsetc.) concentration, proliferation and cell-cycle in the operated-breast region isexplicitly described via an established diffusion transport equation [8]. Follow-ing Moreo et al. [9], fibroblasts are also considered to actively contribute to thecontraction of the wound in the model. Additionally, tissue recovery and angio-genesis is modelled using a simplified non-linear mathematical model. Hence, inthe operated region of the patient-specific breast model, the transport of chem-ical agents (i.e. macrophage-derived growth factor, vascular endothelial growthfactor) regulating inflammation, cell proliferation and angiogenesis is explicitlyspecified using a pair of Kolmogorov-Petrovsky-Piskounov equations [10,11]. Incommon with the early work of Maggelakis [12], neo-vascularisation in the recov-ering tissue, and oxygen-level and nutrient transport is described mathematically

278 B. Eiben et al.

(a) (b) (c) (d)

Fig. 3. Surface alignment procedure in the upright position. The optical surface scan(a) acts as a target to which the upright simulation (b) is adapted. A material parameteroptimisation step followed by a non-linear surface warping (c) is used to update thebiomechanical simulation (d) so that it fits the target surface (a). (Color figure online)

as a diffusion process. The present computational framework has been imple-mented into our open-source finite element project FEB3,2 which incorporatesseveral high-performance numerical libraries, i.e. libmesh, PETSc, and MPICH.

In the grey shaded rectangle of Fig. 1, the workflow of the BCT modellingframework is illustrated, where the core modules of the surgical simulation toolare depicted in boxes. Starting from the FE model (c.f. Sect. 2.2) an inversedeformation analysis [13] is carried out to predict tissue pre-stressing due to theeffect of gravity. Then, conventional forward deformation analysis, to predict thebreast shape in the supine and upright position, is performed. Subsequently, sur-gical planning takes place in which the virtual patient (i.e. the predicted breastshape in the supine setting) is visualised through a graphical user interface by thesurgeon. At this point, annotations of the surgical plan on the virtual patient-specific model are specified, namely the skin incisions, the resected breast tissuevolume (including the lesion and margins) and optional tissue mobilisation. Thefinite element model is then updated accordingly and is fed back to the surgicalsimulation tool, where the wound-healing simulation during a 3-month period ofthe patient’s recovery is carried out. The final prediction of the patient-specificbreast is then transformed into the upright configuration and, subsequently,compared with the corresponding pre-operative breast shape.

3.2 Model-to-Surface Alignment and Outcome Prediction

The FE methodology described in Sect. 3.1 facilitates the transformation of theprone MRI based model to another configuration such as upright. Despite theuse of a rigorous simulation framework, inaccuracies in the model persist and, asa result, the upright simulation does not completely match the optical uprightsurface. Such inaccuracies are for instance caused by (i) an uncertainty about thepatient-specific material parameters and (ii) deformations of the breast in theprone MRI acquisition configuration due to contact between the breast and thescanner. Figure 3 shows an example case where the upright simulation (Fig. 3(b))differs from the upright scan (Fig. 3(a)) especially due to medial indentation ofthe breast during MRI acquisition. Correction of such deformations based on a

2 https://bitbucket.org/vasvav/feb3-finite-element-bioengineering-in-3d/wiki/Home.

Breast Conserving Surgery Outcome Prediction 279

P1 P2 P3 P40

2

4

6

8

10

d[m

m]

(a) (b)

Fig. 4. Validation results. (a) Absolute surface distance d between the simulated surgi-cal outcome predictions for patients P1 to P4 and the corresponding optical follow-upsurface scans. The grey bars represent the mean values, the whiskers the standarddeviations and the diamonds the 95th percentile of the measured distances. (b) Thesimulated surgical outcome for case P1 with the colour coded surface distances, as wellas the post-surgical surface scan as a wire-frame. (Color figure online)

pure mechanical simulation is difficult. Hence we make use of the optical uprightsurface in order to address both issues above in a two-step procedure [14].

In a first step the global alignment between the upright simulation and thetarget surface is improved by optimising the material parameters of the biome-chanical model. After this global alignment, local refinements are carried outby the means of a surface warping step. In this step the skin nodes are driventowards the target surface, while the deformation is regularised by a smooth-ness term and an area-change penalising term. The deformation vector field andthe final alignment result for an example case are shown in Fig. 3(c) and (d)respectively.

The alignment procedure results in an updated biomechanical model whichthen can be further transformed according to the displacements calculated bythe surgical simulation (Sect. 3.1). In combination with a texture transfer fromthe reconstructed optical surface to the biomechanical model, a photo-realisticvisualisation of the surgical outcome prediction becomes possible.

4 Validation

In order to validate the BCT modelling framework, we compared the gener-ated surgical outcome predictions with follow-up surface scans (c.f. bottom mostinput in Fig. 1). The follow-up scans were acquired 6–12 months after surgery,and registered to the simulated outcome using a rigid Iterative Closest Pointalgorithm [15].

The results in terms of the mean absolute surface distance for the first fourpatients with complete data sets are shown in Fig. 4(a). The mean distance inall four cases is computed to be 3.3 mm. Figure 4(b) shows the surgical outcomesimulation as a coloured surface, together with the follow-up acquisition as a

280 B. Eiben et al.

wire-frame. The colours represent the evaluated distance between follow-up andprediction. It can be observed that the mechano-biological simulation predictsthe overall deformity of the breast very well and within clinically useful accuracy.However, due to the relatively large time gap between baseline and follow-upscans, the breast shape might change due to effects unrelated to surgery – suchas weight loss or gain. Such effects, presently not considered by the surgicaloutcome prediction, will increase the surface distance measure.

5 Conclusion

This paper presents an overview of our simulation methodology which enablesbreast shape changes due to breast conserving surgery to be predicted. Thepatient-specific mechano-biological models are derived from magnetic resonanceimages acquired in the prone configuration. In addition, optical surface scansare used to improve model predictions in the upright configuration. Breast sur-face acquisition is carried out using a 3dMD system alongside Microsoft Kinectv1 or v2 devices, with the aim of investigating and developing methodologiesthat allow reliable and high-fidelity surface reconstruction. We have developed anovel surgical simulation tool that incorporates a multiscale mechano-biologicalfinite element solver. The modelling framework can facilitate simulations of largedeformations of the breast due to gravity, but can also predict breast tissue defor-mities and scarring caused by the physiological wound healing process.

The modelling framework was evaluated by performing surgical simulationsusing imaging data acquired from four patients. This data included pre-operativeMRIs, surface scans and the corresponding surgical plans specified by the surgeonwho performed the procedure. The numerical predictions were compared andvalidated against clinical follow-up surface scans acquired 6–12 months afterBCT. This resulted in a mean absolute surface error, over all four cases, of3.3 mm. The Picture project also investigated the use of biomechanical modelsthat do not require a pre-surgical MRI. However, these results will be presentedin future work.

In this contribution, we demonstrate that efficient integration of differentinput modalities and detailed mechano-biological modelling tools can achieveaccurate predictions of complex surgical interventions, such as BCT. In thefuture such predictions could promote improved and visually enriched communi-cation between clinicians and patients. Personalised information about plannedprocedures could facilitate a shared decision making process in which decisionsare made with more confidence.

Acknowledgements. The authors would like to acknowledge the financial support ofthe European FP7 project VPH-PICTURE (FP7-ICT-2011-9, 600948) and the MarieCurie Fellowship project iBeSuP (FP7-PEOPLE-2013-IEF, 627025). The authors arealso indebted to members of the Royal Free Hospital NHS Foundation Trust for theirsupport of this research; in particular Dominic Baxter for patient recruitment, GeorginaBartl for data administration and David Bishop, Emily Appleby, Imogen Ashby andSusan Smart for medical photography.

Breast Conserving Surgery Outcome Prediction 281

References

1. Ferlay, J., Steliarova-Foucher, E., Lortet-Tieulent, J., Rosso, S., Coebergh, J.W.W.,Comber, H., Forman, D., Bray, F.: Cancer incidence and mortality patterns inEurope: estimates for 40 countries in 2012. Eur. J. Cancer 49(6), 1374–1403 (2013)

2. Curran, D., van Dongen, J., Aaronson, N., Kiebert, G., Fentiman, I., Mignolet,F., Bartelink, H.: Quality of life of early-stage breast cancer patients treated withradical mastectomy or breast-conserving procedures: results of EORTC trial 10801.Eur. J. Cancer 34(3), 307–314 (1998)

3. Hill-Kayser, C.E., Vachani, C., Hampshire, M.K., Lullo, G.A.D., Metz, J.M.: Cos-metic outcomes and complications reported by patients having undergone breast-conserving treatment. Int. J. Radiat. Oncol. Biol. Phys. 83(3), 839–844 (2012)

4. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J.,Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: real-time densesurface mapping and tracking. In: 2011 10th IEEE International Symposium onMixed and Augmented Reality (ISMAR), pp. 127–136 (2011)

5. Lacher, R., Hipwell, J., Williams, N., Keshtgar, M., Hawkes, D., Stoyanov, D.:Low-cost surface reconstruction for aesthetic results assessment and prediction inbreast cancer surgery. In: 2015 37th Annual International Conference of the IEEEEngineering in Medicine and Biology Society (EMBC), pp. 5871–5874 (2015)

6. Samani, A., Plewes, D.: A method to measure the hyperelastic parameters of exvivo breast tissue samples. Phys. Med. Biol. 49(18), 4395–4405 (2004)

7. Veronda, D., Westmann, R.: Mechanical characterization of skin - finite deforma-tions. J. Biomech. 3(1), 111–124 (1970)

8. Murray, J.: Mathematical Biology I: An Introduction. Interdisciplinary AppliedMathematics, 3rd edn. Springer, New York (2011)

9. Moreo, P., Garcıa-Aznar, J., Doblare, M.: Modeling mechanosensing and its effecton the migration and proliferation of adherent cells. Acta Biomater. 4(3), 613–621(2008)

10. Sherratt, J., Murray, J.: Mathematical analysis of a basic model for epidermalwound healing. J. Math. Biol. 29(5), 389–404 (1991)

11. Olsen, L., Sherratt, J., Maini, P.: A mechanochemical model for adult dermalwound contraction and the permanence of the contracted tissue displacement pro-file. J. Theor. Biol. 177(2), 113–128 (1995)

12. Maggelakis, S.: A mathematical model of tissue replacement during epidermalwound healing. Appl. Math. Model. 27(3), 189–196 (2003)

13. Vavourakis, V., Hipwell, J., Hawkes, D.: An inverse finite element u/p-formulationto predict the unloaded state of in vivo biological soft tissues. Ann. Biomed. Eng.44(1), 187–201 (2016)

14. Eiben, B., Vavourakis, V., Hipwell, J.H., Kabus, S., Lorenz, C., Buelow, T.,Williams, N.R., Keshtgar, M., Hawkes, D.J.: Surface driven biomechanical breastimage registration. In: Webster, R.J., Yaniv, Z.R. (eds.) Medical Imaging 2016:Image-Guided Procedures, Robotic Interventions, and Modeling. ProceedingsSPIE, vol. 9786, 97860W–97860W-10 (2016)

15. Besl, P., McKay, N.: A method for registration of 3-D shapes. IEEE Trans. PatternAnal. Mach. Intell. 14(2), 239–256 (1992)