problems in biological imaging: opportunities for signal processing jelena kovačević bimagiclab...

Problems in Biological Imaging: Problems in Biological Imaging: Opportunities for Signal ProcessingOpportunities for Signal Processing

Jelena KovačevićJelena Kovačević

bimagicLabbimagicLabCenter for Bioimage InformaticsCenter for Bioimage InformaticsDepartment of Biomedical EngineeringDepartment of Biomedical EngineeringDepartment of Electrical and Computer EngineeringDepartment of Electrical and Computer EngineeringCarnegie Mellon UniversityCarnegie Mellon University

Cast of CharactersCast of Characters

The RoadmapThe Roadmap

Tasks

Issues

Framework

Tools

Revolution in biology

What can we do?

Revolution in BiologyRevolution in Biology

Focus in biologyFocus in biology Vertical to horizontal approachVertical to horizontal approach ““Omics”: genomics, proteomics, …Omics”: genomics, proteomics, …

Fluorescence microscopyFluorescence microscopy Hugely successfulHugely successful Allows for live-cell imagingAllows for live-cell imaging Fluorescent markers, starting with GFPFluorescent markers, starting with GFP Allows for collection of high-dimensional data setsAllows for collection of high-dimensional data sets

2D images and 3D volumes2D images and 3D volumes At multiple time instantsAt multiple time instants Multiple channelsMultiple channels

Analysis and interpretation Analysis and interpretation Cumbersome, nonreproducible, error proneCumbersome, nonreproducible, error prone

GoalGoal

Imaging in systems biologyImaging in systems biology

Use informatics toUse informatics to acquire, store, manipulate acquire, store, manipulate

and share large and share large bioimaging databasesbioimaging databases

Leads toLeads to automated, efficient and automated, efficient and

robust processing robust processing

NeedNeed Host of sophisticated tools Host of sophisticated tools

from many areasfrom many areas

RegistrationMosaicing

SegmentationTracking

AnalysisModeling

PSF h

A/D

Denoising

Deconvolution

RestorationDenoising +

Deconvolution

The RoadmapThe RoadmapIssues

Revolution in biologyNoise levels and typesNoise levels and typesLack of ground truthLack of ground truthLarge deviationsLarge deviationsLow definition and contrastLow definition and contrastWide range of time-frequency featuresWide range of time-frequency features

Noise Levels and TypesNoise Levels and Types

Shift towards noninvasiveShift towards noninvasive Data collected farther from the sourceData collected farther from the source Signals typically corrupted by Signals typically corrupted by

high levels of noisehigh levels of noise Weak biosignalsWeak biosignals Standard SP techniques not used Standard SP techniques not used

but even those will not work well but even those will not work well with such signalswith such signals

Types of noiseTypes of noise Electrical, neuronal, …Electrical, neuronal, … Modeling of noise a problemModeling of noise a problem

Lack of Ground TruthLack of Ground Truth

Shift towards noninvasive Shift towards noninvasive No access to ground truthNo access to ground truth

Large DeviationsLarge Deviations

Humans and/or animals as ``customers'‘Humans and/or animals as ``customers'‘ Wide range of states considered ``normal'‘Wide range of states considered ``normal'‘ Looking for is a range rather than a single stateLooking for is a range rather than a single state Large deviations from the range of normal states may Large deviations from the range of normal states may

characterize what we are looking forcharacterize what we are looking for

normal delayed abnormal

Low Definition and ContrastLow Definition and Contrast

Images typically have low contrast Images typically have low contrast and are poorly definedand are poorly defined Lack of consistent edgesLack of consistent edges

Wide Range of Time- and Frequency-Wide Range of Time- and Frequency-Localized FeaturesLocalized Features

BioimagesBioimages Global behaviors together with spikes and transientsGlobal behaviors together with spikes and transients Puts time-frequency tools to the testPuts time-frequency tools to the test ““Speckled” nature---stochastic representationSpeckled” nature---stochastic representation


Framework


Continuous-domain image processingContinuous-domain image processingFrom continuous to discrete domainFrom continuous to discrete domainDiscrete-domain image processingDiscrete-domain image processing

Continuous-Domain Image ProcessingContinuous-Domain Image Processing

Specimen (object) vs Specimen (object) vs image of it (projection)image of it (projection)

LSI systemsLSI systems Impulse response of the Impulse response of the

microscope: PSFmicroscope: PSF

Fourier viewFourier view FT or FSFT or FS



AnalysisModeling

PSF h

A/D

Denoising

Deconvolution


Deconvolution

From Continuous to DiscreteFrom Continuous to Discrete

Resolution in microscopyResolution in microscopy

Filtering before samplingFiltering before sampling

Sources of uncertaintySources of uncertainty



AnalysisModeling

PSF h

A/D

Denoising

Deconvolution


Deconvolution

Discrete-Domain Image ProcessingDiscrete-Domain Image Processing

LSI system, digital LSI system, digital filteringfiltering

Consider the signal asConsider the signal as Infinite signal with finite Infinite signal with finite

number of nonzero number of nonzero coefficientscoefficients

Finite signalFinite signal

Fourier viewFourier view DTFTDTFT DFTDFT



AnalysisModeling

PSF h

A/D

Denoising

Deconvolution


Deconvolution


Framework


Signal and image representationsSignal and image representationsFourier analysisFourier analysisGabor analysisGabor analysisMultiresolution analysisMultiresolution analysisData-driven representation and analysisData-driven representation and analysis

Tools

t

fDirac basisWPWT

ER

Actin

STFTFT

Signal RepresentationsSignal Representations

““Holy Grail” of signal Holy Grail” of signal analysis/processing analysis/processing Understand the “blob”-like Understand the “blob”-like

structure of the energy structure of the energy distribution in the time-distribution in the time-frequency spacefrequency space

Design a representation Design a representation reflecting thatreflecting that

Data Driven Representation & AnalysisData Driven Representation & Analysis

Use representations based on training data and Use representations based on training data and automated learning approachesautomated learning approaches Wavelet packetsWavelet packets PCA & variationsPCA & variations ICAICA ……

Estimation FrameworkEstimation Framework

Random variations introduced by system noise, Random variations introduced by system noise, artifacts, uncertainty originating from the biological artifacts, uncertainty originating from the biological phenomena lead to statistical methodsphenomena lead to statistical methods

Seek the solution optimal in some probabilistic Seek the solution optimal in some probabilistic sensesense

Optimality criterionOptimality criterion MSE, often depends on unknown parametersMSE, often depends on unknown parameters Bayesian framework, MAP estimatorsBayesian framework, MAP estimators

The RoadmapThe Roadmap

Tasks

Issues

Framework

Tools


AcquisitionAcquisitionDeblurring, denoising, restorationDeblurring, denoising, restorationRegistration and mosaicingRegistration and mosaicingSegmentation, tracing and trackingSegmentation, tracing and trackingClassification and clusteringClassification and clusteringModelingModeling

AcquisitionAcquisition

Issues in acquisition of Issues in acquisition of fluorescence microscope imagesfluorescence microscope images

Increase resolutionIncrease resolution Total data acquisition is reduced, speeding up image acquisitionTotal data acquisition is reduced, speeding up image acquisition Allows a higher frame rate (increased temporal resolution)Allows a higher frame rate (increased temporal resolution) Allows us to spend more time acquiring the regions of interest (which gives increased spatial Allows us to spend more time acquiring the regions of interest (which gives increased spatial

resolution)resolution)

Acquire for longer periodsAcquire for longer periods Acquisition process damages both the signal (photobleaching) and the cell (phototoxicity)Acquisition process damages both the signal (photobleaching) and the cell (phototoxicity) Efficient acquisition reduces the total amount of data acquired, thus reducing damage to the cellEfficient acquisition reduces the total amount of data acquired, thus reducing damage to the cell This allows us to observe cellular processes for longer periodsThis allows us to observe cellular processes for longer periods

Intelligent acquisitionIntelligent acquisition Acquire only Acquire only wherewhere and and whenwhen needed needed adaptivity adaptivity Model driven (microscope model & data model)Model driven (microscope model & data model)

Model-Driven AcquisitionModel-Driven Acquisition

AcquisitionAcquisition Grid acquisitionGrid acquisition MR adaptive acquisitionMR adaptive acquisition Markov Random FieldsMarkov Random Fields Example-based enhancementExample-based enhancement

ReconstructionReconstruction Simple interpolation methodsSimple interpolation methods Wavelet reconstructionWavelet reconstruction Model-based reconstructionModel-based reconstruction

Knowledge Extraction

Reconstruction

Efficient Acquisition

Mo

del

ing

MR AcquisitionMR Acquisition

ProblemProblem Why acquire in areas of Why acquire in areas of

low fluorescence?low fluorescence? Acquire only Acquire only whenwhen and and

wherewhere needed needed

Measure of successMeasure of success Problem dependentProblem dependent Here: Here:

Strive to maintain the Strive to maintain the achieved classification achieved classification accuracyaccuracy

ApproachApproach Mimic “Battleship”Mimic “Battleship” Compression Ratio

Accuracy

[Merryman & Kovačević, 2005][Merryman & Kovačević, 2005]

as well as design intelligent acquisition systems based on those models

Develop a mathematical framework and algorithmsto build accurate models of fluorescence microscope data sets

Efficient Acquisition and Learning of Fluorescence Microscope Data Models

2. Choose acquisition regions that allow us to construct an accurate model in the shortest amount of time

1. Use all the input from the microscope to model the data set

2.Intelligent Acquisition

1.Model Building Model

Model satisfactory?

Yes

No

Efficient Acquisition and Learning of Efficient Acquisition and Learning of Fluorescence Microscope Data Models Fluorescence Microscope Data Models

Predict the distribution of fluorescence in the subsequent Predict the distribution of fluorescence in the subsequent frame and acquire accordinglyframe and acquire accordingly Predict likelihood of object moving to any given position Acquire those positions with the highest likelihood

Too small an acquisition region may not find the object Too large an acquisition region is inefficient

Motion modelsMotion models Three motion models commonly observed in practice

Random walk Constant velocity Constant acceleration

[Jackson, Murphy & Kovačević, 2007][Jackson, Murphy & Kovačević, 2007]

Efficient Acquisition and Learning of Efficient Acquisition and Learning of Fluorescence Microscope Data ModelsFluorescence Microscope Data Models

Learning the motion modelLearning the motion model Prediction: Based on current beliefs about motion model,

find likelihood of each object appearing at any given pixel in the subsequent frame

Acquisition: Acquire the pixels that have the highest overall likelihood of containing an object

Observation: Observe the actual location of each object, if found

Update: Use this information to update our beliefs about the motion models for each object


Known motion modelKnown motion model Single object, random walk of known variance Probability distribution of it appearing in any given location

in the subsequent frame Acquisition regions capture the locations where the object

is expected with the highest probabilities


Known motion modelKnown motion model If the object is detected, repeat, centering the new

acquisition region at the object’s most recent location If the object is not detected, estimate where it is Probability distribution given that the object was not in the

acquisition region


Known motion modelKnown motion model Predict this object’s location in the next frame Probability distribution

1D case: choose two disconnected acquisition regions 2D case: choose to acquire between the two black circles

Deblurring, Denoising & RestorationDeblurring, Denoising & Restoration

Microscope images contain artifactsMicroscope images contain artifacts Blurring caused by a PSFBlurring caused by a PSF Noise from the electronics of digitizationNoise from the electronics of digitization

Deblurring/deconvolutionDeblurring/deconvolution Widefield microscopyWidefield microscopy Effect of depthEffect of depth

DenoisingDenoising

Deconvolution + Denoising = RestorationDeconvolution + Denoising = Restoration

Registration & MosaicingRegistration & Mosaicing

RegistrationRegistration Find spatial relationship and alignment between imagesFind spatial relationship and alignment between images

MosaicingMosaicing Used when fine resolution is needed within a global viewUsed when fine resolution is needed within a global view Stitching together pieces of an imageStitching together pieces of an image Usually requires registration, given overlapping piecesUsually requires registration, given overlapping pieces

Segmentation, Tracing & TrackingSegmentation, Tracing & Tracking

SegmentationSegmentation Methods used: thresholding and watershedMethods used: thresholding and watershed Edge-based, region-based, combinationEdge-based, region-based, combination Active contoursActive contours

TracingTracing Mostly tracing of axonsMostly tracing of axons Typical, path following approachesTypical, path following approaches Fail in the presence of noiseFail in the presence of noise

TrackingTracking Molecular dynamics and cell migrationMolecular dynamics and cell migration Tracking of objects over timeTracking of objects over time

SegmentationSegmentation

Separate objects of interest Separate objects of interest from each other and the from each other and the backgroundbackground

Fundamental step in Fundamental step in microscopymicroscopy

Hand segmentationHand segmentation Not reproducibleNot reproducible Not tightNot tight Piecewise linearPiecewise linear Cannot compute statisticsCannot compute statistics Time-consumingTime-consuming

Current standardCurrent standard Watershed segmentationWatershed segmentation

Active Contour SegmentationActive Contour Segmentation

Active contour algorithmsActive contour algorithms Contour comparable to an elastic stringContour comparable to an elastic string Moved under external and internal forcesMoved under external and internal forces

External: derived from the image (edges)External: derived from the image (edges) Internal: geometric properties of the contour (curvature)Internal: geometric properties of the contour (curvature)

Level Set method: A way to track the contour as it evolvesLevel Set method: A way to track the contour as it evolves

Positive inside the contour Positive inside the contour (mountain)(mountain)

Negative outside the contour Negative outside the contour (valley)(valley)

Zero on the contour, Zero on the contour, C embedded at its zero (sea) levelC embedded at its zero (sea) level

n

Fc > 0

Fc < 0

> 0

< 0

= 0

STACSSTACS Combines energy minimization approach with statistical modelingCombines energy minimization approach with statistical modeling

Model matchingModel matching Pixels inside and outside the contour follow different statistical Pixels inside and outside the contour follow different statistical

modelsmodels Modified STACs for fluorescence microscopy imagesModified STACs for fluorescence microscopy images

No edge informationNo edge information No obvious shape informationNo obvious shape information Segmentation driven by statistics of the image and contour Segmentation driven by statistics of the image and contour

smoothnesssmoothness

MSTACSMSTACS: Our level-set evolution equation: Our level-set evolution equation

Topology needs to be preserved Topology needs to be preserved TPSTACS TPSTACS

TPSTACS: ResultsTPSTACS: Results

SuccessfulSuccessful

ProblemProblem Extremely slowExtremely slow

SolutionSolution MRSTACSMRSTACS

Hand-segmented

TPTACS

[Coulot, Kirschner, Chebira, Moura, Kovačević, Osuna & Murphy, 2006][Coulot, Kirschner, Chebira, Moura, Kovačević, Osuna & Murphy, 2006]

37

MRSTACSMRSTACS Decompose image Decompose image

to L levelsto L levels

Smoothing renders cell Smoothing renders cell easier to discerneasier to discern

Detect cells using Detect cells using morphological operationsmorphological operations

Get coarse version of Get coarse version of contour (TPSTACS)contour (TPSTACS)

Refine contour iteratively Refine contour iteratively faster faster segmentationsegmentation Coarse result < 3 secCoarse result < 3 sec Fine result < 30 minFine result < 30 min

horizontalvertical

↓2g

↓2h↓2h

↓2g

↓2h

↓2g2D Filter bankLevel 1 decomposition

A Critical Review of Active Contours

Flexible

Can be tuned to be accurate

Adapt to topological changes in the image

But… Tuning of parameters is involved Updating the level set function – inefficient What is the ‘contour’ in a digital image? Discrete topological rules – external constraints can cause

abruptness Multiresolution – how do we reconstruct the level set function?

New math needed

Active Mask Framework: No ContoursActive Mask Framework: No Contours

Fluorescence microscope images speckled in natureFluorescence microscope images speckled in nature Estimate densities of bright pixels in local neighborhood Estimate densities of bright pixels in local neighborhood

at different scalesat different scales

Recast computation of force as a transformationRecast computation of force as a transformation No need for the time consuming extension functionNo need for the time consuming extension function

For image f, transform T isFor image f, transform T is

Windowing function Windowing function and scale factor and scale factor aa Different conditions (cell lines, resolution, etc.) Different conditions (cell lines, resolution, etc.) Different Different and and aa TPSTACS: Rectangular TPSTACS: Rectangular , a, a = 1 and suitable operands = 1 and suitable operands

20 40 60 80 100 120 140 160 180 200

20

40

60

80

100

120

140

160

20 40 60 80 100 120 140 160 180 200

20

40

60

80

100

120

140

160

20 40 60 80 100 120 140 160 180 200

20

40

60

80

100

120

140

160

20 40 60 80 100 120 140 160 180 200

20

40

60

80

100

120

140

160

Original ImageOriginal Image A slight blurA slight blur

Enough to discern the cell Enough to discern the cell boundaryboundary

Too much blur – Edges Too much blur – Edges roundedrounded

Active Masks: ResultsActive Masks: Results

SuccessSuccess Initialization: Level set function is identically zeroInitialization: Level set function is identically zero

Iterations: 3Iterations: 3

Time taken: 6.5 sec per iterationTime taken: 6.5 sec per iteration

HeLa cells – Total protein imageHeLa cells – Total protein image HeLa cells – Membrane protein imageHeLa cells – Membrane protein image

Active Masks

Pros Framework suited to digital images Can be made specific with the choice of suitable forces,

windows and scale factors Performance not critically dependent on initialization Easy and fast to compute Translation, dilation and rotation invariance can be

preserved

Cons Topology preservations hard

Multiple active mask framework

Multiple Active Masks

Initialization Random initialization with M»M0 masks,

where M0 = expected number of objects in the image

Evolution: driven by distributor functions

Can incorporate multiresolution/multiscale

Convergence Experimentally Working on a proof

Results of Results of STACS on Different ModalitiesSTACS on Different Modalities

Yeast DIC Cardiac MRI: Endocardium and epicardium

Axial Coronal

True Positive False Positive False Negative

Saggital

Brain fMRI

Classification Problems in BioimagingClassification Problems in Bioimaging

Determination of Determination of protein subcellular location patternsprotein subcellular location patterns[Chebira, Barbotin, Jackson, Merryman, Srinivasa, Murphy & Kovačević, 2007][Chebira, Barbotin, Jackson, Merryman, Srinivasa, Murphy & Kovačević, 2007]

Detection of developmental stages in Detection of developmental stages in DrosophilaDrosophila embryos embryos[Kellogg, Chebira, Goyal, Cuadra, Zappe, Minden & Kovačević, 2007][Kellogg, Chebira, Goyal, Cuadra, Zappe, Minden & Kovačević, 2007]

Classification of histological stem-cell teratomasClassification of histological stem-cell teratomas[Ozolek, Castro, Jenkinson, Chebira,, Kovačević, Navara, Sukhwani, [Ozolek, Castro, Jenkinson, Chebira,, Kovačević, Navara, Sukhwani, Orwig, Ben-Yehudah & Schatten, 2007]Orwig, Ben-Yehudah & Schatten, 2007]

Fingerprint recognitionFingerprint recognition [Hennings, Thornton, Kovačević & Kumar , 2005] [Hennings, Thornton, Kovačević & Kumar , 2005] [Chebira, Coelho, Sandryhalia, Lin, Jenkinson, MacSleyne, Hoffman, Cuadra, [Chebira, Coelho, Sandryhalia, Lin, Jenkinson, MacSleyne, Hoffman, Cuadra, Jackson, Püschel & Kovačević , 2007]Jackson, Püschel & Kovačević , 2007]

Develop an Develop an automated systemautomated system capable of capable of

fast, robust and accurate classificationfast, robust and accurate classification

Multiresolution ClassificationMultiresolution Classification

Hypothesis: Better classification accuracy obtained if we use the space-Hypothesis: Better classification accuracy obtained if we use the space-frequency information lying in the MR subspacesfrequency information lying in the MR subspaces Compute features in the MR-decomposed subspaces (subbands) insteadCompute features in the MR-decomposed subspaces (subbands) instead

Would like to use wavelet packetsWould like to use wavelet packets Do not have an obvious cost measureDo not have an obvious cost measure Do it implicitly insteadDo it implicitly instead

Generic Classification System

FeatureExtraction

ClassificationMRWeightingAlgorithmFE CMR W

shorthand

MR BlockMR Block

Grow full tree to L levelsGrow full tree to L levels

Use all nodes Use all nodes

MR Bases MR Bases DWTDWT DFTDFT DCTDCT … …

MR FramesMR Frames SWTSWT DT-CWTDT-CWT DD-DWTDD-DWT Our design: LTFTOur design: LTFT

FE CMR W

Lapped Tight Frame Transforms

Build MR transforms for these problems

Not many nonredundant ones exist

Seed them from higher-dimensional bases

Feature Extraction and ClassifierFeature Extraction and Classifier

Feature ExtractionFeature Extraction New Haralick texture features (TNew Haralick texture features (T33, 26 features), 26 features)

Morphological features (M, 16 features)Morphological features (M, 16 features) Zernike features (Z, 49 features)Zernike features (Z, 49 features)

ClassifierClassifier Neural networksNeural networks

No hidden layersNo hidden layers

FE CMR W

FE CMR W

Weighting ProcedureWeighting Procedure

Local decisionsLocal decisions Decision vectors for each subband Decision vectors for each subband

of each training image containing C numbersof each training image containing C numbers Goal: combine local decisions into a global oneGoal: combine local decisions into a global one

AlgorithmsAlgorithms Open form (iterative)Open form (iterative) Closed form (analytical)Closed form (analytical)

Per data setPer data set Per classPer class

Pruning criteriaPruning criteria

FE CMR W

Determination of PSL Patterns: Determination of PSL Patterns: ResultsResults

MR significantly MR significantly outperforms outperforms NMRNMR

MRF outperform MRF outperform MRBMRB

Per-Dataset CF Per-Dataset CF slightly slightly outperforms OFoutperforms OF

Trend is flatTrend is flat→ → TT33 set enough set enough

Why Do MR Frames Work?

Looking into classes of signals where bases/frame perform better

Simple example Real plane Two classes Decision rule Union of nonoverlaping parallelograms, bases,

otherwise, frames

Conclusions and OpportunitiesConclusions and Opportunities

Tasks

Issues

Framework

Tools


What can we do?

Conclusions & OpportunitiesConclusions & Opportunities

The “dream”:The “dream”:automated, efficient andautomated, efficient andreliable processing as wellreliable processing as wellas knowledge extractionas knowledge extraction

from large bioimage from large bioimage databasesdatabases

Dig in!Dig in!

Gaps to fillGaps to fill Need tools adapted to Need tools adapted to

specific bioimaging specific bioimaging applications applications

Need to adapt state-of-the-Need to adapt state-of-the-art techniques and/or art techniques and/or come up with new ones for come up with new ones for bioimaging tasksbioimaging tasks

problems in biological imaging: opportunities for signal processing jelena kovačević bimagiclab...

Documents

contrastwide range of

timefrequency spacedesign

signal asinfinite signal

frequency featuresnoise

customerswide range

range of normal states

system noise

training data