1

Supplementary Information for Fast fit-free analysis of fluorescence lifetime imaging via deep learning Jason T. Smith*, Ruoyang Yao, Nattawut Sinsuebphon, Alena Rudkouskaya, Nathan Un, Joseph Mazurkiewicz, Margarida Barroso, Pingkun Yan and Xavier Intes*

Jason T. Smith, Dr. Xavier Intes Email: smithj28@rpi.edu, intesx@rpi.edu, This PDF file includes:

Supplementary text Figures S1 to S25 Tables S1 to S7 SI References

Other supplementary materials for this manuscript include the following:

GitHub Repository: https://github.com/jasontsmith2718/DL4FLI

www.pnas.org/cgi/doi/10.1073/pnas.1912707116

mailto:smithj28@rpi.edumailto:intesx@rpi.eduhttps://github.com/jasontsmith2718/DL4FLI

2

Supplementary Information Text

1. Simulation Data Routine.

The way in which data simulated for MFLI and FLIM analysis differed is detailed on our GitHub repository https://github.com/jasontsmith2718/DL4FLI (along with the corresponding data and MATLAB script), but will be described in brief here. Fig. S2 is given to illustrate an overview of the simulation data procedure used in this work. During the start of each TPSF voxel simulation, an MNIST figure was selected at random. To ensure MNIST figures with high sparsity were not selected, if the sum of all pixels in a MNIST figure was below a certain threshold (

3

2. In silico Performance Validation.

Fig. S21 illustrates an in silico replication of the well-plate experiment performed for Fig. S20. By using the intensity values obtained at both 15 µW and 45 µW illumination power, along with a single lifetime of 0.55 ns, a mono-exponential TPSF was generated at each pixel to obtain two pseudo-MFLI data to explore the effect of photon-count on retrieval of lifetime values via both FLI-Net and LSF versus a known ground-truth. Fig. S21c & Fig. S21d illustrate similar trends of decreasing lifetime values at lower photon-count just as Fig. S20 illustrated. Similarly, the trend observed with the LSF results is much steeper than that obtained via the 3D-CNN. Moreover, Fig. S21k illustrates both distributions of lifetime along with the assigned ground-truth (notated by a vertical dashed line) – illustrating a much narrower distribution more closely centered around ground-truth than with LSF. Fig. S21m is given to further quantify this point, where all averaged lifetimes measured across each well ROI by FLI-Net are observed to match ground-truth values (right-most column) more closely than with LSF at respective wells and laser excitation. For the highest illumination the 3D-CCN was able to retrieve the lifetime with less than 30ps error even in the case of a low fluorophore concentration (well #5). Again, the model used to evaluate this data was trained with TPSFs simulated using maximum p.c. > 500.

3. Macroscopic FLI Performance Validation

3.1. Lifetime Retrieval at Low Photon-Counts

In contrast to Fig. 1d, e and Fig. S4, Fig. S20 illustrates a controlled in vitro experiment performed to investigate the 3D CNN’s capability in reconstructing lifetime values at exceedingly low photon-counts. Five wells containing a serial dilution of AF750 (300 μL, 150 μL, 75 μL, 37.5 μL and 18.75 μL) were prepared and volume normalized to 300 μL/well with PBS for wide-field MFLI. Imaging took place using a 750 nm excitation wavelength, 780 12/25 band-pass filter, at both 15 µW and 45 µW laser output power. Dark-image subtraction was performed before analysis via both approaches.

At 45 µW laser power (intensity illustrated by Fig S20b) the 3D CNN and mono-exponential LSF results (Fig S20(d, f)) are in agreement across all wells – both in average values, variance across each well and with regards to the expected slightly decreasing lifetime trend it may be related to deterioration of the lifetime decay due to lack of photon counts. This lifetime trend is also observed at 15 µW laser power, yet a stark dip in reconstructive accuracy is observed in LSF versus the DNN approach. This can be observed in the given boxplot pair Fig S20c, e, which illustrates examples of failed convergences and wide spreads for LSF in large difference to the 3D CNN-obtained values. Further, one can observe a few notable contrasts between the MFLI reconstructions obtained with both analytic methodologies. As Fig S20i, j illustrates, the lifetime values obtained via the DNN using both laser powers are quite similar. This is in large contrast with those obtained through LSF (shown as Fig S20k, l). Indeed, LSF estimates are much lower at wells #4 and #5, which possess TPSFs with maximum p.c. < 100 (Fig S20g).

As before, the network used to evaluate this data was trained with TPSFs simulated with maximum p.c. > 500. This, along with the previously discussed in silico study (Fig. 1e), provides evidence for the Deep Learning’s capability to perform FLI reconstruction well at exceedingly low photon-count levels without being trained with data possessing such high degrees of noise. Future study will be aimed towards exploiting this valuable characteristic of the presented analytic methodology.

3.2. Lifetime Retrieval in the Case of a Biased Training.

In Deep Learning-based work, it is of paramount important to ask: what would happen in the event that a trained model was applied to process a dataset for which it was not trained to process? Our first experimental examination of this is given in Fig. S17, where six different well plates containing AF700 underwent differing dilution triads over the course of 420 MFLI acquisitions (Table S4). This data was analyzed by both mono-exponential LSF and the 3D-CNN trained for NIR applications (𝜏1, 𝜏2) = ([0.2-0.5] ns, [0.9-1.1] ns)). Fig. S17b, which illustrates the average and standard deviation value of each well-plate’s predicted 𝜏𝑀 over all 420 acquisitions, provides a consistent average value of ~ 1.2 ns for the ‘PNN’. Given that the maximum mean-lifetime of the network during training was 1.1 ns, this result illustrates that even though the network never “saw” a TPSF with lifetimes this high during training, the model extrapolated by > 0.1 ns to make this quantification. The mono-exponential LSF, which was given an upper lifetime bound of 1.5 ns for fitting, calculated similar results for the ‘PNN’ well and all others (Fig. S17d). This is indicative that the network has potential to extrapolate slightly outside of the bounds that it was trained for.

Though the results from Fig. S17 are noteworthy, they are not entirely unexpected given that the degree of extrapolation was just 0.1 ns. For our objective, an example of a more extreme question would be framed as follows: what would happen if a model was trained, using two bounds of both short and long lifetime, that did not encapsulate the ground-truth experimental data? In Fig. S25, a well-plate was prepared using both the NIR FRET pair AF700/AF750 (top row) and

4

AF700/QC1 (bottom row) at decreasing (left-right) acceptor-donor ratios (3:1 – 0:1), which is known to correspond with increasing values of 𝜏𝑀. The model trained for NIR MFLI was used to obtain the 𝜏𝑀 for Case #1 (Table S1), where the lifetime values used during training fully encapsulate those expected during the experiment (0.4 ns and 1.0 ns, respectively). A second model was trained with TPSFs possessing lifetime values outside of these bounds (𝜏1, 𝜏2) = ([0.5-0.85] ns, [1.5-2.5] ns to investigate how the model would behave in such conditions. Further, the LSF approach was used as a comparative method – where in both Cases #1 and #2 the bounds were set to that of the trained network.

Given that each well-plate is prepared to be homogenous, all well-plate lifetime reconstructions should have a more-or-less constant value. As Fig. S25(a, b, e) illustrates, the 3D-CNN and LSF provide results in high agreement – though the standard deviation across each well-plate can be observed as higher for LSF. However, Fig. S25(c, d, f) illustrates the 3D-CNN’s performance after being trained with lifetimes falling well outside of the range of ground-truth, along with the LSF equivalent. Notably, the 3D-CNN and LSF reconstructions of 𝜏𝑀 are still in high agreement. Indeed, the model employed on experimental data it was never trained to correctly reconstruct performs at the level of LSF and still allows for the user to observe a similar trend of changing mean-lifetime with varied FRET acceptor-donor ratios. This is promising, since there is little reason to expect meaningful reconstructive results from the DNN in such a scenario.

Though, as was previously addressed, this is just a single example. Another important question to ask pertains to the DNN’s lifetime reconstruction capability in the presence of tri-exponential data. As one can observe from Fig. 1a, along with the simulation routine illustrated in Fig. S2, the addition of lifetime parameters via additional branches in the network architecture and an additional few lines of script in the simulation data workflow would not be a challenging task. Yet, if the presented 3D-CNN was trained on a bi-exponential model and given tri-exponential TPSF data to reconstruct, it would not be able to retrieve the third lifetime and it is challenging to report on how the presence of a third lifetime would affect all parametric mappings without further experimentation. Indeed, this sort of presence can affect visible FRET quantification if endogenous fluorescence also mixes in with the collected emission light and would be easily observed via phasor. Since with have focused on applications that are typically processed using only a bi-exponential model, we leave the undertaking of this important investigation to future work.

4. Supplementary Information for Figure 5. Comparison of FLI-Net with LSF using MFLI of NIR dyes possessing sub-nanosecond lifetime

The χ value for both approaches was obtained numerically via non-linear fitting (MATLAB) in both Fig. S15b and Fig. S16b. The procedure included obtaining a value for χµ via fitting each 𝜏𝑀 series (corresponding to differing µ, Fig. S15 legend) to Eq. S1 & S2 (χ, 𝜏1 and 𝜏2 were allowed to vary).

𝐴1−1 − 1 = (χµ)−1(𝑣1

−1 − 1)𝜏1

𝜏2 [S1]

𝜏𝑀 = 𝐴1𝜏1 + (1 − 𝐴1)𝜏2 [S2]

As mentioned previously, χ corresponds to a ratio of parameters inherent to both involved fluorescent species (Eq. S3).

χ =𝜎1∅1𝑛1

𝜎2∅2𝑛2 [S3]

Where 𝜎, ∅ and n correspond to absorption cross-section, quantum yield and detection efficiency, respectively.

In this experiment, the relationship between χµ and μ was chosen to such that a linear relationship would be expected. Indeed, χ values ~ 0.43 were obtained via linear regression for both the 3D-CNN and LSF results (Fig. S15b and Fig. S16b, respectively). For those interested, further detail is provided elsewhere.5

5. NIR MFLI-FRET in vivo: Matrigel Plugs.

Fig. S24 illustrates in vivo MFLI FRET quantification of implanted Matrigel plugs – each containing cancerous cells possessing various acceptor-donor ratios of Tf AF700 and AF750 (0:1, 1:1, 2:1 and 3:1, ordered numerically in Fig. S24a). Further detail is provided elsewhere.4

Notably, Fig. S24c shows little overlap in comparison to the FD% obtained through the LSF approach (Fig. S24f). Indeed, as was encountered previously (Fig. 5), LSF illustrated many convergences at either the upper or lower bound of 𝜏1 and 𝜏2. Employing the 3D-CNN approach again resulted in highly accurate central clustering for both lifetime values, demonstrated by the histogram given in Fig S24d. This is yet another example of the presented Deep Learning-based approach’s capability to quantify all three bi-exponential parameters with high accuracy even with the use of fluorophores

5

possessing analytically challenging sub-nanosecond lifetimes. Further, a single trained model, focused on NIR application, was used to process the well-plate datasets from Fig. S17, Fig. S20 and Fig. S24, as well as provide state-of-the-art reconstructive sensitivity for two notable applications of in vivo MFLI (including Fig. 6).

6

Supplementary Figures

Figure S1. Structure of “FLI-Net”. A 4D data voxel of size X × Y (pixels) × TP (time-points) × 1 (a) is convolved with a set of 50 (1 × 1 × 10) kernels (stride represented by k, defaulted to one if unlisted) along the temporal axis along with a subsequent, 3D-residual block (b) to extract point-by-point temporal features within the earliest network layers. To obtain the three 2D maps at the end (size X × Y), a reshape layer is used (depicted in green). Subsequent 2D convolutions with (1 × 1) filters downsample the parametric size without losing the fully-convolutional capability. After two 2D-convolutional residual blocks, the output is split into three independent series of fully convolutional operations (c) to simultaneously reconstruct the images of 𝝉𝟏, 𝝉𝟐 and 𝑨𝑹.

Conv3D(1110), k=(1,1,5)(50)

Reshape (XY(n))

ResBlock3D (1x1x5)(50)

Conv2D(11)(256)

Resblock2D(11)(256)

FC-Downsample FC-Downsample FC-Downsample

Lifetime image t1 Lifetime image t2 Amplitude AR

(x2)

τ1 τ2 AR

(X × Y × TP × 1)

time (ns)

τ1

TPSF

TPSF-Fit

IRF

BN & ReLU()

ReLU()

Conv3D

Conv3D

BN

Conv2D(55)

BN & ReLU()

Conv2D(11)

BN & ReLU()

Conv2D(33)

ReLU()

τ2

7

Figure S2. Illustration of the sequence used for creating simulation TPSF voxels used to train FLI-Net.

τ1

τ2

AR

I

(I, τ1, τ2, AR)

(1928, 0.23 ns, 0.52 ns, 0.95)

(763, 0.31 ns, 0.49 ns, 0.28)

TPSF Ex. 1

TPSF Ex. 2

IRF

time (ns)In

tensity (

norm

aliz

ed)

a b

0

2000

1000

0

1

0.5

0

0.4

0.2

0

0.6

0.3

8

Figure S3. Example TPSFs generated for both NIR-FLIM and NIR-MFLI. a-b, Both NIR-MFLI and NIR-FLIM (low photon-count example emphasized with b) were generated using the same lifetime parameters ( (𝐴𝑅, 𝜏1, 𝜏2) = (0.5, 0.4 ns, 1.0 ns), noted in green) but used a slightly different simulation procedure (detailed in Methods, in prior supplementary section Simulation Data Routine and in our GitHub repository https://github.com/jasontsmith2718/DL4FLI ).

Inte

nsity (

p.c

.)

time (ns)

MF

LI s

top

(6.4

ns

)

MFLI ex. #1

MFLI ex. #2

. 1

. .

(τ1 , τ2, AR) = (0.4 ns, 1.0 ns, 0.5) FOR ALL

a

b

https://github.com/jasontsmith2718/DL4FLI

9

Figure S4. Representative 𝝉𝑴 reconstructions obtained at three photon-count thresholds via the 3D-CNN (b, f, i) and LSF (c, g, k) approach in silico with corresponding ground-truth (a, e, i). (d, h, l) 2D scatter plot of 𝜏𝑀 results obtain via both analytic methodologies versus ground-truth (determined at point of simulation) at varying intensity thresholds. (m, n) 2D scatter plots of calculated 𝐴𝑅 versus ground-truth (GT) at all three intensity thresholds. o, Table of absolute error (𝜏𝑀) given for all three intensity thresholds. The lifetime bounds were set to (𝜏1, 𝜏2)[(0.5-0.75), (2-3)] ns for both techniques.

3D-CNN

LSF

=

3D-CNN

LSF

=

3D-CNN

LSF

=

p.c

. =

[250,5

00]

p.c

. =

[100,2

50]

p.c

. =

[25,1

00]

Ground-Truth τM (ns)

Calc

ula

ted τ

M(n

s)

Calc

ula

ted τ

M(n

s)

Calc

ula

ted τ

M(n

s)

Ground-Truth 3D-CNN LSF

Photon Counts |G.T – LSF| (τM) |G.T – 3D-CNN| (τM)

[25-100] 0.242 ± 0.180 ns 0.083 ± 0.062 ns

[100-250] 0.159 ± 0.118 ns 0.056 ± 0.046 ns

[250-500] 0.099 ± 0.086 ns 0.042 ± 0.035 ns

τM

τM

τM

a b c

gfe

kji l

h

d

o

[50-100]

[100-250]

−

=

Ground-Truth ARGround-Truth AR

Calc

ula

ted A

R

Calc

ula

ted A

R

m n

10

Figure S5. Comparison of FLI-Net with an alternative temporal-information extracting 2D-CNN architecture. a, The best-performing 2D-CNN developed during the beginning phase of this project. b, Two example TPSFs obtained by averaging all TPSFs within the liver and bladder regions of a mouse during the dynamic Tf/TfR experiment at a random time-point. c, Example overlay of all generated TPSFs for a single simulated voxel used for training to compensate for scattering artifacts in vivo. The training and validation MSE curves are shown for a network architecture (a) using only 2D, separable convolutions to extract temporal information (d) versus the use of 3D convolutions (e) along with an alternative side-view of what the generated TPSFs look like at a specific row versus time (f-g).

FC-Downsample FC-Downsample FC-Downsample

Lifetime image t1 Lifetime image t2 Amplitude AR

(x2)

τ1 τ2 AR

(X × Y × TP)

SeparableConv2D(11)(256)

Resblock2D(11)(256)

Conv2D(11)(256)

BN & ReLU()

ReLU()

BN

Conv2D

Conv2D

MS

E

Epoch

MS

E

Training

Validation

Training

Validation

time (ns)

Liver

Bladder

Inte

nsity

Inte

nsity

b

c

d

e

tim

e (

ns)

X (pixel)

X (pixel)

f

g

Alternative Network (2D-CNN)a

11

Figure S6. SSIM and MSE comparison results between FLI-Net and SPCImage 𝝉𝑴 images of NADH (Fig 2).

0 0.80 0.85 0.90 0.95 1.00.75

SSIM

T47D

T47D-NaCN

MDA-NaCN

MDA

MCF10a-NaCN

MCF10a

AU565-NaCN

AU565

T47D

T47D-NaCN

MDA-NaCN

MDA

MCF10a-NaCN

MCF10a

AU565-NaCN

AU565

0.001 0.002 0.003 0.004 0.005 0.006

MSE

12

Figure S7. Further quantification of the visible TCSPC FLIM data included in Fig. 3. a, Averaged lifetime values over all pixels for both the 3D-CNN and SPCImage along with linear-fit overlays. b, 2D scatter plot of the averaged mean-lifetime obtained for every visible FLIM-FRET dataset through both analytic methodologies (p = 1.78e-08).

(A:D) Ratio

τ Mn

s

SPC

Imag

e(τ

M)

3D-CNN (τM)

3D-CNN

SPCImage

Fit Line (3D CNN)

Fit Line (SPCImage)

(0:1)(0.5:1)(1:1)

(2:1)

R2 = 0.969

a b

13

Figure S8. Full-image comparisons of FLI-Net vs. SPCImage using visible TCSPC FLIM microscopy data (Fig 3).

A:D - 0:1 A:D – 0.5:1 A:D - 1:1 A:D - 2:1

FL

I-N

et

A:D - 0:1 A:D – 0.5:1 A:D - 1:1 A:D - 2:1

SP

CIm

ag

e

τM (ns)

0.5 2.71.05 1.6 2.15

a b c d

e f g h

25 mm 25 mm 25 mm 25 mm

25 mm 25 mm 25 mm 25 mm

14

Figure S9. SSIM comparisons of FLI-Net vs. SPCImage using visible TCSPC FLIM microscopy data (Fig 3).

(0:1)

(0.5:1)

(1:1)

(2:1)

0 0.80 0.85 0.90 0.95 1.00.75

SSIM

Acce

pto

r:D

ono

rR

atio

15

Figure S10. Illustration of user-related bias during quantification of NIR FLIM-FRET via SPCImage. (a, b) FD% distributions obtained by locking 𝜏1 at the expected 0.4 ns and 𝜏2 at both 1.0 ns (a) and 1.1 ns (b) for four different acceptor-donor ratios. c, Average FD% values of all calculated values with standard deviation using both 𝜏2 values. (d, e) Illustration of the differing FRET quantification obtained for a (0:1) accepter-donor FLIM voxel using both values of 𝜏2.

Fre

que

ncy

Fre

que

ncy

Calculated FD(%)

τ2 = 1.0ns

τ2 = 1.1ns

(A:D) Incubation

Calc

ula

ted

FD

(%)

b

a c

τ2

= 1

.1 n

sτ

2 =

1.0

ns

(0:1)(0.5:1)(1:1)

(2:1)

FD(%)

0% 50% 100%

τ2 = 1.0ns τ2 = 1.1ns

Calculated FD(%)

d e25 mm

A:D - 0:1 A:D - 0:1

25 mm

16

Figure S11. Example of inherent user bias using SPCImage for FLIM-based quantification of metabolic status in breast cancer cell line AU565. (a-c), NADH 𝜏𝑀 values obtained from the same FLIM voxel calculated using three different starting locations (all other settings left identical). d, Histogram of distributions along with average and standard deviation values (red (a), blue (b) and golden (c) with corresponding starting locations (e)).

25mm 25mm 25mm

1.3

τM (ns)

0.5 0.9

0.5 1 1.5 20

200

400

00

800

1000

1200

1400

1 00

0.9856 ± 0.1885 1.0890 ± 0.1867 1.2223 ± 0.1874

NAD(P)H τM (ns)

Fre

qu

en

cy

a b c

d e

c

a

b

17

.

Figure S12. Full-image comparisons of FLI-Net vs. SPCImage using NIR TCSPC FLIM microscopy data.

0% 80%20% 40% 60%

FD (%)

a b c d

e f g h

A:D - 0:1 A:D – 0.5:1 A:D - 1:1 A:D - 2:1

A:D - 0:1 A:D – 0.5:1 A:D - 1:1 A:D - 2:1

FL

I-N

et

SP

CIm

ag

e

25 mm 25 mm 25 mm 25 mm

25 mm 25 mm 25 mm 25 mm

18

Figure S13. SSIM comparisons of FLI-Net vs. SPCImage using NIR TCSPC FLIM microscopy data.

0 0.80 0.85 0.90 0.95 1.00.75

(0:1)

(0.5:1)

(1:1)

(2:1)

SSIM

Acce

pto

r:D

ono

rR

atio

19

Figure S14. Further quantification of the NIR TCSPC FLIM-FRET data included in Fig 4. a, Averaged FD% values over all pixels for both the 3D-CNN and SPCImage along with linear-fit overlays. b, 2D scatter plot of the averaged mean-lifetime obtained for every visible FLIM-FRET dataset through both analytic methodologies (p = 2.44e-09).

(A:D) Ratio

τ Mn

s

SPC

Imag

e(τ

M)

3D-CNN (τM)

3D-CNN

SPCImage

Fit Line (3D CNN)

Fit Line (SPCImage)

(0:1)(0.5:1)(1:1)

(2:1)

R2 = 0.969

a b

20

Figure S15. Further quantification of Figure 5 (FLI-Net). a, 2D scatter plot of each averaged 𝜏𝑀 value obtained via FLI-Net versus volumetric fraction of HITCI, overlaid with six corresponding mechanistic fits, for six well-plate dilutions. Each well-plate dilution was prepared using a differing initial concentration (notated as µ) corresponding to (HITCI0/ATTO0). b, Linear-fit of χµ (χ corresponding to a ratio of optical properties inherent to both dye species) versus µ calculated for all six well-plate mixtures (described further in the text below). Shaded region corresponds to 95% confidence interval. c, Linear regression of all averaged 𝜏𝑀 values obtained through both analytic methodologies.

R2 = 0.992 ± 3.60e-3

3D-CNN τM (ns)

LSF τ M

(n

s)

c

χ = 0.43

R2 = 0.983

HITCI Volume Fraction (v1)

3D

-CN

N τ

M (n

s)

μχμ

a b

(Mix #1) μ = 0.5

(Mix #2) μ = 1.0

(Mix #3) μ = 1.5

(Mix #4) μ = 2.5

(Mix #5) μ = 5.0

(Mix #6) μ = 7.5

21

Fig. S16. Further quantification of Fig. 5 (LSF). a, 2D scatter plot of each averaged 𝜏𝑀 value obtained via FLI-Net versus volumetric fraction of HITCI, overlaid with six corresponding mechanistic fits, for six well-plate dilutions. Each well-plate dilution was prepared using a differing initial concentration (notated as µ) corresponding to (HITCI0/ATTO0). b, Linear-fit of χµ (χ corresponding to a ratio of optical properties inherent to both dye species) versus µ calculated for all six well-plate mixtures (described further in the text below). Shaded region corresponds to 95% confidence interval.

HITCI Volume Fraction (v1)

LSF τ M

(n

s)

μ

χμ

χ = 0.44

R2 = 0.987

a b

(Mix #1) μ = 0.5

(Mix #2) μ = 1.0

(Mix #3) μ = 1.5

(Mix #4) μ = 2.5

(Mix #5) μ = 5.0

(Mix #6) μ = 7.5

22

Fig. S17. MFLI quantification of AF700 undergoing separate dilution series. a, Well-plate p.c. at acquisition #1. The labels denote the dilution series each well underwent over 420 MFLI acquisitions (‘P’, ‘E’ and ‘N’ corresponding to PBS, Ethanol and none, respectively). b, The 𝜏𝑀 obtained via the 3D-CNN across all 420 MFLI acquisitions. The solid line and shaded area correspond to average and standard-deviation, respectively. c, 2D Stern-Volmer plot3, where a 2nd order fit is given for visualization. d, 2D scatter plot of all average lifetime values obtained through LSF (mono-exponential) versus the 3D-CNN (𝜏𝑀). More information is given in Table S2.

PNN

ENN

PEE

PEP

EPE

EPP

RMSE = 0.0133

R2 = 0.991

PNN

ENN

PEE

PEP

EPE

EPP

3D-C

NN

(τ M

)

3D

-CN

N (

1/τ

M)

LSF (τ)

3D-C

NN

(τ M

)

VPBS/ VTotal

a b

c d

MFLI Acquisition (#)

23

Fig. S18. Histograms of 𝑨𝑹 absolute error. a-b, Absolute error for both techniques using (𝜏1, 𝜏2, 𝑝. 𝑐.) = ([0.2-0.4] ns, [0.45-0.65] ns, [250-500])). c-d, Absolute error for both techniques using (𝜏1, 𝜏2, 𝑝. 𝑐.) = ([0.35-0.6] ns, [0.9-1.1] ns, [500-2000])).

AR < 0.05

AR > 0.95

AR < 0.05

AR > 0.95

|Predicted – G.T.|

fre

que

ncy

fre

qu

en

cy

AR < 0.05

AR > 0.95

AR < 0.05

AR > 0.95

FLI-NetLSF

|Predicted – G.T.|

a

c d

b

AR < 0.05

AR > 0.95

24

Fig. S19. Diagram illustrating the timeline of in vivo MFLI acquisition and injection (Fig. 6).

Donor-Tf

(Tail vein injection)

Acceptor-Tf

(Retro-orbital injection)

Data Acquisition

15 minutes 105 minutes120 minutes

Mouse Donor-Tf Acceptor-Tf

Control 20 µg -

FRET 20 µg 40 µg

25

Fig. S20. In vitro MFLI reconstruction of AF750 serial dilution at varying degrees of photon-count. a,b Intensity of both well-plate acquisitions at 15µW and 45µW laser power. c-f, Boxplots of lifetime values obtained over all wells through both analytic approaches. g-h, TPSF overlay of a representative FLI decay obtained at each of the five well locations using both laser powers. i-l, Mean-lifetime value maps obtained via both approaches at both laser powers. m, Table of averaged mean-lifetime values at each well ROI with corresponding standard deviations.

15 µ

W45 µ

W

3D-CNN LSF

τ(n

s)

τ(n

s)

1

15 µ

W45 µ

W

Inte

nsity (

p.c

.)

time (ns)

Inte

nsity (

p.c

.)

τ (ns)

123

45

a

b

c

d

e

g

h

j

i k

l

2 3 4 5

well-plate#

Well1Well2Well3Well4Well5

τM well1 τM well2 τM well3 τM well4 τM well5

LSF (15 µW) 0.498 ± 4.12e-2 ns 0.487 ± 5.51e-2 ns 0.464 ± 2.79e-2 ns 0.418 ± 3.90e-2 ns 0.358 ± 4.88e-2 ns

LSF (45 µW) 0.523 ± 4.18e-2 ns 0.517 ± 5.52e-2 ns 0.508 ± 1.64e-2 ns 0.493 ± 2.20e-2 ns 0.476 ± 2.86e-2 ns

3D-CNN (15 µW) 0.535 ± 2.18e-2 ns 0.521 ± 2.97e-2 ns 0.492 ± 5.34e-2 ns 0.477 ± 4.05e-2 ns 0.461 ± 4.90e-2 ns

3D-CNN (45 µW) 0.564 ± 1.65e-2 ns 0.556 ± 1.89e-2 ns 0.538 ± 4.57e-2 ns 0.518 ± 2.45e-2 ns 0.505 ± 3.31e-2 nsm

τ(n

s)

f1 2 3 4 5

well-plate#

τ(n

s)

26

Fig. S21. Simulation of MFLI well-plate serial experimental dilution containing AF700 and AF750 at two laser excitation powers. Each generated TPSF was set to have ground-truth lifetime values (𝐴𝑅, 𝜏1, 𝜏2) = (0.5, 0.4 ns, 1.0 ns) where only intensity varied from pixel-to-pixel and across well ROIs. a,b Intensity of both well-plate acquisitions obtained for Fig. S19 at both 15µW and 45µW illumination power. The numerical labeling along with the average intensity across each well is overlaid in a. c-f, Boxplots of lifetime values obtained over all wells through both analytic approaches. g-h, Mean-lifetime maps obtained using both techniques at both laser powers. k-l, Histograms of all 𝜏𝑀 values obtained using both techniques at both laser powers along with the ground-truth mean-lifetime (illustrated as a vertical black dashed line). m, Table of all average 𝜏𝑀 values obtained at each well ROI, corresponding standard deviations as well as the ground-truth listed in the right-most column.

Inte

nsit

y (p

.c.)

Inte

nsit

y (

p.c

.)

37.5 67.5

118.5 177.0 242.5

123

45

15 µ

W45 µ

W

τM

(ns

)τ

M(n

s)

3D-CNN LSF

LS

F3D

-DN

N

3D-CNN

LSF

. .

3D-CNN

LSF

. .

frequency (

45

µW

)

τM (ns)

frequency (

15

µW

)

τM well1 τM well2 τM well3 τM well4 τM well5 Ground-Truth τM

LSF (15 µW) 0.518 ± 4.61e-2 0.502 ± 5.38e-2 0.488 ± 5.92e-2 0.456 ± 7.51e-2 0.428 ± 7.86e-2 0.55 ns

LSF (45 µW) 0.534 ± 2.96e-2 0.531 ± 3.19e-2 0.529 ± 3.41e-2 0.506 ± 4.85e-2 0.484 ± 6.12e-2 0.55 ns

3D-CNN (15 µW) 0.534 ± 2.91e-2 0.530 ± 3.04e-2 0.528 ± 3.40e-2 0.515 ± 4.36-2 0.507 ± 5.49e-2 0.55 ns

3D-CNN (45 µW) 0.551 ± 1.99e-2 0.547 ± 2.19e-2 0.543 ± 2.38e-2 0.536 ± 2.72e-2 0.525 ± 3.41e-2 0.55 ns

15 µW 45 µW

15 µW 45 µW

a

b

i

g

j

hk

l

m

τM

(ns)

τM

(ns)

c e

well-plate#well-plate#

τM

(ns)

τM

(ns)

d f

1 2 3 4 51 2 3 4 5

1 2 3 4 5 1 2 3 4 5

27

Fig. S22. Illustration of how the selection of “n” in SPCImage dictates the amount of spatial binning used for analysis.

28

Fig. S23. NADH 𝝉𝑴 maps obtained at various photon-count levels (a-f) using SPCImage (d-f) and our approach (a-c). Maximum photon-count images (g-i) and a corresponding 25-pixel averaged TPSF (j, ROI notated by yellow arrow) is given.

1.3

τM (ns)

a b

0.5 0.9

(3x3) (5x5) (11x11)(9x9)

1.3

τM (ns)

0.90.5

d e f

3D

-CN

NS

PC

Imag

e

(3x3) (4x4)

c

(9x9)

b

(5x5)

(3x3) (5x5) (9x9)

g h i

0

100

150

200

250

300

350

50

0

100

150

200

250

300

350

50

0

100

150

200

250

300

350

50

j

Ph

oto

n-C

ou

nts

time (ns)

I II

Ma

x P

ho

ton

-Co

un

t

TPSF via bin (3x3)

TPSF via bin (5x5)

TPSF via bin (9x9)

29

Fig. S24. FRET quantification of subcutaneously implanted Matrigel plugs in vivo via wide-field NIR MFLI possessing various acceptor-donor ratios of AF700 and AF750. a, Maximum photon-counts at each matrigel ROI with numeric labeling (numbers 1-4 corresponding to (0:1, 1:1, 2:1 and 3:1) A:D ratio). b-d, Visual aid (b) and histogram depiction of FD% (c) and lifetime (d) obtained via the presented 3D-CNN. e-g, Visual aid (e) and histogram depiction of FD% (f) and lifetime (g) obtained via LSF bounded to the same lifetime values used during network training.

0.204 ± 0.090

0.346 ± 0.069

0.480 ± 0.061

0.606 ± 0.050

0.088 ± 0.050

0.309 ± 0.054

0.509 ± 0.047

0.623 ± 0.031

0.402 ± 0.035 1.01 ± 0.042

0.289 ± 0.0951.01 ± 0.071

τ1τ2

τ1τ2

ROI1ROI2ROI3ROI4

ROI1ROI2ROI3ROI4

FD

%F

D%

time (ns)

time (ns)

FD%

FD%

Fre

quency

Fre

quency

Fre

quency

Fre

quency

Inte

nsit

y (p

.c.)

3D-CNN

LSF

a

c

d

f

g

1

2

4

3

b

e

30

Fig. S25. Example of FLI-Net 𝝉𝒎 reconstruction after training with lifetime values encompassing those used for a well-plate experiment (b, (𝝉𝟏, 𝝉𝟐) = ([0.2-0.5] ns, [0.9-1.1] ns)) versus after training with lifetime values outside of the range of training (d, (𝝉𝟏, 𝝉𝟐) = ([0.5-0.85] ns, [1.5-2.5] ns)). Well-plates were prepared with AF700/AF750 (top row) and AF700/QC1 (bottom row) FRET pairs at varying concentrations and imaged via wide-field MFLI. Corresponding LSF results obtained using the same bounds that FLI-Net was trained with are given (a, c). (e, f) A scatter plot of average 𝜏𝑀 and corresponding standard deviation error bars is provided for further illustration.

Ca

se

#1

Ca

se

#2

LSF 3D-CNN

τ M(n

s)

τ M(n

s)

Well-Plate #

τM

τM

3D-CNN

LSF

3D-CNN

LSF

1 3 5 7 9

2 4 6 8 10

3:1 2:1 1:1 0.5:1 0:1 3:1 2:1 1:1 0.5:1 0:1

a b

c d

e

f

31

Fig. S26. FLI-Net architecture depicted in detail for clarity.

32

Tables

Table S1. Values used during simulation of TPSFs for each trained model (top row) along with the results obtained by each (second row).

NIR MFLI Visible FLIM NIR FLIM NIR MFLI (FRET) Experimental

Training Parameters Fig 1, 5 & S17 Fig 2 & 3 Fig 4 Fig 6, S15, S19,

S20a, S21

Fig S20b

τ1 0.2-0.4 ns 0.4-0.7 ns 0.2-0.5 ns 0.2-0.5 ns 0.5-1.0 ns

τ2 0.45-0.65 ns 2-3 ns 0.9-1.1 ns 0.9-1.1 ns 1.5-2.5 ns

AR 0-1 0-1 0-1 0-1 0-1

Time-Points 176 256 155 160 160

p.c. 500-2000 250-1500 250-1500 500-2000 500-2000

33

Table S2. Initial concentrations and concentration ratios used for preparation of each well-plate dataset used for Fig. 5.

Mix # 1 2 3 4 5 6

Concentration ratio 0.5 1 1.5 2.5 5 7.5

ATTO 740 (μM) 10 10 10 10 10 10

HITCI (μM) 5 10 15 25 50 75

34

Table S3. Fluorescent dye volumes as prepared for MFLI Fig. 5.

Well # 1 2 3 4 5 6 7 8 9 10 11

HITCI volume fraction 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

ATTO 740 volume (μL) 300 270 240 210 180 150 120 90 60 30 0

HITCI volume (μL) 0 30 60 90 120 150 180 210 240 270 300

35

Table S4. Volumetric information pertaining to dilution series experiment illustrated and explained in Fig. S19. Each well (top row), along with the corresponding volumes of both PBS and Ethanol (EtOH) for each MFLI measurement (first column) are given.

EPP EPE PEP PEE ENN PNN

Data PBS EtOH PBS EtOH PBS EtOH PBS EtOH PBS EtOH PBS EtOH

0 0 100 0 100 100 0 100 0 0 200 200 0

21 20 100 20 100 100 20 100 20 0 200 200 0

41 40 100 40 100 100 40 100 40 0 200 200 0

61 60 100 60 100 100 60 100 60 0 200 200 0

81 80 100 80 100 100 80 100 80 0 200 200 0

101 80 100 80 100 100 80 100 80 0 200 200 0

181 100 100 80 120 120 80 100 100 0 200 200 0

201 120 100 80 140 140 80 100 120 0 200 200 0

221 140 100 80 160 160 80 100 140 0 200 200 0

241 160 100 80 180 180 80 100 160 0 200 200 0

261 180 100 80 200 200 80 100 180 0 200 200 0

281 200 100 80 220 220 80 100 200 0 200 200 0

301 220 100 80 240 240 80 100 220 0 200 200 0

321 220 100 80 240 240 80 100 220 0 200 200 0

420 220 100 80 240 240 80 100 220 0 200 200 0

36

To assess the 3D-CNN’s capability in accurately identifying mono-exponential decays, two tests were performed. Two different sets of TPSF data were simulated between differing bi-exponential lifetime and intensity bounds. The absolute errors were calculated using only pixels possessing ground-truth 𝐴𝑅 values of either below 0.05 (bottom 5%) or above 0.95 (top 5%) for both FLI-Net and LSF.

Table S5. Average absolute error |Predicted – G.T.| obtained using both techniques. n corresponds to number of total pixels evaluated (500 simulated TPSF voxels, (𝜏1, 𝜏2, 𝑝. 𝑐.) = ([0.2-0.4] ns, [0.45-0.65] ns, [250-500])).

Threshold FLI-Net LSF

𝐴𝑅 < 5% (n = 7460) 0.0437 ± 0.0372 0.259 ± 0.112

𝐴𝑅 > 95% (n = 7415) 0.0135 ± 0.0124 0.0684 ± 0.0529

37

Table S6. Average absolute error |Predicted – G.T.| obtained using both techniques. n corresponds to number of total pixels evaluated (450 simulated TPSF voxels, (𝜏1, 𝜏2, 𝑝. 𝑐.) = ([0.35-0.6] ns, [0.9-1.1] ns, [500-2000])).

Threshold FLI-Net LSF

𝐴𝑅 < 5% (n = 6323) 0.0353 ± 0.0294 0.0480 ± 0.0545

𝐴𝑅 > 95% (n = 6393) 0.0136 ± 0.0136 0.0684 ± 0.0529

38

References

1. R Yao, M Ochoa, X Intes, P Yan, “Deep Compressive Macroscopic Fluorescence Lifetime Imaging,” arXiv preprint arXiv:1711.06187

2. Chollet, François. "Xception: Deep learning with depthwise separable convolutions." arXiv preprint (2017): 1610-02357.

3. Keizer, Joel. "Nonlinear fluorescence quenching and the origin of positive curvature in Stern-Volmer plots." Journal of the American Chemical Society 105.6 (1983): 1494-1498.

4. N Sinsuebphon, Alena Rudkouskaya, M Barroso and X Intes, “Comparison of illumination geometry for lifetime-based measurements in whole-body preclinical imaging,” Journal of Biophotonics, e201800037 (2018).

5. Chen, Sez‐Jade, et al. "In vitro and in vivo phasor analysis of stoichiometry and pharmacokinetics using short‐lifetime near‐infrared dyes and time‐gated imaging." Journal of biophotonics 12.3 (2019): e201800185.