bio-image analysis, bio-statistics and machine learning · fourier transforms, convolutions, and...

Fourier Transforms, Convolutions, and Deconvolution

Bio-image analysis, bio-statistics and machine learning

Martin Weigert, Myers Lab, CSBD/MPI-CBG

June 4th, 2019

Outline

• Fourier Transform • Convolutions • Deconvolution

… in the context of image analysis

Fourier-Transforms

https://xkcd.com/26/

History: Solving the Diffusion problem

• Concentration of a chemical in solution • Thermal conduction (flow of heat) <latexit sha1_base64="9dXkl10ZToFGRiGUS5+e9FutdMs=">AAACH3icbVDLSgMxFE1aH7W+Wl26CRahgpQZEXRZdOOygn1AO5RMmmljM8mQZMQ69B/c6s6vcSdu+zdm2lnY1gsXDufcy7n3+BFn2jjODObyG5tb24Wd4u7e/sFhqXzU0jJWhDaJ5FJ1fKwpZ4I2DTOcdiJFcehz2vbHd6nefqZKMykezSSiXoiHggWMYGOpFqm+XJjzfqni1Jx5oXXgZqACsmr0yzDfG0gSh1QYwrHWXdeJjJdgZRjhdFrsxZpGmIzxkHYtFDik2kvm507RmWUGKJDKtjBozv7dSHCo9ST07WSIzUivain5n9aNTXDjJUxEsaGCLIyCmCMjUfo7GjBFieETCzBRzN6KyAgrTIxNaMklwtwGJOTyJ5p4Seoa4VerFG1s7mpI66B1WXOdmvtwVanfZgEWwAk4BVXggmtQB/egAZqAgCfwBt7BB/yEX/Ab/ixGczDbOQZLBWe/JX+h8Q==</latexit>

<latexit sha1_base64="xbvFwBdG4RUhZcioIgdPCO40jJY=">AAACbHicbVDBSiNBEO2M6xqzuiauNxGaDSsuLGEmCHoRZN3DHg0YFZIx1HR6tElPT9NdI4nDfJff4sGr3vYbticJaNSChsd7r6q6XqSlsOj7DxVv6dPy55Xqau3L2vrXjXpj89ymmWG8y1KZmssILJdC8S4KlPxSGw5JJPlFNDop9YtbbqxI1RlONA8TuFYiFgzQUYN6px8bYHlfg0EBMmd741/4syheGCwKekT/0EXjVfsD67i4aheDetNv+dOi70EwB00yr9NBo7LUH6YsS7hCJsHaXuBrDPNyKJO8qPUzyzWwEVzznoMKEm7DfHp7QX84Zkjj1LinkE7Z1x05JNZOksg5E8Ab+1YryY+0XobxYZgLpTPkis0WxZmkmNIySDoUhjOUEweAGeH+StkNuIzQxb2wRYN0aat08RLLwrzcquHOKTUXW/A2pPfgvN0K/FbQ2W8e/54HWCXb5DvZIwE5IMfkLzklXcLIPXkkT+S58s/b8ra9nZnVq8x7vpGF8nb/A6u8vqE=</latexit>

Diffusion (Heat) Equation

Change in time Curvature in space (Laplace operator)

+ + +

Enter Fourier

Jean-Baptiste Joseph Fourier“Théorie analytique de la chaleur” (1822)



Plane waves decomposition of initial distribution <latexit sha1_base64="DRS0vGZj/o/I4eAAq7QnSMp3d8Q=">AAACTXicbZBPSwJBGMZntdLsn9axy5AUCiG7EtRFkLp0NMg/oIvMjqMOzs4sM7OhLX6PPk3XunXug3SLaFz3kNoLAw/P73155328gFGlbfvTSqW3tncy2d3c3v7B4VG+cNxSIpSYNLFgQnY8pAijnDQ11Yx0AkmQ7zHS9iZ3C95+IlJRwR/1LCCuj0acDilG2lj9fBX3J6Xppa5BuwwvalBRXppMy7DXyyUE1mwYIyxUjPr5ol2x44KbwklEESTV6BesdG8gcOgTrjFDSnUdO9BuhKSmmJF5rhcqEiA8QSPSNZIjnyg3io+bw3PjDOBQSPO4hrH7dyJCvlIz3zOdPtJjtc4W5n+sG+rhjRtRHoSacLxcNAwZ1AIukoIDKgnWbGYEwpKav0I8RhJhbfJc2RIgZuLkYvUShd1osTVAz4bkTGzOekibolWtOHbFebgq1m+TALPgFJyBEnDANaiDe9AATYDBC3gFb+Dd+rC+rG/rZ9maspKZE7BSqcwvpvautA==</latexit>

If diffusion is simple for a single component…



Single mode has an easy time dynamic (damped)

<latexit sha1_base64="+I2pqxC5oG+oeWXEwih+bNgEZcc=">AAACOnicbVBNTwIxFOyKH4hfoEcPNhINJEp2iYleTIh68KiJKAkspFsKNNvtbtq3Btxw9Nd41Zt/xKs349UfYEEOgk7SZDLzXqZvvEhwDbb9Zs2l5hcWl9LLmZXVtfWNbG7zVoexoqxKQxGqmkc0E1yyKnAQrBYpRgJPsDvPPx/5d/dMaR7KGxhEzA1IV/IOpwSM1Mru0JZf6B9AEe+fYs1lwe8XMWsmhxfYh2Z52Mrm7ZI9Bv5LnAnJowmuWjkr1WiHNA6YBCqI1nXHjsBNiAJOBRtmGrFmEaE+6bK6oZIETLvJ+JIh3jNKG3dCZZ4EPFZ/byQk0HoQeGYyINDTs95I/M+rx9A5cRMuoxiYpD9BnVhgCPGoFtzmilEQA0MIVdz8FdMeUYSCKW8qJSLCdCfD6Us0dZNRakQejJMxtTmzJf0lt+WSY5ec66N85WxSYBpto11UQA46RhV0ia5QFVH0iJ7QM3qxXq1368P6/BmdsyY7W2gK1tc3HRWq4g==</latexit>

+ + +

….then it can be solved for a several components



So the full time dynamics can be found via superposition!

The Fourier-Transform

Decomposition of an n-dimensional signal (audio, image, volume…) into its plane waves components

Also called Fourier spectrum

The (Discrete) Fourier-Transform


Fourier domain (Frequency domain)

<latexit sha1_base64="8W9N2AaIE41+0eY/nwU98jMz5WA=">AAACHHicbVDLSgMxFE2sj1pfrS7dBIvgqsyIoMuiG5cV7QPaoWTSTBuaSUKSEevQT3CrO7/GnbgV/Bsz7Sxs64ULh3Pu5dx7QsWZsZ73A9cK6xubW8Xt0s7u3v5BuXLYMjLRhDaJ5FJ3QmwoZ4I2LbOcdpSmOA45bYfjm0xvP1JtmBQPdqJoEOOhYBEj2Drq/qkv+uWqV/NmhVaBn4MqyKvRr8BCbyBJElNhCcfGdH1P2SDF2jLC6bTUSwxVmIzxkHYdFDimJkhnt07RqWMGKJLatbBoxv7dSHFszCQO3WSM7cgsaxn5n9ZNbHQVpEyoxFJB5kZRwpGVKHscDZimxPKJA5ho5m5FZIQ1JtbFs+CiMHfpCLn4iSFBmrkq/OyUkovNXw5pFbTOa75X8+8uqvXrPMAiOAYn4Az44BLUwS1ogCYgYAhewCt4g+/wA37Cr/noGsx3jsBCwe9fy5+hTA==</latexit>

Image domain (Spatial domain)

Fourier Transform<latexit sha1_base64="u8d+vOLjfw2toTEdqT3soCMq9C8=">AAACWnicbZDLSgMxFIYz473e6mXnJlgEXbTMiKAbURTElSjYC3TGIZNmbGgmE5KMWMM8jk/jVheCD2NaZ2GrBwI//38OJ+eLBaNKe96n487Mzs0vLC5VlldW19arG5stleUSkybOWCY7MVKEUU6ammpGOkISlMaMtOPB5ShvPxGpaMbv9VCQMEWPnCYUI22tqHrWiQbwFF5d3e8/R/zAykDlaWT4qVc8mJu6X0DrQ/Jg6kEiETaHgaCQFuamgAPIi6ha8xreuOBf4ZeiBsq6jTacmaCX4TwlXGOGlOr6ntChQVJTzEhRCXJFBMID9Ei6VnKUEhWa8aUF3LNODyaZtI9rOHZ/TxiUKjVMY9uZIt1X09nI/C/r5jo5CQ3lIteE459FSc6gzuAIG+xRSbBmQysQltT+FeI+skC0hTuxRSBm2fJs8hKFQzPaKtCLTSoWmz8N6a9oHTZ8r+HfHdXOL0qAi2AH7IJ94INjcA6uwS1oAgxewRt4Bx/Ol+u6S+7yT6vrlDNbYKLc7W8VqbO3</latexit>

<latexit sha1_base64="+VRqe9Sza5vD/c1c4dYpeJQrMGA=">AAACWHicbZDNSgMxFIVvx//6V3XpJlgEXVhmRNBNQRTElShYLXTGIZNmNEwmE5KMWMM8jU/jVlf6NKa1C6teCBzOuZeb+yWSM218/6PmTU3PzM7NL9QXl5ZXVhtr6ze6KBWhHVLwQnUTrClngnYMM5x2paI4Tzi9TbLTYX77SJVmhbg2A0mjHN8LljKCjbPiRvspFqiN2NnZ9U43znZRqMs8tlnbr+7sxV5Qoa5roHc2TBUmdj+UDLHKXlQoQ6KKG02/5Y8K/RXBWDRhXJfxWm0q7BekzKkwhGOte4EvTWSxMoxwWtXDUlOJSYbvac9JgXOqIzu6s0LbzumjtFDuCYNG7s8Ji3OtB3niOnNsHvTvbGj+l/VKkx5FlglZGirI96K05MgUaAgN9ZmixPCBE5go5v6KyAN2QIxDO7FFYu7IimLyEk0iO9wq8bNL6g5b8BvSX3Gz3wr8VnB10Dw+GQOch03Ygh0I4BCO4RwuoQMEXuAV3uC99umBN+ctfLd6tfHMBkyUt/4FQW+zXw==</latexit>

inverse Fourier Transform

<latexit sha1_base64="ztW35HABb3Rcv3KkmmNjY+pHQZE=">AAACNXicbVDLSgMxFM20Pmp9tbrTTbAIFaHMiKAboejGZQX7gHYsd9K0Dc0kQ5IR6lDwa9zqzm9x4U7c+gumj4VtPRA4nHMvJ/cEEWfauO6Hk0qvrK6tZzaym1vbO7u5/F5Ny1gRWiWSS9UIQFPOBK0aZjhtRIpCGHBaDwY3Y7/+SJVmUtybYUT9EHqCdRkBY6V27oA+JAzDCF9hInURTk4Z1kxYgtu5gltyJ8DLxJuRApqh0s476VZHkjikwhAOWjc9NzJ+Asowwuko24o1jYAMoEeblgoIqfaTyREjfGyVDu5KZZ8weKL+3Ugg1HoYBnYyBNPXi95Y/M9rxqZ76SdMRLGhgkyDujHHRuJxI7jDFCWGDy0Bopj9KyZ9UECM7W0uJQJuaxNy/hJN/GScGsGTdbK2Nm+xpGVSOyt5bsm7Oy+Ur2cFZtAhOkJF5KELVEa3qIKqiKBn9IJe0Zvz7nw6X873dDTlzHb20Rycn19nqKkH</latexit>

k is called wave number or wave vector (inverse of the wave length)

<latexit sha1_base64="/bLyYkVkKl2OBWXV5zYUu23op+U=">AAACJ3icbVDLSgMxFM1YH7W+Wl26CRahbsqMCLoRikJxWaEvaMchk2ba0EwmJBmxDv0Pt7rza9yJLv0TM+0sbOuBwOGcezk3xxeMKm3b39Zabn1jcyu/XdjZ3ds/KJYO2yqKJSYtHLFIdn2kCKOctDTVjHSFJCj0Gen449vU7zwSqWjEm3oiiBuiIacBxUgb6aHrjSG8rteblSePn3nFsl21Z4CrxMlIGWRoeCUr1x9EOA4J15ghpXqOLbSbIKkpZmRa6MeKCITHaEh6hnIUEuUms7On8NQoAxhE0jyu4Uz9u5GgUKlJ6JvJEOmRWvZS8T+vF+vgyk0oF7EmHM+DgphBHcG0AzigkmDNJoYgLKm5FeIRkghr09RCikDMFMWjxZ8o7CZpqkDPximY2pzlklZJ+7zq2FXn/qJcu8kKzINjcAIqwAGXoAbuQAO0AAYSvIBX8Ga9Wx/Wp/U1H12zsp0jsADr5xdNWKSK</latexit>

The Fourier-Transform


Works not only in 1D….

The Fourier-TransformWhy plane waves?


Wave Equation General solution to wave equations in free space!

• Acoustics • Optics • Quantum mechanics • ….

Anatomy of the Fourier Domain

Input Fourier Spectrum (“k-space”)

<latexit sha1_base64="dbonh5/On+u8/UCMALPBzsBgSKA=">AAACHHicdVDLSgMxFM20Pmp9tbp0EyyCq5IUse2u6MZlRfuAdiiZNNOGZjJDkhHr0E9wqzu/xp24FfwbM20FK3rhwuGcezn3Hi8SXBuEPp1Mdm19YzO3ld/e2d3bLxQP2jqMFWUtGopQdT2imeCStQw3gnUjxUjgCdbxJpep3rljSvNQ3pppxNyAjCT3OSXGUjeTwf2gUEJlhBDGGKYAV8+RBfV6rYJrEKeSrRJYVnNQdLL9YUjjgElDBdG6h1Fk3IQow6lgs3w/1iwidEJGrGehJAHTbjK/dQZPLDOEfqhsSwPn7M+NhARaTwPPTgbEjPVvLSX/0nqx8WtuwmUUGybpwsiPBTQhTB+HQ64YNWJqAaGK21shHRNFqLHxrLhERNh0ZLj6iaZukrpG5MEqeRvbdzbwf9CulDEq4+uzUuNiGWAOHIFjcAowqIIGuAJN0AIUjMAjeALPzovz6rw574vRjLPcOQQr5Xx8AR30oX8=</latexit>

<latexit sha1_base64="4JM1R0lwz67WHkOW5GYwayUHG6k=">AAACHHicdVBNSwMxEM22ftT61erRS7AInkpSxLa3ohePFW0ttEvJptk2NJtdkqywLv0JXvXmr/EmXgX/jdm2ghUdGHi8N8ObeV4kuDYIfTq5/Nr6xmZhq7i9s7u3XyofdHUYK8o6NBSh6nlEM8El6xhuBOtFipHAE+zOm15m+t09U5qH8tYkEXMDMpbc55QYS91Mh8mwVEFVhBDGGGYA18+RBc1mo4YbEGeSrQpYVntYdvKDUUjjgElDBdG6j1Fk3JQow6lgs+Ig1iwidErGrG+hJAHTbjq/dQZPLDOCfqhsSwPn7M+NlARaJ4FnJwNiJvq3lpF/af3Y+A035TKKDZN0YeTHApoQZo/DEVeMGpFYQKji9lZIJ0QRamw8Ky4RETYdGa5+oqmbZq4RebBK0cb2nQ38H3RrVYyq+Pqs0rpYBlgAR+AYnAIM6qAFrkAbdAAFY/AInsCz8+K8Om/O+2I05yx3DsFKOR9fH6qhgA==</latexit>

FFT

wave vector <latexit sha1_base64="Uo87QHikAD5bg57DfcVQZfmMJsQ=">AAACLnicbVDLSsNAFJ20Pmp9tXbpZrAIFaQkIuhGKLpxWcE+oA1hMp20QyaTMDMpxtBvcas7v0ZwIW79DCdtFrb1wMDhnHs5d44bMSqVaX4aheLG5tZ2aae8u7d/cFipHnVlGAtMOjhkoei7SBJGOekoqhjpR4KgwGWk5/p3md+bEiFpyB9VEhE7QGNOPYqR0pJTqQ2nBKf+DN7Ahu88nftOcuZU6mbTnAOuEysndZCj7VSN4nAU4jggXGGGpBxYZqTsFAlFMSOz8jCWJELYR2My0JSjgEg7nV8/g6daGUEvFPpxBefq340UBVImgasnA6QmctXLxP+8Qay8azulPIoV4XgR5MUMqhBmVcARFQQrlmiCsKD6VognSCCsdGFLKRFiui8eLv9EYjvNUiP0rJ2yrs1aLWmddC+altm0Hi7rrdu8wBI4BiegASxwBVrgHrRBB2CQgBfwCt6Md+PD+DK+F6MFI9+pgSUYP7/FJqdU</latexit>

• direction = direction of plane wave component • magnitude = inverse of wavelength

(small -> slowly varying, large -> fast varying)





Slowly varying structures (small magnitude of k)





Fast varying structures (large magnitude of k)





Oriented features (k orthogonal to orientation )





magnitude of k





orientation of k

Quiz

Which one is the Fourier transform?

A B C

Historic aside: X-Ray Crystallography

Photo 51 Franklin, Gosling “Molecular Configuration in Sodium Thymonucleate" Nature (1953)

X Ray Diffraction pattern (~Fourier Transform)DNA

Historic aside: X-Ray Crystallography

Photo 51 Franklin, Gosling “Molecular Configuration in Sodium Thymonucleate" Nature (1953)

DNA X Ray Diffraction pattern (~Fourier Transform)

Fourier Transform and Algorithmic Complexity

Cooley, Tukey "An algorithm for the machine calculation of complex Fourier series" (1965)

Complexity of an algorithm:

How does the numerical cost (operations) increases with increasing input (e.g. number of pixels N)?

• Linear complexity

• Quadratic complexity

• Exponential Complexity

• Log-Linear complexity

<latexit sha1_base64="mi5LVt1nt1WlL9KGE5lHPqpsSoA=">AAACM3icbVBNSwMxEM1aP2r9avUieAkWoV7Krgh6LHrxpBXsB7RLmU3TNjSbLElWqEv9NV715o8Rb+LV/2C27cG2Dgw83pvhzbwg4kwb1/1wVjKra+sb2c3c1vbO7l6+sF/XMlaE1ojkUjUD0JQzQWuGGU6bkaIQBpw2guF1qjceqdJMigcziqgfQl+wHiNgLNXJHxKpDW5rFuJ2CGZAgCd349LtaSdfdMvupPAy8GagiGZV7RScTLsrSRxSYQgHrVueGxk/AWUY4XSca8eaRkCG0KctCwWEVPvJ5IUxPrFMF/eksi0MnrB/NxIItR6FgZ1Mz9SLWkr+p7Vi07v0Eyai2FBBpka9mGMjcZoH7jJFieEjC4AoZm/FZAAKiLGpzblEwG1oQs5/oomfpK4RPFklZ2PzFkNaBvWzsueWvfvzYuVqFmAWHaFjVEIeukAVdIOqqIYIekYv6BW9Oe/Op/PlfE9HV5zZzgGaK+fnF4FbqcQ=</latexit>

<latexit sha1_base64="6sZfSp6FzgKk3PqmDx7POiPbAdQ=">AAACNXicbVDLSgNBEJyN7/hK9KaXwSDES9gVQY9BL558gHlAdgm9k0kyZHZmmZkV4hLwa7zqzW/x4E28+gvOJjmYxIaGoqqb6q4w5kwb1/1wckvLK6tr6xv5za3tnd1Cca+uZaIIrRHJpWqGoClngtYMM5w2Y0UhCjlthIOrTG88UqWZFA9mGNMggp5gXUbAWKpdOPA1i7AfgekT4OntqHyDfS57+OakXSi5FXdceBF4U1BC07prF50lvyNJElFhCAetW54bmyAFZRjhdJT3E01jIAPo0ZaFAiKqg3T8xAgfW6aDu1LZFgaP2b8bKURaD6PQTmbX6nktI//TWonpXgQpE3FiqCATo27CsZE4SwR3mKLE8KEFQBSzt2LSBwXE2NxmXGLgNjYhZz/RJEgz1xierJK3sXnzIS2C+mnFcyve/VmpejkNcB0doiNURh46R1V0je5QDRH0jF7QK3pz3p1P58v5nozmnOnOPpop5+cXapGqKw==</latexit>

<latexit sha1_base64="ICgQmELtP+CpGIuaF/ZnAnly5oc=">AAACMHicbVDLTgIxFO2AD8QXaFy5aSQmuCEzxESXRDeuFBN5JDCSO6VAQ6czaTsmOOFj3OrOr9GVcetX2IFZCHiTJifn3JtzeryQM6Vt+9PKZNfWNzZzW/ntnd29/ULxoKmCSBLaIAEPZNsDRTkTtKGZ5rQdSgq+x2nLG18neuuJSsUC8aAnIXV9GAo2YAS0oXqFo65iPu76oEcEeHw3Ld8+Vs96hZJdsWeDV4GTghJKp94rWtluPyCRT4UmHJTqOHao3RikZoTTab4bKRoCGcOQdgwU4FPlxrP8U3xqmD4eBNI8ofGM/XsRg6/UxPfMZhJULWsJ+Z/WifTg0o2ZCCNNBZkbDSKOdYCTMnCfSUo0nxgARDKTFZMRSCDaVLbgEgI3jYlg8SeKuHHiGsKzUfKmNme5pFXQrFYcu+Lcn5dqV2mBOXSMTlAZOegC1dANqqMGIihGL+gVvVnv1of1ZX3PVzNWenOIFsb6+QXFO6hd</latexit>

<latexit sha1_base64="m4ta9s3DkA8TRNpSea66qg/EHak=">AAACMHicbVDLTgIxFO2AD8QXaFy5aSQmuCEzxESXRDeuFBN5JDCSO6VAQ6czaTsmOOFj3OrOr9GVcetX2IFZCHiTJifn3JtzeryQM6Vt+9PKZNfWNzZzW/ntnd29/ULxoKmCSBLaIAEPZNsDRTkTtKGZ5rQdSgq+x2nLG18neuuJSsUC8aAnIXV9GAo2YAS0oXqFo65iPu76oEcEeHw3LVcfb896hZJdsWeDV4GTghJKp94rWtluPyCRT4UmHJTqOHao3RikZoTTab4bKRoCGcOQdgwU4FPlxrP8U3xqmD4eBNI8ofGM/XsRg6/UxPfMZhJULWsJ+Z/WifTg0o2ZCCNNBZkbDSKOdYCTMnCfSUo0nxgARDKTFZMRSCDaVLbgEgI3jYlg8SeKuHHiGsKzUfKmNme5pFXQrFYcu+Lcn5dqV2mBOXSMTlAZOegC1dANqqMGIihGL+gVvVnv1of1ZX3PVzNWenOIFsb6+QXFA6hd</latexit>

Fast Fourier Transform (FFT)

Naive Fourier Transform

Linear Log-Linear Quadratic

What is the complexity of multiplying every pixel of an image with a constant number?

<latexit sha1_base64="u8d+vOLjfw2toTEdqT3soCMq9C8=">AAACWnicbZDLSgMxFIYz473e6mXnJlgEXbTMiKAbURTElSjYC3TGIZNmbGgmE5KMWMM8jk/jVheCD2NaZ2GrBwI//38OJ+eLBaNKe96n487Mzs0vLC5VlldW19arG5stleUSkybOWCY7MVKEUU6ammpGOkISlMaMtOPB5ShvPxGpaMbv9VCQMEWPnCYUI22tqHrWiQbwFF5d3e8/R/zAykDlaWT4qVc8mJu6X0DrQ/Jg6kEiETaHgaCQFuamgAPIi6ha8xreuOBf4ZeiBsq6jTacmaCX4TwlXGOGlOr6ntChQVJTzEhRCXJFBMID9Ei6VnKUEhWa8aUF3LNODyaZtI9rOHZ/TxiUKjVMY9uZIt1X09nI/C/r5jo5CQ3lIteE459FSc6gzuAIG+xRSbBmQysQltT+FeI+skC0hTuxRSBm2fJs8hKFQzPaKtCLTSoWmz8N6a9oHTZ8r+HfHdXOL0qAi2AH7IJ94INjcA6uwS1oAgxewRt4Bx/Ol+u6S+7yT6vrlDNbYKLc7W8VqbO3</latexit>

Image Filtering in the FFT Domain

Input Fourier Spectrum OutputFFT iFFT


Low-Pass Filter




High-Pass Filter



Band-Pass Filter



Directional Filter

In practice

Python ImageJ

import numpy as np

# fft of imgimg_f = np.fft.fftn(img)

# inverse fft of img_fimg = np.fft.ifftn(img_f)

run("FFT");

run("Inverse FFT”);

Convolutions and Filters

Convolution - Recap

<latexit sha1_base64="Tukaz1qrn9XhXGvjiZW+N/Q4kvo=">AAACGnicbVDLSgMxFE2sj1pfrS7dBIsgLsqMCLosunHZgn1AO5RMmmlDM0lIMkId+gVudefXuBO3bvwbM+0sbOuFC4dz7uXce0LFmbGe9wM3Cptb2zvF3dLe/sHhUbly3DYy0YS2iORSd0NsKGeCtiyznHaVpjgOOe2Ek/tM7zxRbZgUj3aqaBDjkWARI9g6qnk5KFe9mjcvtA78HFRBXo1BBRb6Q0mSmApLODam53vKBinWlhFOZ6V+YqjCZIJHtOegwDE1QTq/dIbOHTNEkdSuhUVz9u9GimNjpnHoJmNsx2ZVy8j/tF5io9sgZUIllgqyMIoSjqxE2dtoyDQllk8dwEQzdysiY6wxsS6cJReFuctGyOVPDAnSzFXhZ6eUXGz+akjroH1V872a37yu1u/yAIvgFJyBC+CDG1AHD6ABWoAACl7AK3iD7/ADfsKvxegGzHdOwFLB71+b0qAd</latexit>

Input f h

<latexit sha1_base64="3m7J4vrhzaOzkZmevxjLJqbp68U=">AAACGnicbVDLSgMxFE2sj1pfrS7dBIvgqsyIoBuh6MZlC/YB7VAyaaYNzSQhyQh16Be41Z1f407cuvFvzLSzsK0XLhzOuZdz7wkVZ8Z63g/cKGxube8Ud0t7+weHR+XKcdvIRBPaIpJL3Q2xoZwJ2rLMctpVmuI45LQTTu4zvfNEtWFSPNqpokGMR4JFjGDrqObtoFz1at680Drwc1AFeTUGFVjoDyVJYios4diYnu8pG6RYW0Y4nZX6iaEKkwke0Z6DAsfUBOn80hk6d8wQRVK7FhbN2b8bKY6Nmcahm4yxHZtVLSP/03qJjW6ClAmVWCrIwihKOLISZW+jIdOUWD51ABPN3K2IjLHGxLpwllwU5i4bIZc/MSRIM1eFn51ScrH5qyGtg/ZlzfdqfvOqWr/LAyyCU3AGLoAPrkEdPIAGaAECKHgBr+ANvsMP+Am/FqMbMN85AUsFv38BvFSgMA==</latexit>

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Result g

Transposer? Kernel Folder?

How is the green object h called?

Convolution is an operation on an image where the same linear operation is applied to every pixel in the same sliding widow way

Convolution


Input f Kernel h

80 85 89 70 55 …

70 70 85 99 65 …

87 25 57 89 54 …

90 54 90 96 87 …

66 69 76 76 65 …

… … … … … ….


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Result g

Convolution


Input f Kernel h

80 85 89 70 55 …

70 70 85 99 65 …

87 25 57 89 54 …

90 54 90 96 87 …

66 69 76 76 65 …

… … … … … ….

72


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Result g

80/9+85/9+89/9+ 70/9+70/9+85/9+ 87/9+25/9+57/9

Convolution


Input f Kernel h

80 85 89 70 55 …

70 70 85 99 65 …

87 25 57 89 54 …

90 54 90 96 87 …

66 69 76 76 65 …

… … … … … ….

72 74


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Result g

85/9+89/9+70/9+ 70/9+85/9+99/9+ 25/9+57/9+89/9+

Mean Filter (Boxfilter, uniform filter)


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Input f Kernel h Result g


Gaussian Filter




0.01 0.06 0.1 0.06 0.01

0.06 0.36 0.6 0.36 0.06

0.1 0.6 1 0.6 0.1

0.06 0.36 0.6 0.36 0.06

0.01 0.06 0.1 0.06 0.01

Edge Filter





-1 -1 -1

-1 8 -1

-1 -1 -1

Median Filter Sobolev Filter Laplace Filter

How is that Filter called (and why)?

Convolution with single pixel image


Input f Kernel h

0 0 0 0 0 …

0 0 0 0 0 …

0 0 1 0 0 …

0 0 0 0 0 …

0 0 0 0 0 …

… … … … … ….

0 0 0 0 0

0 1/9 1/9 1/9 0

0 1/9 1/9 1/9 0

0 1/9 1/9 1/9 0

0 0 0 0 0


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Result g

kernel = impulse response (Point spread function)

Separable Convolutions

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Kernel h

1/3 1/3 1/3

1/3

1/3

1/3

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

“Separable”

hx

hy

f <latexit sha1_base64="Tukaz1qrn9XhXGvjiZW+N/Q4kvo=">AAACGnicbVDLSgMxFE2sj1pfrS7dBIsgLsqMCLosunHZgn1AO5RMmmlDM0lIMkId+gVudefXuBO3bvwbM+0sbOuFC4dz7uXce0LFmbGe9wM3Cptb2zvF3dLe/sHhUbly3DYy0YS2iORSd0NsKGeCtiyznHaVpjgOOe2Ek/tM7zxRbZgUj3aqaBDjkWARI9g6qnk5KFe9mjcvtA78HFRBXo1BBRb6Q0mSmApLODam53vKBinWlhFOZ6V+YqjCZIJHtOegwDE1QTq/dIbOHTNEkdSuhUVz9u9GimNjpnHoJmNsx2ZVy8j/tF5io9sgZUIllgqyMIoSjqxE2dtoyDQllk8dwEQzdysiY6wxsS6cJReFuctGyOVPDAnSzFXhZ6eUXGz+akjroH1V872a37yu1u/yAIvgFJyBC+CDG1AHD6ABWoAACl7AK3iD7/ADfsKvxegGzHdOwFLB71+b0qAd</latexit> h = (f hx) hy<latexit sha1_base64="Tukaz1qrn9XhXGvjiZW+N/Q4kvo=">AAACGnicbVDLSgMxFE2sj1pfrS7dBIsgLsqMCLosunHZgn1AO5RMmmlDM0lIMkId+gVudefXuBO3bvwbM+0sbOuFC4dz7uXce0LFmbGe9wM3Cptb2zvF3dLe/sHhUbly3DYy0YS2iORSd0NsKGeCtiyznHaVpjgOOe2Ek/tM7zxRbZgUj3aqaBDjkWARI9g6qnk5KFe9mjcvtA78HFRBXo1BBRb6Q0mSmApLODam53vKBinWlhFOZ6V+YqjCZIJHtOegwDE1QTq/dIbOHTNEkdSuhUVz9u9GimNjpnHoJmNsx2ZVy8j/tF5io9sgZUIllgqyMIoSjqxE2dtoyDQllk8dwEQzdysiY6wxsS6cJReFuctGyOVPDAnSzFXhZ6eUXGz+akjroH1V872a37yu1u/yAIvgFJyBC+CDG1AHD6ABWoAACl7AK3iD7/ADfsKvxegGzHdOwFLB71+b0qAd</latexit>


A (2D) Filter is called separable, if the rows of the kernel are multiples of each other

The full 2D kernel h can then be composed as an outer product of two 1D kernels hx and hy


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Kernel h

1/3 1/3 1/3

1/3

1/3

1/3

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

“Separable”

hx

hy



0.01 0.06 0.1 0.06 0.01

0.06 0.36 0.6 0.36 0.06

0.1 0.6 1 0.6 0.1

0.06 0.36 0.6 0.36 0.06

0.01 0.06 0.1 0.06 0.01

Yes No

Is the Gaussian Filter separable?

Gaussian Filter


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Kernel h

Examples of separable filters: • Gaussian Filter (Blur) • Mean Filter

1/3 1/3 1/3

1/3

1/3

1/3

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

“Separable”

hx

hy



Not separable filters: • Laplace Filter • Median Filter….

Convolutions in the Fourier Domain


1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9



Fourier Convolution Theorem:

Efficient way to compute convolutions (esp for large kernels)

Convolution in image domain is multiplication in fourier domain!

<latexit sha1_base64="LbrL/KOdtKZMygY+F31MCbfRhUI=">AAACN3icdVDLSgMxFM34tr5aXeoiWAS7KUkR2y4EURCXCq0K7VAyaaYNzSRDkhHq4Mavcas7P8WVO3HrH5hpK1jRC4Fzz7mXk3uCWHBjEXr1Zmbn5hcWl5ZzK6tr6xv5wuaVUYmmrEmVUPomIIYJLlnTcivYTawZiQLBroPBaaZf3zJtuJINO4yZH5Ge5CGnxDqqk985O2vs90rwCGYgLME27So7avqlTr6IygghjDHMAK4eIgfq9VoF1yDOJFdFMKmLTsGbbXcVTSImLRXEmBZGsfVToi2ngt3n2olhMaED0mMtByWJmPHT0Rn3cM8xXRgq7Z60cMT+3EhJZMwwCtxkRGzf/NYy8i+tldiw5qdcxollko6NwkRAq2CWCexyzagVQwcI1dz9FdI+0YRal9yUS0yEC06q6UsM9dPMNSZ3Tsm52L6zgf+Dq0oZozK+PCgen0wCXALbYBfsAwyq4BicgwvQBBQ8gEfwBJ69F+/Ne/c+xqMz3mRnC0yV9/kFYfKo8w==</latexit>

Filter Implementations

Python

from scipy.signal import fftconvolve

response = fftconvolve(img, kernel)

Fiji

from scipy.ndimage.filters import gaussian_filter, uniform_filter

response = gaussian_filter(img,sigma = 3)

Aside: Image processing on the GPU

Convolutions can be implemented very fast on Graphical Processing Units (GPUs)!

https://clij.github.io/https://github.com/maweigert/gputools/by Robert Haase

Convolutions/Correlations for template matching

Hubel & Wiesel 1959

(Convolutional) Filter

Convolutions for template matching (Correlations)

Strong positive response

Strong negative response

No / weak response

Kernel (Template)

Convolutions for template matching

Strong positive response

Strong negative response

No / weak response

Convolutional Filter

Sneak Peak: Convolutional neural networks

1. How many filters shall we use?2. Which filter kernel shall we use?

1. A LOT!2. Learn them!

Convolutional Neural Networks (CNN)

Deconvolution

Blur in Images

• Kernel (Point Spread Function, PSF) of microscope • Motion blur if camera moves (photography)

Blur in ImagesInput f Kernel h Result g


Data: Corinna Blasse, Natalie Dye (MPI-CBG)

Deconvolution: Recover the input from the blurry result (given the kernel)

Methods: Linear Inverse Filtering (Wiener)

Inverse Filtering (Wiener Deconvolution)



<latexit sha1_base64="tO+03WXbAg1S7Ouyw5hzg4yRgsw=">AAACOHicbVDLSgMxFM20Pmp9tboUIViEdlNmRNCNUBSKywp9QTuUTJppQzPJkGSEOnTl17jVnX/izp249QvMtLOwrRcC555zLyf3eCGjStv2h5XJbmxubed28rt7+weHheJRW4lIYtLCggnZ9ZAijHLS0lQz0g0lQYHHSMeb3CV655FIRQVv6mlI3ACNOPUpRtpQg8Jpvd4sjyrwBibAr8A+Hgo9b8YVOCiU7Ko9L7gOnBSUQFqNQdHK9ocCRwHhGjOkVM+xQ+3GSGqKGZnl+5EiIcITNCI9AzkKiHLj+R0zeG6YIfSFNI9rOGf/bsQoUGoaeGYyQHqsVrWE/E/rRdq/dmPKw0gTjhdGfsSgFjAJBQ6pJFizqQEIS2r+CvEYSYS1iW7JJUTMJMfF8iUKu3HiGqIno+RNbM5qSOugfVF17KrzcFmq3aYB5sAJOANl4IArUAP3oAFaAINn8AJewZv1bn1aX9b3YjRjpTvHYKmsn19uJqjn</latexit>

<latexit sha1_base64="u3RWhbhvpt9qawQAR+SUXOHfheA=">AAACPXicbZBNS8MwGMdT58ucb5sevYQNYbuMVgQ9DoXhccLeYC0jzdItLE1Lkgqz9O6n8ao3P4cfwJt49Wra9eA2Hwj58//n4cnzc0NGpTLND2OrsL2zu1fcLx0cHh2flCunfRlEApMeDlgghi6ShFFOeooqRoahIMh3GRm487s0HzwSIWnAu2oREsdHU049ipHS1rhc9aAtqQ9pu92t255AOE7VtJFk96yRNMblmtk0s4KbwspFDeTVGVeMgj0JcOQTrjBDUo4sM1ROjISimJGkZEeShAjP0ZSMtOTIJ9KJs2USeKGdCfQCoQ9XMHP/dsTIl3Lhu/qlj9RMrmep+V82ipR348SUh5EiHC8HeRGDKoApGTihgmDFFlogLKj+K8QzpIkozW9lSoiYxseD1U0kduJ0aoiedFLS2Kx1SJuif9m0zKb1cFVr3eYAi+AcVEEdWOAatMA96IAewOAZvIBX8Ga8G5/Gl/G9fLpl5D1nYKWMn18pRqxs</latexit>

idea:

Since FFT(h) typically has zeros:

<latexit sha1_base64="3WYd56dgPXBoZpjS+g93AJlREZ4=">AAACWHicbZDNSgMxFIVvx//6V3XpJliEVqHMFEE3giiISwWrQmcsd9JMG5rJDElGqGOfxqdxqyt9GjNtF1a9EPg45yY394Sp4Nq47mfJmZtfWFxaXimvrq1vbFa2tu90kinKWjQRiXoIUTPBJWsZbgR7SBXDOBTsPhxcFP79E1OaJ/LWDFMWxNiTPOIUjZU6ldOI+JrHhF9e3tb8SCHNC+rXHw9IAb36KH+ZKC+PTXJIfGEf7yKpDeqjeqdSdRvuuMhf8KZQhWldd7ZKc343oVnMpKECtW57bmqCHJXhVLBR2c80S5EOsMfaFiXGTAf5eM8R2bdKl0SJskcaMlZ/3sgx1noYh7YzRtPXv71C/M9rZyY6CXIu08wwSSeDokwQk5AiNNLlilEjhhaQKm7/SmgfbVjGRjszJUVhk5XJ7CaaBnkxNcVn65RtbN7vkP7CXbPhuQ3v5qh6dj4NcBl2YQ9q4MExnMEVXEMLKLzCG7zDR+nLAWfJWZm0OqXpnR2YKWf7G5YKsUs=</latexit>

<latexit sha1_base64="rRCMjCCUNdDsDQWqU/Cp+6JLgwU=">AAACPXicbZBNS8MwGMdT58ucb5sevYQNYbuMVgQ9DoXhccLeYC0jzdItLE1Lkgqz9O6n8ao3P4cfwJt49Wra9eA2Hwj58//n4cnzc0NGpTLND2OrsL2zu1fcLx0cHh2flCunfRlEApMeDlgghi6ShFFOeooqRoahIMh3GRm487s0HzwSIWnAu2oREsdHU049ipHS1rhcbbe7da8BbUl9aHsC4Th1po0ku2eNBI7LNbNpZgU3hZWLGsirM64YBXsS4MgnXGGGpBxZZqicGAlFMSNJyY4kCRGeoykZacmRT6QTZ8sk8EI7E+gFQh+uYOb+7YiRL+XCd/VLH6mZXM9S879sFCnvxokpDyNFOF4O8iIGVQBTMnBCBcGKLbRAWFD9V4hnSBNRmt/KlBAxjY8Hq5tI7MTp1BA96aSksVnrkDZF/7JpmU3r4arWus0BFsE5qII6sMA1aIF70AE9gMEzeAGv4M14Nz6NL+N7+XTLyHvOwEoZP7+jUqwj</latexit>

Good: • Fast

Bad: • Bad performance if images contain noise (always the case)

• Ringing artifacts

Methods: Richardson-Lucy DeconvolutionInput f Kernel h Result g


Richardson-Lucy (RL) Deconvolution

Good: • Incorporates correct noise mode • Non negative output

Bad: • Using large number of iterations will amplify noise

<latexit sha1_base64="K1urc1l9cB/5pPRZVQfeINJzbKg=">AAACSHicbZDLSgMxFIYzrdd6a3XpJlgEL1hmRNCNILpxqWBboR2HTJppQzPJkJwR6jBP4dO41Z1v4Fu4E3dmahdWPZDw8/85nJwvTAQ34LpvTqk8Mzs3v7BYWVpeWV2r1tZbRqWasiZVQunbkBgmuGRN4CDYbaIZiUPB2uHwosjb90wbruQNjBLmx6QvecQpAWsF1YMoyOS+l+NTHAUSd2lPAd7pRprQrJ9nhbeHB3lx3cFuUK27DXdc+K/wJqKOJnUV1Jxyt6doGjMJVBBjOp6bgJ8RDZwKlle6qWEJoUPSZx0rJYmZ8bPxXjnetk4PR0rbIwGP3Z8dGYmNGcWhfRkTGJjfWWH+l3VSiE78jMskBSbp96AoFRgULiDhHteMghhZQajm9q+YDohlAhbl1JSECEtSqulNDPWzYmpCHmxSsdi835D+itZhw3Mb3vVR/ex8AnABbaIttIM8dIzO0CW6Qk1E0SN6Qs/oxXl13p0P5/P7acmZ9GygqSqVvgDPNq+j</latexit>

Iterative algorithm that finds best estimate under a realistic Poisson noise model

-> estimates f_n should become better and better (in theory)Lucy "An iterative technique for the rectification of observed distributions." (1974) Richardson "Bayesian-based iterative method of image restoration.” (1972)

Deconvolution Tools

from skimage.restoration import unsupervised_wiener, richardson_lucy

Python

ImageJCommercial: • Huygens • Autoquant

• DeconvolutionLab • ImageJ-ops

• scikit-image • flowdec (tensorflow)

Exercises

Exercise 1

Exercise 2

Exercise 2b*

Apply filters and remove stripe artefacts from an image

Investigate several classic deconvolution methods on a test image

Implement Richardson-Lucy deconvolution (optional)

https://git.mpi-cbg.de/rhaase/lecture_applied_bioimage_analysis

10_example_code_and_data

Exercises

Exercise 1

Image A high frequency artefacts

=+

You are given Image B (“image_with_stripes.tif”)

Image B

• Apply some filters (gaussian, uniform) to B with different paramters - can you get rid of the stripes? • Look at the fourier spectrum of B and try if you can spot the source of the stripes - can you remove it?

Apply filters and remove stripe artefacts from an image

Try not to use the original A!

Exercises

Exercise 2

• Apply Wiener and Richardson Lucy deconvolution to image B (for several noise levels) • Play with the parameters of RL (iterations) - what do you observe? • What happens for images with larger noise level (e.g. “image_blurred_sigma_30.tif”) ?

Investigate several classic deconvolution methods on a test image


=

Image A Image BMotion blur

You are given the blurred Image B (“image_blurred_$s.tif”) with different noise level s

Exercises

Exercise 2b Implement Richardson-Lucy

<latexit sha1_base64="K1urc1l9cB/5pPRZVQfeINJzbKg=">AAACSHicbZDLSgMxFIYzrdd6a3XpJlgEL1hmRNCNILpxqWBboR2HTJppQzPJkJwR6jBP4dO41Z1v4Fu4E3dmahdWPZDw8/85nJwvTAQ34LpvTqk8Mzs3v7BYWVpeWV2r1tZbRqWasiZVQunbkBgmuGRN4CDYbaIZiUPB2uHwosjb90wbruQNjBLmx6QvecQpAWsF1YMoyOS+l+NTHAUSd2lPAd7pRprQrJ9nhbeHB3lx3cFuUK27DXdc+K/wJqKOJnUV1Jxyt6doGjMJVBBjOp6bgJ8RDZwKlle6qWEJoUPSZx0rJYmZ8bPxXjnetk4PR0rbIwGP3Z8dGYmNGcWhfRkTGJjfWWH+l3VSiE78jMskBSbp96AoFRgULiDhHteMghhZQajm9q+YDohlAhbl1JSECEtSqulNDPWzYmpCHmxSsdi835D+itZhw3Mb3vVR/ex8AnABbaIttIM8dIzO0CW6Qk1E0SN6Qs/oxXl13p0P5/P7acmZ9GygqSqVvgDPNq+j</latexit>

<latexit sha1_base64="K2ArnkDVtXTbon5jU8KpnNArsZ0=">AAACHHicbVDLSgMxFE2sj1pfrS7dBIvgqsyIoMuiG5cV7QPasWTSTBuaSUKSEerQT3CrO7/GnbgV/Bsz7Sxs64ULh3Pu5dx7QsWZsZ73A9cK6xubW8Xt0s7u3v5BuXLYMjLRhDaJ5FJ3QmwoZ4I2LbOcdpSmOA45bYfjm0xvP1FtmBQPdqJoEOOhYBEj2DrqfvRo++WqV/NmhVaBn4MqyKvRr8BCbyBJElNhCcfGdH1P2SDF2jLC6bTUSwxVmIzxkHYdFDimJkhnt07RqWMGKJLatbBoxv7dSHFszCQO3WSM7cgsaxn5n9ZNbHQVpEyoxFJB5kZRwpGVKHscDZimxPKJA5ho5m5FZIQ1JtbFs+CiMHfpCLn4iSFBmrkq/OyUkovNXw5pFbTOa75X8+8uqvXrPMAiOAYn4Az44BLUwS1ogCYgYAhewCt4g+/wA37Cr/noGsx3jsBCwe9fuKyhQQ==</latexit>

( is the transposed/flipped kernel)

bio-image analysis, bio-statistics and machine learning · fourier transforms, convolutions, and...

Documents