Upload File

visual dynamics: probabilistic future frame synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... ·...

  • Home
  • Documents
  • Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
26
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks Tianfan Xue* Jiajun Wu* Katie Bouman Bill Freeman NIPS 2016 VGG Reading Group, 24 Feb 2017 Ankush Gupta

Upload: hoangduong

Post on 07-Sep-2018

216 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

VisualDynamics:ProbabilisticFutureFrameSynthesisviaCrossConvolutionalNetworks

TianfanXue* JiajunWu* KatieBouman BillFreeman

NIPS2016

VGGReadingGroup,24Feb2017AnkushGupta

Page 2: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Frame2

Task:futureframeprediction

Page 3: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Frame1 Frame2Deterministicneuralnetwork

Deterministicpredictionsfailtomodeluncertainty

Page 4: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Frame1 Deterministicneuralnetwork

Deterministicpredictionsfailtomodeluncertainty

Prediction

Whatistheproblem?

Page 5: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Frame1 Deterministicneuralnetwork

Deterministicpredictionsfailtomodeluncertainty

Prediction

Whatistheproblem?

Page 6: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

SynthesisnetworkInputframe Sampledfutureframe

Sampledifferentfutureframes

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Inputrandommotionvector𝑧~𝑝$(𝑧)

Page 7: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

SynthesisnetworkInputframe

Sampledifferentfutureframes

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Inputrandommotionvector𝑧~𝑝$(𝑧)

Sampledfutureframe

Page 8: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Inputframe Anothersampledfutureframe

Segments Transformedsegments

Inputrandommotionvector𝑧~𝑝$(𝑧)

Synthesizeusingdifferenttransformations

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 9: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Sampledfutureframe

Motionvector𝑧

SynthesisnetworkInputframe

Encodingnetwork

Futureframe(groundtruth)

Training

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 10: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Motionvector𝑧

Encodingnetwork

Synthesisnetwork Futureframe

(prediction)Trainingsamples

(Label-free)

Training

Inputframe

Futureframe(groundtruth)

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 11: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Futureframe𝐼()*(prediction)

Motionvector𝑧

Encodingnetwork

Synthesisnetwork

Training

Futureframe𝐼+,(groundtruth)

Inputframe

Objectivefunction:𝐼()* − 𝐼+, + 𝐷01(𝒛||𝑁(𝟎, 𝐈))

Reconstructionloss

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 12: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Futureframe𝐼()*(prediction)

Futureframe𝐼+,(groundtruth)

Inputframe

Encodingnetwork

Synthesisnetwork

Training Objectivefunction:𝐼()* − 𝐼+, + 𝐷01(𝒛||𝑁(𝟎, 𝐈))

KL-divergenceloss

Motionvector𝑧

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Variational Autoencoder[Kingma andWelling,2014]

Page 13: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Futureframe𝐼()*(prediction)

Synthesisnetwork

Testing

Futureframe𝐼+,(groundtruth)

Encodingnetwork

Inputframe

Inputframe

Mainidea NetworkstructureOutline Whatthenetworklearns Result

u

Inputrandommotionvector𝑧~𝑝$(𝑧)

Realoutputfromournetwork

Page 14: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Inputframe Futureframe

TransformsegmentsFindsegments

Inputrandommotionvector𝑧

Synthesizebytransformingsegments

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Imagesegments Convolution

Page 15: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

0 0 0

0 1 0

0 0 0

0 0 1

0 0 0

0 0 0

Movementcanbesynthesizedthroughconvolution

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 16: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Imagesegments

Applyingmotiontoeachsegment

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Motionkernels

Thedecodingnetworkgeneratesamotionkernelforeachcorrespondingsegment

Decodingnet

Motionvector𝑧

[Brabandere etal.2016][Finnetal.2016]

Page 17: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Motionvector𝑧

Inputframe

Futureframe

Synthesisnetwork

Futureframe

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Whatisencodedinthemotionvector?

Encodingnetwork

Page 18: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Motionvector𝑧 Upwardmotionwhenchangingthisdimension

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Eachdimensionencodesatypeofmotion

Page 19: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Motionvector𝑧 Legmotionwhenchangingthisdimension

Eachdimensionencodesatypeofmotion

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 20: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

• Simulatedshapes

• Trainingsamples

Results:toyexample

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 21: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Input

Learnedsegments

Networkautomaticallydetectssegments

Triangles

Circles

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 22: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Input SamplednextframeGroundtruthdistribution

Sampledistribution

Networklearnsthecorrelationbetweenappearanceandmotion

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 23: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Input Sampledfutureframes

Results:real-worldimages

Mainidea NetworkstructureOutline Whatnetworklearns Result

Page 24: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Challenge:largemotion

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Input TwosampledfutureframesArtifactsappearwhenmotionislarge

Page 25: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Baseline:Transferflow 25.5%Ourmethod 31.3%

Labeledasreal

MechanicalTurkstudytoassesssynthesisquality

Idealsynthesisalgorithmachieves50%

Mainidea NetworkstructureOutline Whatthenetworklearns Result

Page 26: Visual Dynamics: Probabilistic Future Frame Synthesis …vgg/rg/slides/vgg_rg_23_feb_2017... · Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

• Samplemultiplefutureframesthatareconsistentwiththeinput

• Synthesizeframesbytransformingsegments

• Learnamotionrepresentationwithoutsupervision

…

Contributions


Module 4 Pronunciation & prosody - speech.zone · Today’s topics - Module 4: pronunciation & prosody Theory Application Speech Signal processing Probabilistic modelling Speech Synthesis

Atmospheric Synthesis of Superhydrophobic TiO2 Nanoparticle Deposit in Single Step Using Liquid Frame Spray

ImprovedReproductionofStopsinNoiseReductionSystemswith ......based analysis-synthesis system using overlap-add for signal synthesis, the delay is given by the frame length of the synthesis

PROBABILISTIC RELIABILITY ENGINEERING - …gnedenko-forum.org/library/Ushakov/Probabilistic Reliability...Probabilistic Reliability Engineering, Boris Gnedenko and Igor Ushakov. PROBABILISTIC

PROBABILISTIC MODELING OF BOWING GESTURES FOR GESTURE ...hedvig/publications/smac_13.pdf · PROBABILISTIC MODELING OF BOWING GESTURES FOR GESTURE-BASED VIOLIN SOUND SYNTHESIS Akshaya

SYNTHESIS FROM PROBABILISTIC COMPONENTS 1. Introduction

A Proof-Directed Parallelization Synthesis by Separation LogicProof-Directed Parallelization Synthesis by Separation Logic A:3 (2)We leverage abduction and frame inference to de ne

A Probabilistic Model for Component-Based Shape Synthesissidch/docs/siggraph2012_synthesis.pdf · Graphics—Computational Geometry and Object Modeling; Keywords: shape synthesis,

Probabilistic mapping - Probabilistic Robotics DT4051 ...130.243.105.49/Research/Learning/courses/probrob/... · Probabilistic mapping Probabilistic Robotics DT4051, spring 2015 Martin

Probabilistic Circuits: A New Synthesis of Logic and Machine Learningweb.cs.ucla.edu/~guyvdb/slides/KULeuven18.pdf · Probabilistic and Logistic Circuits: A New Synthesis of Logic

A Probabilistic Model for Component-Based Shape Synthesis

A Probabilistic Model for Component-Based Shape Synthesiskalo/papers/Shape... · Keywords: shape synthesis, shape structure, probabilistic graphi-cal models, machine learning, data-driven

Lectures on PROBABILISTIC LOGICS AND THE SYNTHESIS …

Probabilistic Robotics Probabilistic Motion Models

SRL Approaches: Frame-based Probabilistic models February 11, 2005

Bayesian Synthesis of Probabilistic Programs for …Bayesian Synthesis of Probabilistic Programs for Automatic Data Modeling 37:3 given data, but still require users to write probabilistic

PROBABILISTIC ANALYSIS OF REINFORCED CONCRETE ...ethesis.nitrkl.ac.in/6887/1/Probabilistic_Prusty_2015.pdfnonlinear analysis of RC plane frame structures including both structural

Bayesian Synthesis of Probabilistic Programs for Automatic ...people.csail.mit.edu/rinard/paper/popl19.pdf · Bayesian Synthesis of Probabilistic Programs for Automatic Data Modeling

Parameter Synthesis for Probabilistic Timed Automata …qav.comlab.ox.ac.uk/papers/jk17.pdf · Parameter Synthesis for Probabilistic Timed Automata Using Stochastic Game Abstractions

Video Frame Synthesis Using Deep Voxel Flow...Video Frame Synthesis using Deep Voxel Flow Ziwei Liu1 Raymond A. Yeh2 Xiaoou Tang1 Yiming Liu3∗ Aseem Agarwala4 1The Chinese University

Visual Dynamics: Probabilistic Future Frame Synthesis via ...visualdynamics.csail.mit.edu/visualDynamics16.pdf · and present an analysis of the learned network representations. 2

Applying Combinatory Logic Synthesis To Work With Existing … · 2017-10-17 · Combinatory Logic Synthesis (CLS) then is our solution to the large frame-work problem. Application

Automated Verification and Strategy Synthesis for ...qav.comlab.ox.ac.uk/talks/marta-atva13inv.pdf · Automated Verification and Strategy Synthesis for Probabilistic Systems Marta

Application of the Probabilistic Dynamic Synthesis Method ... · Application of the Probabilistic Dynamic Synthesis Method to Realistic ... structure,suchasahollowed-outturbineblade

Minimization of the Probabilistic p-frame Potential...Minimization of the Probabilistic p-frame Potential M. Ehlera,b,, K. A. Okoudjoub aNational Institutes of Health, National Institute

A Probabilistic Model for Component-Based Shape Synthesis

Event-driven Video Frame Synthesis - arXiv

LTL Synthesis on Probabilistic Systems

Nearly Perfect Detection of Continuous F0 Contour and ... · Contour and Frame Classification for TTS Synthesis ... ANN-based frame classification for prosody ... Speech and Signal

Noise Processing for Simple Laplacian Pyramid Synthesis Based on Dual Frame Reconstruction

Probabilistic Seismic Vulnerability assessment of RC Frame

Foundations of Probabilistic Programming/Program SynthesisOverview Foundations of Probabilistic Programming Probabilistic Programs Probabilistic programs extend sequential programs

Video Frame Synthesis using Deep Voxel Flow - arXiv · PDF fileVideo Frame Synthesis using Deep Voxel Flow ... 3D optical flow vector across space and time in the ... flow from the

A Probabilistic Genome-Wide Gene Reading Frame Sequence Modeliwbbio.ugr.es/2014/papers/IWBBIO_2014_paper_40.pdf · frame-sequence bias, but a pronounced bias is detectable even within

Multivariate Probabilistic Synthesis of Cellular Networks