deep learning meets flow visualization2018/11/26 · deep learning meets flow visualization chaoli...
TRANSCRIPT
![Page 1: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/1.jpg)
Deep Learning Meets Flow Visualization
Chaoli Wang
Big Data Challenges for Predictive Modeling of Complex Systems Symposium
University of Hong KongNovember 26, 2018
![Page 2: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/2.jpg)
Outline of talk
• Deep learning + Visualization• Flow visualization• FlowNet• Vector field reconstruction• Future directions
![Page 3: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/3.jpg)
Flow visualization
![Page 4: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/4.jpg)
Flow fields
![Page 5: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/5.jpg)
• A vector field: F(U) = V • U: field domain (x, y) in 2D; (x, y, z) in 3D • V: vector (u, v) in 2D; (u, v, w) in 3D
• Like scalar fields, vectors are defined at discrete points
Flow (vector) field
![Page 6: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/6.jpg)
• Rendering each individual vector in the field• Hedgehogs: oriented
lines spread over the volume, indicating the orientation and magnitude of the flow; do not show directional information
• Glyphs, arrows: icons that show directions, but
Glyph-based methods
![Page 7: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/7.jpg)
• Shading every pixels/voxels in the visualization• Line integral convolution
(LIC)• 3D LIC volume is
problematic due to the dense texture
Texture-based methods
![Page 8: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/8.jpg)
• Rendering primitives built from particle trajectories• 1D: streamlines • Line variations: tubes,
ribbons• 2D: stream surfaces• 3D: flow volumes
Geometry-based methods
![Page 9: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/9.jpg)
Streamlines• A family of curves that are instantaneously tangent to the
velocity vector of the flow• Show the direction a fluid element will travel in at any
point in time• Streaklines, pathlines, timelines: extension streamlines
to unsteady data
FlowVisual https://sites.nd.edu/chaoli-wang/flowvisual/
![Page 10: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/10.jpg)
Examples of flow lines and surfaces
![Page 11: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/11.jpg)
FlowNet
![Page 12: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/12.jpg)
Outline of approach• Goal
– A single deep learning approach for identifying representative flow lines or flow surfaces
• Key ideas– Leverage an autoencoder to automatically
learn line or surface feature descriptors– Apply dimensionality reduction and interactive
clustering for exploration and selection
![Page 13: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/13.jpg)
FlowNet user interface
![Page 14: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/14.jpg)
Video demo
![Page 15: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/15.jpg)
• Encoder-decoder framework • 3D voxel-based binary representation as input• Feature descriptor learning in the latent space
FlowNet architecture
![Page 16: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/16.jpg)
Why voxel-based approach• Manifold-based
– Suitable for 3D mesh manifold (genus zero or higher genus surface)
– Does not work for flow lines or surfaces (non-closed)• Multiview-based
– Represent 3D shape with images rendered from different views
– Flow surfaces could be severely self-occluded• Voxel-based
– No precise line or surface is required for loss function computation and reconstruction quality evaluation
– Currently limited to a low resolution (e.g., 643)
![Page 17: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/17.jpg)
FlowNet details
• The encoder consists of four convolutional (CONV) layers with batch normalization (BN) added in between, one CONV layer w/o BN, followed by two fully-connected layers
• The decoder consists of five CONV layers and four BN layers
• Apply the rectified linear unit (ReLU) at the hidden layers and the sigmoid function at the output layer
• Consider three loss functions: binary cross entropy, mean squared error (MSE), and Dice loss
![Page 18: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/18.jpg)
Dimensionality reduction (DR) and object clustering
• Consider three DR methods: t-SNE (neighborhood-preserving), MDS and Isomap(distance-preserving)
• Consider three clustering methods: DBSCAN (density-based), k-means (partition-based), and agglomerative clustering (hierarchy-based)
• Finally choose t-SNE + DBSCAN• Compare three distance measures: FlowNet
feature Euclidean distance, streamline MCP distance, and streamline Hausdorff distance
![Page 19: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/19.jpg)
Parameter setting and performance
![Page 20: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/20.jpg)
Qualitative evaluation
Training set only Test set only Training set + test set
![Page 21: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/21.jpg)
Quantitative evaluation
• Use representative streamlines to reconstruct the vector field using gradient vector flow (GVF)
![Page 22: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/22.jpg)
FlowNetresults (1)
![Page 23: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/23.jpg)
FlowNetresults (2)
![Page 24: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/24.jpg)
FlowNet results (3)
![Page 25: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/25.jpg)
Vector field reconstruction
![Page 26: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/26.jpg)
Outline of approach• Goal
– Target streamline-based flow field representation and reduction in in-situ visualization setting
– Outperform the de facto method of gradient vector flow (GVF) for vector field reconstruction
– Achieve higher reconstruction quality compared with using compressed vector field
• Key ideas– Compress streamlines as sequence data using
SZ compression– Reconstruct vector field using deep learning
![Page 27: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/27.jpg)
Two-stage vector field reconstruction
Stage 1 (S1) Stage 2 (S2)
![Page 28: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/28.jpg)
S1: Low-resolution initialization• Starting from a randomly initialized vector field
Vlow
• Iteratively update the low-resolution vector field Vlow until it produces the same velocities as the input streamlines
Original S1: Random tracing
S1: Fixed tracing
![Page 29: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/29.jpg)
S2: High-resolution refinement• Borrow the ideas of image super-resolution and
image completion• Build a CNN to upscale Vlow to Vhigh
• Fill the empty voxels through a nonlinear combination of their neighborhoods
Original S1: Random tracing
S2: Random tracing
![Page 30: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/30.jpg)
Verification of two-stage reconstruction
Original
S2: Random tracing
Direct fixed tracingDirect random tracing
S1: Random tracing
S1: Fixed tracing
![Page 31: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/31.jpg)
CNN architecture
The CNN includes seven deconvolutional layers and four residual blocks
An example of a residual block
![Page 32: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/32.jpg)
Reduction rate and reconstruction PSNR (different numbers of lines)
![Page 33: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/33.jpg)
Compression rate and reconstruction PSNR (different error bounds)
![Page 34: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/34.jpg)
Visualization of reconstruction errorsG
VFO
ur m
etho
d
![Page 35: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/35.jpg)
Comparison against direct compression of vector field
Ground truth
SZ compression Our method
![Page 36: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/36.jpg)
• Scientific simulation data– From steady to unsteady data– From single variable to multivariate data
• Deep learning techniques– From CNN & RNN to GNN (graph neural network)– Adversarial learning and transfer learning
• Goals– Generalized 3D feature learning framework– Target scalar and vector fields, time-varying
multivariate data– Data augmentation for scientific visualization
Future directions
![Page 37: Deep Learning Meets Flow Visualization2018/11/26 · Deep Learning Meets Flow Visualization Chaoli Wang Big Data Challenges for Predictive Modeling of Complex Systems Symposium University](https://reader035.vdocuments.mx/reader035/viewer/2022081400/5f14161f6565b54157769b56/html5/thumbnails/37.jpg)
• Team members– Jun Han, Hao Zheng, Dr. Jun Tao (now at Sun
Yat-sen University), Martin Imre• Collaborators
– Drs. Hanqi Guo, Danny Chen• Funding
– NSF IIS-1455886, CNS-1629914, and DUE-1833129
– NVIDIA GPU Grant Program
Acknowledgments