![Page 1: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/1.jpg)
1 © 2015 The MathWorks, Inc.
Embedded Vision on FPGAs
![Page 2: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/2.jpg)
2
Enhanced Edge Detection in MATLAB Test bench
Read Image from File
Add noise
Median Filter
Edge Detect
Video Display
Frame To
Pixel
Pixel To
Frame
Design
![Page 3: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/3.jpg)
3
Computer Vision Ø Feature matching, and extraction Ø Object detection and recognition Ø Object Tracking and motion estimation Ø Dynamic resolution scaling Ø Focus assessment
Video Processing Ø Video in and out Ø Gamma correction Ø Color balancing Ø Noise removal Ø Image sharpening
Video Processing and Computer Vision
![Page 4: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/4.jpg)
4
A Workflow for Video Image Processing
Concept Development
Algorithm Development Prototyping Architecture
design Prototyping Chip design
Image/Video Engineer
Pixel based
HW Engineer
Frame based
![Page 5: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/5.jpg)
5
Vision HDL Toolbox, R2015a New Product
§ Customer Problems – Prototyping video and image processing algorithms on FPGA/SoC
§ Our Solution – Pixel-streaming algorithms for the design and implementation of vision systems on
FPGAs and ASICs – A diverse set of frame sizes and frame rates
§ Product Detail – Price: €5000/$5500 – Platforms: MATLAB and Simulink – Dependency: MATLAB for modeling, HDL Coder for code generation
![Page 6: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/6.jpg)
6
Vision HDL Toolbox, Key Benefits
§ Modeling bit-true and cycle-accurate behavior of video image processing algorithms – Pixel-based functions and blocks – Optimized for HDL code generation – Conversion between frame and pixel – Standard and custom frame sizes and frame rates
§ Prototyping video and image processing algorithms on FPGA/SoC – (With HDL Coder) Efficient and readable HDL code – (With HDL Verifier) FPGA-in-the-loop testing and acceleration
![Page 7: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/7.jpg)
7
Pixel Based Video Image Algorithms
§ Analysis & Enhancement – Edge Detection, Median Filter
§ Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table
§ Filter – Image Filter, Median Filter
§ Morphological Operations – Dilation, Erosion, – Opening, Closing
§ Statistics – Histogram – Image Statistics
§ I/O Interfaces – Frame to Pixels, Pixels to
Frame, FIL versions
§ Utilities – Pixel Control Bus Creator – Pixel Control Bus Selector
![Page 8: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/8.jpg)
8
A broader solution for embedded vision
Product Capabilities Vision HDL Toolbox (“VHT”)
Design and simulate image processing, video, and computer vision systems for FPGAs and ASICs
HDL Coder Provide code generation capability for the functions and blocks in Vision HDL Toolbox
HDL Verifier Provide FPGA-in-the-loop capability for Vision HDL Toolbox
Computer Vision System Toolbox (“CVST”)
Provide frame based computer vision functions and blocks as well as image and video I/O capability
Image Processing Toolbox
Provide image processing and analysis functions
![Page 9: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/9.jpg)
9
CVST and VHT, complimentary products Scopes
Vision algorithms Ø Feature matching, and extraction Ø Object detection and recognition Ø Object Tracking and motion estimation Ø Dynamic resolution scaling Ø Focus assessment
CVST
![Page 10: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/10.jpg)
10
CVST and VHT, complimentary products Scopes
Camera pipeline Ø Video in and out Ø Gamma correction Ø Color balancing Ø Noise removal Ø Image sharpening
![Page 11: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/11.jpg)
11
CVST and VHT, complimentary products Scopes
CVST
VHT 1.0 VHT future releases
Camera Pipeline
Vision Algorithm
image image information
![Page 12: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/12.jpg)
12
CVST and VHT, complementary products Product details
VHT CVST Image Processing Pixel based Frame based Workflow System prototyping Algorithm design
Algorithms Image filtering Color space conversion Edge detection Statistics & histogram Morphological operations
(Most of the VHT algorithms, plus) Object detection & tracking Feature extraction and matching Stereo vision Camera calibration Image registration
I/O Frame-to-pixel Pixel-to-frame FIL
File I/O Video display
Code gen HDL (with HDL Coder) C (with ML Coder or SL Coder)
![Page 13: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/13.jpg)
13
CVST and VHT, complementary products MBD workflow for embedded vision
MBD workflow 1. Build behavior model with CVST blocks and simulate 2. Build prototyping model with VHT blocks and simulate 3. Generate HDL code using HDL Coder 4. Run HDL code on FPGA and run testbench in SL (FIL)
![Page 14: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/14.jpg)
14
CVST and VHT, complementary products MBD workflow for embedded vision
![Page 15: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/15.jpg)
15
Enhanced Edge Detection: Simulink Demo
![Page 16: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/16.jpg)
16
Streaming Pixel Interface Full Frame vs Pixel Stream
full-frame source
full-frame sink
à
à
pixel-stream processing
![Page 17: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/17.jpg)
17
Streaming Pixel Interface Full Frame vs Pixel Stream
full-frame source
full-frame sink
à
à
pixel-stream processing
![Page 18: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/18.jpg)
18
Streaming Pixel Interface Pixel control signals
pixelcontrol bus Ø hStart, hEnd, vStart, vEnd, and valid Ø Consistent block interface
![Page 19: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/19.jpg)
19
Frame To Pixels and Pixels To Frame
![Page 20: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/20.jpg)
20
Frame To Pixels and Pixels To Frame
![Page 21: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/21.jpg)
21
Examples: Starting Points for Your Models
![Page 22: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/22.jpg)
22
Image Filtering Keywords: Median Filter, Image Filter, PSNR
![Page 23: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/23.jpg)
23
Gamma Correction Example Keywords: Gamma, PSNR
![Page 24: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/24.jpg)
24
Histogram Equalization Keywords: Histogram, Linear Equalization, External Frame Delay
0 255 0 255
![Page 25: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/25.jpg)
25
Edge Detection and Image Overlay Keywords: Sobel, Align Video, Alpha Mix, PSNR
![Page 26: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/26.jpg)
26
Edge Detection with Impaired Frame Keywords: Sobel, Align Video, Alpha Mix, PSNR
![Page 27: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/27.jpg)
27
Multi-Zone Metering Keywords: ROI, Image Statistics, Mean
Source Mask Selected ROI
![Page 28: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/28.jpg)
28
Multi-Zone Metering Keywords: ROI, Image Statistics, Mean
Source Mask Selected ROI
![Page 29: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/29.jpg)
29
Challenge Accelerate the implementation of advanced thermal imaging filters and algorithms on FPGA hardware Solution Use MATLAB to develop, simulate, and evaluate algorithms, and use HDL Coder to implement the best algorithms on FPGAs Results § Time from concept to field-testable prototype
reduced by 60% § Enhancements completed in hours, not weeks § Code reuse increased from zero to 30%
“With MATLAB and HDL Coder we are much more responsive to marketplace needs. We now embrace change, because we can take a new idea to a real-time-capable hardware prototype in just a few weeks. There is more joy in engineering, so we’ve increased job satisfaction as well as customer satisfaction.”
Nicholas Hogasten
FLIR Systems Link to user story
Raw image (left) and image after applying filter developed with HDL Coder (right).
FLIR Accelerates Development of Thermal Imaging FPGA
![Page 30: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/30.jpg)
30
From Algorithm to FPGA Implementation
Implement Design
Map
Place & Route
Synthesis
Verification
Static Timing Analysis
Timing Simulation
Functional Simulation
Back Annotation
HDL Verifier QuestaSim
HDL Co-Simulation
HDL Coder
RTL Creation
Behavioral Simulation
MATLAB® and Simulink®
Algorithm and System Design Model Refinement for Hardware
HDL Verifier
FPGA-in-the-Loop
![Page 31: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/31.jpg)
31
A complete solution
Frame based
Concept Development
Algorithm Development Prototyping Architecture
design Prototyping Chip design
Pixel based
MATLAB
Vision HDL Toolbox
MATLAB Coder
HDL Coder
Fixed Pt Designer HDL Verifier
Image Processing
Toolbox
Computer Vision System Toolbox
![Page 32: Embedded Vision on FPGAs€¦ · – Edge Detection, Median Filter " Conversions – Chroma Resampling, Color-Space Converter – Demosaic Interpolator, Gamma Corrector, Look-up Table](https://reader034.vdocuments.mx/reader034/viewer/2022050113/5f4a3a20a4728f6ee052b40f/html5/thumbnails/32.jpg)
32
ROI: Customer Adoption Of Model-Based Design Time spent on FPGA implementation
§ Shorter implementation time by 48% (total project 33%) § Reduced FPGA prototype development schedule by 47% § Shorter design iteration cycle by 80% 1st FPGA Prototype 2nd FPGA Prototype
1st FPGA Prototype