an#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · an#automa)c#video#monitoring#...
TRANSCRIPT
![Page 1: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/1.jpg)
An automa)c video monitoring system for es)ma)ng dri4wood
discharge in large rivers
P. Lemaire1, H. Piégay1, B. MacVicar2, L. Vaudor1, C. Mouquet-‐Noppe1, L. Tougne3
1Université de Lyon, CNRS-‐UMR 5600, ENS de Lyon, Lyon, France
2Department of Civil and Environmental Engineering, University of Waterloo, Waterloo, Ontario, Canada
3Université de Lyon, CNRS-‐UMR 5205, LIRIS, Université Lyon 2, Lyon, France
![Page 2: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/2.jpg)
Why should we monitor driFwood discharge automaGcally?
• Major need in wood budgeGng and wood risk monitoring – Measure wood discharge (volume of wood passing through a secGon per unit Gme)
– Different strategies • Reservoir / raF censing => only basin esGmate • At a staGon (for river reaches – in between two staGons)
– RelaGvely sparse flux => need for video (to catch the flux) – ExhausGng and Gme consuming to do it by visual analysis – Need to automaGze the process
![Page 3: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/3.jpg)
Our objecGves • Being able to detect automaGcally driFwood as best as we can: – From any size? – From any part in the secGon? – In any weather, water aspect and lighGng condiGons?
• Being able to extract data about wood blocks : size, speed, rotaGon speed, lateral posiGon, form…
• If a complete automated system does not provide accurate enough results, it must facilitate manual annotaGon
![Page 4: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/4.jpg)
• An example of a video that we are trying to analyse – At a gauge staGon on the Ain river
![Page 5: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/5.jpg)
![Page 6: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/6.jpg)
![Page 7: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/7.jpg)
Outline
• Method – Combining probability masks… – … and examining components movement… – … to extract and characterize wood blocks
• EvaluaGon and results
• Associated works and future developments
![Page 8: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/8.jpg)
Outline
• Method – Combining probability masks… – … and examining components movement… – … to extract and characterize wood blocks
• EvaluaGon and results
• Associated works and future developments
![Page 9: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/9.jpg)
Method
• Combining probability masks… – Pixels can belong to either background (most probably, water) or driFwood
– Classifying pixels is equivalent to studying their probability of belonging to either class (driFwood or background)
– In our works, this probability is a combinaGon of sta)c probability and dynamic probability, and makes use of temporal and spa)al informa)ons
![Page 10: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/10.jpg)
Combining probability masks…
• Sta)c probability mask : – For every pixel, we answer the following quesGon :
“Given the pixel’s colour and intensity, is it likely to belong to dri4wood or background?”
– We used mainly a Gaussian distribuGon model on pixel intensity in grayscale images (which means that we expect every driFwood occurrence to have the same intensity)
![Page 11: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/11.jpg)
![Page 12: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/12.jpg)
![Page 13: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/13.jpg)
![Page 14: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/14.jpg)
Combining probability masks…
• Dynamic probability mask : – For every pixel, we answer the following quesGon :
“Given the pixel’s past and current values, is it likely to belong to dri4wood or background?”
– We used a transfer funcGon where pixels get excited when they become suddenly and significantly darker
![Page 15: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/15.jpg)
![Page 16: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/16.jpg)
![Page 17: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/17.jpg)
Combining probability masks…
• We make the approximaGon that both probabiliGes are independent from each other
• Combining both probability masks is similar to mulGplying each mask value, pixel per pixel
• Combined images are filled with few high intensity values
![Page 18: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/18.jpg)
Combined mask Dynamic mask
Sta)c mask Original image
![Page 19: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/19.jpg)
![Page 20: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/20.jpg)
Outline
• Method – Combining probability masks… – … and examining components movement… – … to extract and characterize wood blocks
• EvaluaGon and results
• Associated works and future developments
![Page 21: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/21.jpg)
… and examining components movements
• We group pixels of high value into blobs (called connex components).
• We remove the smallest components due to background noise
• Now, we are looking forward to classifying connex components as driFwood or noise
![Page 22: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/22.jpg)
… and examining components movements
• We match components from frame to frame, expecGng certain movement quanGGes and direcGons
• Once we observe a sufficient number of matches, and a smooth trajectory, we assume that we are observing a wood block
![Page 23: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/23.jpg)
![Page 24: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/24.jpg)
![Page 25: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/25.jpg)
Outline
• Method – Combining probability masks… – … and examining components movement… – … to extract and characterize wood blocks
• EvaluaGon and results
• Associated works and future developments
![Page 26: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/26.jpg)
… to extract and characterize wood blocks
• When a wood block is extracted, it consists in a set of views along the image
• We build a skeleton for each view. It allows us to find boundaries and extract the radius (in pixels) at each point
• Finally, the image is ortho-‐recGfied thanks to coordinates acquired relaGve to the camera. Pixels are translated to metrics data.
![Page 27: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/27.jpg)
Outline
• Method – Combining probability masks… – … and examining components movement… – … to extract and characterize wood blocks
• EvaluaGon and results
• Associated works and future developments
![Page 28: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/28.jpg)
EvaluaGon and results
• The presented system is fully automaGc • We need to compare it to manually annotated data in order to know how accurate our algorithm and its sedngs are
• During 2007 and 2008, 3 floods were annotated by an operator (MacVicar and Piégay, 2012)
• We matched human annotaGons with our soFware output
![Page 29: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/29.jpg)
EvaluaGon and results
• For each block, 3 outputs are possible :
• DetecGon rate (or TP rate) is the number of TP cases over the whole populaGon (TP+FP+FN)
Detected Annotated
True PosiGve (TP) YES YES
False PosiGve (FP) YES NO
False NegaGve (FN) NO YES
![Page 30: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/30.jpg)
y
x upstream downstream
• We want to know how performances are affected by: – wood block posiGon in the image? – wood block size?
• We kept the best view (that is, when the size is bigger) as an indicator to where the detecGon occurred. Most of the Gme, it is coherent with where the tracking actually took place.
FRONT END
REAR END
Right bank
LeF bank
![Page 31: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/31.jpg)
• TP rate in funcGon of the posiGon (in pixels) within the image – X increases from upstream to downstream – Y increases from the front end to the rear end
![Page 32: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/32.jpg)
• FP rate in funcGon of the posiGon (in pixels) within the image – X increases from upstream to downstream – Y increases from the front end to the rear end
![Page 33: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/33.jpg)
• FP and FN rates in funcGon of the size of blocks – Size is an indicator based on the maximal length and the maximal radius.
![Page 34: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/34.jpg)
UPSTR DOWNSTR
REAR END
FRONT END
![Page 35: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/35.jpg)
What does the evaluaGon show? • As expected, bigger blocks are beler detected, although we can miss big blocks at the rear end of the image. Reliability is greater than 80% when size > 15 Liters.
• There is room for improvement in this experiment : we can see palerns, in parGcular in FPs, and we can probably filter out detecGons so that results are improved
• Turbulent structures caused by the bridge pile probably impact our performances => feedback to improve camera locaGon
![Page 36: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/36.jpg)
Plan
• Method – Combining probability masks… – … and examining components movement… – … to extract and characterize wood blocks
• EvaluaGon and results
• Associated works and future developments
![Page 37: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/37.jpg)
Associated works and future developments
• We developed an User Interface to annotate videos. – AutomaGc video characterizaGon can be used as a basis for annotaGons
• We want to base our parameters and algorithms more on learnt data (annotaGon-‐driven). We are currently invesGgaGng means of improving the performances in general, including the robustness to partly submerged blocks
![Page 38: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/38.jpg)
Associated works and future developments
• Tests with other setups: – We are currently tesGng the soFware with videos provided by Maxime Boivin. • More fps and resoluGon impact the performances. • Those videos embed ice as well as wood. First results are encouraging.
![Page 39: An#automa)c#video#monitoring# system#foresmangdri4wood ...€¦ · An#automa)c#video#monitoring# system#foresmangdri4wood# discharge#inlarge#rivers!! P.!Lemaire 1,!H.!Piégay ,!B.!MacVicar2,!L.!Vaudor](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e1abb2e28b300da74b9a2/html5/thumbnails/39.jpg)
Thank You for Your alenGon & QuesGons?