working(with(real(time(data...
TRANSCRIPT
![Page 1: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/1.jpg)
Working with Real Time Data Streams for IoT Devices
Lachlan Blackhall (@lblackhall) PyconAU, 2016
![Page 2: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/2.jpg)
Smart Homes
![Page 3: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/3.jpg)
Smart CiJes
![Page 4: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/4.jpg)
Data!!!
![Page 5: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/5.jpg)
Stream Processing
![Page 6: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/6.jpg)
A Typical IoT Device
![Page 7: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/7.jpg)
What you hope your data looks like
![Page 8: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/8.jpg)
What it actually looks like
![Page 9: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/9.jpg)
Filtering to the Rescue
![Page 10: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/10.jpg)
A Simple Example
![Page 11: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/11.jpg)
Noisy Temperature Data
![Page 12: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/12.jpg)
Moving Average Filter class MovingAverageFilter(object): def __init__(self, window): self.window = window self.data = [] def step(self, measurement): self.data.append(measurement) if len(self.data) > self.window: self.data.pop(0) def current_state(self): return sum(self.data) / len(self.data)
![Page 13: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/13.jpg)
Kalman Filter class SingleStateKalmanFilter(object): def __init__(self, A, B, C, x, P, Q, R): self.A = A # Process dynamics self.B = B # Control dynamics self.C = C # Measurement dynamics self.current_state_esJmate = x # Current state esJmate self.current_prob_esJmate = P # Current probability of state esJmate self.Q = Q # Process covariance self.R = R # Measurement covariance
![Page 14: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/14.jpg)
Kalman Filter cont. def current_state(self): return self.current_state_esJmate def step(self, control_input, measurement): # PredicJon step predicted_state_esJmate = self.A * self.current_state_esJmate + self.B * control_input predicted_prob_esJmate = (self.A * self.current_prob_esJmate) * self.A + self.Q # ObservaJon step innovaJon = measurement -‐ self.C * predicted_state_esJmate innovaJon_covariance = self.C * predicted_prob_esJmate * self.C + self.R # Update step kalman_gain = predicted_prob_esJmate * self.C * 1 / float(innovaJon_covariance) self.current_state_esJmate = predicted_state_esJmate + kalman_gain * innovaJon # eye(n) = nxn idenJty matrix. self.current_prob_esJmate = (1 -‐ kalman_gain * self.C) * predicted_prob_esJmate
![Page 15: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/15.jpg)
IniJalising the Filters # IniJalise the Kalman Filter A = 1 # No process innovaJon C = 1 # Measurement B = 0 # No control input Q = 0.005 # Process covariance R = 1 # Measurement covariance x = 18 # IniJal esJmate P = 1 # IniJal covariance filter = SingleStateKalmanFilter(A, B, C, x, P, Q, R) # IniJalise two moving average filters with different window lengths ma5 = MovingAverageFilter(5) ma50 = MovingAverageFilter(50)
![Page 16: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/16.jpg)
SimulaJng a Data Stream # Empty lists for capturing filter esJmates predicJons = [] wavg5 = [] wavg50 = [] # Simulate the data arriving sequenJally for data in random_data: ma5.step(data) wavg5.append(ma5.current_state()) ma50.step(data) wavg50.append(ma50.current_state()) filter.step(0, data) predicJons.append(filter.current_state())
![Page 17: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/17.jpg)
Noisy Data
![Page 18: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/18.jpg)
Moving Average (5 samples)
![Page 19: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/19.jpg)
Moving Average (50 Samples)
![Page 20: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/20.jpg)
Kalman Filter
![Page 21: Working(with(Real(Time(Data Streams(for(IoTDeviceslachlanblackhall.com/.../2016/08/...PYCONAU-2016.pdf · Working(with(Real(Time(Data Streams(for(IoTDevices Lachlan(Blackhall((@lblackhall)](https://reader034.vdocuments.mx/reader034/viewer/2022052020/6034627e3eb240298a4a90c4/html5/thumbnails/21.jpg)
Resources • hjp://www.bzarg.com/p/how-‐a-‐kalman-‐filter-‐works-‐in-‐
pictures/ • hjps://en.wikipedia.org/wiki/Kalman_filter • hjps://en.wikipedia.org/wiki/Moving_average