drone data flowing through apache nifi

1 ©HortonworksInc.2011– 2016.AllRightsReserved

DroneDataFlowingThroughApacheNiFiTimothySpann2016FutureofData– PrincetonMeetup


hdfs dfs -get/drone/meta/Bebop2_20160920085308-0400.json{"date":"2016-09-20T08:53:08","Compression":"JPEG","ExifVersion":"2.10","ComponentsConfiguration":"YCbCr","file.group":"root","CompressionType":"Baseline","Image Description":"{\"product_id\":\"090C\",\"uuid\":\"CE112B4276E13339AFBAE10E9ED794E3\",\"run_date\":\"2016-09-20T083537-0400\",\"filename\":\"Bebop_2_2016-09-20T085308-0400_CE112B.jpg\",\"media_date\":\"2016-09-20T085308-0400\"}","NumberofComponents":"3","Component2":"Cbcomponent:Quantizationtable1,Samplingfactors1horiz/1vert","Focal Length":"1.83mm","Component 1":"Ycomponent:Quantizationtable0,Samplingfactors2horiz/2vert","YCbCr Positioning":"Center ofpixelarray","tiff:ResolutionUnit":"Inch","uuid":"9576a956-ec32-4314-bc8b-bd5bb43af4f8","Date/TimeOriginal":"2016:09:2008:53:08","ShutterSpeedValue":"1/249sec","X Resolution":"72dotsperinch","tiff:Make":"PARROT","path":"/","PhotometricInterpretation":"YCbCr","Component3":"Crcomponent:Quantizationtable1,Samplingfactors1horiz/1vert","Unique ImageID":"[32bytes]","F-Number":"F2.3","modified":"2016-09-20T08:53:08","FocalLength35":"6mm","tiff:BitsPerSample":"8","ExposureProgram":"Program action(high-speedprogram)","GPSVersionID":"2.200","GPSLatitudeRef":"N","meta:creation-date":"2016-09-20T08:53:08","exif:FNumber":"2.2999999166745724","GPSAltitudeRef":"Sea level","Exposure Time":"8597231/2147483647sec","GPS Longitude":"-73° 4'50.41\"","Creation-Date":"2016-09-20T08:53:08","ISOSpeedRatings":"426","Make":"PARROT","Orientation":"Top,leftside(Horizontal/normal)","MeteringMode":"(Other)","tiff:Orientation":"1","GPSLongitudeRef":"W","tiff:Software":"Dragon3.9.0","exif:FocalLength":"1.8300001180412324","filename":"Bebop2_20160920085308-0400.jpg","XMPValueCount":"9","geo:long":"-73.080669","file.owner":"root","Software":"Dragon3.9.0","ExifImageHeight":"1088pixels","tiff:YResolution":"72.0","YResolution":"72dotsperinch","GPSLatitude":"40° 45'16.51\"","dc:description":"{\"product_id\":\"090C\",\"uuid\":\"CE112B4276E13339AFBAE10E9ED794E3\",\"run_date\":\"2016-09-20T083537-0400\",\"filename\":\"Bebop_2_2016-09-20T085308-0400_CE112B.jpg\",\"media_date\":\"2016-09-20T085308-0400\"}","geo:lat":"40.754585","FlashPixVersion":"1.00","DataPrecision":"8bits","White Balance":"Flash","tiff:ImageLength":"1088","description":"{\"product_id\":\"090C\",\"uuid\":\"CE112B4276E13339AFBAE10E9ED794E3\",\"run_date\":\"2016-09-20T083537-0400\",\"filename\":\"Bebop_2_2016-09-20T085308-0400_CE112B.jpg\",\"media_date\":\"2016-09-20T085308-0400\"}","dcterms:created":"2016-09-20T08:53:08","dcterms:modified":"2016-09-20T08:53:08","Last-Modified":"2016-09-20T08:53:08","file.permissions":"rwxrwxrwx","exif:ExposureTime":"0.00400339765660623","Last-Save-Date":"2016-09-20T08:53:08","GPSAltitude":"21metres","absolute.path":"/opt/demo/dronedata/","ColorSpace":"Undefined","File Size":"853159bytes","meta:save-date":"2016-09-20T08:53:08","file.creationTime":"2016-09-20T12:53:10+0000","Date/TimeDigitized":"2016:09:2008:53:08","FileName":"apache-tika-941370357006559178.tmp","Content-Type":"image/jpeg","Aperture Value":"F2.3","X-Parsed-By":"org.apache.tika.parser.DefaultParser,org.apache.tika.parser.jpeg.JpegParser","FileModifiedDate":"Tue Sep2023:33:24UTC2016","tiff:XResolution":"72.0","file.lastModifiedTime":"2016-09-20T12:53:10+0000","exif:DateTimeOriginal":"2016-09-20T08:53:08","Date/Time":"2016:09:2008:53:08","ExifImageWidth":"1920pixels","Image Height":"1088pixels","Image Width":"1920pixels","Unknown tag(0xc62f)":"[19bytes]","ResolutionUnit":"Inch","tiff:Model":"Bebop2","exif:IsoSpeedRatings":"426","MaxApertureValue":"F2.3","ExposureMode":"Auto exposure","Model":"Bebop 2","file.lastAccessTime":"2016-09-20T23:33:24+0000","tiff:ImageWidth":"1920","WhiteBalanceMode":"Auto whitebalance"}


pythonclassify_image.py --image_file /opt/demo/dronedataold/Bebop2_20160920083655-0400.jpgsolardish,solarcollector,solarfurnace(score=0.98316)windowscreen(score=0.00196)manholecover(score=0.00070)radiator(score=0.00041)doormat,welcomemat(score=0.00041)

bazel-bin/tensorflow/examples/label_image/label_image --image=/opt/demo/dronedataold/Bebop2_20160920083655-0400.jpgtensorflow/examples/label_image/main.cc:204]solardish(577):0.983162Itensorflow/examples/label_image/main.cc:204]windowscreen(912):0.00196204Itensorflow/examples/label_image/main.cc:204]manholecover(763):0.000704005Itensorflow/examples/label_image/main.cc:204]radiator(571):0.000408321Itensorflow/examples/label_image/main.cc:204]doormat(972):0.000406186

Local TensorFlow via Python or C++ Binary


Google Vision


Google VisionGoogle Vision


IBM


Microsoft Computer Vision API


NIFITimothy Spann


Installation

Download the binary from here: http://hortonworks.com/downloads/#dataflowOr here:https://nifi.apache.org/download.htmlOr on Mac:brew install nifi

https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#starting-nifi

bin/nifi.sh start


TheFutureofData:ActionableIntelligence

D A T A I N M O T I O N

STO

RA

GE

STO

RA

GE

GROUP 2GROUP 1

GROUP 4GROUP 3

D A T A A T R E S T

INTERNETOF

ANYTHING

Hortonworks’uniqueapproachtodata-in-motionanddata-at-restpowersActionableIntelligence


DATAATREST

DATAINMOTION

ACTIONABLEINTELLIGENCE

MODERNDATAAPPLICATIONS

ActionableIntelligencefromConnectedDataPlatforms

Capturingperishableinsightsfromdatainmotion

Ensuringrich,historicalinsightsondataatrest

Necessaryformoderndataapplications

HortonworksDataFlow

HortonworksDataPlatform


HDFProvides“DataPlanofControl”byManagingIoT Dataflows

ConstrainedHigh-Latency

Localized Context

Hybrid – Cloud/On-PremiseLow-Latency

Global Context

Datasourceagnosticcollectionofdataacrossheterogeneousenvironments


RapidEcosystemAdoption:170+Processors

HTTP

Syslog

Email

HTML

Image

Hash Encrypt

Extract

TailMerge

Evaluate

Duplicate Execute

Scan

GeoEnrich

Replace

ConvertSplit

Translate

HL7

FTP

UDP

XML

SFTP

RouteContent

RouteContext

RouteText

ControlRate

DistributeLoadAMQP


Data-in-motion:HortonworksDataFlowPoweredbyApacheNiFi

Collect,conductandcuratereal-timedata

End-to-endsecuritywithencryptionandrules

Traceabilityandreal-timeprovenance

DeliversInstant,PerishableInsights

HortonworksDataFlow

Data-in-motion


Accumulate,Analyze,ActonAllData

CentralizedArchitectureforMulti-tenancy

EnterpriseOperations,GovernanceandSecurity

DeliversRichHistoricalInsights

Data-at-rest:HortonworksDataPlatformPoweredbyApacheHadoop

HortonworksDataPlatform

Data-at-rest


Ã https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/

Ã https://github.com/jfrazee/awesome-nifi

Ã https://dzone.com/articles/getting-started-with-apache-nifi-and-hdf

Ã https://nifi.apache.org/docs.html

Ã https://community.hortonworks.com/articles/4356/getting-started-with-nifi-expression-language-and.html

LearningMore


Contact:

[email protected]/futureofdata-princeton

community.hortonworks.com/users/9304/tspann.html


HortonworksCommunityConnection

Read access for everyone, join to participate and be recognized

• FullQ&APlatform(likeStackOverflow)

• KnowledgeBaseArticles

• CodeSamplesandRepositories

drone data flowing through apache nifi

Technology