apache nifi meetup - princeton nj 2016

1 ©HortonworksInc.2011–2016.AllRightsReserved

ApacheNiFiOverviewTimothySpannSolutionsEngineerJuly14,20166:00PMFutureofData– PrincetonMeetup


Agenda

• 6:00pm – 6:45pm Registration and Food• 6:45pm – 7:00pm Introduction and Welcome• 7:00pm – 8:00pm Introduction to Apache NiFi by Tim Spann

• WhatisNiFi?• Demo• Hands-On

• 8:00pm – 8:45pm Creating a Custom Apache NiFi Processor by Milind Pandit

• 8:45pm – 9:00pm Wrap-up, Upcoming Kafka Talk


NIFITimothy Spann


Terminology

FlowFile• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)Processor• Performsthework,canaccessFlowFilesConnection• Linksbetweenprocessors• QueuesthatcanbedynamicallyprioritizedProcessGroup• Setofprocessorsandtheirconnections• Receivedataviainputports,senddataviaoutputports


Installation

Download the binary from here: http://hortonworks.com/downloads/#dataflowOr here:https://nifi.apache.org/download.htmlOr on Mac:brew install nifi

https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#starting-nifi

bin/nifi.sh start


TheFutureofData:ActionableIntelligence

D A T A I N M O T I O N

ST

OR

AG

EST

OR

AG

E

GROUP 2GROUP 1

GROUP 4GROUP 3

D A T A A T R E S T

INTERNETOF

ANYTHING

Hortonworks’uniqueapproachtodata-in-motionanddata-at-restpowersActionableIntelligence


DATAATREST

DATAINMOTION

ACTIONABLEINTELLIGENCE

MODERNDATAAPPLICATIONS

ActionableIntelligencefromConnectedDataPlatforms

Capturingperishableinsightsfromdatainmotion

Ensuringrich,historicalinsightsondataatrest

Necessary formoderndataapplications

HortonworksDataFlow

HortonworksDataPlatform


Data-in-motion:HortonworksDataFlowPoweredbyApacheNiFi

Collect,conductandcuratereal-timedata

End-to-endsecuritywithencryptionandrules

Traceabilityandreal-timeprovenance

DeliversInstant,PerishableInsights

HortonworksDataFlow

Data-in-motion


Accumulate,Analyze,ActonAllData

CentralizedArchitectureforMulti-tenancy

EnterpriseOperations,GovernanceandSecurity

DeliversRichHistoricalInsights

Data-at-rest:HortonworksDataPlatformPoweredbyApacheHadoop

HortonworksDataPlatform

Data-at-rest


ApacheNiFi• Powerfulandreliablesystemtoprocessanddistributedata

• Directedgraphsofdataroutingandtransformation

• Web-basedUserInterfaceforcreating,monitoring,&controllingdataflows

• Highlyconfigurable- modifydataflowatruntime,dynamicallyprioritizedata

• DataProvenancetracksdatathroughentiresystem

• Easilyextensiblethroughdevelopmentofcustomcomponents

[1]https://nifi.apache.org/


NiFi - Terminology

Ã FlowFile• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)

Ã Processor• Performsthework,canaccessFlowFiles

Ã Connection• Linksbetweenprocessors• Queuesthatcanbedynamicallyprioritized

Ã ProcessGroup• Setofprocessorsandtheirconnections• Receivedataviainputports,senddataviaoutputports


NiFi - UserInterface

• Draganddropprocessorstobuildaflow• Start,stop,andconfigurecomponentsinrealtime• Viewerrorsandcorrespondingerrormessages• Viewstatisticsandhealthofdataflow• Createtemplatesofcommonprocessor&connections


NiFi - Provenance

• Tracksdataateachpointasitflowsthroughthesystem

• Records,indexes,andmakeseventsavailablefordisplay

• Handlesfan-in/fan-out,i.e.mergingandsplittingdata

• Viewattributesandcontentatgivenpointsintime


NiFi - QueuePrioritization

• Configureaprioritizer perconnection

• Determinewhatisimportantforyourdata– timebased,arrivalorder,importanceofadataset

• Funnelmanyconnectionsdowntoasingleconnectiontoprioritizeacrossdatasets

• Developyourownprioritizer ifneeded


NiFi - Extensibility

Ã Builtfromthegroundupwithextensionsinmind

Ã Service-loaderpatternfor…• Processors• ControllerServices• ReportingTasks• Prioritizers

Ã ExtensionspackagedasNiFi Archives(NARs)• DeployNiFi libdirectory andrestart• ProvidesClassLoader isolation• Samemodelasstandardcomponents


NiFi - Architecture

OS/Host

JVM

FlowController

WebServer

Processor1 ExtensionN

FlowFileRepository

ContentRepository

ProvenanceRepository

LocalStorage

OS/Host

JVM

FlowController

WebServer


FlowFileRepository

ContentRepository


LocalStorage

OS/Host

JVM

NiFiClusterManager– RequestReplicator

WebServer

MasterNiFiClusterManager(NCM)

OS/Host

JVM

FlowController

WebServer


FlowFileRepository

ContentRepository


LocalStorage

SlavesNiFiNodes


RapidEcosystemAdoption:130+Processors

HTTP

Syslog

Email

HTML

Image

Hash Encrypt

Extract

TailMerge

Evaluate

Duplicate Execute

Scan

GeoEnrich

Replace

ConvertSplit

Translate

HL7

FTP

UDP

XML

SFTP

RouteContent

RouteContext

RouteText

ControlRate

DistributeLoadAMQP


Addprocessorfordataintake.Time:1minute1 Draganddropprocessoriconfromthetopmenu


Ã https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/

Ã https://github.com/jfrazee/awesome-nifi

Ã https://dzone.com/articles/getting-started-with-apache-nifi-and-hdf

Ã https://nifi.apache.org/docs.html

Ã https://community.hortonworks.com/articles/4356/getting-started-with-nifi-expression-language-and.html

LearningMore


Contact:

[email protected]/futureofdata-princeton

community.hortonworks.com/users/9304/tspann.html


HortonworksCommunityConnection

Read access for everyone, join to participate and be recognized

• FullQ&APlatform(likeStackOverflow)

• KnowledgeBaseArticles

• CodeSamplesandRepositories


WePowertheEcosystem

1600+ Partners

3000+members

11,000+Weekly visitors

apache nifi meetup - princeton nj 2016

Technology