connecting to streaming data sources; adapters, client

Connecting to Streaming Data Sources;

Adapters, Client Connectivity and

Feed Simulations

© Copyright 2000-2017 TIBCO Software Inc.

2

Agenda


• Data Connectivity Overview

• Adapters

• Client Connectivity

• Feed Simulation

3


Streaming Data

Connectivity

4

A Systems of Insight Architecture


Drag & Drop connectivity to streaming and enterprise data

CONNECT

Monitor campaign KPI’s and alert on model

efficacy

TRACK

Automate actions based on real-time data using

generated model

AUTOMATE

Load Model

Persist

Absorb enterprise & streaming data; cleanse,

normalize, enrich, bin

INGEST

Interactive Spotfire application to discover and model predictive analytics

DISCOVER

Model Feedback

Action

5


StreamBase

Adapters

6

Points of Connectivity


7 © Copyright 2000-2017 TIBCO Software Inc.

StreamBase Adapters

7

Provide:

• Connectivity to external systems

• Translation or conversion from some API or protocol to stream of tuples

Streaming Input Adapter Examples

• Financial market data feeds

• Sensor data

• Bus provider of any type of data


StreamBase Adapters Cont’d

8

Some adapters address external, static data and convert to stream form.

Example: each line in CSV file becomes a tuple, and the lines are streamed in the order read

Output Adapters convert from StreamBase tuple format to external resources

Output Adapter Examples

• XML file

• Live Datamart


StreamBase Adapters

9

Drag onto canvas to

invoke selection dialog:


Configuring StreamBase Adapters

10

• Once dropped onto canvas, must configure

• Each adapter configuration is unique

• Example: Twitter • Calls Twitter4J library

• Twitter returns JSON-encoded entities

• Requires Oauth consumer and access tokens

• Tokens go into streambase.conf:


Configuring Twitter Adapter (cont’d)

11

Default

Configuration:

Configured to access all available information:


Example: RSS Adapter

12

Items exposed for configuration vary by adapter and can be conditional (Proxy Host/Port)

13


StreamBase

Client Connectivity


Client Connectivity

14

• Client Types

• Enqueue (producer)

• Dequeue (consumer)

• Client Language Options • Java

• C++

• .NET

• Python

• File->New->StreamBase Client

• Wizard creates code stubs for Java

15


StreamBase

Feed Simulations


Why Feed Simulations

16

• Use to simulate live data

• Especially useful for development phase

• Customizable • Random generation of fields in tuples

• Read data from file

• Custom generation

• Enums, ranges, conditional

• Generation rate control


How to Create Feed Simulations

17

• File->New->StreamBase Feed Simulation

• Pick parent folder / project

• Give it a name

• Generates <Name>.sbfs in project

• Invokes configuration wizard


Configuring Feed Simulations (1 of 3)

18

• Select stream

• Generates tuples that match defined schema on selected stream



19

• Select generation method:



20

• Configure processing options

• Data rate

• Rate can be changed in manual input mode

• Loop - esp useful in conjunction with reading from a data file


Feedsim Data File

21

• Choose data file

• Header or not?

• Preview

• Column mapping

• Delimiter


Custom Fields

22

• Default generation: random

• Custom

• Separately, independently or conditionally based on other fields

• Customization types depend on data type:


Conditional Fields (1 of 2)

23

• Base conditions on known set

• Enumerated

• Incremented

• Strict equality only test

• No StreamBase expressions are available to be used to evaluate conditions

• Example: Items will be conditional based on category


Conditional Fields (2 of 2)

24

Default value • Provide constant for when no

other conditions match

Set up each case • For each known value, add a case

with a generation type


Running EventFlow Fragments with a Feedsim

25

26


Lab A

Exercise 3:

Feed Simulations

connecting to streaming data sources; adapters, client

Documents