a presentation by w h inmon analyzing text in the big data environment

A presentation byW H Inmon

ANALYZING TEXT IN THEBIG DATA ENVIRONMENT

U n stru c tu re dte x t

E m a il

Tw e ets L o g rec ord s

M o v ie s

A na lo g

much of the information found inthe Big Data environment is text

Unstructuredtext

S im pleQ uery

Unstructuredtext

AnalyticalQ uery

the queries of the world can be divided into two classes – - simple search - analytical query

Sim pleQ uery

Rawdata

AnalyticalQ uery

RefinedTransform edData

there are many differences between a simple search and an analytical query one of those differences is the state of the data that the query is made against

I th ink your a irline is the best that I have flow n on. It is certa inly as good as S ingapore A ir, which sets the standard for quality.

P lane was late. M essed up m y schedule. A irline sucks.

Flt 462 D NV to O RD lst bgs CN 235-PO -908M erci pour le service m erveilleux sur le vol 232 de Washington a Paris . L’hostesse de l’a ir vra im ent a lle de sa facon de fa ire du vol confortable et inoubliable.

Vuelo 398 desde E l Paso hasta el Cabo de m iercoles 18 Noviem bre de 1998 fue e l vuelo m as m iserable he tenido servic io fue terrib le, ya era tarde, y m e perdi m iconexion a lineas C unard. Creo que voy a dem andar.

U Bill Inm on 00015427302 LAX-SYD flt 4634 O ct 26 2011 c-b st 16Ca

Dear UnitedI d id not appreciate being m oved from First class to coach involuntarily on flight 216 Denver to San FranciscoNov 5, 2011. I paid for first class and I want a refund.

Am 14 juni 2010 flog s ie m ein hund Fluffy von M unchen nach New York. F luffy kam wohlbehalten v ielen dank fur d ie betruung von fluffy

some raw text

I th in k yo u r a ir lin e is th e b e s t th a t I h a ve flow n on . It is ce rta in ly a s g o o d a s S ing a p ore A ir, w h ich se ts th e s ta n da rd fo r q u a lity.

To n eg o od

A ir ca rrie rS ing a p ore A ir

the tone of the message is good. Another carrier that is mentionedis Singapore Air

Plane w as late. M essed up m y schedule. A irline sucks.

O n tim elate

Tonevery bad

the tone of the message is very bad. The message mentions a lateflight. The message is not formed in proper English, as might be foundin a tweet or IM

F lt 4 62 D N V to O R D ls t bg s C N 23 5 -P O -9 08

F lig h t 4 6 2 D e n ve r to C h ic ag o lo s t b ag s c la im n u m b er 23 5 -P O -9 08

F lig h t 4 6 2 D e n ve r to C h ic ag o lo s t b ag s c la im n u m b er 23 5 -P O -9 08

F lig h tn u m be r

C ity C ity A c tiv ityO pe ra n d C la im n u m b er To n e

n o neF lig h t typ eU S

this cryptic message must first be expanded to understandable wordsThere is no tone to this message. It is purely informational. Other types ofdata that have been extracted include flight number, city, activity, operand,and claim number. Note that two cities have been used to determine thatthe flight type is a US domestic flight

M e rc i p o u r le se rv ice m e rve illeu x su r le vo l 2 3 2 d e W as h in g to n a P a ris . L’h o s te ss e d e l’a ir v ra im e n t a lle d e sa fa co n d e fa ire du vo l co n fo rta b le e t in o u b lia b le .

To n eg o od

To n eve ry g o o d

C ity

C ity

P ers on n e l F lig h t typ eo ve rse as

this message written in French has a good tone and a very good tone.Where there are two tones the higher is designated as the official toneof the message. Other types of information found include city, anda reference to personnel. Note that two cities are used to determine that this is an overseas flight.

City

Vuelo 398 desde E l Paso hasta el Cabo de m iercoles 18 Noviem bre de 1998 fue el vuelo m as m iserable he tenido servic io fue terrib le, ya era tarde, y m e perdi m iconexion a lineas Cunard. C reo que voy a dem andar.

Flightnum ber

C ity

DateTonevery bad

ServiceO n tim e

Connection

CarrierLawsuit

F light typeinternational

this message is written in Spanish. The tone is very bad. City, service,on time, and connection are found in the message. A lawsuit is mentionednote that two cities are used to determine that this in international flightbut not an overseas flight

U Bill Inm on 00015427302 LAX-SYD flt 4634 O ct 26 2011 c-b st 16Ca

Upgrade B ill Inm on 00015427302 Los Angeles to Sydney fight 4634 O ct 26 2011 coach to business seat 16C ais le

Upgrade B ill Inm on 00015427302 Los Angeles to Sydney fight 4634 O ct 26 2011 coach to business seat 16C ais le

ActionToneneutral

PassengerPassenger Id

C ity C ityFlight typeoverseas

Flightnum ber

DateUpgradetype

Seatnum ber

A islew indowm iddle

Dear UnitedI d id not appreciate being m oved from First class to coach involuntarily on flight 216 Denver to San FranciscoNov 5, 2011. I paid for firs t c lass and I want a refund.

F light typeUS

Financialactiv ity

Tonevery bad

Downgrade

Flightnum ber

C ity C ity

Date

the tone of this message is very bad. The flight number,date, and cityare mentioned. city is used twice to determine that this was a domesticUS flight.

U n stru c tu re dte x t

Te x tua lE T L

E m a il

Tw e et

Tran sc rip tio n s

Te le ph o n e

L e tte rs

the source of the text can be any electronic source

U n stru c tu re dte x t

Te x tua lE T L

Sta n da rdre la tio n a ld a ta b a se

after transformation occurs, the results are placed in astandard relational data base

Unstructuredtext

TextualETL

Standardanalytica ltechnology

and once a standard relational data base is created, there areliterally hundreds of analytical tools that can operate againstthe data base

Unstructuredtext

TextualETL

Let m e go findsom e raw text

Let m e look at data organized by any or all of -

- tone of m essage - in ternational dom estic flight - date - lost luggage - upgrades - personnel - c ity - arrival tim e - carrier - ..................

I think your airline is the best that I have flown on. It is certainly as good as Singapore Air, which sets the standard for quality.

Plane was late. Messed up my schedule. Airline sucks.

Flt 462 DNV to ORD lst bgs CN 235-PO-908Merci pour le service merveilleux sur le vol 232 de Washington a Paris. L’hostesse de l’air vraiment alle de sa facon de faire du vol confortable et inoubliable.

Vuelo 398 desde El Paso hasta el Cabo de miercoles 18 Noviembre de 1998 fue el vuelo mas miserable he tenido servicio fue terrible, ya era tarde, y me perdi miconexion a lineas Cunard. Creo que voy a demandar.

U Bill Inmon 00015427302 LAX-SYD flt 4634 Oct 26 2011 c-b st 16Ca

Dear UnitedI did not appreciate being moved from First class to coach involuntarily on flight 216 Denver to San FranciscoNov 5, 2011. I paid for first class and I want a refund.

Am 14 juni 2010 flog sie mein hund Fluffy von Munchen nach New York. Fluffy kam wohlbehalten vielen dank fur die betruung von fluffy

if you are ananalyst, what kindof data do youwant to operateon