a presentation by w h inmon analyzing text in the big data environment
TRANSCRIPT
A presentation byW H Inmon
ANALYZING TEXT IN THEBIG DATA ENVIRONMENT
U n stru c tu re dte x t
E m a il
Tw e ets L o g rec ord s
M o v ie s
A na lo g
much of the information found inthe Big Data environment is text
Unstructuredtext
S im pleQ uery
Unstructuredtext
AnalyticalQ uery
the queries of the world can be divided into two classes – - simple search - analytical query
Sim pleQ uery
Rawdata
AnalyticalQ uery
RefinedTransform edData
there are many differences between a simple search and an analytical query one of those differences is the state of the data that the query is made against
I th ink your a irline is the best that I have flow n on. It is certa inly as good as S ingapore A ir, which sets the standard for quality.
P lane was late. M essed up m y schedule. A irline sucks.
Flt 462 D NV to O RD lst bgs CN 235-PO -908M erci pour le service m erveilleux sur le vol 232 de Washington a Paris . L’hostesse de l’a ir vra im ent a lle de sa facon de fa ire du vol confortable et inoubliable.
Vuelo 398 desde E l Paso hasta el Cabo de m iercoles 18 Noviem bre de 1998 fue e l vuelo m as m iserable he tenido servic io fue terrib le, ya era tarde, y m e perdi m iconexion a lineas C unard. Creo que voy a dem andar.
U Bill Inm on 00015427302 LAX-SYD flt 4634 O ct 26 2011 c-b st 16Ca
Dear UnitedI d id not appreciate being m oved from First class to coach involuntarily on flight 216 Denver to San FranciscoNov 5, 2011. I paid for first class and I want a refund.
Am 14 juni 2010 flog s ie m ein hund Fluffy von M unchen nach New York. F luffy kam wohlbehalten v ielen dank fur d ie betruung von fluffy
some raw text
I th in k yo u r a ir lin e is th e b e s t th a t I h a ve flow n on . It is ce rta in ly a s g o o d a s S ing a p ore A ir, w h ich se ts th e s ta n da rd fo r q u a lity.
To n eg o od
A ir ca rrie rS ing a p ore A ir
the tone of the message is good. Another carrier that is mentionedis Singapore Air
Plane w as late. M essed up m y schedule. A irline sucks.
O n tim elate
Tonevery bad
the tone of the message is very bad. The message mentions a lateflight. The message is not formed in proper English, as might be foundin a tweet or IM
F lt 4 62 D N V to O R D ls t bg s C N 23 5 -P O -9 08
F lig h t 4 6 2 D e n ve r to C h ic ag o lo s t b ag s c la im n u m b er 23 5 -P O -9 08
F lig h t 4 6 2 D e n ve r to C h ic ag o lo s t b ag s c la im n u m b er 23 5 -P O -9 08
F lig h tn u m be r
C ity C ity A c tiv ityO pe ra n d C la im n u m b er To n e
n o neF lig h t typ eU S
this cryptic message must first be expanded to understandable wordsThere is no tone to this message. It is purely informational. Other types ofdata that have been extracted include flight number, city, activity, operand,and claim number. Note that two cities have been used to determine thatthe flight type is a US domestic flight
M e rc i p o u r le se rv ice m e rve illeu x su r le vo l 2 3 2 d e W as h in g to n a P a ris . L’h o s te ss e d e l’a ir v ra im e n t a lle d e sa fa co n d e fa ire du vo l co n fo rta b le e t in o u b lia b le .
To n eg o od
To n eve ry g o o d
C ity
C ity
P ers on n e l F lig h t typ eo ve rse as
this message written in French has a good tone and a very good tone.Where there are two tones the higher is designated as the official toneof the message. Other types of information found include city, anda reference to personnel. Note that two cities are used to determine that this is an overseas flight.
City
Vuelo 398 desde E l Paso hasta el Cabo de m iercoles 18 Noviem bre de 1998 fue el vuelo m as m iserable he tenido servic io fue terrib le, ya era tarde, y m e perdi m iconexion a lineas Cunard. C reo que voy a dem andar.
Flightnum ber
C ity
DateTonevery bad
ServiceO n tim e
Connection
CarrierLawsuit
F light typeinternational
this message is written in Spanish. The tone is very bad. City, service,on time, and connection are found in the message. A lawsuit is mentionednote that two cities are used to determine that this in international flightbut not an overseas flight
U Bill Inm on 00015427302 LAX-SYD flt 4634 O ct 26 2011 c-b st 16Ca
Upgrade B ill Inm on 00015427302 Los Angeles to Sydney fight 4634 O ct 26 2011 coach to business seat 16C ais le
Upgrade B ill Inm on 00015427302 Los Angeles to Sydney fight 4634 O ct 26 2011 coach to business seat 16C ais le
ActionToneneutral
PassengerPassenger Id
C ity C ityFlight typeoverseas
Flightnum ber
DateUpgradetype
Seatnum ber
A islew indowm iddle
Dear UnitedI d id not appreciate being m oved from First class to coach involuntarily on flight 216 Denver to San FranciscoNov 5, 2011. I paid for firs t c lass and I want a refund.
F light typeUS
Financialactiv ity
Tonevery bad
Downgrade
Flightnum ber
C ity C ity
Date
the tone of this message is very bad. The flight number,date, and cityare mentioned. city is used twice to determine that this was a domesticUS flight.
U n stru c tu re dte x t
Te x tua lE T L
E m a il
Tw e et
Tran sc rip tio n s
Te le ph o n e
L e tte rs
the source of the text can be any electronic source
U n stru c tu re dte x t
Te x tua lE T L
Sta n da rdre la tio n a ld a ta b a se
after transformation occurs, the results are placed in astandard relational data base
Unstructuredtext
TextualETL
Standardanalytica ltechnology
and once a standard relational data base is created, there areliterally hundreds of analytical tools that can operate againstthe data base
Unstructuredtext
TextualETL
Let m e go findsom e raw text
Let m e look at data organized by any or all of -
- tone of m essage - in ternational dom estic flight - date - lost luggage - upgrades - personnel - c ity - arrival tim e - carrier - ..................
I think your airline is the best that I have flown on. It is certainly as good as Singapore Air, which sets the standard for quality.
Plane was late. Messed up my schedule. Airline sucks.
Flt 462 DNV to ORD lst bgs CN 235-PO-908Merci pour le service merveilleux sur le vol 232 de Washington a Paris. L’hostesse de l’air vraiment alle de sa facon de faire du vol confortable et inoubliable.
Vuelo 398 desde El Paso hasta el Cabo de miercoles 18 Noviembre de 1998 fue el vuelo mas miserable he tenido servicio fue terrible, ya era tarde, y me perdi miconexion a lineas Cunard. Creo que voy a demandar.
U Bill Inmon 00015427302 LAX-SYD flt 4634 Oct 26 2011 c-b st 16Ca
Dear UnitedI did not appreciate being moved from First class to coach involuntarily on flight 216 Denver to San FranciscoNov 5, 2011. I paid for first class and I want a refund.
Am 14 juni 2010 flog sie mein hund Fluffy von Munchen nach New York. Fluffy kam wohlbehalten vielen dank fur die betruung von fluffy
if you are ananalyst, what kindof data do youwant to operateon