solving the unstructured data problem – case study the unstructured data problem – case study...

© 2017 IBM Corporation

Solving the Unstructured Data Problem – Case Study

Subodh Manicka, Exec IT Architect, IBM Corporation

Randall Ho, Senior IT Architect, IBM Corporation

Function of a Government

• A primary function of government is to protect citizens from harm

� Personal safety and security

� Unfair trade practice

� Etc.

• We expect our government to

� Keep the citizens safe from bad people

� Regulate automobile and consumer product safety

� Regulate drug and medical device safety

� Regulate medical device safety

� Regulate unfair trade practice

1

How does the Government address this?

• Through Laws and Regulations

• Investigating safety issues or concerns reported by citizens

• Defending the country from enemies of the state

2

3

Headlines Related to Automobile Safety

4

Headlines Related to Consumer Product Safety

5

Headlines Related to Citizens Comments to Regulations

6

Impact to Business

Examples of Government Agencies• USA

� Federal Communications Commission

� Consumer Product Safety Commission

� National Highway Traffic Safety Agency

� Food and Drug Administration

• Canada

� Canadian Radio-television and Telecommunications Commission

� Office of Consumer Affairs

� Transport Canada’s Motor Vehicle Safety Directorate

� Health Canada

• Australia

� Australian Communications and Media Authority

� Australian Competition and Consumer Commission

� Vehicle Safety Standards Branch, Department of Infrastructure and Regional Development

� ACPM, ARGMD, Therapeutic goods Administration

7

• UK

� Office of Communication

� Office of Fair Trading

� Driver and Vehicle Standards Agency

� Medicines & Healthcare products Regulatory Agency

• Germany

� Federal Network Agency (Bundesnetzagentur)

� Verbraucherschutzminister

� Federal Ministry of Transport, Building and Urban Development

� Federal Institute for Drugs and Medical Devices

• Austria

� Austrian Regulatory Authority for Broadcasting and Telecommunications

� Federal Ministry of Labour, Social Affairs and Consumer Protection

� Austrian Ministry for Transport, Innovation and Technology

Process and Action

• Citizens discover safety issue or other concerns

• Citizens voice their concern by submitting complaints

• Citizens provide feedback on upcoming regulations

• To fulfil its mission, government agencies analyze data that they receive and from various sources including citizens and manufactures

• Following investigation, government agencies act on the concern through product recall or change in regulations

8

Sample Forms

US Automobile – Safecar.gov

9

Sample Forms

UK Automobile Report Form

10

Sample Forms

Australia Automobile

11

Sample Forms

US Consumer Product Safety

12

Sample Forms

Australian Consumer Product Safety

13

Sample Forms

14

Sample Forms

15

Challenges with the Data

16

• Data volume fluctuates with significant spikes at times

• Citizens misspelling words

• Information is typically a mix of structured data and unstructured text

• Meaning of words may depend on context

� Examples:

• Lead – the element lead in paint that is poisonous or the government regulation that leads to higher consumer cost.

• “Engine caught fire” vs. “Fire department suspects transmission fuel leak”

• Fruit flies like banana

• “Shot” a bear – with a camera vs with a gun

The Architecture

18

IT Context Diagram

Cognos BI

Complaints / Issues

Unstructured Data

Analytics

Data Warehouse

1

3

2

4

Case Solution

Solution Software Stack

• IBM Watson Explorer Advanced Edition (Content Analytics Components)

• IBM Datacap (Path #1)

• IBM Content Collector (Path #1)

• IBM Content Classification

• IBM PureData System (Path #2)

• IBM Case Manager (Path #3)

• Cognos BI (Path #4)

19

Logical Architecture of Solution

20

Enhancements with Watson Services

• Data Ingest Challenges

� Wide variety of formats – text, image, audio etc.

• Watson APIs

� Speech to Text

� Visual Recognition

� Language Translator

� Natural Language Classifier

� Natural Language Understanding

21

Potential Applications

• Potential areas of application

� Citizen Services

• Consumer product safety (Ex: CPSC)

• Automobile safety (Ex: NHTSA)

• Drug and Medical devices Safety (Ex: FDA)

• Health and Medical Research (Ex: NIH)

• Enterprise unstructured data processing

� Analyzing comments on regulations (Ex: FCC)

� Seized media analysis (Ex: Intel and Defense)

• Any place where volumes of unstructured or semi structured contents need to be analyzed for insight

22

23