topic modeling with spark

OPEN DATA SCIENCE CONFERENCE

Boston | May 20-22, 2016

Natural Language Processing and Topic Modeling with Spark

Frank D. Evans

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery. But recovery is not enough. If we are to prevail in the long run, we must expand the long-run strength of our economy.

america enjoy twenty-two month uninterrupted economy recovery recovery not enough prevail long run expand long-run strength economy

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery. But recovery is not enough. If we are to prevail in the long run, we must expand the long-run strength of our economy.

america enjoy twenty-two month uninterrupted economy recovery recovery not enough prevail long run expand long-run strength economy

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery in car sales.

After the car accident, he made a heroic full recovery.

america enjoy twenty-two months uninterrupted economic recovery car sales

After car accident made heroic full recovery

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery in car sales.

After the car accident, he made a heroic full recovery.

(america has), (has enjoyed), (enjoyed twenty-two), (twenty-two months), (months of), (of uninterrupted), (uninterrupted, economic), (economic recovery), (recovery in), (in car), (car sales)

(after the), (the car), (car accident), (accident he), (he made), (made a), (a heroic), (heroic full), (full recovery)

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery in car sales.

After the car accident, he made a heroic full recovery.

(america has), (has enjoyed), (enjoyed twenty-two), (twenty-two months), (months of), (of uninterrupted), (uninterrupted, economic), (economic recovery), (recovery in), (in car), (car sales)

(after the), (the car), (car accident), (accident he), (he made), (made a), (a heroic), (heroic full), (full recovery)

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery in car sales.

After the car accident, he made a heroic full recovery.

(america has), (has enjoyed), (enjoyed twenty-two), (twenty-two months), (months of), (of uninterrupted), (uninterrupted, economic), (economic recovery), (recovery in), (in car), (car sales)

(after the), (the car), (car accident), (accident he), (he made), (made a), (a heroic), (heroic full), (full recovery)

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery in car sales.

After the car accident, he made a heroic full recovery.

Raw Text Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery in car sales.

After the car accident, he made a heroic full recovery.

america enjoy twenty-two months uninterrupted [economic recovery] car sales

After [car accident] made heroic full recovery

Frequency Merge

Raw Text Wrangle Model Extract Visualize

Fed Chairman Ben Bernanke led from Washington with the help of the bank's current $3.6T balance sheet. He's accompanied by Mario Draghi at the European Central Bank.

Raw Text Wrangle Model Extract Visualize

Fed Chairman Ben Bernanke led from Washington with the help of the bank's current $3.6T balance sheet. He's accompanied by Mario Draghi at the European Central Bank.

E:”Ben Bernanke”(Person) E:”Washington”(City) E:”Mario Draghi”(Person, Politician) E:”Federal Reserve”(Organization) E:”United States”(Country) E:”European Central Bank”(Organization) E:”central banks”(terminology) C:”monetary policy” C:”economics”

Entity/Concept Recognition

exaptive.com/blog

Frank D. Evans@frankdevans

@exaptive

slideshare.net/frankdevansgithub.com/frankdevans

Raw Text Wrangle Model Extract Visualize