WebNLP – An Integrated Web-Interfacefor Python NLTK and Voyant

Manuel Burghardt, Julian Porsch, Bianca Tirlea, & Christian WolffMedia Informatics Group, University of Regensburg

{manuel.burghardt,christian.wolff}@ur.de{julian.poersch,bianca.tirlea}@stud.uni-regensburg.de

Abstract

We present WebNLP, a web-based toolthat combines natural language process-ing (NLP) functionality from Python NLTKand text visualizations from Voyant in anintegrated interface. Language data can beuploaded via the website. The results ofthe processed data are displayed as plaintext, XML markup, or Voyant visualiza-tions in the same website. WebNLP aimsat facilitating the usage of NLP tools forusers without technical skills and experi-ence with command line interfaces. It alsomakes up for the shortcomings of the pop-ular text analysis tool Voyant, which, upto this point, is lacking basic NLP featuressuch as lemmatization or POS tagging.

1 Introduction

Modern corpus linguistics has been on the risesince the late 1980s (Hardie, 2012), largely be-cause of the availability of vast amounts of dig-ital texts and computer tools for processing thiskind of data. Since then, corpus linguistics hasproduced a number of important subfields, suchas web as a corpus (cf. Kilgarriff and Grefen-stette, 2003; Baroni et al., 2009), language in thesocial media (cf. Beißwenger and Storrer, 2009)or using language data for sentiment and opinionmining (cf. Pak and Paroubek, 2010). More re-cently it has been claimed that the mass of dig-

This work is licensed under a Creative Commons At-tribution 4.0 International License (CC BY 4.0). Pagenumbers and proceedings footer are added by the orga-nizers. License details: http://creativecommons.org/licenses/by/4.0/

ital text available for automatic analysis consti-tutes a new research paradigm called culturomics(Michel et al., 2010) and that the recent arrival ofthe digital humanities opens up additional fieldsof application for corpus linguistics and text min-ing. Taking the increased amount of digital textdata which is readily available into consideration,Gregory Crane has asked the well justified ques-tion “what to do with a million books” (Crane,2006). The question is partially answered byMoretti (2013), who introduces the idea of dis-tant reading of texts, as opposed to the more tradi-tional, hermeneutic close reading, which is partic-ularly popular in the field of literary studies. Theidea of distant reading suggests to interpret liter-ary texts on a more generic level by aggregatingand analyzing vast amounts of literary data.

All these novel types of applications require ba-sic NLP analysis such as tokenization, lemmatiza-tion, POS tagging, etc. Currently, there is no lackof adequate tools than can be used to process largeamounts of text in different languages. Promi-nent examples are GATE (General Architecturefor Text Engineering)1 or the UIMA framework(Unstructured Information Management Infras-tructure)2. However, most of these tools can becharacterized as having a fairly high entry bar-rier3, confronting non-linguists or non-computerscientists with a steep learning curve, due to the

1Available at https://gate.ac.uk; all web re-sources described in this article were last accessed on May4, 2014.

2Available at http://uima.apache.org3Hardie (2012) gives a short overview of the develop-

ment of corpus analysis tools while at the same time dis-cussing their usability requirements.

fact that available tools are far from offering asmooth user experience (UX). This may possiblybe caused by complex interaction styles typicallyencountered in command line interfaces, by sub-optimal interface design for graphical user inter-faces (GUIs) or by the necessity of bringing to-gether disparate tools for a specific task.

Nowadays, a decent UX is a basic requirementfor the approval of any application such as of-fice tools or smartphone apps (Nielsen and Budiu,2013). At the same time, a large and well ac-cepted body of knowledge on usability and usercentered design (cf. Shneiderman, 2014) is at ourdisposal. However, tools developed for scientificpurposes like corpus linguistics or text mining donot seem to take advantage of these knowledgesets: It appears that many tools are designed byscientists who may have acquired the necessaryprogramming and software engineering skills, butwho are lacking experience and training in userinterface design and usability engineering. As aresult, many tools are functionally perfect, but anobvious mess as far as usability aspects are con-cerned.

In the following, we will not introduce yet an-other tool, but we rather try to provide an inte-grated, easy-to-use interface to existing NLP andtext analysis tools.

2 Tools for NLP and text analysis

There are a number of available tools that can beused for NLP tasks and quantitative text analy-sis (cf. the notion of distant reading). This sec-tion introduces some of the most prominent tools,and also makes the case for the newly createdWebNLP prototype.

2.1 Python NLTK

Python NLTK4 (Bird, 2006) is a widely usedtoolkit that allows the user to perform sophisti-cated NLP tasks on textual data and to visual-ize the results. One drawback of NLTK, how-ever, is its command line interface. Also, a ba-sic understanding of the programming languagePython is necessary for using it. Depending onthe target platform, setting up the NLTK environ-ment can be rather cumbersome. For these rea-

4Available at http://www.nltk.org/

sons, many humanities scholars who are lackingtechnical skills in Python and command line in-terfaces may refrain from using NLTK as a meansfor NLP.

2.2 TreeTagger

TreeTagger5 (Schmid, 1994), another widely usedNLP tool, tries to address this issue by providinga GUI (only available for Microsoft Windows)6.The output of the tool can however not be visual-ized in the same GUI.

2.3 Voyant Tools

Voyant7 (cf. Ruecker et al., 2011) is a web-basedtool that is very popular in the digital humanitiescommunity. It allows the user to import text doc-uments and performs basic quantitative analysisof the data (word count, term frequency, concor-dances, etc.). The results of this analysis are vi-sualized in the browser, e.g. as KWIC lists, wordclouds or collocation graphs. While the tool iseasy to use via a modern web browser, Voyant islacking a feature to perform basic NLP operations(e.g. lemmatization) on the data before it is ana-lyzed.

2.4 The case for WebNLP

It shows that many of the existing tools are ei-ther not accessible to non-technical users due totheir technical complexity, or that they are lack-ing important functionality. The goal of this workis to provide an easy-to-use interface for the im-port and processing of natural language data that,at the same time, allows the user to visualize theresults in different ways. We suggest that NLPand data analysis should be combined in a sin-gle interface, as this enables the user to experi-ment with different NLP parameters while beingable to preview the outcome directly in the visu-alization component of the tool. We believe thatthe immediate visualization of the results of NLPoperations makes the procedure more transparentfor non-technical users, and will encourage themto utilize NLP methods for their research.

5Available at http://www.cis.uni-muenchen.de/˜schmid/tools/TreeTagger/

6Available at http://www.smo.uhi.ac.uk/˜oduibhin/oideasra/interfaces/winttinterface.htm

7Available at http://voyant-tools.org/

Figure 1: WebNLP architecture and main components.

In order to achieve this goal, we integrate twoexisting tools (Python NLTK and Voyant) in acombined user interface named WebNLP8.

3 WebNLP

In this section we describe the basic architectureof WebNLP and explain the main functions andinterface components of the tool.

3.1 Tool architectureWe decided to implement the interface as a webservice for several reasons:

• No installation or setup of Python NLTK andrelated Python modules by the user is re-quired.

• Previous experience and familiarity of non-technical users with web services and inter-active elements such as form fields, radiobuttons, etc.

8WebNLP is currently available as a prototype athttp://dh.mi.ur.de/

• Seamless integration of the existing web toolVoyant, which allows the user to quicklyanalyze and visualize language data in thebrowser.

• Opportunities for future enhancements of thetool, e.g. collaboration with other users,sharing of data and results, etc.

WebNLP uses a client-server architecture to pro-vide an easy-to-use interface via modern webbrowsers, while the NLP functions are executedon our server (cf. Figure 1). The interface onthe client side is structured in three main areas(cf. Figure 2) which will be explained in moredetail in the next section. All interface logic isimplemented by means of JavaScript, the pagelayout utilizes a template from the popular front-end framework Bootstrap9. The communicationbetween client and server is realized by means ofPHP and AJAX.

9Bootstrap is available at http://getbootstrap.com/.

Figure 2: WebNLP interface with three main areas: input, options, results.

A number of Python NLTK scripts (e.g. fortokenization, lemmatization, etc.) can be calledfrom the client interface and are then executedon the server. The results are displayed on theclient side by calling different visualization formsof the web service Voyant, which is embedded inthe WebNLP interface as an HTML iframe. Atthe same time, the NLTK processed data is storedon the server as plain text or as text with XMLmarkup, which are both available for downloadon the client side.

3.2 Input: Upload of natural language data

The input field allows the user to upload text doc-uments to the NLP application on the server. Datamay either be entered directly in the text area formfield, or by making use of the file upload dialog.Currently, only files in plain text format (.txt) canbe processed by the NLTK tools on our server.Another restriction for the current implementa-tion of the tool is concerned with the language ofthe text documents: At the moment, only NLTKscripts for processing English language data havebeen integrated into the tool. However, the sys-tem architecture is designed in a modular fash-ion that allows the administrators to add moreNLTK scripts for other languages at a later point

in time. Once the data has been uploaded to theserver, a first NLTK pre-processing of the data isexecuted, analyzing the overall number of tokens,types and sentences in the file. This informationis displayed at the bottom of the input area afterthe upload of the file has been completed.

3.3 Options: NLP and visualizationparameters

The second area in the interface contains optionsfor the NLP and visualization of the uploadeddata. The first set of options selects Python NLTKscripts on the server, that are then executed on thedata. In the current tool version, the followingmain functions are available:

• Stop word filter; can be combined with anyother parameter (a list of all stop words maybe looked up in the interface)

• Tokenizer (words and punctuation marks)

• Part of speech tagger (tokenization implied)

• Lemmatizer (tokenization implied)

• No NLP (used if no additional NLP process-ing is needed)

The second group of options allows the user toselect a visualization style for the processed datafrom Voyant. The following visualization10 op-tions are available in the current WebNLP proto-type:

• Wordcloud

• Bubblelines

• Type frequency list

• Collocation clusters

• Terms radio

• Scatter plot

• Type frequency chart

• Relationships

• No visualization

Due to the internal NLP workflow on the server,currently only one NLP and one visualization op-tion can be selected at a time. We are planningto implement a more flexible solution in the nextversion of WebNLP.

A short evaluation with a sample of five textdocuments with different file sizes indicates an al-most linear increase of processing time related totext size. The smallest of the test documents had asize of 50 kB (approx. 11.000 tokens), the largestdocument had a size of 4230 kB (approx. 920.000tokens). POS tagging for the smallest documenttook 18 seconds, lemmatization took 20 seconds.For the largest document, POS tagging took ap-prox. 24 minutes, lemmatization took approx. 25minutes. These results indicate that WebNLP inits current implementation is well-suited for smallto medium sized corpora, but may be too slow forlarger text collections.

3.4 Results: Client-side visualizations anddownload formats

The third interface area displays the results of thechosen NLP options in the selected Voyant visual-ization (e.g. word cloud view). The user may also

10A detailed description of the different Voyant visualiza-tion types can be found at http://hermeneuti.ca/voyeur/tools.

switch to plain text or XML markup view of theresults (these formats are also available for down-load).Plain text view (original NLTK output):

( VBN , come )...

XML view (custom WebNLP format):

<root><token>

<pos>VBN</pos><word>come</word>

</token>...

</root>

4 Conclusions

Our tool provides access to existing NLP and vi-sualization tools via a combined interface, thusacting as a GUI wrapper for these applications.While a thorough usability evaluation is stillmissing, we are confident that NLP functionalityfrom the Python NLTK becomes more accessiblethrough WebNLP, and that the combination withvisualizations from the Voyant set of tools willbe attractive for many applications of text tech-nology. In its current implementation, WebNLPshould be treated as a prototype that illustrateshow a web-based interface to basic NLP and textvisualization functions can be realized by meansof standard web technologies. We are, however,planning to implement more NLTK functions,and to improve the performance as well as the in-terface of the service in the future.

References

Marco Baroni, Silvia Bernardini, Adriano Fer-raresi, and Eros Zanchetta. The wacky wideweb: a collection of very large linguisticallyprocessed web-crawled corpora. Language re-sources and evaluation, 43(3):209–226, 2009.

Michael Beißwenger and Angelika Storrer. Cor-pora of Computer-Mediated Communication.In Anke Ludeling and Kyto Merja, editors,Corpus Linguistics. An International Hand-book, pages 292–308. Mouton de Gruyter,Berlin, New York, 2009.

Steven Bird. Nltk: the natural language toolkit.In Proceedings of the COLING/ACL on Inter-active presentation sessions, pages 69–72. As-sociation for Computational Linguistics, 2006.

Gregory Crane. What do you do with a millionbooks? D-Lib Magazine, 12(3), 2006.

Andrew Hardie. Cqpweb – combining power,flexibility and usability in a corpus analysistool. International Journal of Corpus Linguis-tics, 17(3):380–409, 2012.

Adam Kilgarriff and Gregory Grefenstette. In-troduction to the special issue on the web ascorpus. Computational linguistics, 29(3):333–347, 2003.

Jean-Baptiste Michel, Yuan Kui Shen, Aviva P.Aiden, Adrian Veres, Matthew K. Gray, TheGoogle Books Team, Joseph P. Pickett, DaleHoiberg, Dan Clancy, Peter Norvig, Jon Or-want, Steven Pinker, Martin A. Nowak, andErez Lieberman Aiden. Quantitative analysisof culture using millions of digitized books.Science, 331(6014):176–182, 2010.

Franco Moretti. Distant reading. London: Verso,2013.

Jakob Nielsen and Raluca Budiu. Mobile usabil-ity. New Riders, Berkeley, CA, 2013.

Alexander Pak and Patrick Paroubek. Twitter asa Corpus for Sentiment Analysis and OpinionMining. In Proceedings of the LREC, pages1320–1326, 2010.

Stan Ruecker, Milena Radzikowska, and StefanSinclair. Visual interface design for digital cul-tural heritage: A guide to rich-prospect brows-ing. Ashgate Publishing, Ltd., 2011.

Helmut Schmid. Probabilistic part-of-speech tag-ging using decision trees. In Proceedings of in-ternational conference on new methods in lan-guage processing, pages 44–49. Manchester,UK, 1994.

Ben Shneiderman. Designing the user interface:strategies for effective human-computer inter-action. Pearson, 5th edition, 2014.

!!!!!!!!!!!!!!!!

Web

NLP

– A

n In

tegr

ated

Web

-Inte

rfac

e fo

r Py

thon

NLT

K a

nd V

oyan

t

1. U

ploa

d na

tura

l lan

guag

e da

ta to

th

e se

rver

. 2.

Set

NLP

(Pyt

hon

NLT

K) a

nd

visu

aliz

atio

n (V

oyan

t) pa

ram

eter

s.

3. V

isual

ize

resu

lts in

the

brow

ser o

r dow

nloa

d th

em

as p

lain

text

/ xm

l file

s.

Man

uel B

urgh

ardt

, Jul

ian

Pörs

ch, B

ianc

a Ti

rlea,

and

Chr

istia

n W

olff

Med

ia In

form

atic

s an

d In

form

atio

n Sc

ienc

e G

roup

, Uni

vers

ity o

f Reg

ensb

urg

Try

out

Web

NLP

: htt

p://d

h.m

i.ur.d

e/

Cont

act:

man

uel.b

urgh

ardt

@ur

.de

D

igit

al H

uman

itie

s Re

gens

burg

: htt

p://d

hreg

ensb

urg.

de

Abs

trac

t –

We

pres

ent W

ebN

LP, a

web

-bas

ed to

ol th

at

com

bine

s na

tura

l lan

guag

e pr

oces

sing

(NLP

) fun

ctio

nalit

y fr

om P

ytho

n N

LTK

and

text

visu

aliz

atio

ns fr

om V

oyan

t in

an in

tegr

ated

inte

rfac

e. L

angu

age

data

can

be

uplo

aded

vi

a th

e w

ebsit

e. T

he re

sults

of t

he p

roce

ssed

dat

a ar

e di

spla

yed

as p

lain

text

, XM

L m

arku

p, o

r Voy

ant v

isual

i-za

tions

in th

e sa

me

web

site.

Web

NLP

aim

s at

faci

litat

ing

the

usag

e of

NLP

tool

s fo

r use

rs w

ithou

t tec

hnic

al s

kills

an

d ex

perie

nce

with

com

man

d lin

e in

terf

aces

. It a

lso

mak

es u

p fo

r the

sho

rtco

min

gs o

f the

pop

ular

text

an

alys

is to

ol V

oyan

t, w

hich

, up

to th

is po

int,

is la

ckin

g ba

sic N

LP fe

atur

es s

uch

as le

mm

atiz

atio

n or

PO

S ta

ggin

g.