Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 314–323,Seoul, South Korea, 5-6 July 2012. c©2012 Association for Computational Linguistics

Markov Logic Networks forSituated Incremental Natural Language Understanding

Casey Kennington David Schlangen

CITEC Dialogue Systems Group and Faculty of Linguistics and Literary StudiesUniversitat Bielefeld, Bielefeld, Germany

ckennington@cit-ec.uni-bielefeld.dedavid.schlangen@uni-bielefeld.de

Abstract

We present work on understanding natural lan-guage in a situated domain, that is, languagethat possibly refers to visually present enti-ties, in an incremental, word-by-word fashion.Such type of understanding is required in con-versational systems that need to act immedi-ately on language input, such as multi-modalsystems or dialogue systems for robots. Weexplore a set of models specified as MarkovLogic Networks, and show that a model thathas access to information about the visual con-text of an utterance, its discourse context, aswell as the linguistic structure of the utter-ance performs best. We explore its incremen-tal properties, and also its use in a joint pars-ing and understanding module. We concludethat MLNs offer a promising framework forspecifying such models in a general, possiblydomain-independent way.

1 Introduction

We speak situated in time and space. Speech by ne-cessity unfolds sequentially in time; and in a conver-sation, all speech but that of the opening utterance ispreceded by other speech belonging to the same con-versation. In many, if not most, conversational situa-tions speaker and addressee are co-located in space,and their speech may refer to their shared situation.

Most current spoken dialogue systems attempt toabstract from this fact, however. They work in do-mains where physical co-location is not necessary,such as information look-up, and they quantize timeinto discrete turn units by endpointing utterances

(see discussion in (Aist et al., 2007; Schlangen andSkantze, 2009)).

In this paper we present our current work on over-coming these abstractions for the task of natural lan-guage understanding (NLU). We have created a sta-tistical model that can be trained on conversationaldata and which can be used as an NLU module foran incremental, situated dialogue system (such asthat described in (Buß et al., 2010)). We show thatthis model beats baseline approaches by a wide mar-gin, and that making available the full set of infor-mation comprising visual context, discourse context,and linguistic structure gives significantly better re-sults than any subset of these information sources ontheir own.

The paper is structured as follows: we first dis-cuss related work and introduce some background,and then describe in detail our set of experiments,and present and analyse our results. We close with ageneral discussion of this work and possible futureextensions.

2 Related Work and Background

The work in this paper builds on, connects and ex-tends several strands of research: grounded seman-tics (Roy, 2005), which worries about the connec-tion between language and the situation in whichit is used, but often does not go beyond the wordlevel to include linguistic structure information anddoes not work incrementally;1 statistical NLU (seee.g. (Zettlemoyer and Collins, 2009; Liang et al.,

1But see (Spranger et al., 2010); for recent attempts that par-tially overcome these limitations.

314

2011)), which tries to infer linguistic structures au-tomatically, but normally stops at generating, not in-terpreting semantic representations, and works with(the text of) full utterances and not incrementally onspeech data; and incremental NLU, which is a lessintensely studied field, but where previous contri-butions (such as (DeVault et al., 2009; Devault etal., 2011; Aist et al., 2007; Schlangen and Skantze,2009)) have not dealt with learned grounded seman-tics.

We go beyond this earlier work in that we studya model that is incremental, can use linguistic struc-ture, and learns from conversational data a semanticsthat connects the utterance to its visual and discoursecontext. We have looked at individual componentsof this before (grounded semantics in (Siebert andSchlangen, 2008); incremental reference resolutionin (Schlangen et al., 2009); incremental general NLU

in (Heintze et al., 2010); interaction between incre-mental parsing and reference resolution in (Peldszuset al., 2012)), but use a more sophisticated model inthis work and show that tackling these tasks jointlyimproves performance.

MLNSystem

Context/World

Language/RMRS

Context/Discourse

Prediction:actionobjectresult

Figure 1: NLU Data Flow

We apply Markov Logic Networks (MLNs,(Richardson and Domingos, 2006)) as the machinelearning technique in our experiments. MLNs haverecently received attention in language processingfields like co-reference resolution (Chen, 2009), se-mantic role labeling (Meza-Ruiz and Riedel, 2009),spoken (albeit neither situational nor incremental)NLU (Meurs et al., 2008), and web information ex-traction (Satpal et al., 2011). The framework of-fers a convenient way of specifying factor functionson sets of random variables for undirected graphicalmodels (Markov Random Fields, see (Kindermannand Snell, 1980)), in such a way that the factorscorrespond to weighted first order formulae and thejoint distribution of random variables corresponds to

probabilities of groundings of formulae. In this way,MLNs offer a helpful bridge between symbolic rep-resentation and stochastic inference. Weights of for-mulae can be specified by hand or learned from data;we used the latter capability.

Figure 1 shows data flow in our task. We use com-binations of situated context, previous context, andlinguistic information as evidence to an MLN, andinfer what action is to be taken, what object is to beacted upon, and specifications of the manner of exe-cution.

3 Experiments

We will now describe our experiments with usingMarkov Logic Networks for situated incrementalnatural language understanding.

3.1 Data and TaskFor our experiments, we used task-oriented con-versational data from the Pentomino domain(Fernandez et al., 2007); more specifically, weworked with the corpus also used recently in(Heintze et al., 2010) and (Peldszus et al., 2012).This corpus was collected in a Wizard-of-Oz study,where the user goal was to instruct the computer topick up, delete, rotate or mirror puzzle tiles on arectangular board (as in Figure 2), and place themonto another one. For each utterance, the corpusrecords the state of the game board before the utter-ance, the immediately preceding system action, andthe intended interpretation of the utterance (as un-derstood by the Wizard) in the form of a semanticframe specifying action-type and arguments, wherethose arguments are objects occurring in the descrip-tion of the state of the board. The language of thecorpus is German.

Figure 2: Example Pentomino Board

For this study, we were interested in the potentialcontribution of linguistic structure to the NLU task.

315

To this end, we produced for each utterance an in-cremental sequence of parses and corresponding se-mantic representations (as RMRS structures (Copes-take, 2007), i.e. underspecified semantic representa-tions), using the parser described in (Peldszus et al.,2012). These representations were not further man-ually checked for appropriateness, and hence do notnecessarily represent ground truth.

As in (Peldszus et al., 2012), we discarded ut-terances without clear semantic alignments. Onemajor difference from them is that we do includethe 661 utterances that used pronouns to refer topieces, leaving us with 1687 utterances, 5.43 wordsper utterance (sd 2.36), with a vocabulary of 237 dis-tinct words. These were transcribed utterances andnot automatic speech recognition output, so our re-sults represent an upper-bound on real world perfor-mance.

The task that we wanted our model to tackle canthen be stated as follows: given information aboutthe current state of the world (i.e., the game board),the previous system action, and about the (possiblystill not-yet completed) utterance, predict an inter-pretation for the utterance, in the form of such aframe. The elements of the frame may be speci-fied separately; as argued in (Heintze et al., 2010),this is the most appropriate format for incrementalprocessing since it provides a rough alignment be-tween parts of the utterance and parts of its inter-pretation. Figure 3 illustrates such a desired outputfrom the model. In more general terms, what wewant our model to learn then is how, in a given dis-course context, language connects to the world. Toexplore what information contributes to this, we willsystematically vary in our experiments what is avail-able to the learner.

3.2 Representation

As mentioned above, Markov Logic allows the spec-ification of knowledge bases through first order for-mulae. A straightforward representation of the gameboard would simply assert salient properties of theindividual objects such as their colour, shape, po-sition, etc.; for the topmost object in Figure 2 thiscould be colour(yellow) ∧ shape(g) ∧ pos(2, 1).However, in pre-experiments on held-out data, wefound that a more parsimonious representation ac-tually worked better, in which there is only one

n word interpretation1 rotate action:rotate2 the ...3 yellow argument:yellow objects4 piece argument:yellow pieces5 next ...6 to ...7 the ...8 yellow argument:yellow pieces

by yellow objects9 plus argument:yellow piece

next to unique yellow plus10 clockwise option:clockwise

Figure 3: Incremental interpretation of a 10-word utter-ance. Only changes to the frame are shown, e.g. whenpredictions about different frame elements are made. Forillustration, sets of objects are represented by descrip-tions; in the system, these would be sets of object identi-fiers.

abstract property that only implicitly does a typ-ing into different features of the objects; again, forthe topmost piece from the figure this would bepiece(p)∧ property(p, yellow)∧ property(p, g)∧property(p, row0)∧property(p, col1). This repre-sentation follows a Davidsonian form of represent-ing the relations between predicates.

The properties of the objects that we representedin this way were colour, shape, its row and column,horizontal percentage from the center and verticalpercentage from the center.

The utterance itself forms another source of in-formation about the situation. In the simplest form,it could be represented just through assertions ofthe words which are part of it, e.g. word(rotate) ∧word(the) ∧ word(yellow) ∧ . . . . As mentionedabove, we were interested in whether a more de-tailed linguistic analysis could provide more usefulinformation to a model of situated semantics; werepresented this information by extracting some ofthe relations of the RMRS representation for each ut-terance (-prefix) and converting them to a slightlysimpler form. Figure 4 shows the RMRS representa-tion of an example utterance and the correspondingsimplified representation that we derive from it (la-bels as defined by RMRS and quotes required by andthe MLN are removed for simplicity). We representwords as RMRS EPs (elementary predicates); i.e., by

316

their lemma and with additional identifiers as argu-ments, which can be used to relate the EP to otherRMRS structure. In the variants of the model thatonly look at words, the other arguments can sim-ply be ignored in the MLN template. The final ar-gument for EP is the board identifier, which remainsunchanged during an utterance.

RMRSa33:yellow(e34)a19:NN(x14)ARG1(a49,x14)ARG2(a49,x53)a49:nextto(e50)BV(a52,x53)RSTR(a52,h60)BODY(a52,h61)a52:def()ARG1(a72,x53)a72:yellow(e73)a58:plus(x53)

MLNEP(a33,yellow,e34,1)EP(a19,NN,x14,1)RMRS(ARG1,a49,x14,1)RMRS(ARG2,a49,x53,1)EP(a49,nextto,e50,1)RMRS(BV,a52,x53,1)EP(a52,def,,1)RMRS(ARG1,a72,x53,1)EP(a72,yellow,e73,1)EP(a58,plus,x53,1)

Figure 4: RMRS and MLN for yellow piece next to theyellow plus

Finally, the previous system action and, duringlearning but not testing, the interpretation that isto be predicted needs to be represented. This isdone through predicates action(), argument() andoption() for the interpretation of the current utter-ances and corresponding predicates for that of theprevious one.

To summarise, each problem instance is hencerepresented as a conjunction of predicates encodinga) the (world) situational context (the state of thegame board), b) the discourse context (in the formof the previous action), and c) the (possibly as-yetpartial) utterance, linguistically analysed.

3.3 Model and Decision RuleThe actual model is now formed by the MLN tem-plates that specify the relations between the predi-cates; in particular those between those representingthe available information (evidence) and the predi-cates that represent the information that is to be pre-dicted (or, in MLN terminology, whose most likelyvalues are to be inferred). Figure 5 illustrates graph-ically how our model makes these connections, sep-arately for each frame element that is to be predicted.

These graphs show that for action andoption, we assume an influence both of the words

Action

Argument

Option

PrevAction PrevOptionEP

RMRS

Property

EP

EP

Property

EP

Figure 5: MLN relations between predicates

present in the utterance (denoted by EP; see above)and of the previous value of these slots on the cur-rent one. The previous context that is used for train-ing and evaluation is taken from the corpus anno-tation files. The structure for argument is some-what more complicated; this is where the linguis-tic information coming from the RMRSs comes intoplay, and also where the connection between lan-guage and properties of the visual scene is made.The actual template that defines our MLN is shownin Figure 6.

1 EP (a1, a2,+w, a3, b) ⇒ Action(+a, b)2 PrevAction(+a, b) ⇒ Action(+a, b)3 EP (a1, a2,+w, a3, b)) ⇒ Option(+o, b)4 PrevOption(+o, b) ⇒ Option(+o, b)5 EP (a1, a2,+w, a3, b)) ∧ Property(p,+pr, b)

⇒ Argument(p, b)6 EP (a1, a2, w1, a3, b) ∧RMRS(+t, a4, a3, b)

∧RMRS(+t, a4, a5, b) ∧ EP (a5, a6, w2, a5, )∧Property(p,+pr, b) ⇒ Argument(p, b)

Figure 6: The MLN template specifying our model

Our MLN system gives us probability distribu-tions over all possible groundings of the frame pred-icates, but as we are interested in single best candi-dates (or the special value unknown, if no guesscan be made yet), we applied an additional deci-sion rule to the output of the MLN component. Ifthe probability of the highest candidate is below athreshold, unknown is returned, otherwise that can-didate is returned. Ties are broken by random se-lection. The thresholds for each frame element /predicate were determined empirically on held-outdata so that a satisfactory trade-off between lettingthrough wrong predictions and changing correct re-

317

Type Class Acc.Action majority put 33.55Argument majority tile-3 20.98Option majority na 27.08Frame majority take, tile-3, na 3.67Action Contextual 42.24

Table 1: Majority class and Action contextual baselines

sults to unknown was achieved.

3.4 Parameter Training Procedure, Baselines,Metrics

All results reported below were obtained by aver-aging results of a 10-fold validation on 1489 Pentoboards (i.e., utterances + context). We used a sep-arate set of 168 boards for small-scale, held-outexperiments. For learning and inference we usedthe Alchemy system (Domingos et al., 2006), us-ing the discriminative training option (Singla andDomingos, 2005).2 Inference was performed on theAction, Argument, and Option predicates; a sin-gle answer was derived from the distributions deliv-ered by alchemy in the way described in the previoussection.

To be able to assess our results, we devised twokinds of baselines for the full utterance. The sim-plest is just the majority class. Table 1 shows ac-curacy when choosing the majority class, both forthe frame elements individually (where this baselineis quite high) and for the most frequent full frame(which, unsurprisingly, only reaches a very low ac-curacy). Action can be predicted with somewhatmore accuracy if not the overall most frequent valueis chosen but that given the previous action (i.e.,when Action is conditioned on PreviousAction).The accuracy for this method, where the conditionaldistribution was determined on the 1489 boards andtested on the remaining 168 boards, is shown in theTable under “action contextual”.

We give our results below as f-score, slot accuracyand frame accuracy based on comparison to a goldrepresentation. To compute the f-score, we count aprediction of unknown as a false negative (since forour test utterance a value should always have beenpredicted) and a wrong prediction as a false posi-

2http://alchemy.cs.washington.edu/

tive; i.e., a frame with one correct slot and the rest asunknown has perfect precision, but only 1/3 recall.Slot accuracy counts the number of slots that arecorrect, and frame accuracy only counts fully cor-rect frames. Hence, these metrics are successivelymore strict. Which one most accurately predicts per-formance of the model in the context of a dialoguesystem depends on properties of the further compo-nents: if they can act on partial frames, then an f-score that start highs and continually improves as theutterance goes on is desired; if not, then what’s rel-evant is when in the utterance high frame accuracycan be reached.

Using the best model variant, we further com-pare two parsing/NLU feedback strategies, where thefeedback is to provide aid to the syntactic/RMRS

parser as to which parses to prune (as in (Peldszuset al., 2012)). If a candidate parse does not resolveto anything, then the parse score is degraded. (Peld-szus et al., 2012) use a rule-based reference resolu-tion component to provide this feedback signal. Weexplore what the effects are of exchanging this fora learned feedback strategy using our MLN model.This model, however, does not provide discrete ref-erent sets, but instead gives a probability distributionover all possible pieces. We therefore simply mul-tiplied each parse by the probability of the highestprobable piece, so that low probabilities effectivelyresult in pruning a parse.

On the incremental level, we followed Schlangenet al. (2009) by using a subset of their incrementalmetrics, with a modification on the edit overhead:first correct: how deep into the utterance do wemake the first correct guess?first final: how deep into the utterance do we makethe correct guess, and don’t subsequently change ourminds?edit overhead: ratio of unnecessary edits / sentencelength, where the only necessary edit is that goingfrom unknown to the final, correct result anywherein the sentence)

We also follow their assumption that as the sen-tence progresses incrementally, the earlier the frameprediction can be made, the better. This is an impor-tant part of our threshold decision rule, because wealso assume that no decision is better than a bad de-cision. A comparison between first correct and firstfinal would reveal how well this assumption is real-

318

W E R P FScore Slot Frame5 5 5 5 92.18 88.88 74.76 1

{86.76} {81.61} {61.21}5 5 5 81.06 72.59 34.36

{68.20} {58.61} {19.19}5 5 5 91.63 88.03 72.68 2

{86.47} {80.69} {58.18}5 5 75.44 65.72 22.555 5 5 72.29 61.61 24.565 5 18.15 12.10 0.05 5 72.34 61.67 24.635 18.32 12.21 0.0

5 5 5 90.68 85.68 63.75 4

5 5 68.94 56.26 0.05 5 90.67 85.68 63.89 3

5 69.10 56.39 0.05 5 72.29 61.61 24.565 18.15 12.10 0.0

5 72.30 61.63 24.6918.15 12.10 0.0

Table 2: Comparison of combinations using World, EPs(words), RMRS and Previous context. Number in brack-ets are for tests on automatically transcribed speech.

ized. A good model would have the two numbersfairly close together, and the prediction would bebest if both were lower, meaning good predictionsearlier in the sentence. The edit overhead furthersheds light on this distinction by showing what per-centage of the time edits were made unnecessarilythroughout a sentence.

The procedure on the incremental level is simi-lar to the full utterance procedure, except that forincremental evaluation the f-score, slot accuracy,and frame accuracies were calculated word for wordagainst the final gold representation.

3.5 Results

Since we were interested in the relative contributionsof our different kinds of information sources (visualcontext, discourse context, words, linguistic struc-ture), we trained and tested variant of the model de-scribed above that had access to only parts of the fullinformation (by removing the appropriate predicatesfrom the MLN template). We report results in Table 2for these different variants; here just as results afterthe final word of the utterance, i.e., we’re not yet

Feedback Predictor FScore Slot FrameHC HC 38.2HC Full 92.26 88.94 74.69none Full 92.18 88.88 74.76Full Full 92.29 89.01 74.96

Table 3: Feedback strategies comparison for hard-coded(HC), automatic (MLN) and no feedback (none)

looking at the incremental performance. For easierreference, some lines are indexed with their rank ac-cording to frame accuracy. The tope three lines alsocontain a bracketed entry which represents automat-ically transcribed utterances (also trained on manu-ally transcribed data as in (Peldszus et al., 2012)).

First, it should be pointed out that the full model(which has access to all information types) performsrather well, giving a fully correct interpretation for74% of all frames. As the somewhat higher f-scoreindicates, some of the loss of frame accuracy is notdue to wrong predictions but rather to staying unde-cided (choosing unknown)—a behaviour that couldbe advantageous in some applications.

The next line shows that much of the informa-tion required to reach this accuracy comes not fromthe visual context or an analysis of the language butfrom the discourse context; without access to it, ac-curacy drops to 22%. However, the advantage ofhaving access to discourse context only really comesout when access to the utterance is given as well(rows indexed with 3 and 4, and 1 and 2). The modelthat just goes by previous context can only achievean accuracy of 24%

Connecting discourse context to language aloneonly brings accuracy to around 65% (rows 3 and 4);only when the visual context is provided as well canthe best accuracy be reached. This is a pleasing re-sult, as it shows that the model is indeed capableof making the desired connection between languageand world; as none of it was not explicitly given,which words and linguistic structure linked to whichproperties was completely learned by the discrimi-native training.

For the automatically transcribed results, all ver-sions take a hit especially with regards to frame ac-curacy. These also show that previous context andlinguistic structure contribute to increased perfor-mance.

319

action 1-6 7-8 9-14first correct (% into utt.) 4.43 9.17 6.80first final (% into utt.) 29.47 31.57 28.47edit overhead 4.28argument 1-6 7-8 9-14first correct (% into utt.) 12.12 11.14 8.08first final (% into utt.) 38.26 36.10 30.84edit overhead 5.72option 1-6 7-8 9-14first correct (% into utt.) 7.62 27.75 26.73first final (% into utt.) 45.13 56.68 59.36edit overhead 13.96

Table 4: Incremental Results for Action, Argument, andOption with varying sentence lengths

3.5.1 Feedback Results

Table 3 shows the various feedback strategies. HCrefers to the hard-coded version of feedback as in(Peldszus et al., 2012). None means no feedbackwas used, which is the setting of the parser as it wasused for the RMRS structures used in Table 2. MLNrefers using our learned model to provide feedback.The column “Predictor” shows what model was usedto make the final prediction at the end of the utter-ance. Overall, MLN performed much better on pre-dicting the frame than the HC system (first row vs theother rows); but one should keep in mind that muchof that improvement is presumably due to it havingaccess to discourse context.

The last three lines show that, as (Peldszus etal., 2012) observed, providing feedback during pars-ing does offer benefits; both HC-MLN and MLN-MLN significantly improve over NONE-MLN (for f-score: one-sided t(1489) = -3.313, p-value < 0.001,and t(1489) = -3.67, p-value < 0.001, respectively;significance-level Bonferroni corrected for multiplecomparisons; similar numbers for other metrics).There was no significance when comparing HC withMLN. This is an interesting result, indicating thateven though our model performs better at accuratelypicking out referents, it provides a less useful feed-back signal. This may be due to the way we com-pute this signal; we leave further exploration to fu-ture work.

3.5.2 Incremental Results

Table 4 shows the incremental results. Rows in-volving first correct and first final represent aver-age percentage into the utterance, where the utter-ances were binned for lengths 1-6, 7-8, and 10-17(“short”, “normal”, “long” utterances, respectively).The boundaries of the bins were determined by look-ing at the distribution of utterance lengths, whichlooked like a normal distribution with 7 and 8-wordutterances having the highest representation. Ourmodel makes very early predictions (low first cor-rect), but those predictions don’t always remain sta-ble, and there is an edit overhead which leads to afinal correct decision only later in the sentence (firstfinal). For action and argument, the final deci-sion is typically made within the first third of the ut-terance. For option, it comes between the first andsecond third of the sentence; this reflects typical ut-terance structure, where the words that describe theoption (“spiegle es horizontal”; mirror it horizon-tally) usually come later in the sentence.

A final way to show incremental progress is inFigures 7 and 8 for sentences of “normal” length(7-8 words). These show how accurate the pre-diction was for each incremental step into the sen-tence, both for the model with and that without ac-cess to discourse context. Where first correct andfirst final help identify specific points in the process-ing of an utterance, for this graph each incrementalstep is compared with the gold result. Figure 8, forthe model variant without access to discourse con-text, shows that there is little impact on predictionof action or option, but a significant and con-stant impact on the quality of predicting argument(i.e., of doing reference resolution); this is due tosome extent to the presence of anaphoric referenceswhich simply cannot be resolved without access tocontext.

Taken together, the incremental statistics help de-termine an “operating point” for later modules thatconsume NLU output. Under the assumption that theongoing utterance will be one of normal length (thisof course cannot be known in advance), the strengthwith which a decision of the predictor can be be-lieved at the current point into the utterance can beread off the graphs.

Some discussion on speed efficiency: Using

320

●

●

●

●●

●

●●

●●

●●

●●

0.0 0.2 0.4 0.6 0.8

0.0

0.2

0.4

0.6

0.8

1.0

% into sentence

accu

racy

● actionargumentoptionfscore

Figure 7: incremental accuracies

●

●

●●

●●

●

●

●

●

●●

●●

0.0 0.2 0.4 0.6 0.8

0.0

0.2

0.4

0.6

0.8

1.0

% into sentence

accu

racy

● actionargumentoptionfscore

Figure 8: incremental accuracies, no discourse context

MLNs did not introduce any noticeable speed effi-ciency reduction in non-feedback models. In feed-back models which used Auto, many more callsto MLN were used, which greatly slowed down themodel.

3.6 Model AnalysisExamining the utterances that were not correctly in-terpreted, we found that words dealing with the ar-gument occured most frequently, specifically wordsinvolving spatial language where the argument wasdescribed in relation to another piece. This is some-what disappointing, as we were hoping that RMRS

structure might help learn such constructions.However, basic spatial expressions were learned

successfully, as can be illustrated by Figure 9. Itshows shows the probability distributions for the ut-terances left and bottom right, on a 5x5 board wegenerated for analysis, where each field was filledwith the same kind of piece of the same colour

(thus making these properties non-distinguishing).The darker the gradient in the Figure the higher theprobability. The Figure shows that model success-fully marks the fields closer to the left (or bottom-right, respectively) as having higher probability. In-terestingly, “left” seems to have some confusabilitywith “right” for the model, indicating perhaps thatit picked up on the general type of description (“farside”). Further investigation of model properties isleft to future work, however.

left bottom right

Figure 9: probability gradient for left and bottom right

4 Conclusions

Markov logic networks are effective in expressingmodels for situated incremental natural language un-derstanding in a domain like Pentomino. We haveshown that various aspects of situated language use,like previous context and the current state of theworld, all play a role in NLU. We have also shownthat semantic representations like RMRS can im-prove performance, and we further verified that in-cremental feedback between parser and NLU can im-prove performance (Peldszus et al., 2012). MLNsalso provide an easy-to-read trained model whichcan be easily analyzed. However, there is a trade-offin that MLNs take some time to design, which still isan intellectual task. Furthermore, inference in MLNsis still not as efficient as other methods, which cancause a slowdown in applications where very manyinference steps are required, such as the feedbackmodel.

In future work, we will further explore how to bestintegrate linguistic information from the RMRSs,specifically in spatial language; as well as look intoimprovements in speed performance. Future workwill focus on interaction with live ASR. We will alsoinvestigate using this setup for automatically trainednatural language generation.

321

Acknowledgements: Thanks to Andreas Peld-szus for help with data and to the reviewers.

References

Gregory Aist, James Allen, Ellen Campana, Car-los Gomez Gallo, Scott Stoness, Mary Swift, andMichael K. Tanenhaus. 2007. Incremental under-standing in human-computer dialogue and experimen-tal evidence for advantages over nonincremental meth-ods. In Proceedings of Decalog 2007, the 11th Inter-national Workshop on the Semantics and Pragmaticsof Dialogue, Trento, Italy.

Okko Buß, Timo Baumann, and David Schlangen. 2010.Collaborating on utterances with a spoken dialoguesystem using an isu-based approach to incrementaldialogue management. In Proceedings of the SIG-dial 2010 Conference, pages 233–236, Tokyo, Japan,September.

Fei Chen. 2009. Coreference Resolution with MarkovLogic. Association for the Advancement of ArtificialIntelligence.

Ann Copestake. 2007. Semantic composition with (ro-bust) minimal recursion semantics. In Proceedings ofthe Workshop on Deep Linguistic Processing - DeepLP’07, page 73, Morristown, NJ, USA. Association forComputational Linguistics.

D. DeVault, Kenji Sagae, and David Traum. 2009. Can Ifinish?: learning when to respond to incremental inter-pretation results in interactive dialogue. In Proceed-ings of the SIGDIAL 2009 Conference: The 10th An-nual Meeting of the Special Interest Group on Dis-course and Dialogue, number September, pages 11–20. Association for Computational Linguistics.

David Devault, Kenji Sagae, and David Traum. 2011.Incremental Interpretation and Prediction of UtteranceMeaning for Interactive Dialogue. Dialoge & Dis-course, 2(1):143–170.

Pedro Domingos, Stanley Kok, Hoifung Poon, andMatthew Richardson. 2006. Unifying logical and sta-tistical AI. American Association of Artificial Intelli-gence.

Raquel Fernandez, Tatjana Lucht, and David Schlangen.2007. Referring under restricted interactivity condi-tions. In Proceedings of the 8th SIGdial Workshop onDiscourse and Dialogue, pages 136–139.

Silvan Heintze, Timo Baumann, and David Schlangen.2010. Comparing local and sequential models for sta-tistical incremental natural language understanding. InProceedings of the 11th Annual Meeting of the SpecialInterest Group on Discourse and Dialogue, pages 9–16. Association for Computational Linguistics.

Ross Kindermann and J. Laurie Snell. 1980. Markovrandom fields and their applications. In In Practice,volume 1 of Contemporary Mathematics, page 142.American Mathematical Society.

Percy Liang, Michael Jordan, and Dan Klein. 2011.Learning Dependency-Based Compositional Seman-tics. In Proceedings of the 49th Annual Meeting ofthe Association for Computational Linguistics: Hu-man Language Technologies, pages 590–599, Port-land, Oregon. Association for Computational Linguis-tics.

Marie-jean Meurs, Frederic Duvert, Fabrice Lefevre, andRenato De Mori. 2008. Markov Logic Networks forSpoken Language Interpretation. Information SystemsJournal, (1978):535–544.

Ivan Meza-Ruiz and Sebastian Riedel. 2009. Jointlyidentifying predicates, arguments and senses usingMarkov logic. In Proceedings of Human LanguageTechnologies: The 2009 Annual Conference of theNorth American Chapter of the Association for Com-putational Linguistics on - NAACL ’09, number June,page 155, Morristown, NJ, USA. Association forComputational Linguistics.

Andreas Peldszus, Okko Buß, Timo Baumann, and DavidSchlangen. 2012. Joint Satisfaction of Syntacticand Pragmatic Constraints Improves Incremental Spo-ken Language Understanding. In Proceedings of the13th Conference of the European Chapter of the As-sociation for Computational Linguistics, pages 514–523, Avignon, France, April. Association for Compu-tational Linguistics.

Matthew Richardson and Pedro Domingos. 2006.Markov logic networks. Machine Learning, 62(1-2):107–136.

Deb Roy. 2005. Grounding words in perception and ac-tion: computational insights. Trends in Cognitive Sci-ences, 9(8):389–396, August.

Sandeepkumar Satpal, Sahely Bhadra, S SundararajanRajeev, and Rastogi Prithviraj. 2011. Web Infor-mation Extraction Using Markov Logic Networks.Learning, pages 1406–1414.

David Schlangen and Gabriel Skantze. 2009. A General,Abstract Model of Incremental Dialogue Processing.In Proceedings of the 12th Conference of the Euro-pean Chapter of the ACL (EACL 2009), number April,pages 710–718, Athens, Greece. Association for Com-putational Linguistics.

David Schlangen, Timo Baumann, and Michaela At-terer. 2009. Incremental Reference Resolution: TheTask, Metrics for Evaluation, and a {B}ayesian Filter-ing Model that is Sensitive to Disfluencies. In Pro-ceedings of the SIGDIAL 2009 Conference, numberSeptember, pages 30–37, London, UK. Association forComputational Linguistics.

322

Alexander Siebert and David Schlangen. 2008. A Sim-ple Method for Resolution of Definite Reference in aShared Visual Context. In Proceedings of the 9th SIG-dial Workshop on Discourse and Dialogue, numberJune, pages 84–87, Columbus, Ohio. Association forComputational Linguistics.

Parag Singla and Pedro Domingos. 2005. Discrimina-tive Training of Markov Logic Networks. Computing,20(2):868–873.

Michael Spranger, Martin Loetzsch, and Simon Pauw.2010. Open-ended Grounded Semantics. In Euro-pean Conference on Artificial Intelligence 2010, Lis-bon, Portugal. Volume 215 Frontiers in Artificial In-telligence and Applications.

Luke S. Zettlemoyer and Michael Collins. 2009. Learn-ing context-dependent mappings from sentences tological form. Proceedings of the Joint Conference ofthe 47th Annual Meeting of the ACL and the 4th Inter-national Joint Conference on Natural Language Pro-cessing of the AFNLP: Volume 2 - ACL-IJCNLP ’09,2:976.

323