seeking consistent stories

Lydia L. Chen

04/21/23 Lydia L. Chen <[email protected]>

Seeking Consistent StoriesSeeking Consistent StoriesBy Reinterpreting or Discounting Evidence: By Reinterpreting or Discounting Evidence:

An Agent-Based ModelAn Agent-Based Model

Decision Consortium May Conference

13-Mar-2009

04/21/23 Lydia L. Chen <[email protected]>“Seeking Consistency” - 2

AgendaAgenda

1. The Phenomenon

2. Agent-Based Model (ABM) Primer

3. The Model

4. Sample Run

5. Experiments


Real-Life Scenario: Bench TrialReal-Life Scenario: Bench TrialProsecution () Defense Lawyer

“Guilty!

”“Innocen

t!”


Sequential Evidence – What’s NormativeSequential Evidence – What’s Normative

Evidence 1

…

Evidence 2

Evidence 3

Evidence N

Official Deliberatio

n

Story/ Verdict


Empirical Literature on JDMEmpirical Literature on JDM Form “coherent” stories that support option/verdict

(Pennington & Hastie, 1988)

Confidence in option/verdict increases with “coherence” (Glockner et al., under review)

Decision threshold: “sufficiently strong” (supported by many consistent evidence) or “sufficiently stronger” than other stories (review by Hastie, 1993

Narrative coherence: consistency causality completeness

“Consistency” aspect of “good” stories consistency between evidences in a story consistency of evidence with favored story


Example Case (Pennington & Hastie, 1988)Example Case (Pennington & Hastie, 1988)

Scenario: Defendant Frank Johnson stabbed and killed Alan Caldwell

Evidence := facts or arguments given in support of a story/verdict Facts— “Johnson took knife with him” ; “Johnson pulled out his knife”

Arguments—”Johnson pulled out knife because he wanted revenge” vs. “Johson pulled out knife because he was afraid”

Story := set of evidence supporting a given verdict

Same evidence can be framed to support multiple verdicts/stories!

P D

Desired Verdict “Guilty” “Innocent”

Story “Premeditated murder” “Self-defense”

Interpretation of Evid

“Johnson was angry” “Johnson was afraid”


Sequential Evidence – More DescriptiveSequential Evidence – More Descriptive

Evidence 1

Compare, Deliberate, Interpret

…

Evidence 2

Evidence 3

Evidence N

Official Deliberatio

n

Story/ Verdict

Premature Story/

Verdict @ Evid n < N

(Brownstein, 2003; Russo et al., 2000)


How People Deal with Incoming EvidenceHow People Deal with Incoming Evidence

People don’t just take evid @ face value, but are selective!

Possible Reactions to New Evid in Light of Old:

“reinforce” each other

“reinterpret” less plausible one (Russo et al., etc.); e.g., misremember the info

“discount” less plausible one (Winston)

actively “seek” more evidence (not modeled here)

Existing evidence A: “Johnson was not carrying a knife.”

New evidence B1: “Johnson is nonviolent.”

Inconsistent new evid B2: “Johnson pulled a knife.”

Reinterpret B2: “Johnson grabbed a knife from Caldwell.” (i.e., explain it was Caldwell’s knife, not Johnson’s)

Discount B2: “Witness must be mistaken.”

Judge asks layers follow-up questions


MotivationsMotivations

Study emergence of consistent stories via reinforcement, reinterpretation, and discounting mechanisms

Process, functions, consequences

Adaptive? aid consistency? speed-accuracy tradeoff? avoid indecisiveness

increases convergence rates? order effects hurt accuracy?


AgendaAgenda

1. The Phenomenon2. Agent-Based Model (ABM) Primer

[1:43]

3. The Model

4. Sample Run

5. Experiments


What is Agent-Based Modeling?What is Agent-Based Modeling? agents + interactions^

start simple; build up^

Key terminology agents system dynamics

agent births and deaths interactions/competitions

parameters

A System of Agents

A

A

A

A A

A


(Contributions ABMs Can Make)(Contributions ABMs Can Make)

Symbiotic relationship:

Behavioral Experiments

ABM

parsimony:

demo emergence of seemingly complex

phenomen from small set of simple rulespredictions:

new observations/predictions

understanding:

study processes in detail

Test

Inputdescription:

informs base assumptions


What’s new and different?


(Other Models (Hastie, 1993))(Other Models (Hastie, 1993))


Algebraic (additive), Bayesian (multiplicative): “single meter” of overall plausibility

ABM allows: revisiting and reconsidering previously-processed

evidence

interaction/competition between individual evidences (not just conglomerate)

(Contrast with Bayesian & Algebraic Models)(Contrast with Bayesian & Algebraic Models)

conglomerate of all previous evidence

new evidence


Contrast with Story Model & ECHOContrast with Story Model & ECHO Explanatory Coherence Model := Thagard’s Theory

of Explanatory Coherence (TEC) + Story Model

Only implemented discounting, not reinterpretation

Local evidence-agent-level consistency, as opposed to global story-level consistency

Unlike previous ABMs, model agents within individual as system


AgendaAgenda

1. The Phenomenon

2. Agent-Based Model (ABM) Primer3. The Model [1:46]

4. Sample Run

5. Experiments


GoalGoal

Model consistency-seeking process in story formation

Present evidence-agents to judge-system

Judge-system compares evidence-agents => keep, reinterpret, or discount evidence (agents “interact” & “compete”)

Until sufficiently strong/stronger story emerges


Evidence-agents, composed of “features”

Operationalize consistency /b/ agents: “similarity” in abstract features (Axelrod’s culture ABM,

1997) “inverse Hamming distance” := % feature matches

AgentsAgents

G

x

x

89%

Evid 1

N

x

y

34%

Evid 2

…

Verdict (“G”,”N”,…)

Abstract features (binary)

Plausibility index (0%-

100%)

N

x

x

14%

Evid 1


(Parameters of “Base Case”)(Parameters of “Base Case”)Parameter Variabl

eBase

# agents to present

- Initial A0 3

- Additional A 20

# possible verdicts V 2

# features/evidence (excluding verdict) F 2

% feature matches for:

Sufficiently consistent C 100%

Close enough for reinterpretation N < C 50%

# interactions /b/ evidence-agent births I (A0+A -1)

Change rate of agent plausibility (increase upon winning, decrease upon losing)

D 0.1

Rule(s) and threshold(s) for winning story-- “sufficiently strong” and/or “sufficiently stronger”

S, Sd Both — 0.5, 0.2


(Java GUI)(Java GUI)


System & Agent BirthsSystem & Agent Births

Judge-system represents judge’s mind

Evidence presentation = “agent birth”:

Initialization of N0 Agents: Set up randomly-generated agents, OR…

uniform distributions—even for plausibility index due to prior beliefs (Kunda, 1990) or knowledge (Klein, 1993)

User-specified


(Topology)(Topology)

results\sysout_081118_0455.log

Printing 8 agents:

01 02 03 04 05 06 07 08

N G I G N I N N

y y y y y y y y

y y y y y y y y

100% 100% 100% 100% 100% 100% 100% 80%

Dead/Rejected Evidence--Printing 9 agents:

01 02 03 04 05 06 07 08 09

I G G I G I N N N

y y x x y x x x y

y x x x y x x y y

00% 00% 00% 00% 00% 00% 00% 00% 00%

To keep track of evidence-agents and their order of presentation:

lists of agents in order of birth/presentation to the system.

latest-born agent always appears at the end of a list.

no geographical “topology”


StoriesStories:= {evidence-agents supporting a verdict}

Consistency = inverse Hamming distance amongst evidence (:= evidencePairs # feature matches / # evidencePairs / F )

Plausibility Index (“Strength”) = average of plausibility indices

G

x

x

89%

Evid 1N

x

y

51%

Evid 2

N

x

x

14%

Evid 1

Story promoting

“Guilty” verdict

Story promoting “Innocent”

verdict

strength = 89%

strength = 99% / 3

= 33%

N

y

y

34%

Evid 3


Births & InteractionsBirths & Interactions1. If time period t = k * I, where k is some constant, birth new

agent.

2. Random selection of agent to compare with newest-born.

3. Interaction. Compare agents and compute consistency. Depending upon consistency [next slide]:

“reward consistency” “increase consistency” or “punish consistency”

Agents with plausibility = 0% => death & removal from system

4. Gather stories in system. Check strengths.

“Winning story found” if 1 ! story with strength >= S and/or |strength-strength | >= Sd for all competing stories; stop run early.


Operationalizing Consistency--ExamplesOperationalizing Consistency--Examples

Possible Reactions to New Evid :

Completely consistent (e.g., 100% features match) => => both are “winners” => “reinforce” both; “reward consistency”

Inconsistent…

but salvageable (50% features match) => “reinterpret” less plausible “loser” (by plausibility); “increase inconsistency”

not salvageable (0% features match) => “discount” “loser”; “punish inconsistency”

=> plaus(Evid1) > plaus(Evid2) => Evid2 “loser”

x

y

51%

Evid 1

x

y

34%

Evid 2

x

y

61%

Evid 1

x

y

44%

Evid 2

x

y

51%

Evid 1

y

y

34%

Evid 2

x

y

51%

Evid 1

x

y

34%

Evid 2

x

x

51%

Evid 1

y

y

34%

Evid 2

x

x

51%

Evid 1

x

y

24%

Evid 2


AgendaAgenda

1. The Phenomenon


3. The Model4. Sample Run [1:54]

5. Experiments


Sample Output^Sample Output^ Live Evidence--Printing 8 agents:

01 02 03 04 05 06 07 08

N G I G N I N N

y y y y y y y y

y y y y y y y y

100% 100% 100% 100% 100% 100% 100% 80%


01 02 03 04 05 06 07 08 09

I G G I G I N N N

y y x x y x x x y

y x x x y x x y y

00% 00% 00% 00% 00% 00% 00% 00% 00%

3 stories found:

-Verdict N supported by 4 evidence, with 1.00 consistency => 0.48 strength

-Verdict G supported by 2 evidence, with 1.00 consistency => 0.25 strength

-Verdict I supported by 2 evidence, with 1.00 consistency => 0.25 strength

*** Found winning story! Verdict N supported by 4 evidence, with 1.00 consistency => 0.48 strength


Judge-System can get Stuck…Judge-System can get Stuck…results\sysout_STUCK.log

Live Evidence--Printing 17 agents:

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17

N G G I G I I I I N I G G N N N G

x x x x x x x x x x x x x x x x x

y y y x y y y y y y y y y y y y y

100% 100% 100% 90% 100% 100% 100% 100% 100% 100% 84% 100% 79% 100% 100% 98% 100%


01 02 03 04 05 06 07 08 09 10

I I I I N I I G N G

y x y y y y y y y y

y x x x x x x y x x

00% 00% 00% 00% 00% 00% 00% 00% 00% 00%

3 stories found:

-Verdict N supported by 5 evidence, with 1.00 consistency => 0.29 strength

-Verdict G supported by 6 evidence, with 1.00 consistency => 0.34 strength

-Verdict I supported by 6 evidence, with 1.00 consistency => 0.34 strength

No winning story found.


Outputs of a Run (DVs)^Outputs of a Run (DVs)^

GGG GGGGGGGG GGGGGGG GG

5 10 15 20

04

8

N N N N N N N N N N N N N N N N N N N N

5 10 15 20

04

8

Supporting Evidence (#)

Time (presentation of agents)

GGGGGGG GGGGGGG GGGGGG

5 10 15 20

0.0

0.6

N NN

N N N N N N N N N N N N N N N N N

5 10 15 20

0.0

0.6

Plausibility (%)


GGGGG

GGGG

G GGGGGGG GG

5 10 15 20

0.0

0.6

N NN N N

N N N N N N N NN N

N NN N N

5 10 15 20

0.0

0.6

Consistency (%)



5 10 15 20

0.0

0.4

0.8

N N NN N N N N N N N N N N N N N N N N

5 10 15 20

0.0

0.4

0.8

Strength (%)



AgendaAgenda

1. The Phenomenon


3. The Model

4. Sample Run5. Experiments [1:56]


Controlled Experiments in SimulationControlled Experiments in Simulation “control group”

vary IV => various “treatment groups”

Compare and contrast control group and treatment groups

“base case”

vary parameter => various other cases

Compare and contrast base case and other cases

Adv: Don’t have to worry about assumptions b/c study inter-group differences rather than absolute

outputs


5 Experiments5 Experiments

Experiment 1: Emergence of consistency

Decision Speed Experiment 2: Speedup Experiment 3: Accuracy tradeoff

Decision Accuracy—Order Effects Experiment 4: Emergence of order effects Experiment 5: Extending deliberation


Obtaining Consistent Stories – Q1Obtaining Consistent Stories – Q1

Q1: Evidence-level consistency sufficient? Which of the 3 mechanisms?

Implementation: No stopping rules

DV: Consistency of stories


Obtaining Consistent Stories – Q1 ResultsObtaining Consistent Stories – Q1 Results

Reinforce Reinterpret Discount Median consistency

0.9

0.8

0.7

0.6

0.6

0.6

0.5

Reinterpret > Discount > Reinforcement


Speed-Accuracy Tradeoff – Q2Speed-Accuracy Tradeoff – Q2

Q2: Reinterpretation & Discounting increase speed?

Prediction: Reinterpretation & Discounting allow faster convergence

DV: Time to Converge, Max nEvid (Max Consistency)

Implementation: Both rules; all cases have reinforce


Speed-Accuracy Tradeoff – Q2 ResultsSpeed-Accuracy Tradeoff – Q2 Results

Results of 10 Runs—Time to First Convergence, Maximum Number of Evidence,

Maximum Story Consistency

Run1 2 3 4 5 6 7 8 9 10

W/o Reinterp, W/o Discount results/sysout_081215_1938.log

NoC160.5

NoC130.5

NoC120.5

NoC120.5

NoC120.5

NoC140.5

NoC150.5

NoC120.5

NoC120.5

NoC130.5

W/o Reinterp, W/ Discountresults/sysout_081215_1937.log

NoC9

0.8

64

1.0

18110.6

NoC9

0.6

NoC100.5

NoC140.5

NoC110.6

NoC110.7

178

1.0

NoC9

0.5

W/ Reinterp, W/o Discountresults/sysout_081215_1935.log

NoC131.0

NoC120.8

NoC140.6

54

1.0

86

1.0

NoC120.6

NoC130.8

NoC120.9

21140.9

54

1.0

W/ Reinterp, W/ Discountresults/sysout_081215_1946.log

NoC100.8

NoC131.0

43

1.0

NoC8

0.8

126

1.0

43

1.0

NoC8

0.8

86

1.0

NoC9

0.8

19101.0


Speed-Accuracy Tradeoff – Q2 ResultsSpeed-Accuracy Tradeoff – Q2 Results

Medians of 10 Runs—Time to First Convergence, Maximum Number of Evidence,

Maximum Story Consistency

Reinterpret > Discount > Reinforcement only

W/o Discounting W/ Discounting

W/o Re-interpretation Converged 0% of runs nEvid = 12.5

(0.5)

Converged 30% of runs, time = 17,

8(1.0)

W/ Re-interpretation Converged 40% of runs, time = 6.5, nEvid = 5

(1.0)

Converged 50% of runs, time = 8

6 (1.0)


Speed-Accuracy Tradeoff – Q3Speed-Accuracy Tradeoff – Q3 Q3: What would happen if allow process to continue even after having

found winner? Any point to "holding off judgment" until all evidence presented?

DV: Which story wins? (Strength)

Prediction: Leader will only be strengthened; competing stories never get a foothold.

Implementation: Allow system to continue running even if found winner

Run 1 2 3 4 5 6 7 8 9 10

Winner once conditions met (G/N)

Winner after additional evidence presented (G/N)

Changed winner? (Y/N)


Speed-Accuracy Tradeoff – Q3 Results^Speed-Accuracy Tradeoff – Q3 Results^

Over 20 runs, # runs same winner:# runs different winner = 15:1

=> Good heuristic to stop deliberation, for less time & effort


5 10 15 20

04

8


5 10 15 20

04

8




5 10 15 20

0.0

0.6

N NN

N N N N N N N N N N N N N N N N N

5 10 15 20

0.0

0.6

Plausibility (%)


GGGGG

GGGG

G GGGGGGG GG

5 10 15 20

0.0

0.6

N NN N N

N N N N N N N NN N

N NN N N

5 10 15 20

0.0

0.6

Consistency (%)



5 10 15 20

0.0

0.4

0.8

N N NN N N N N N N N N N N N N N N N N

5 10 15 20

0.0

0.4

0.8

Strength (%)


Figure 2. Example run where winner switches


Order Effects – Q4Order Effects – Q4

Heuristic may be ok only if randomized evidence…what if biased evidence?

Q4: Is there an Order Effect?

Took {20 randomly-generated evidence} and then “doctored” it; % D win = “accuracy”

IV: Presentation order--P goes first, followed by D vs. interwoven evidence

Prediction: earlier, weaker side (e.g., P) beats out later, stronger side (e.g., D); "Accuracy" of D…P… > PDPDPD… > P…D…

DV: Time to Converge, (Projected) Winner


Order Effects – Q4 ResultsOrder Effects – Q4 Results


5 10 15 20

04

8


5 10 15 20

04

8



GGGGGGG G

GGGGGG GGGGGG

5 10 15 20

0.0

0.6


5 10 15 20

0.0

0.6

Plausibility (%)


G

GG GGGGGG

GG GGGGGGG GG

5 10 15 20

0.0

0.6 N N N

N N N NN N N N N N N N N N N

N N

5 10 15 20

0.0

0.6

Consistency (%)



5 10 15 20

0.0

0.4

0.8

N NN N N N N N N N N N N N

N N N N N N

5 10 15 20

0.0

0.4

0.8

Strength (%)


Figure 3. Sample random presentation order run.



5 10 15 20

04

8


5 10 15 20

04

8



GGGGGGG GGGGGG

G GGGGGG

5 10 15 20

0.0

0.6

N N N N N N N N N N N NN N N N N N N N

5 10 15 20

0.0

0.6

Plausibility (%)


GGG GGGGGGGG GGGGG

GG GG

5 10 15 20

0.0

0.6

N N

N N N N N N N N N N N N N N N N N N

5 10 15 20

0.0

0.6

Consistency (%)



5 10 15 20

0.0

0.4

0.8

NN

N N N N N N N N N NN N N N N N N N

5 10 15 20

0.0

0.4

0.8

Strength (%)



Figure 4. Sample D…P… biased presentation order run.



Figure 5. Sample P…D… biased presentation order run.


5 10 15 20

06

12


5 10 15 20

06

12



GGGG

GGG GGGGGGG GGGGGG

5 10 15 20

0.0

0.6

NN N N N N N N N N N N N N N N N N N N

5 10 15 20

0.0

0.6

Plausibility (%)


GGGG

GGGGGGG GGGGGGG GG

5 10 15 20

0.0

0.6 N

N N N N N N

N NN N N N N N N N

N N N

5 10 15 20

0.0

0.6

Consistency (%)


GGG

GGGG GGGGGGG GGGGGG

5 10 15 20

0.0

0.4

0.8


5 10 15 20

0.0

0.4

0.8

Strength (%)




Table 4. Results of 10 Runs Varying Presentation Order of Evidence

Presentation Order

Median Time to Converge% Runs that Defense Won

% Runs Resulting in Switch in Winner

PDPDPD… 7D won 90%, NoC 10%

0%

D… P… 5D won 90%, NoC 10%

0%

P... D… 8P won 80%, NoC 20%

30%

• Strong primacy effect

• Exper3 conclusion no longer holds; longer deliberation DOES help!


(Increasing Deliberation – Q5)(Increasing Deliberation – Q5)

Q5: Can deliberating more often between births reduce order effects (i.e., increase "accuracy")?

Implementation: Use P…D… model from Exper4

IV: Varied I (I = 0 => wait till end to deliberate)

DV: % runs that P wins


(Increasing Deliberation – Q5 Results)(Increasing Deliberation – Q5 Results)

Run 1 2 3 4 5 6 7 8 9 10 % P Wins

I = 0.8 * (# agents – 1)

60%

I = 1.0 * 80%

I = 1.2 * 100%

Too much lag time during trials can be detrimental!


Summary of Key FindingsSummary of Key Findings Why reinterpret and discount evidence?

1. Maximizes consistency (Experiment 1)2. Hastens convergence on decision (Experiment 2)

Reinterpret > Discount > Reinforcement

Speed-accuracy tradeoff? Yes… Accuracy ok if evidence balanced (Experiment 3) Not ok/primacy effect if biased (Experiment 4) => Important to interweave evidence, like in real trials!

Can reduce primacy effect by decreasing premature deliberation (Experiment 5)

All achieved by modeling consistency @ evidence level, not story level! more parsimonious & realistic(?)


Other Simplifying AssumptionsOther Simplifying Assumptions Features

Any combination of abstract features can be framed (e.g., by the lawyers) to support a verdict. All evidence are automatically categorizable into a verdict.

Interactions Interactions only take place between the newest agent and another agent.

Computing Consistency Verdict feature neither considered nor included in interactions

Comparing Stories judge forms 1 story / verdict

If only 1 story is found at time t, use the "sufficiently strong" rule as opposed to the "sufficiently stronger" rule. (i.e., no “holding out” by judge)

Which layer drives consistency-seeking? consistency at the individual agent interaction level, AND/OR consistency at the story level


(Future Expansions)(Future Expansions) Q6: What happens when introduce bias toward

certain verdicts? Prediction: Verdict-driven process takes less time to

converge DV: Time to Converge (Consistency of Stories) Implementation: Add favoredVerdict

Q7: In general, what conditions lead to indecision?

seeking consistent stories

Documents