Preference Elicitation

in Scheduling Problems Ulaş Bardak

Ph.D. Thesis ProposalCommittee

Jaime Carbonell, Eugene Fink, Stephen Smith,

Sven Koenig (University of Southern California)

Outline

Introduction

Example

Preliminary results

Plan of work

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Motivation

Improve resource planning by reducing uncertainty of the available knowledge.

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Hypothesis

By asking the questions with the highest

potential to reduce uncertainty, we can improve the quality of the resource plan while minimizing the cost of elicitation.

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Missing info:• Invited talk: – Projector need• Poster session: – Room size – Projector need

Assumptions:• Invited talk: – Needs a projector• Poster session: – Small room is OK

– Needs no projector

Initial scheduleAvailable rooms:

Roomnum.

Capacity

Projector

123

500 100 80

YesNoYes

Requests:• Invited talk, 9–10am: Needs big room• Poster session, 9–11am: Needs a room

1 2

3

Initial schedule:

Talk

Posters

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Choice of questions

1

Initial schedule:

Talk

Posters

Candidate questions:• Invited talk: Needs a projector?• Poster session: Needs a larger room? Needs a projector?

Requests:• Invited talk, 9–10am: Needs a large room• Poster session, 9–11am: Needs a room

Useless info: There are no large rooms w/o a projector

×

Useless info: There are no unoccupied larger rooms×Potentially useful info

√

2

3

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Improved scheduleRequests:• Invited talk, 9–10am: Needs a large room• Poster session, 9–11am: Needs a roomInfo elicitation:System:Does the poster sessionneed a projector?

User:A projector may be useful,but not really necessary.

Posters

1

Initial schedule:

Talk

Posters2

3

1

New schedule:

Talk

2

3

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Architecture

ElicitorNatural Lang. Optimizer

Ask user and get answers

Chooseand sendquestions

Updateresource

allocation

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Inside the Elicitor

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Get list of questions

For each question i getutilities for possible answers

Return top N questions

, costutility iScore i i Get question score

Each uncertain variable is a potential question

Plug in possible answers to the utility functionto get change in utility.

Optimizer

Uses hill climbing to allocate resources

Searches for an assignment of resources with the greatest expected utility

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Related Work

Example critiquing [Burke et al.]Have users tweak result set

Collaborative filtering [Resnick], [Hill et al.]Have the user rank related items

Similarity-based heuristics [Burke]Look at past similar user ratings

Focusing on targeted use [Stolze]

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Related Work

Clustering utility functions [Chajewska] Decision tree [Stolze and Ströbel] Min-max regret [Boutilier]

Choose question that reduces max regret Auctions [Smith], [Boutilier],

[Sandholm]

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

What is different?

No bootstrapping Continuous variables Large number of uncertain variables Tight integration with the optimizer Integration of multiple approaches Dynamic elicitation costs

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Example Domain

Assigning rooms to conference sessions

Rooms have properties. Sessions have preferences,

constraints, and importance values. Each preference is a function from a

room property to utility.

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Rooms have properties.

Sessions have preferences, constraints, and importance values.

Each preference is a function from a room property to utility.

Example Domain

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Room 1 can accommodate 200 people.Room 1 can accommodate 200 people.Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance

Rooms have properties.

Sessions have preferences, constraints, and importance values.

Each preference is a function from a room property to utility.

Example Domain

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance

Invited talk cannot be before 2 p.m.Invited talk is more important than poster session.Invited talk cannot be before 2 p.m.Invited talk is more important than poster session.Invited talk very important : 40% chanceInvited talk moderately important : 60% chanceInvited talk very important : 40% chanceInvited talk moderately important : 60% chance

Rooms have properties.

Sessions have preferences, constraints, and importance values.

Each preference is a function from a room property to utility.

Example Domain

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Capacity of Room 1 is 200.Capacity of Room 1 is 200.

Capacity preference: 150 people is minimum,200 people is acceptable, 250 people is best.Capacity preference: 150 people is minimum,200 people is acceptable, 250 people is best.

Invited talk very important : 40% chanceInvited talk moderately important : 60% chanceInvited talk very important : 40% chanceInvited talk moderately important : 60% chance

Capacity preference is [150, 200, 250] : 40% chanceCapacity preference is [50, 100, 150] : 60% chanceCapacity preference is [150, 200, 250] : 40% chanceCapacity preference is [50, 100, 150] : 60% chance

Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance

Experiments

Evaluation of RADAR 15 room properties 88 rooms 84 sessions 2500 variables 700 uncertain valuesSystem asked to provide 50 top

questions.

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

0.72

0.78

Utility

10 3020 40 50No. of Questions

0.58

CertainIncrementalOptimizerestimate

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Incremental elicitation

Completed work

Questions based on potential reduction of uncertainty

Empirical evaluation

Integration with RADAR

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Contributions

Fast computation of expected impact for potential questions

Use of the optimizer for calculating more accurate question weights.

Use of past elicitation results to improve the elicitation process.

Unifying different elicitation strategies.

√

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Search for optimal questions

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

100-150: 40% chance151-200: 60% chance

160-200: 50% chance100-160: 50% chance

100-130: 25% chance130-160: 25% chance

h=20, max utility increase = 20

h=10, max utility increase = 30h=10

h=15, max utility increase = 100

h=15

Best-first search with the optimizer used as the heuristic function.

Example: Uncertain room size

Elicitation rules

Encoding of elicitation heuristics

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

mean 10,000

std-dev 5,000

type room

size room

size room

Conditions: Auditorium

elicit size roomAction:

rule Uncertain-Auditorium-Size(room)

Learning of elicitation rulesDerive rules based on past elicitations

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

mean 1

100

type room

projector room

importance event

Conditions: Auditorium

elicit projector roomAction:

rule Learned-Rule(room,event)

… EventImp.

Room Prop.

Mean Prop.

Elicit. Result

… 110 Proj. 0.5 +

… 115 Size 250 -

… 105 Proj. 0.9 +

… 90 Proj. 0.5 -

… 200 Size 100 +

… 150 Proj. 0.3 +

… SessionType

RoomType

EventImportance

Room Property

MeanValue

ElicitationResult

… Invited Talk

Auditorium

110 Proj. 0.5 +

… Posters Meeting R.

115 Size 250 -

… Best Paper

Auditorium

105 Proj. 0.9 +

… Posters Classroom 90 Proj. 0.5 -

… Talk Auditorium

200 Size 100 +

… Keynote Auditorium

150 Proj. 0.3 +

Dynamic question costs

Same cost for all questions Different cost for different question

types Learning of the question costs for

each type Learning of the question costs for

each information source

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

√

√

Compare different approaches: Current system Search for optimal questions Hand coded elicitation rules Learned elicitation rules Unified system Human elicitorMeasure utility gain after each answer; also

evaluate running time

Experiments

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

TimelineBest-First Search

Syntax for rules

Learning of rules

Experiments

Aug 2

007

Mar

200

7

Nov 2

006

July 2

006

Mar

200

6Unified System

Dec 2

007

Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions

Writing

Learning of costs

Addendum

Outline – Introduction – Example – Preliminary Results – Plan of Work – Outline – Introduction – Example – Preliminary Results – Plan of Work – QuestionsQuestions

0

0.05

0.1

0.15

0 10 20 30 40 50

Question number (more important to less important)

Abs

olut

e ch

ange

in s

ched

ule

qual

ity

0

0.05

0.1

0.15

0 10 20 30 40 50

Question number (more important to less important)

Abs

olut

e ch

ange

in s

ched

ule

qual

ity