preference elicitation in scheduling problems ulaş bardak ph.d. thesis proposal committee jaime...
Post on 21-Dec-2015
218 Views
Preview:
TRANSCRIPT
Preference Elicitation
in Scheduling Problems Ulaş Bardak
Ph.D. Thesis ProposalCommittee
Jaime Carbonell, Eugene Fink, Stephen Smith,
Sven Koenig (University of Southern California)
Outline
Introduction
Example
Preliminary results
Plan of work
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Motivation
Improve resource planning by reducing uncertainty of the available knowledge.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Hypothesis
By asking the questions with the highest
potential to reduce uncertainty, we can improve the quality of the resource plan while minimizing the cost of elicitation.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Missing info:• Invited talk: – Projector need• Poster session: – Room size – Projector need
Assumptions:• Invited talk: – Needs a projector• Poster session: – Small room is OK
– Needs no projector
Initial scheduleAvailable rooms:
Roomnum.
Capacity
Projector
123
500 100 80
YesNoYes
Requests:• Invited talk, 9–10am: Needs big room• Poster session, 9–11am: Needs a room
1 2
3
Initial schedule:
Talk
Posters
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Choice of questions
1
Initial schedule:
Talk
Posters
Candidate questions:• Invited talk: Needs a projector?• Poster session: Needs a larger room? Needs a projector?
Requests:• Invited talk, 9–10am: Needs a large room• Poster session, 9–11am: Needs a room
Useless info: There are no large rooms w/o a projector
×
Useless info: There are no unoccupied larger rooms×Potentially useful info
√
2
3
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Improved scheduleRequests:• Invited talk, 9–10am: Needs a large room• Poster session, 9–11am: Needs a roomInfo elicitation:System:Does the poster sessionneed a projector?
User:A projector may be useful,but not really necessary.
Posters
1
Initial schedule:
Talk
Posters2
3
1
New schedule:
Talk
2
3
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Architecture
ElicitorNatural Lang. Optimizer
Ask user and get answers
Chooseand sendquestions
Updateresource
allocation
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Inside the Elicitor
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Get list of questions
For each question i getutilities for possible answers
Return top N questions
, costutility iScore i i Get question score
Each uncertain variable is a potential question
Plug in possible answers to the utility functionto get change in utility.
Optimizer
Uses hill climbing to allocate resources
Searches for an assignment of resources with the greatest expected utility
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Related Work
Example critiquing [Burke et al.]Have users tweak result set
Collaborative filtering [Resnick], [Hill et al.]Have the user rank related items
Similarity-based heuristics [Burke]Look at past similar user ratings
Focusing on targeted use [Stolze]
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Related Work
Clustering utility functions [Chajewska] Decision tree [Stolze and Ströbel] Min-max regret [Boutilier]
Choose question that reduces max regret Auctions [Smith], [Boutilier],
[Sandholm]
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
What is different?
No bootstrapping Continuous variables Large number of uncertain variables Tight integration with the optimizer Integration of multiple approaches Dynamic elicitation costs
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Example Domain
Assigning rooms to conference sessions
Rooms have properties. Sessions have preferences,
constraints, and importance values. Each preference is a function from a
room property to utility.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Rooms have properties.
Sessions have preferences, constraints, and importance values.
Each preference is a function from a room property to utility.
Example Domain
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Room 1 can accommodate 200 people.Room 1 can accommodate 200 people.Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance
Rooms have properties.
Sessions have preferences, constraints, and importance values.
Each preference is a function from a room property to utility.
Example Domain
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance
Invited talk cannot be before 2 p.m.Invited talk is more important than poster session.Invited talk cannot be before 2 p.m.Invited talk is more important than poster session.Invited talk very important : 40% chanceInvited talk moderately important : 60% chanceInvited talk very important : 40% chanceInvited talk moderately important : 60% chance
Rooms have properties.
Sessions have preferences, constraints, and importance values.
Each preference is a function from a room property to utility.
Example Domain
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Capacity of Room 1 is 200.Capacity of Room 1 is 200.
Capacity preference: 150 people is minimum,200 people is acceptable, 250 people is best.Capacity preference: 150 people is minimum,200 people is acceptable, 250 people is best.
Invited talk very important : 40% chanceInvited talk moderately important : 60% chanceInvited talk very important : 40% chanceInvited talk moderately important : 60% chance
Capacity preference is [150, 200, 250] : 40% chanceCapacity preference is [50, 100, 150] : 60% chanceCapacity preference is [150, 200, 250] : 40% chanceCapacity preference is [50, 100, 150] : 60% chance
Room 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chanceRoom 3 has one projector : 80% chanceRoom 3 has no projectors : 20% chance
Experiments
Evaluation of RADAR 15 room properties 88 rooms 84 sessions 2500 variables 700 uncertain valuesSystem asked to provide 50 top
questions.
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
0.72
0.78
Utility
10 3020 40 50No. of Questions
0.58
CertainIncrementalOptimizerestimate
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Incremental elicitation
Completed work
Questions based on potential reduction of uncertainty
Empirical evaluation
Integration with RADAR
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Contributions
Fast computation of expected impact for potential questions
Use of the optimizer for calculating more accurate question weights.
Use of past elicitation results to improve the elicitation process.
Unifying different elicitation strategies.
√
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Search for optimal questions
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
100-150: 40% chance151-200: 60% chance
160-200: 50% chance100-160: 50% chance
100-130: 25% chance130-160: 25% chance
h=20, max utility increase = 20
h=10, max utility increase = 30h=10
h=15, max utility increase = 100
h=15
Best-first search with the optimizer used as the heuristic function.
Example: Uncertain room size
Elicitation rules
Encoding of elicitation heuristics
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
mean 10,000
std-dev 5,000
type room
size room
size room
Conditions: Auditorium
elicit size roomAction:
rule Uncertain-Auditorium-Size(room)
Learning of elicitation rulesDerive rules based on past elicitations
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
mean 1
100
type room
projector room
importance event
Conditions: Auditorium
elicit projector roomAction:
rule Learned-Rule(room,event)
… EventImp.
Room Prop.
Mean Prop.
Elicit. Result
… 110 Proj. 0.5 +
… 115 Size 250 -
… 105 Proj. 0.9 +
… 90 Proj. 0.5 -
… 200 Size 100 +
… 150 Proj. 0.3 +
… SessionType
RoomType
EventImportance
Room Property
MeanValue
ElicitationResult
… Invited Talk
Auditorium
110 Proj. 0.5 +
… Posters Meeting R.
115 Size 250 -
… Best Paper
Auditorium
105 Proj. 0.9 +
… Posters Classroom 90 Proj. 0.5 -
… Talk Auditorium
200 Size 100 +
… Keynote Auditorium
150 Proj. 0.3 +
Dynamic question costs
Same cost for all questions Different cost for different question
types Learning of the question costs for
each type Learning of the question costs for
each information source
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
√
√
Compare different approaches: Current system Search for optimal questions Hand coded elicitation rules Learned elicitation rules Unified system Human elicitorMeasure utility gain after each answer; also
evaluate running time
Experiments
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
TimelineBest-First Search
Syntax for rules
Learning of rules
Experiments
Aug 2
007
Mar
200
7
Nov 2
006
July 2
006
Mar
200
6Unified System
Dec 2
007
Outline – Introduction – Example – Preliminary Results – Plan of Work – Questions
Writing
Learning of costs
Addendum
Outline – Introduction – Example – Preliminary Results – Plan of Work – Outline – Introduction – Example – Preliminary Results – Plan of Work – QuestionsQuestions
0
0.05
0.1
0.15
0 10 20 30 40 50
Question number (more important to less important)
Abs
olut
e ch
ange
in s
ched
ule
qual
ity
0
0.05
0.1
0.15
0 10 20 30 40 50
Question number (more important to less important)
Abs
olut
e ch
ange
in s
ched
ule
qual
ity
top related