atr presentation
TRANSCRIPT
![Page 1: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/1.jpg)
Pain Avoidance Learning “Model-based” and “Model-free” systems
Oliver WangDepartment of Cognitive Neuroscience
![Page 2: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/2.jpg)
Project Concepts
● Creation of a cognitive map of the space● Learning the values of actions as well as states
● Only learning the values of each action
![Page 3: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/3.jpg)
Model-Based System Model-Free System
![Page 4: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/4.jpg)
Project Basis● Neural Computations Underlying Arbitration between Model-based and Model-free Learning
by Sang Wan Lee, Shinsuke Shimojo, and John P. O’Doherty (2014)
● We hypothesized that a similar arbitration method might exist in aversion learning as it does in reward learning.
![Page 5: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/5.jpg)
LiteratureMinimal research has been done on pain aversion learning
● Hendersen and Graham (Avoidance of Heat by Rats, 1979)● Prevost and O’Doherty (Pavlovian Aversive Learning, 2013)● Gillan and Robbins (Enhanced Avoidance Habits, 2014)
![Page 6: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/6.jpg)
Task Design: Two-layer Markov Decision Task
● Training session followed by 2 sessions, each with 48 blocks, each with on average 5 trials ● Sequential 2-choices (L/R) to final state● Following states are determined by the choice and the probabilities of each branch at that time● 4 block conditions:
o Flexible, high uncertaintyo Flexible, low uncertaintyo Specific, high uncertaintyo Specific, low uncertainty
![Page 7: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/7.jpg)
Block ConditionsFlexible Final state values are set and you receive the number of shocks indicatedEncourages a Model-Free strategy
SpecificBin color must match final state color to receive the number of shocks indicated. Otherwise you receive 4, the maximum number, of shocks.Encourages a Model-Based strategy
Example of 1 trial, flexible condition, high uncertainty
![Page 8: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/8.jpg)
Probability: UncertaintyHigh uncertainty vs. Low uncertainty
1. High uncertainty refers to a (.5,.5) chance between the 2 resulting states.
2. Low uncertainty refers to a (.9,.1) chance, and thus a state (left state in the diagram to the right) is much more highly favorable.
*Uncertainty is maintained throughout each block
![Page 9: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/9.jpg)
Behavioral Results
Participants: 16 subjects
![Page 10: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/10.jpg)
Behavioral Results con’tObservation● Significantly higher proportion
observed in flexible, low uncertainty condition
Conclusion● Some difference in the arbitrator
must exist
![Page 11: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/11.jpg)
Model-free Simulation
16 Simulated Subjects Alpha = .03, Beta = 1
MF Learning is able to replicate only the results of the flexible condition
![Page 12: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/12.jpg)
Subject Choices: MB/MFLeft
● Refers to the flexible condition
● MB system is not adequate
Right● Refers to the specific
condition● MB system is
adequate
![Page 13: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/13.jpg)
Parameters
Observation● Parameter for “learning rate for the estimate of absolute reward prediction error” is much
greater in our pain aversion task
Interpretation● Suggests a more dynamic arbitration system exists
*Pain Aversion Parameters (left bar) and Reward Based Parameters (right bar) (Lee)
![Page 14: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/14.jpg)
Conclusion1. Both Model-free and Model-based systems
exist in aversion learning.
2. Although the arbitration process between the two systems share many similarities to reward-based learning, there exists subtle differences between the two.
![Page 15: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/15.jpg)
Next Steps1. fMRI 2. Modeling
![Page 16: ATR Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022042502/5888b4cf1a28ab80248b5f85/html5/thumbnails/16.jpg)
ありがとうございますThank you for listening.