towards understanding long short term memory...
TRANSCRIPT
![Page 1: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/1.jpg)
Towards Understanding Long Short Term Memory Networks
HMI
1/28/2019
Jordan Rodu
Department of Statistics
University of Virginia
![Page 2: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/2.jpg)
Towards Understanding Long Short Term Memory Networks
HMI
1/28/2019
Jordan Rodu
Department of Statistics
University of Virginia with Joao SedocUniversity of PennsylvaniaDepartment of Computer Science
![Page 3: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/3.jpg)
Mapping LSTMs to state space models
• Goal is not to interpret results on specific data
• Rather, map LSTM onto reasonable models- understand the space of sequences captured by LSTMs
• Preliminary work, basic ideas
![Page 4: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/4.jpg)
Hidden Markov Models
t t+1 t+2
![Page 5: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/5.jpg)
Hidden Markov Models
t t+1 t+2
![Page 6: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/6.jpg)
Hidden Markov Models
t t+1 t+2
T T
![Page 7: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/7.jpg)
Hidden Markov Models
t t+1 t+2
T T
𝑝 𝑥𝑡+1 𝑥1:𝑡) = 𝑝 𝑥𝑡+1 𝑥𝑡)
![Page 8: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/8.jpg)
Hidden Markov Models
t t+1 t+2
O O O
![Page 9: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/9.jpg)
Hidden Markov Models
t t+1 t+2
O O O
𝑝 𝑦𝑡 𝑥𝑡)
![Page 10: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/10.jpg)
Hidden Markov Models- flavors
• Output• Discrete• Continuous• Low dimensional• High dimensional
• States• Discrete• Continuous
• Low dimensional• High dimensional
• Time• Discrete• Continuous
![Page 11: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/11.jpg)
Hidden Markov Models- flavors
• Output• Discrete• Continuous• Low dimensional• High dimensional
• States• Discrete• Continuous
• Low dimensional• High dimensional
• Time• Discrete• Continuous
![Page 12: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/12.jpg)
Hidden Markov Models1⋮0
0⋮1
0⋮0
![Page 13: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/13.jpg)
Hidden Markov Models
𝑏1⋮𝑏𝑘
![Page 14: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/14.jpg)
Hidden Markov Models
𝑏1⋮𝑏𝑘
![Page 15: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/15.jpg)
Hidden Markov Models
𝑏1⋮𝑏𝑘
𝑝𝑥1|𝑦⋮
𝑝𝑥𝑘|𝑦
![Page 16: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/16.jpg)
Hidden Markov Models
𝑏1⋮𝑏𝑘
𝑝𝑥1|𝑦⋮
𝑝𝑥𝑘|𝑦
෨𝑏1⋮෨𝑏𝑘
![Page 17: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/17.jpg)
Hidden Markov Model- belief states
![Page 18: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/18.jpg)
Hidden Markov Model- belief states
![Page 19: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/19.jpg)
Hidden Markov Model- belief states
![Page 20: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/20.jpg)
A few related architectures
⋮ ⋮ ⋮
![Page 21: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/21.jpg)
A few related architectures
⋮ ⋮ ⋮
![Page 22: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/22.jpg)
A few related architectures
⋮ ⋮ ⋮
![Page 23: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/23.jpg)
A few related architectures
⋮ ⋮ ⋮
![Page 24: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/24.jpg)
A few related architectures
⋮ ⋮ ⋮
![Page 25: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/25.jpg)
A few related architectures
Τ𝑡 Τ𝑡+1 Τ𝑡+2
![Page 26: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/26.jpg)
A few related architectures
Τ𝑡 Τ𝑡+1 Τ𝑡+2
𝑂𝑡 𝑂𝑡+1 𝑂𝑡+2
![Page 27: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/27.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
![Page 28: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/28.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
![Page 29: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/29.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
![Page 30: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/30.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
![Page 31: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/31.jpg)
Hidden Markov Models-reminder
𝑏1⋮𝑏𝑘
𝑝𝑥1|𝑦⋮
𝑝𝑥𝑘|𝑦
෨𝑏1⋮෨𝑏𝑘
![Page 32: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/32.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
![Page 33: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/33.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
prior (T)
![Page 34: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/34.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
posterior (T and O)
![Page 35: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/35.jpg)
![Page 36: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/36.jpg)
LSTM
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
![Page 37: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/37.jpg)
Hidden states, 2𝑘 states for k hidden nodes
⋮ ⋮ ⋮
![Page 38: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/38.jpg)
Dependencies specified by weights from 𝑊ℎ
⋮ ⋮ ⋮
![Page 39: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/39.jpg)
Incorporating memory cell
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
![Page 40: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/40.jpg)
Incorporating memory cell
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡𝐶𝑡 = 𝑓𝑡 ∗ 𝐶𝑡−1 + 𝑖𝑡 ∗ ෩𝐶𝑡
![Page 41: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/41.jpg)
Incorporating memory cell
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡𝐶𝑡 = 𝑓𝑡 ∗ 𝐶𝑡−1 + 𝑖𝑡 ∗ ෩𝐶𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
![Page 42: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/42.jpg)
⋮ ⋮ ⋮
![Page 43: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/43.jpg)
⋮ ⋮ ⋮−1, 0, 1
![Page 44: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/44.jpg)
⋮ ⋮ ⋮−1, 0, 1
State space size 3𝑘 with special symmetry thatmodulates excitation andinhibition
![Page 45: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/45.jpg)
Partially Observable Markov Decision Process
t t+1 t+2T T
*relaxed visualization
O O O
T
𝑎𝑡 𝑎𝑡+1 𝑎𝑡+2
![Page 46: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/46.jpg)
POMDP representation
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
![Page 47: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/47.jpg)
POMDP representation
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
![Page 48: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/48.jpg)
POMDP representation
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
policy
![Page 49: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/49.jpg)
POMDP representation
𝜎 𝜎 tanh 𝜎
x
x
+
x
tanh
𝑦𝑡
ℎ𝑡
𝜎(𝑊ℎℎ𝑡−1 +𝑊𝑦𝑦𝑡 + 𝑏)
policy
Recall that ourgoal here is notto learn a POMDPor to approximatea POMDP usingLSTMs, rather towrap LSTMs inmodels for which we have a morerobust understanding.
![Page 50: Towards Understanding Long Short Term Memory Networks”hmi.virginia.edu/wp-content/uploads/2018/01/jordan-rodu-slides.pdf · Towards Understanding Long Short Term Memory Networks](https://reader031.vdocuments.mx/reader031/viewer/2022013021/5ec6957ac098f5714b46ec69/html5/thumbnails/50.jpg)
Thanks!