1. Checkpointing for VirtualPlatforms and SystemC-TLM-2.0 Mrius Montn Directors:Dr.Mark Burton Dr. Jordi Carrabina December2010 Univ. Autnoma Barcelona

2. Outline

Introduction

3. Objectives 4. Virtual Platforms and SystemC 5. Checkpointing for SystemC 6. Conclusions 7.

Introduction

8. Objectives 9. Virtual Platforms and SystemC 10. Checkpointing for SystemC 11. Conclusions Introduction 12. Introduction Virtual Platforms

Functional models of physical platforms

13. Target SW unable to distinguish virtual platform from real HW 14. Run SW or OS on Virtual HW 15. Develop SW for non-existing HW 16. Simulate complex system interconnectivity 17. Introduction Virtual Platforms 18. Introduction Virtual Platforms 19. Introduction Virtual Platforms 20. Introduction Virtual Platforms 21. Virtual Platform Design Hardware Software Integration & Test time Engineering Resources WindRiver (Virtutech) 22. Virtual Platform Design Hardware Software Integration & Test time Engineering Resources time Engineering Resources Resources Time to Market Hardware Software Integration & Test WindRiver (Virtutech) 23. Generic Virtual Platform Diagram Virtual Platform ISS ISS VP Back-end functions Devices Devices Memories Target Drivers Boot code Target Operating System User User User 24. Introduction SystemC & TLM-2

SystemC language for systems description (HW mainly)

25. OSCI simulator 26. TLM-2 standarizes communication model

Sockets to emulate any memory-mapped bus

De facto standard for system modeling 27. Introduction Checkpointing

Ability to stop a execution and save it to disk

28. Retrieve the execution from a disk 29. Useful for simulations of large systems

avoid repetitive tasks (boot OS)

30. bug fixing (modify a simulation value & continue) 31. reverse execution 32. multi-site development teams 33. Introduction VP Languages

Different Virtual Platforms uses different languages (C, C++, DLM, ...) and APIs

34. HW engineers know (or should) SystemC, not other languages 35. Different VP -> Rewrite own models 36. No interoperability 37.

Introduction

38. Objectives 39. Virtual Platforms and SystemC 40. Checkpointing for SystemC 41. Conclusions Objectives 42.

Introduction

43. Objectives 44. Virtual Platforms and SystemC 45. Checkpointing for SystemC 46. Conclusions Objectives 47. Objectives

Add SystemC-TLM to any Virtual Platform

Different strategies to add two simulators

48. Add SystemC-TLM support to an open-sourced VP Add checkpointing support to SystemC

C++ not checkpointable

49. Overcome limitations 50. Generic Virtual Platform Diagram Virtual Platform ISS ISS VP Back-end functions Devices Devices Memories Target Drivers Boot code Target Operating System User User User SystemC SystemC Devices Sync 51.

Introduction

52. Objectives 53. Virtual Platforms and SystemC 54. Checkpointing for SystemC 55. Conclusions Virtual Platforms and SystemC 56.

Introduction

57. Objectives 58. Virtual Platforms and SystemC 59. Checkpointing for SystemC 60. Conclusions Virtual Platforms and SystemC 61. Virtual Platforms and SystemC

Link together two simulators

62. Synchronization strategy 63. Generic bridge 64. Support for generic TLM-2 devices

LT, AT, DMI...

65.

Link together two simulators

Virtual Platforms and SystemC SynchVPSystemCVirtual time 66. Virtual Platforms and SystemC Execution time SynchVPSystemCVirtual execution 67. QEMU-SC

Transition from RTL to TLM

68. Generic fabric bus (TLM)->any architecture 69. QEMU simul. master and SystemC slave

QEMU manages simulation

Focus on few SystemC devices 70. QEMU-SC Linux Driver VP SystemC module SC_Bridge SC_Link TLM Socket Application 71. Synchronization

Only synchronize when needed ( sc_start() )

When SystemC devices are accessed

72. When pending event in current simulation time

SystemC events list

When I/O to/from SystemC device

Capture or notify all I/O in SystemC device

QuantumKeeper asks to

To adhere to standard

73. Communication

QEMU works with zero-delay communication

CPU accesses one device and the device responds immediately

Fit to LT devices 74. Need to manage AT devices

Special synchronization

75. Finish all protocol phases before return to VP 76. Communication - LT 77. Communication - AT 78. Synchronization

Manage external I/O from/to SC device

Devices use QEMU callbacks for I/O

79. Bridge knows when a I/O is performed

Synchronize simulators

Continue SystemC simulation when

VP time arrives to first SC event

80. Every quantum time (TLM-2) Advance SystemC time until transaction ends 81. QBox

Change simulation manager

QEMU becomes simulator slave

SystemC manages simulation 82. QEMU is a TLM-2 Initiator module 83. Easy integration 84. Focus on many SystemC 85. models 86. QBox 87. QBox internal architecture 88. QBox complex example QEMUWrapper NIC TILE TILE TILE QEMU NoC 89. Test & results

Different tests and examples

Validate implementation

90. Extract performance metrics Results for performance

relatives to same system in native language

C in QEMU & Qbox

91. QEMU-SC Test System TLM Socket QEMU SystemC iDCT acc. SC_Bridge SC_Link DMI Pointer Reg. File IRQ IRQLinux Driver MPEG accelerator irq_event MPEG-2 decoder 92. QEMU-SC Results

Noacc.

93. No acc. w/ drivers 94. QEMU style 1 acc. 95. SystemC 1 acc. 96. QEMU style 2 acc. 97. SystemC 2 acc. 98. SystemC skel. (1 acc.) 99. SystemC skel. (2 acc.) 100. QEMU-SC Results

Noacc.

101. No acc. w/ drivers 102. QEMU style 1 acc. 103. SystemC 1 acc. 104. QEMU style 2 acc. 105. SystemC 2 acc. 106. SystemC skel. (1 acc.) 107. SystemC skel. (2 acc.) Penalty8% ~ 14% 108. Complex QBox system DMI Pointer TLM 2 Socket SignalSocket (IRQ) QEMU Wrapper GreenRouter TLM 2 Socket (Ethernet) Linux QEMU NE2000 NE2000 Testbench 109. QBox Test System TLM Socket SystemC iDCT acc. Reg. File IRQSignalSocket (IRQ) QEMU Wrapper GreenRouter Linux Driver irq_event QEMU MPEG-2 decoder 110. QBox Results

Noacc.

111. No acc. w/ drivers 112. QEMU style 1 acc. 113. SystemC 1 acc. 114. QEMU style 2 acc. 115. SystemC 2 acc. 116. SystemC skel. (1 acc.) 117. SystemC skel. (2 acc.) 118. QBox Results

Noacc.

119. No acc. w/ drivers 120. QEMU style 1 acc. 121. SystemC 1 acc. 122. QEMU style 2 acc. 123. SystemC 2 acc. 124. SystemC skel. (1 acc.) 125. SystemC skel. (2 acc.) Penalty~ 100%!!! 126. QBox Test System TLM Socket SystemC iDCT acc. Reg. File IRQSignalSocket (IRQ) QEMU Wrapper GreenRouter Linux Driver irq_event QEMU MPEG-2 decoder Penalty ~ 100%!!! 127.

Introduction

128. Objectives 129. Virtual Platforms and SystemC 130. Checkpointing for SystemC 131. Conclusions

Checkpointing for SystemC

132.

Introduction

133. Objectives 134. Virtual Platforms and SystemC 135. Checkpointing for SystemC 136. Conclusions

Checkpointing for SystemC

137.

Checkpointing for SystemC

Add checkpointing capabilities to SystemC language

138. Two approaches

Process save and restore

139. State-variables save and restore 140. Process save checkpointing Save Restore O.S. HW O.S. HW VP VP 141. Process save checkpointing Save Restore O.S. HW O.S. HW HW VP VP 142. Process save checkpointing Save Restore O.S. HW O.S. HW HW VP VP 143. Process save checkpointing Save Restore O.S. HW O.S. HW HW O.S. VP VP 144. Process save checkpointing Save Restore O.S. HW O.S. HW HW O.S. VP VP 145. State save checkpointing Save Restore O.S. HW O.S. HW VP VP chkp_mng chkp_mng 146. State save checkpointing Save Restore O.S. HW O.S. HW VP VP chkp_mng chkp_mng 147. State save checkpointing Save Restore O.S. HW O.S. HW HW VP VP chkp_mng chkp_mng 148. State save checkpointing Save Restore O.S. HW O.S. HW HW O.S. VP VP chkp_mng chkp_mng 149. Checkpointing for SystemC

Inspection to OSCI kernel to get/set:

Dynamic & static SC_METHOD events list

Non-destructive access

150. Text conversion for easy management Channels (SC_FIFO, SC_MUTEX, etc.)

state

151. stored values 152. Checkpointing for SystemC

User code needs to give its internal state

153. Implicit

Designer use special typesto publish state variables

gs_param from GreenSocs

Explicit

Designer use special callbacks to publish state variables

do_checkpoint()

154. do_restore() 155. Checkpoint manager

New class to OSCI kernel

156. Offers simulation checkpoint to external control

checkpoint()

157. restore() 158. chk_mng::checkpoint()

Collects data from:

Module state variables

159. Channels state and data 160. Pending events lists (for SC_METHOD)

Static sensibility

161. Dynamic sensibility Converts al l data to text

Portable

162. Editable 163. chk_mng::restore()

Restores all data, converting from text to:

module state variables

164. channels state and data 165. events (for SC_METHOD)

Static sensibility

166. Dynamic sensibility Simulation state ready to continue

sc_start() again

167. Checkpointing for SystemC

Restrictions to device modeling style

SC_METHOD only

168. SC_THREAD left to designer Guidelines

To model FSM using SC_METHOD

169. Avoid usage of SC_THREAD (in literature) Enough for VP modeling 170. Checkpointing for SystemC VP Device Reg. File

Member Variables

gs_param

171. Callbacks functions 172. Checkpointing for SystemC VP Device Reg. File

Member Variables

gs_param

173. Callbacks functions 174. Checkpointing for SystemC VP Device Reg. File

FSM

SC_THREAD to SC_METHOD

Member Variables

S3 S1 S2 175. Checkpointing for SystemC VP Device Reg. File

FSM

SC_THREAD to SC_METHOD

Member Variables

S3 S1 S2 176. Checkpointing for SystemC VP Device Reg. File IRQ

FSM

Member Variables

Events

from OSCI events lists

IRQgenerator S3 S1 S2 177. Checkpointing for SystemC VP Device Reg. File IRQ

FSM

Member Variables

Events

I/O

Save Channels

IRQgenerator I/O manag S3 S1 S2 178. Simulation time

No performance penalty on OSCI kernel

New functions only used when doing checkpointing

Main performance loss is due to overhead in gs_param

Mosts cases less than 1%

179. Other cases about a 50% 180. Checkpoint data size

Checkpoint data depends on

state variable

181. pending events 182. channels dataUsually order of ~kB vs ~GB other solutions

Easily portable

183. Fast saving to disk time 184. Reverse execution 185. Checkpoint data # OSCI data 55000 60000;Initiator.initiator_func; # FIFO my_fifoTarget.my_fifo=14;Target.my_fifo=0; Target.my_fifo=0.5;Target.my_fifo=1; ... Target.my_fifo=6.5; # gs_params Initiator.initiator_func_fsm_param 0 Initiator.int_param 6 Target.data_gs_param {"0" "0.5" "1" "1.5" "2" "2.5" "0" "0" "0" "0"} Target.data_gs_paramB.0 0 Target.data_gs_paramB.1 0.5 Target.data_gs_paramB.2 1 Target.data_gs_paramB.3 1.5 Target.data_gs_paramB.4 2 ... Target.data_gs_paramB.9 0 OSCI kernel data Channels data User modules data 186.

Introduction

187. Objectives 188. Virtual Platforms and SystemC 189. Checkpointing for SystemC 190. Conclusions Conclusions 191.

Introduction

192. Objectives 193. Virtual Platforms and SystemC 194. Checkpointing for SystemC 195. Conclusions Conclusions 196.

Joined SystemC to two Virtual Platforms

197. Tested two different strategies for joining two simulators

QEMU-SC

198. QBox Minor performance impact SystemC bridge 199. Published as open-sourced projects www.greensocs.com Conclusions SystemC 200. Conclusions SystemC Simulation Manager Penalty System for QEMU-SC Yes 10% SW QBox No 25~30% HW 201. Conclusions Simics Virtutech

Add SystemC capabilities to a commercial VP

202. Close source -> new device manages SystemC 203. Same strategy than QEMU-SC 204. Conclusions Simics Results

SystemC QT = 1 sec.

205. SystemC QT = 1 ms. 206. SystemC QT = 1 s. 207. Simics DML Microbenchmark 208. Conclusions Simics Results

SystemC QT = 1 sec.

209. SystemC QT = 1 ms. 210. SystemC QT = 1 s. 211. Simics DML Microbenchmark Penalty ~ 5% 212. ConclusionsIndustry

R&D from university to industry

213. Industry project involves PhD

Commercial tool: Simics

Other vendors using our work

Synopsys Innovator and QBox

214. ConclusionsCheckpointing

Added checkpointing to SystemC

215. Saving only necessary data, not the entire process 216. Focused on VP modeling 217. With limitations but enough for VP devices 218. No simulation speed penalty due to Checkpointing

Use of callbacks

219. Inspect OSCI kernel when checkpointing 220. Future Work (light)

Automagic configuration of QBox systems

Manage map-address in QEMU, Router, BIOS, etc.

Enhance QEMU time management

Hard to measure virtual time in QEMU

Add multiple instances from QEMU

Current QBox library allows one

Explore QEMU user mode

Simplified version, only ISS, run applications

221. Future Work (medium)

Merge SystemC methods into QEMU kernel

Remove OSCI simulator and write an API SystemC QEMU

Merge QEMU functionality into OSCI kernel

Make QEMU a truly SystemC ISS

Both merges increase simulation speed due to removed synchronization 222. Future Work (hard)

Develop an automatic tool for SC_THREAD to SC_METHOD translation

223. Build a SystemC Virtual Machine

allows native checkpointing

224. performance issues 225. Thank you! Questions? 226. BACKUP SLIDES 227. Conclusions Simics Virtutech 228. QEMU ecosystem