stackless python in eve
DESCRIPTION
Stackless Python in EVE. Kristján Valur Jónsson [email protected] CCP Games inc. EVE. MMORPG Space game Client / server Single shard massive server 120.000 active players, >24.000 concurrent users World concurrency record on a shard Relies on Stackless Python. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/1.jpg)
Stackless Python in EVE
Kristján Valur Jónsson
CCP Games inc.
![Page 2: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/2.jpg)
EVE
• MMORPG Space game
• Client / server
• Single shard massive server
• 120.000 active players, >24.000 concurrent users
• World concurrency record on a shard
• Relies on Stackless Python
![Page 3: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/3.jpg)
The Tranquility cluster• 400 GHz CPU / 200 Gb
RAM• 2 Routers (CISCO Alteon)• 14 Proxy servers (IBM
Blade) • 55 Sol servers (IBM x335)• 2 DB servers (clustered,
IBM Brick x445)• FastT600 Fiber, 56 x FC
15k disks, DS4300 + 3*EXP700
• Windows 2000, MS SQL Server
• Currently being upgraded• AMD x64
![Page 4: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/4.jpg)
EVE Architecture
• COM-like basic architecture
• Python tighly integrated at an early stage
• Home-grown wrapping of BLUE objects
python modulesblue modules
ExeFile.exe
Netclient.dll
Trinity.dll
...
blue.dll
Pythonlib.libautoexec.py
...
bar.py
foo.py
![Page 5: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/5.jpg)
Stackless Python
• Tasklets
• Threads of execution. Not OS threads
• Lightweight
• No pre-emption
• Channels
• Tasklet rendezvous point
• Data passing
• Scheduling
• Synchronization
![Page 6: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/6.jpg)
Stackless?
• No C stack
• Python stack in linked
frame objects
• Tasklet switching by
swapping frame chain
• Compromise
• stackless where
possible.
• C stack whisked away
if necessary
main()
PyStackless_CallMethod_Main(“MyMain”)
def MyMain(self): cmodule.Frobnicate(self.Callback)
PyFrobnicate() { PyObject_CallObject(cb, …); }
def Callback(self): self.channel.send(42)
Stack switch
![Page 7: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/7.jpg)
Channels
Channel
Readers Writers
Tasklet 1
Tasklet 2
![Page 8: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/8.jpg)
Channel semantics
• Send on a channel with no receiver blocks tasklet.
• Send on a channel with a (blocked) receiver,
suspends tasklet and runs receiver immediately.
Sender runs again in due course.
• Symmetric wrt. Send and Receive.
• “balance”, can have a queue of readers or writers.
• Conceptually similar to Unix pipes
![Page 9: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/9.jpg)
Channel semantics, cont.
• Scheduling semantics are precise:
• A blocked tasklet is run immediately
• Usable as a building block:
• semaphores
• mutex
• critical section
• condition variables
![Page 10: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/10.jpg)
Stackless in EVE
• BLUE foundation: robust, but cumbersome
• RAD
• Stackless Python: Python and so much more
• EVE is inconceivable without Stackless
• Everyone is a programmer
![Page 11: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/11.jpg)
The main loop
• Establish stackless context
int WinMain(...) {
PyObject *myApp = new EveApp();
PyObject *r = PyStackless_CallMethod_Main(MyApp,
“WinMain”, 0);
return PyInt_AsLong( r );
![Page 12: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/12.jpg)
PyObject* EveApp::WinMain(PyObject *self, PyObject *args) { PyOS->ExecFile("script:/sys/autoexec.py"); MSG msg; while(PeekMessage(&msg, 0, 0, 0, PM_REMOVE)){
TranslateMessage(&msg);DispatchMessage(&msg);
} for (TickIt i = mTickers.begin(; i !=
mTickers.end(); i++)i->mCb->OnTick(mTime, (void*)taskname);
}
• Regular Windows message loop• Runs in Stackless context• The “Main Tasklet”
The main loop cont.
![Page 13: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/13.jpg)
Autoexec.py
import blue
def Startup(): import service srvMng = service.ServiceManager() run = ["dataconfig", "godma", “ui", …] srvMng.Run(run)
#Start up the client in a tasklet!if CheckDXVersion(): import blue blue.pyos.CreateTasklet(Startup, (), {})
![Page 14: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/14.jpg)
Tickers
• Tickers are BLUE modules:
• Trinity (the renderer)
• Netclient
• DB (on the server)
• Audio
• PyOS (special python services)
• …
![Page 15: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/15.jpg)
The PyOS tick:
• Runs fresh tasklets
• (sleepers awoken elsewhere)
Tick() {
…
mSynchro->Tick() PyObject *watchdogResult; do {
watchdogResult = PyStackless_RunWatchdog(20000000);
if (!watchdogResult)PyFlushError("PumpPython::Watchdog");
Py_XDECREF(watchdogresult); } while (!watchdogResult);
![Page 16: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/16.jpg)
blue.pyos.synchro
• Synchro:
• Provides Thread-like tasklet utilities:
• Sleep(ms)
• Yield()
• BeNice()
![Page 17: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/17.jpg)
1. Sleep: A python script makes the call
blue.pyos.Sleep(200)
2. C++ code runs:
1. Main tasklet check
2. sleeper = New Sleeper();
mSleepers.insert(sleeper);
PyObject *r =
PyChannel_Receive(sleeper-
>mChannel);
3. Another tasklet runs
blue.pyos.synchro cont.
![Page 18: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/18.jpg)
blue.pyos.synchro, ctd.4. Main tasklet in windows loop enters PyOS::Tick()5. mSleepers are examined for all that are due we
do:mSleepers.remove(sleeper);PyChannel_Send(sleepers.mChannel,
Py_NONE);6. Main tasklet is suspended (but runnable), sleeper
runs.
![Page 19: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/19.jpg)
Points to note:
• A tasklet goes to sleep by calling
PyChannel_Receive() on a channel which has
no pending sender.
• It will sleep there (block) until someone sends
• Typically the main tasklet does this, doing
PyChannel_Send() on a channel with a reader
• Ergo: The main tasklet may not block
![Page 20: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/20.jpg)
Socket Receive
• Use Windows asynchronous file API
• Provide a synchronous python API. A python script
calls Read().
• Tasklet may be blocked for a long time, (many
frames) other tasklets continue running.
• Do this using channels.
![Page 21: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/21.jpg)
Receive, cont.
1. Python script runs:
foo, bar = socket.Read()
2. C code executes the request:
Request *r = new Request(this);WSAReceive(mSocket, …);mServe->insert( r );PyChannel_Receive(r->mChannel);
3. Tasklet is suspended
![Page 22: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/22.jpg)
Receive, cont.
4. Socket server is ticked from main loop
5. For all requests that are marked completed, it
transfers the data to the sleeping tasklets:
PyObject *r =
PyString_FromStringAndSize(req->mData,
req->mDataLen);
PyChannel_Send(req->mChannel, r);
Py_DECREF(data);
delete req;
4. The sleeping tasklet wakes up, main tasklet is
suspended (but runnable)
![Page 23: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/23.jpg)
Receive completed
EveApp::WinMain()
Netclient::OnTick()
PyChannel_Send(mReceiver, data);
Stack switch
Cmodule::Work(PyObject *cb)
def TaskletMain():
Stack switch
def ProcessStuff:
def Callback():
Socket::Receive(PyObject *cb)
d = PyChannel_Receive(mReceiver);
![Page 24: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/24.jpg)
Main Tasklet• The one running the windows loop• Can be suspended, allowing other tasklets to run• Can be blocked, as long as there is another tasklet to
unblock it (dangerous)• Is responsible for waking up Sleepers, Yielders, IO
tasklets, etc. therefore cannot be one of them• Is flagged as non-blockable
(stackless.get_current().block_trap = True)
![Page 25: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/25.jpg)
Channel magic• Channels perform the stackless context switch.• If there is a C stack in the call chain, it will magically
swap the stacks.• Your entire C stack (with C and python invocations)
is whisked away and stored, to be replaced with a new one.
• This allows stackless to simulate cooperative multi-threading
![Page 26: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/26.jpg)
Co-operative multitasking
• Context is switched only at known points.
• In Stakcless, this is channel.send() and
channel.receive()
• Also synchro.Yield(), synchro.Sleep(), BeNice(),
socket and DB ops, etc.
• No unexpected context switches
• Almost no race conditions
• Program like you are single-threaded
• Very few exceptions.
• This extends to C state too!
![Page 27: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/27.jpg)
Tasklets• Tasklets are cheap• Used liberally to reduce perceived lag
• UI events forked out to tasklets
• A click can have heavy consequences.• Heavy logic• DB Access• Networks access
• special rendering tasks forked out to tasklets.• controlling an audio track
• “tasklet it out”• Use blue.pyos.synchro.BeNice() in large loops
![Page 28: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/28.jpg)
Example: UI Event:
• Main tasklet receives window messages such as
WM_CLICK
• Trinity invokes handler on UI elements or global
handler
• Handler “tasklets out” any action to allow main
thread to continue immediately.
def OnGlobalUp(self, *args): if not self or self.destroyed: return mo = eve.triapp.uilib.mouseOver if mo in self.children: uthread.new(mo._OnClick) class Action(xtriui.QuickDeco): def _OnClick(self, *args): pass
![Page 29: Stackless Python in EVE](https://reader038.vdocuments.mx/reader038/viewer/2022102818/5681349e550346895d9b93cb/html5/thumbnails/29.jpg)
That’s all
• For more info:
• http://www.ccpgames.com
• http://www.eve-online.com
• http://www.stackless.com