jozef ve providing scalability for pirates, lizards and zombies at #doxlon

of 50/50
Providing scalability for Pirates, Lizards and Zombies. August 2014 Amanda Pla)orm for distributed services

Post on 15-Jan-2015

457 views

Category:

Technology

3 download

Embed Size (px)

DESCRIPTION

Video: https://www.youtube.com/watch?v=OGjaXtauO-c Jozef will talk about building a new generation of distributed services framework called Amanda at MPC. MPC is a leading visual FX company that works with extremely high volumes of data and a vast array of computing power. This talk gives a behind the scenes (sorry) look at how art assets get processed at scale. And like all good talks it includes plenty of metrics!

TRANSCRIPT

  • 1. Providing scalability forPirates, Lizards andZombies.August 2014Amanda&Pla)orm&for&distributed&services

2. Who am ISoftware developer at MPC!Python since 2009!Loves services!Obsessed by [email protected] on twitter 3. Who is MPCWe create Visual Effects!AdvertisingFeature films 4. We create Visual Effects!Across 8 sitesFully integrated crosssite pipeline 5. What are visual effects?Visual effects involve the integration of live-action footageand generated imagery to create environments which lookrealistic, but would be dangerous, expensive, impractical, orsimply impossible to capture on film.(source wikipedia) 6. What are visual effects? 7. What are visual effects? 8. What are visual effects? 9. What are visual effects? 10. Production workflow. 11. Production considerations Management Assets, Production, Users Various sources Used by: In house tools 3rd party applications Multiple environments Users Artists Developers 12. MPCs approach - Amanda Service based architecture Platform as a service Multi protocol, transport and concurrency Ecosystem to write a service for developers of any level 1st generation 2007 2nd generation 2012-2013 13. Some statistics.! Godzilla 450 years if rendered on a single machine 14. Some statistics.! Godzilla 450 years if rendered on a single machine Or, 15 years after Henry 8th died 15. Some statistics.! Godzilla 650 TB of data 16. Some statistics.! Godzilla 650 TB of data Or, 41000 mobile phones with 16GB drives. 17. Some statistics.! Godzilla 250.000 Amanda req/min. 18. Some statistics.! Godzilla 250.000 Amanda req/min. Or, 4 times Germany VS Brazil tweets 19. Amanda - building the platform 20. Amanda - building the platform 21. Amanda - building the platformServicesclass MakeMovieService(object):[email protected]!def greet_director(self, director_name):!return "hello %s" % (director_name,)! 22. Amanda - building the platformWe have a service, but lets make it do something useful 23. Amanda - building the platformInfrastructuresclass MakeMovieService(object):!storage = Dependency(storage")!def _check_data_on_disk(self, files):!""" Check data is available!"""!return self.storage.check_exists(files)! 24. Amanda - building the platformInfrastructuresclass MakeMovieService(object):!storage = Dependency(storage")!def _check_data_on_disk(self, files):!""" Check data is available!"""!return self.storage.check_exists(files)!def _get_shot(self, shot):!""" Get shot data!"""!query = ""!return self._db.execute(query, (shot,))!! 25. Amanda - building the platformInfrastructuresclass MakeMovieService(object):!storage = Dependency(storage")!def _check_data_on_disk(self, files):!""" Check data is available!"""!return self.storage.check_exists(files)!def _get_shot(self, shot):!""" Get shot data!"""!query = ""!return self._db.execute(query, (shot,))[email protected]!def make_movie_magic(self, shotInfo):!""" Do some magic!"""!shot = self._get_shot(shotInfo)!if not self._check_data_on_disk(shot["files"]):!self._log.error(files not found')!raise DataNotAvailable()!self._do_magic(shot["files"], self._config.production.jobsite()) 26. Amanda - building the platformOk now it is doing something but that is one user,one service and not very useful at scale 27. Amanda - building the platform 28. Amanda - building the platformService providersservices = ServiceProvider(!make_movies=MakeMovieService,!storage=StorageService!config=ConfigService!logging=LoggingService!__infrastructure__={!"_db": DbInfrastructure,!"_config": ConfigInfrastructure,!"_xmesh": XmeshInfrastructure,!"_log": LoggingInfrastructure!}!)!services.make_movies.make_movie_magic("shot1")!services.logging.change_logging_level("make_movies", "debug")! 29. Amanda - building the platform 30. Amanda - building the platformProxiesclass Proxy(object):!!def __getattr__(self, method):!def method_stand_in(*args, **kwargs):!return self._transport.publish(self._service, method, *args, **kwargs)!return method_stand_in!!services = ServiceProvider(!make_movies=Proxy("make_movies", transport)!)!services.make_movies.do_movie_magic(shot1)!! 31. Amanda - building the platform 32. Amanda - building the platformOk, at this point we can scale a bit but what aboutconcurrency/parallel processing? 33. Amanda - building the platform 34. Amanda - building the platform 35. Amanda - building the platform 36. Amanda - building the platformThis gives us all the building blocks to assemble adistributed system with the possibility to use thebest technologies for the given tasks 37. Amanda - assembling & maintaining the platform 38. Amanda - assembling & maintaining the platformSetting up for a cross language pipeline. 39. Amanda - assembling & maintaining the platform 40. Amanda - assembling & maintaining the platform 41. Amanda - assembling & maintaining the platformSetting up for a cross language, fault tolerantpipeline. 42. Amanda - assembling & maintaining the platformSetting up for cross language, fault tolerant,monitorable pipeline. 43. Amanda - assembling & maintaining the platformrequest: userAdmin, getUserIdByUserName, [u'jozef'], {}request id: 1393351037.307669-userAdmin-staffdirectory-3500,by jozef from staffdirectoryexception: , Nonehandled by: amanda-host-1_28368-GeventConcurrencyPool_worker_37stats:sent on Tue Feb 25 17:57:17 2014received on Tue Feb 25 17:57:17 2014frontend->backend delivery took 0.003s`internal queuing took 0.0000015srequest execution took 0.005stotal time: 0.011s 44. Amanda - assembling & maintaining the platforminternalqueuing:0.0000015sfrontend->backend: 0.003stotal time: 0.011srequestexecution:0.005s 45. Amanda - assembling & maintaining the platform Maintenance. Salt Extended with an Amanda module. Salt client wrapped inside a service so we can access it from anywhere notjust the master node. Predefined, repeatable configuration. 46. Amanda Adaptable Extendable Configurable Best of breeds 47. Amanda Looking ahead Containerisation Auto scaling Open sourcing possibilities 48. In ProductionMontral 49. Were Now Hiring!!Opportunities across all our Studios and across alldisciplines!Check out www.moving-picture.com for vacancies!Or email [email protected] 50. AmandaThank you, any questions?Jozef van Eenbergen - twitter: @JozefVanE