optimizing key parts of the dunia pipelinegood state: all-in ! early test show potential wins. 4...

Engine Architect - FarCry 4Ubisoft Montréal @azagoth

Rémi QUENIN

Optimizing key parts of the

DUNIA PIPELINEFast iteration for

THE DUNIA PIPELINE 250 GiB of package per Day

Official + Test Maps

Over 3.7 Million Lines Of Code2.5 M. C++ Runtime

6 studios, 450 people in MontréalGetting data everyday

~500 checkins per day350 data, 150 code (+sound)

DATA FLOW IN THE DUNIA PIPELINE

DATA FLOW IN THE DUNIA PIPELINE

DATA FLOW IN THE DUNIA PIPELINE

BinarizeBinarize

AUTOMATED TESTS IN THE DUNIA PIPELINE

BinarizeBinarize

FC3 WORST CASE SCENARIO

BinarizeBinarize

<1 min

<5 min

~40 min

~45 min

~15 min

FC3 WORST CASE SCENARIO

<1 min

2/15 min

~25 min

OPTIMIZING

1 Optimizing Compilation Time

2 Optimizing Nightly Builds

3 Optimizing Package Synchronization

4 Optimizing Local Change Testing

The Pipeline

COMPILATION TIME: FASTBUILDFrom 40 min to 4 min

1

HISTORY Editor DLL : ~40 min

3.7 Million LOC

Unity/Blob builds : ~20 minBad Iteration

Engine Architect pet projectFASTBuild by Franta Fulin

Good State: All-in !Early test show potential wins

4POINTS

PARALLELIZATIONMaximize local resources

CACHINGShare compilation results

BLOBBINGImprove iteration

FASTBuild

Share HW resourcesDISTRIBUTION

WIN

PARALLELIZATION Compiling/Linking : DLL

PARALLELIZATION Before

After

Compiling : Static Libs

WIN

PARALLELIZATION

PARALLELIZATION

Visual Studio: Context Switching File Cache eviction

32 Core Machine: >1000 Processes!

PARALLELIZATION

FASTBuild: NUM_PROCESSORS

CACHING

Write

Read

CACHING

Write

Read WIN

CACHING Before

After

BLOBBING

WIN

Reducing header compilation

BLOBBING

WIN

Edit

Modified File

StableWIN

Iteration

DISTRIBUTION

WIN

MISCELLANEOUS

Link Dependencies Prevent useless relink

BatchingBuild machine, pre-submit checks

REAL TIMINGS Editor-X64-Release DLL (before)

Editor-X64-Release DLL BLOB (before)

Incredibuild

FASTBuild

~40 mins

<4 mins (cache/distribution)

~20 mins

~10 mins (local PC!)

PS4: <1 min (cache/distribution)

~12 mins

EDIT&CONTINUE

Too Much Code: Takes minutes +Microsoft dropping support

DLL with Hot ReloadAnother story: lot of great content available online

Incremental LinkingWhere it actually helps

NIGHTLY BUILDS4 Times Faster

2

Improvements

6 hours

1.5 hours

45 Worlds

75 Worlds

PROFILINGUnderstand, optimize

RESOURCE CACHINGShare

HOW ?

Profiling Multi-Process

Process isolation

Multi-Machine Cluster of Workers

Interactions Working together

Big PictureUnderstand

Remote Profiling

Remote Profiling

On Top of Regular Tool Custom Performance Analyzer

Processes Shown as Threads Analyze Interactions

Fix ! Address Inefficiencies

Improvements: CPU Usage

Resource Caching

?

Resource Caching

Share Prevent useless work

Build MachinePopulate at night

EditorJIT Compilation (Just In Time)

Complex SystemKey: Understanding

Prevent useless workHardware is crucial

NIGHTLIES

Prevent Editor UsageCurrent Bottleneck

More IncrementalSome parts are not

FUTURE WORK

OPTIMIZE PACKAGE SYNC.From 20min to 1sec

3

FARCRY3 AC3 Network

2012 SUMMER

NETWORK

OPTIMIZE PACKAGE SYNC.3

SOFTWARE: RTPAL3.1

HARDWARE: ASSET STORE

3.2

SYNCH. PACKAGE: RTPALSoftware

3.1

PC BUILD

Differential Uploads

Sliced ISO

~50% Saving

Upload

STEAM PIPE

WE KNOW OUR DATA

Big Files File boundaries

Redundancy Same Resource, Several Worlds

Amount of new Data ?Out of 20GB package

WE KNOW OUR DATA

Textures: Main Resource80% of a package

Textures: Revisions Average 3 revisions

SOLUTION

Slice BigFiles into Parts

Hash Parts using MD5

Produce a Manifest

Package

SLICINGInto Parts

SOLUTION

LINK

SOLUTIONVERSION 1 VERSION 2

?

LINK

RESULTS

1GB new per 22GB package

40% Shrink Within Package

Reuse

95%2 Versions

RESULTS

FC3: Few weeks + Milestones

FC4: >10 000 Manifests

History

1 YEARFull Prod. Proof

THIS IS GOOD, BUT…

Regular Workflow…Gym/Test Map

20% of the package20% of 5% = 1%

THIS IS GOOD, BUT…

Linking…20 GB to write

Deploy…zzzzz…..

ON DEMAND

STREAM parts on demand

OPEN WORLD game

NETWORK is faster than crappy

HDD

FILE SYSTEM

VIRTUALIZE File Access

Regular DISK

MANIFEST Content

FILE SYSTEM STACK

FILE SYSTEM

?

FILE SYSTEM STACK

NETWORK FS

?

FILE SYSTEM STACK

MANIFEST FS

NETWORK FS

?

?

FILE SYSTEM STACK

MANIFEST FS

NETWORK FS

CACHE FS

?

?

RTPAL

FILE SYSTEMFILE SYSTEMFILE SYSTEM

RTPAL

STORAGE AND TRANSFER REDUCTIONHistory, Transfers

UNIFIED WORKFLOWFor all platforms

INSTANT PACKAGE SYNC1 click

RTPAL

Re-Invents Package Distribution

By Getting RID of Package Distribution

RTPAL

SYNCH. PACKAGE: ASSET STOREHardware

3.2

HIGH PERFORMANCEIOPS/Throughput

QUICKLY SCALABLE, HIGH CAPACITYAdapt to workload

ROBUSTFailure tolerance

NEEDS

2012 SOLUTIONS

GLUSTER FS

Distributed file systemCluster of PCs

Replication between nodesUp to 3 copies of a file

2 Network interfacesOne for internal data exchange

GLUSTER FS

8 x High End PCsHPZ420 (32GB RAM, 6 HT CPUs 3.2GHz)

8 x High End SDDOn Sata3 RAID0 Controller card

2 x Network Interfaces10 Gb/s

TESTGLUSTER FS – 8 NodesTHROUGHPUT – 30 users, 14 GiB

IOPS – 100k x 1KiB

18MiB/s/user (13min)SAN/EMC

GLUSTER 73MiB/s/user (3min)

SAN/EMC

GLUSTER 544 000 IOPS

250 000 IOPS

GLUSTER - COST

0 500 1000 1500 2000 2500 3000

Speed (MiB/s)

050100150200250300350400

Cost (k$)

GLUSTER

SAN VIOLIN

SAN EMC

USAGE SCHEMES

RTPalPackage distribution

Resources cachingTransformed assets

Code compilation cacheFASTBuild

Re-Evaluate CEPHImproved since 2012

Improve Healing ProcessIf still with Gluster

FUTURE WORK

LOCAL ITERATION: DEVPATCHERFrom a Day to Minutes

4

FARCRY3 Fixes Nightmare

2012 SUMMER

WHAT IF…

Detect CHANGES

Handles COMPILATION

Handles BIG FILE creation

Introducing

PATCHINGas a Workflow

3STAGES

DETECTING CHANGECompile Dep.

PACKAGING CHANGEPatch BigFile

APPLYING CHANGEBigFile Stack

CREATE PATCH

DETECTING CHANGES

INPUTS

1

COMPILE DEP.

PACKAGING CHANGES2

NIGHTLY BUILD

PACKAGING CHANGES

OK

OK

FAIL

2

APPLYING CHANGES?

3

RTPal Compile Dep. STACK FS

DEVPATCHER & RTPAL

DUNIA

1 FASTBuild

2 Fast Nightly Builds

3 RTPal & Asset Store

4 DevPatcher

The Pipeline

THANKS & CREDITS

Franta Fulin / FASTBuild (http://fastbuild.org/) Jean-Francois Cyr / DevPatcher Olivier Deschamps / DevPatcher Laurent Chouinard / AssetStore Jonhatan Chin / AssetStore Jocelyn Hotte / AssetStore

&

QUESTIONS?Thank you!

@azagothremi.quenin@ubisoft.com

Rémi QUENIN