attila research group

1

Attila Research Group

attila.ac.upc.edu

Computer Architecture DepartmentUniv Politècnica de Catalunya (UPC)

2

Attila Project

• Started 2003• Research on GPUs

– Focus on the microarchitecture– Use real games as workloads– Analyze bandwidth/latency/threading tradeoffs

• Spent large fraction of time developing tools• Currently three PhDs in progress

• Funding from– CICYT / Ministry of Education, Spain (2)– Intel (1)

• 2 Students spent 6 months with ATI

3

Attila Team

• Faculty – Agustin Fernandez

• 3 Ph.D. Students– Victor Moya -- Hired by Intel / VCG ’06– Carlos González -- 6 months internship at ATI (Jun’07)– Jordi Roca -- 6 months internship at ATI (Jun’07)

• Master Thesis– Chema Solis – DX9 Driver Development

• Alumni– David Abella – DX9 Player and PIX reader– Christian Perez – Color Compression in Attila

• Industrial Advisor– Roger Espasa, Intel VCG

Attila Facts

• Simulation time– 1 frame @1280x1024 per hour

• Lines of code– Simulator: 142697 lines– Library, driver and trace tools: 217266 lines

• ACDL : 37791 lines• OpenGL : 35960 lines• D3D9: 17348 lines

5

Attila Publications• Conference Papers

– Workload Characterization of 3D Games.Jordi Roca, Victor Moya, Carlos González, Chema Solis, Agustín Fernández and Roger Espasa.IEEE International Symposium on Workload Characterization (IISWC-2006), pp. - , January 2006.

– ATTILA: A Cycle-Level Execution-Driven Simulator for Modern GPU Architectures.Víctor Moya, Carlos González, Jordi Roca, Agustín Fernández and Roger Espasa.IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS-2006), March 2006.

– Shader Performance Analysis on a Modern GPU Architecture.Víctor Moya, Carlos González, Jordi Roca, Agustín Fernández and Roger Espasa.The 38th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO-38), November 2005.

– A Single (Unified) Shader GPU Microarchitecture for Embedded Systems.Víctor Moya, Carlos González, Jordi Roca, Agustín Fernández and Roger Espasa.2005 International Conference on High Performance Embedded Architectures & Compilers (HiPEAC 2005), November 2005.

• Master Thesis– Caracterización e implementación de algoritmos de compresión en la GPU ATILA (Text in Spanish)

Christian Perez. Master Thesis for the Graduate Studies, January 2008.

– Extensión a Direct3D del driver de un simulador de GPU (Text in Spanish) Chema SolisMaster Thesis for the Graduate Studies, July 2007.

– Librería Direct3D (Text in Catalan) David Abella Master Thesis for the Graduate Studies, July 2007– Shader generation and compilation for a programmable GPU (Text in Spanish) Jordi Roca.

Master Thesis for the Graduate Studies, July 2005. – Support tools for a 3D graphics processor simulation framework (Text in Spanish) Carlos González.

Master Thesis for the Graduate Studies, June 2004.

6

Outline

• Attila Tracing Environment

• Attila Architecture & Simulator

• Current Research– Shaders– Memory Hierarchy– Micropolygons– DX9 Driver Development

7

Prey

Riddick Quake 4 Doom 3 UT2004

Half Life 2

Supported workloads

and upcoming D3D games …

8

Collect Verify Simulate Analyze

OGL/D3D App

OGL/D3DCapturer

Vendor OGL/D3DDriver

Trace

ATI R600/NVIDIA G80

Framebuffer

OGL/D3DPlayer

µ-ArchStatistics

Signal Traffic

or Microsoft PIX Capturer

or Attila Pix Player

Framebuffer

ATI R600/NVIDIA G80

Vendor OGL/D3DDriver

Framebuffer

ATTILA Simulator

ATTILA OGL/D3D Driver

Signal TraceVisualizer

Internal traces(mem,$,…)

Detailed cycle-to-cycle visualization

CHECK CHECK

API Stats

9

CollectCollect Verify Simulate Analyze

OGL/D3D App

OGL/D3DCapturer

Vendor OGL/D3DDriver

Trace

ATI R600/NVIDIA G80

Framebuffer

OGL/D3DPlayer

µ-ArchStatistics

Signal Traffic

or Microsoft PIX Capturer

or Attila Pix Player

Framebuffer

ATI R600/NVIDIA G80

Vendor OGL/D3DDriver

Framebuffer

ATTILA Simulator

ATTILA OGL/D3D Driver

Signal TraceVisualizer

Internal traces(mem,$,…)

Detailed cycle-to-cycle visualization

CHECK CHECK

API Stats

API Capturers• Capture API calls from a real game• Gather API level statistics

10

Simulate Analyze

OGL/D3D App

OGL/D3DCapturer

Vendor OGL/D3DDriver

Trace

ATI R600/NVIDIA G80µ-Arch

Statistics

Signal Traffic

or Microsoft PIX Capturer

Framebuffer

ATTILA Simulator

ATTILA OGL/D3D Driver

Signal TraceVisualizer

Internal traces(mem,$,…)

Detailed cycle-to-cycle visualization

CHECK CHECK

API Stats

API Players• Trace checking/integrity• Batch-to-batch playing (helps debug)

OGL/D3DPlayer

or Attila Pix Player

Framebuffer

ATI R600/NVIDIA G80

Vendor OGL/D3DDriver

Framebuffer

Collect VerifyVerify

11

SimulateSimulate Analyze

OGL/D3D App

OGL/D3DCapturer

Vendor OGL/D3DDriver

Trace

ATI R600/NVIDIA G80µ-Arch

Statistics

Signal Traffic

or Microsoft PIX Capturer

Signal TraceVisualizer

Internal traces(mem,$,…)

Detailed cycle-to-cycle visualization

CHECK CHECK

API Stats

Simulation• Attila Drivers

• AOGL (90%)• AD3D9 (60%)

• Attila Simulator• Detailed cycle-to-cycle simulation • 20 Boxes modeling 100-deep pipeline• Execute@Execute: Functionality embedded at each pipeline stage

Framebuffer

Collect

OGL/D3DPlayer

or Attila Pix Player

ATI R600/NVIDIA G80

Vendor OGL/D3DDriver

Verify

Framebuffer

ATTILA Simulator

ATTILA OGL/D3D Driver

Framebuffer

12

AnalyzeAnalyze

OGL/D3D App

OGL/D3DCapturer

Vendor OGL/D3DDriver

ATI R600/NVIDIA G80

or Microsoft PIX Capturer

CHECK CHECK

API Stats

Framebuffer

Collect

OGL/D3DPlayer

or Attila Pix Player

ATI R600/NVIDIA G80

Vendor OGL/D3DDriver

Verify Simulate

Framebuffer

ATTILA Simulator

ATTILA OGL/D3D Driver

µ-ArchStatistics

Signal Traffic

Signal TraceVisualizer

Internal traces(mem,$,…)

Detailed cycle-to-cycle visualization

Framebuffer

Trace

Simulation output• Micro-architectural statistics• Traffic for cache, mem, …• Signal trace (input for STV tool)

• Debug simulation performance

13

Attila Drivers• OpenGL driver

– 200 API calls supported.

– 80% OpenGL 2.0 fixed functionality

• DirectX9 driver– About 50 calls

supported.– 60% API

functionality.

ATTILA Architecture

HAL

Attila OpenGL Driver(GLLIB)

Attila DX9 Driver(D3DLIB)

14

Unified Driver Architecture

• Currently stalled due to lack of resources

• Runs basics traces– Non-textured torus with simple vtx shader.

ATTILA Architecture

HAL

ACDLX ACDL

AOGL* AGL/ES ADX9* ADX10 AREY

15

Outline

• Attila Tracing Environment

• Attila Architecture & Simulator

• Current Research– Shaders– Memory Hierarchy– Micropolygons– DX9 Driver Development

16

Attila Architecture

Me

mo

ryC

on

tro

ller

Me

mo

ryC

on

tro

ller

Me

mo

ryC

on

tro

ller

Me

mo

ryC

on

tro

ller

RO

PR

OP

RO

PR

OP

Shader

Shader

Shader

Shader

Vertex Fetch

Primitive Assembly

Clipping

Triangle Setup

Rasterization

HierarchicalZ

Scheduler

Distributor

Unified shaders, multithreaded … GDDR4 detailed protocol, selectable memory schedulers…

17

Attila Simulator ImplementationUsing Boxes & Signals

StreamerFetch

StreamerOutputCache

StreamerCommit

StreamerLoader

PrimitiveAssembly

ClipperTriangleSetup

FragmentGenerator

HierarchicalZ

ShaderFetch

ShaderDecodeExecute

TextureUnit

FragmentFIFO

Interpolator

Z StencilTest

ColorWrite

DAC

CommandProcessor

MemoryController

STREAMER/VERTEX FETCH

SHADER

Data-driven & cycle-accurate

18

Lots of configurable parameters

GPU Unit Params Examples

COMMAND PROCESSOR 1 Batch pipelining

MEMORY CONTROLLER 42Size, channels and banks (number and interleaving).

STREAMER 13 Fetched indices and attributes per cycle

PRIMITIVE ASSEMBLY 4 Assembled triangles per cycle

CLIPPER 5 Clipping latency

SETUP + RASTERIZER 43 MSAA samples/cycle, Enabled HZ

UNIFIED SHADER UNIT 39 Fetch Instrs/cycle, temp regs, scalar ALU

TEXTURE CACHE 19 Line size, ways, port width

ROP (Z + COLOR) 47 Compression, cache size.

DAC 9 Refresh rate

TOTAL 222

19

Statistics – High Level• API level

• µ-arch level

• “Workload Characterization of 3D Games”, IEEE International Symposium on WC 2006

Quake 4 Vertex Cache Hit Rate

0

0.2

0.4

0.6

0.8

1

1 101 201 301 401 501 601 701 801 901 1001Frames

Hit

Rat

e

UT2004 Mem BW Consumed per Unit

020406080

100120140160

1 101 201 301 401Frames

MB

s

color ztexture vertexdac

Doom3 Unit Utilization

0

0.2

0.4

0.6

0.8

1

1 101 201 301 401 501 601 701 801Frames

mem shExectxALU rop

Prey Average Shader Instructions

0

5

10

15

20

25

30

1 235 469 703 937 1171 1405 1639 1873 2107 2341 2575 2809

Frame

inst

ruct

ions

Vertex instructionsFragment instructionsTexture instructions

FEAR Texture Filter

0

0.2

0.4

0.6

0.8

1

1 50 99 148 197 246 295 344 393 442 491 540 589 638 687

Frame%

filt

er

NEARESTNEAREST_MIPMAP_LINEARANISO BILINEARANISO TRILINEAR

Half Life 2 Triangles per frame

0

100000

200000

300000

400000

500000

1 167 333 499 665 831 997 1163 1329 1495 1661Frame

tria

ngle

s

20

Statistics – Zooming InDoom3 - Frame 0836 Unit Utilization detail

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1

Texture ZStencil Shader

Stencil pass Shading pass Stencil pass Shading pass

Light 0 Light 1

• Fine-grain stats at configurable fractions of i.e: 100, 1K, 10K or 100K execution cycles.

21

Statistics – Cycle Level

22

Outline

• Attila Tracing Environment

• Attila Architecture & Simulator

• Current Research– Shaders– Memory Hierarchy– Micropolygons– DX9 Driver Development

GPU Memory Hierarchy Optimizations

Carlos González

cgonzale@ac.upc.edu

Previous Work

1. Initial Attila’s Boxes & Signals framework2. Tracing Framework

– GLInterceptor & GLPlayer Tools– OpenGL Driver for Attila– Signal Trace Visualizer tool

3. New highly-detailed Memory Controller for Attila4. Internship at ATI (6 months, 07’)

– Work mainly focused on the MC block– Analysis of bandwidth and latency by means of simulation

techniques– Some contributions to the initial system

• Mechanisms to pinpoint sources of latency and analyze bandwidth over time slices

Today’s GPUs remarks

• Tremendous bandwidth available– Core 2: 12 GB/sec VS NVIDIA G80 > 100 GB/sec

• But…– Dozens of clients accessing memory simultaneously– Unbalance and inefficient scheduling of memory

transactions can lead to poor performance• Workload unbalance

– Total available BW decreases• Inefficient scheduling

– Latency increases (DDR protocol overhead)

• Overall performance degradation

Thesis Goals

1. Optimize bank mapping and load balancing among memory channels. Also, propose multiple separated address spaces (per client)

2. Propose efficient memory controller scheduling algorithms

• Also: Measure DRAM chips consumption of our proposals

3. Propose new cache hierarchies for ROP and Texture units

4. Research in interconnection topologies

Some experiments…Doom 3 Total execution time

5

6

7

8

9

10

11

12

13

14

15

64 128 256 512 1024 2048 4096Channel interleaving (bytes)

Execu

tio

n t

ime (

mil

lio

ns o

f cycle

s)

Doom3 - Scheduler Performance Analysis

0

2

4

6

8

10

12

14

16

18

229 240 260 325 350 375

Frames

Ex

ec

uti

on

tim

e (

in m

illio

ns

of

cy

cle

s)

17’8%

17’8%

17’6% 17’1

%13’6%

17’1%

Channel Interleaving Analysis

Some config. parameters of the simulation

• 4 channels of 64-bit

• 8 banks per 32-bit IO chip

• Channel interleaving = 256 bytes

• Bank interleaving = 1024 bytes

• 4 unified shaders (4x)

• Texture cache line (L1) = 64 bytes

• Texture cache ways (L1) = 16

• Texture cache lines (L1) = 16

• Color and Zstencil caches:

4 ways

• line size = 256 bytes - 16 cache lines

Some config. parameters of the experiment

• 8 channels of 32-bit

• 8 banks per 32-bit IO chip

• Bank interleaving fixed to 256

• 4 unified shaders (4x)

Memory Scheduling Analysis

28

Micropolygon Rendering

Jordi Roca

jroca@ac.upc.edu

29

Past work1. OpenGL Fixed Function to ARB vp/fp 1.0 translator.

2. Workload Characterization of 3D Games (IISWC´06):– Extensive analysis of current games in terms of both

API call and µarchitectural level stats.

3. Multi-GPU performance evaluation project (at ATI 2007´s internship):– Hybrid SFR/AFR modes.– Alternatives for RTT surface synchronization.– Scaling of current PCIe BW.

(Related paper is currently submitted at the IISWC 2008).

30

Micropolygon rendering• Understanding and characterizing the pipeline backend

unbalance due to very small polygons.– Newer games tend to render outsides, thus projecting polygons

of a few pixels size.

Synthetic micropolygon test:

Fills the screen with 1 pixel aligned quads:

Raster Input: 1 triangle/clockRaster Output: 15/16 empty slots/clock

(high-end cards).

Unit utilization

0

0.2

0.4

0.6

0.8

1

Mem Shader Raster ROP

31

Research on:• Proposal #1: µpolygon grid traversal scheme:

– An alternative rasterization path to detect and efficiently traverse grids of adjacent pixel-size primitives:

• Fill backend slots combining fragments of different primitives.

• Reuse triangle setup and traversal computations for pixel proximate primitives.

• Proposal #2: Dynamic balancing of rasterization workload: – Assign & schedule shader threads for rasterization.

32

DX9 Driver Development

Chema Solís

csolis@ac.upc.edu

Project target

• Project target is to use D3D9 games as workload for ATTILA GPU simulator.

• Two main tasks:– Trace D3D9 calls executed by the games.– Build a D3D9 driver on top of GPU simulator.

D3D application

Microsoft D3D9 ATTILA D3D9 driver

D3D9 Trace

PixRun Player

• Executes traces of calls to D3D9 captured by Microsoft PIX.• Analyse how the game is using D3D9.

D3D9 Driver

• D3D9 functionality is being added progressively.• The driver is close to support commercial games.

36

Unified Shader Architecture

Victor Moya

vmoya@ac.upc.edu

Unified Shader Architecture

• Evaluated performance of an unified vertex and fragment shader architecture on legacy applications– Evaluated area vs performance

• Evaluated the performance of implementing Triangle Setup on the shader for embedded GPU architectures

• Evaluated bottleneck of GPU architectures with high shader ALU to texture.

Current Research

• Evaluate thread and resource scheduling in an unified shader architecture

• Implementation blending on the shader