Introduction to CUDAheterogeneous programming

Katia Oleinikkoleinik@bu.edu

Scientific Computing and Visualization

Boston University

CUDA• Architecture• C Language extensions• Terminology

CUDA Basics• Hello, World!• CUDA kernels• Blocks and

threads overview

GPU memory• Memory management• Parallel kernels• Threads synchronization• Race conditions and atomic

operations

Architecture

NVIDIA Tesla M2070:

Core clock: 1.15GHz Single instruction 448 CUDA cores 1.15 x 1 x 448 = 515 Gigaflops double precision (peak)

1.03 Tflops single precision (peak)

3GB total dedicated memory

Delivers performance at about 10% of the cost and 5% the power of CPU

Architecture

CUDA:

• Compute Unified Device Architecture

• General Purpose Parallel Computing Architecture by NVIDIA

• Supports traditional OpenGL graphics

Architecture

Memory Bandwidth: the rate at which data can be read from or stored into memory, expressed in bytes per second

Intel Xeon X5650: 32 GB/s Tesla M2070: 148 GB/s

Architecture

Tesla M2070 Processor: • Streaming Multiprocessors (SM): 14• Streaming Processors on each SM: 32

Total: 14 x 32 = 448 Cores

Each Streaming Multiprocessor supports 1024 threads.

Architecture

CUDA:

SIMT philosophy: Single Instruction Multiple Thread

Computationally intensive—The time spent on computation significantly exceeds the time spent on transferring data to and from GPU memory. Massively parallel—The computations can be broken down into hundreds or thousands of independent units of work.

Architecture

# Copy tutorial filesscc1 % cp –r /scratch/katia/cuda .

# Request interactive session on the node with GPUscc1 % qrsh –l gpus=1

# Change directoryscc1-ha1 % cd deviceQuery

# Set Environment variables to link to CUDA 5/0 scc1-ha1 % module load cuda/5.0

# Execute deviceQuery programscc1-ha1 % ./deviceQuery

Architecture

CUDA Driver Version / Runtime Version 5.0 / 5.0

CUDA Capability Major/Minor version number: 2.0

Total amount of global memory: 5375 MBytes

(14) Multiprocessors x ( 32) CUDA Cores/MP: 448 CUDA Cores

Total amount of constant memory: 65536 bytes

Total amount of shared memory per block: 49152 bytes

Total number of registers available per block: 32768

Information that we will need later in this tutorial:

CUDA Architecture

Warp size: 32

Maximum number of threads per multiprocessor: 1536

Maximum number of threads per block: 1024

Maximum sizes of each dimension of a block: 1024 x 1024 x 64

Maximum sizes of each dimension of a grid: 65535 x 65535 x 65535

Information that we will need later in this tutorial:

CUDA Architecture

# Change directoryscc1-ha1 % cd bandwidthTest

# Execute bandwidthTest programscc1-ha1 % ./bandwidthTest

Query device capabilities and measure GPU/CPU bandwidth.This is a simple test program to measure the memcopy bandwidth of the GPU and memcpy bandwidth across PCI-e

CUDA Terminology

CUDA:

Device The GPU and its memory (device memory)

Host The CPU and its memory (host memory)

CUDA: C Language Extensions

CUDA:

• Based on industry-standard C

• Language extensions allow heterogeneous programming

• APIs for memory and device managing

Hello, Cuda!

CUDA: Basic example HelloCuda1.cu

#include <stdio.h>int main(void){

printf("Hello, Cuda! \n");

return(0);}

To build the program, use nvcc compiler:

scc-he1: % nvcc -o helloCuda1 helloCuda1.cu

Hello, Cuda!

Function to be executed on the device (GPU) and called from host code

__device__ void foo(){ . . . }

CUDA Language closely follows C/C++ syntax with minimum set of extensions:

NVCC compiler will compile the function that run on the device and host compiler (gcc) will take care about all other functions that run on the host (e.g. main() )

Hello, Cuda!

CUDA: Basic example HelloCuda2.cu

#include <stdio.h>

__global__ void cudakernel(void){ printf("Hello, I am CUDA kernel ! Nice to meet you!\n");}

Hello, Cuda!

CUDA: Basic example HelloCuda2.cu

int main(void){

printf("Hello, Cuda! \n");

cudakernel<<<1,1>>>(); cudaDeviceSynchronize();

printf("Nice to meet you too! Bye, CUDA\n");

return(0);}

Hello, Cuda!

CUDA: Basic example HelloCuda2.cu

cudakernel<<<N,M>>>();

cudaDeviceSynchronize();

Triple angle brackets indicate that the function will be executed on the device (GPU).This function is called kernel.

Kernel is always of type void.Program returns immediately after launching the kernel. To prevent program to finish before kernel is completed, we have call cudaDeviceSynchronize().

CUDA: C Language Extensions

There is a number of cuda functions:

Device management:cudaGetDeviceCount(), cudaGetDeviceProperties()

Error management:cudaGetLastError(), cudaSafeCall(), cudaCheckError()

Device memory management:cudaMalloc(), cudaFree(), cudaMemcpy()

Hello, Cuda!

CUDA: Basic example HelloCuda2.cu

To build the program, use nvcc compiler:

scc-he1: % nvcc -o helloCuda2 helloCuda2.cu –arch sm_20

The ability to print from within the kernel was added in a later generation of architectural evolution. To request the support of Compute Capability 2.0, we need to add this option into compilation command line.

Hello, Cuda!

CUDA: Basic example HelloCudaBlock.cu

#include <stdio.h>

__global__ void cudakernel(void){ printf("Hello, I am CUDA block %d !\n", blockIdx.x);}

int main(void){

. . . cudakernel<<<16,1>>>(); . . .}

To simplify compilation process we will use Makefile:

% make HelloCudaBlock

CUDA: C Language Extensions

CUDA provides special variable for thread identification in the kernal:

dim3 threadIdx; // thread ID within the block

dim3 blockIdx; // block ID within the grid

dim3 blockDim; // number of threads per block

dim3 gridDim; // number of blocks in the grid

In the simple 1-dimentional case, we use only the first component of each variable, e.g. threadIdx.x

CUDA: Blocks and Threads

Serial Code

Serial Code

Kernel A

Kernel B

Host

Host

Device

Device

CUDA: C Language Extensions

CUDA: Basic example HelloCudaThread.cu

#include <stdio.h>

__global__ void cudakernel(void){ printf("Hello, I am CUDA thread %d !\n", threadIdx.x);}

int main(void){

. . . cudakernel<<<1,16>>>(); . . .}

CUDA: Blocks and Threads

• One kernel is executed on the device at a time

• Many threads execute each kernel

• Each thread execute the same code (SPMD)

• Threads are grouped into thread blocks

• Kernel is a grid of thread blocks

• Threads are scheduled as sets of warps

• Warp is a group of 32 threads

• SM executes same instruction on all threads in the warp

• Blocks cannot synchronize and can run in any order

Vector Addition Example

CUDA: vectorAdd.cu

__global__ void vectorAdd(const float *A, const float *B, float *C, int numElements){

int i = blockDim.x * blockIdx.x + threadIdx.x; if (i < numElements) { C[i] = A[i] + B[i]; }}

Vector Addition Example

CUDA: vectorAdd.cu

1 2 3 4 5 6 70 1 2 3 4 5 6 70 1 2 3 4 5 6 70 1 2 3 4 5 6 70

threadIdx.x threadIdx.x threadIdx.x threadIdx.x

blockIdx.x = 0 blockIdx.x = 1 blockIdx.x = 2 blockIdx.x = 3

int i = blockDim.x * blockIdx.x + threadIdx.x;

Unlike blocks, threads have mechanisms to communicate and synchronize

Vector Addition Example

CUDA: vectorAdd.cu device memory allocation

int main(void) { . . . float *d_A = NULL; err = cudaMalloc((void **)&d_A, size);

float *d_B = NULL; err = cudaMalloc((void **)&d_B, size);

float *d_C = NULL; err = cudaMalloc((void **)&d_C, size); . . .}

Vector Addition Example

CUDA: vectorAdd.cu

int main(void) {

. . . // Copy input values to the device cudaMemcpy(d_A, &A, size, cudaMemcpyHostToDevice);

cudaMemcpy(d_A, &A, size, cudaMemcpyHostToDevice);

. . .}

Vector Addition Example

CUDA: vectorAdd.cu

int main(void) { . . . // Launch the Vector Add CUDA Kernel int threadsPerBlock = 256; int blocksPerGrid =(numElements + threadsPerBlock - 1) / threadsPerBlock;

vectorAdd<<<blocksPerGrid, threadsPerBlock>>>(d_A, d_B, d_C, N); err = cudaGetLastError(); . . .}

Vector Addition Example

CUDA: vectorAdd.cu

int main(void) {

. . . // Copy result back to host cudaMemcpy(&C, d_C, size, cudaMemcpyDeviceToHost); // Clean-up cudaFree(d_A); cudaFree(d_B); cudaFree(d_C); . . .}

Timing CUDA kernel

CUDA: vectorAddTime.cu

float memsettime; cudaEvent_t start, stop;

// initialize CUDA timer cudaEventCreate(&start); cudaEventCreate(&stop); cudaEventRecord(start,0);

// CUDA Kernel . . .

// stop CUDA timer cudaEventRecord(stop,0); cudaEventSynchronize(stop); cudaEventElapsedTime(&memsettime,start,stop); printf(" *** CUDA execution time: %f *** \n", memsettime); cudaEventDestroy(start); cudaEventDestroy(stop);

Timing CUDA kernel

CUDA: vectorAddTime.cu

scc-ha1 % make

// specify the number of threads per block scc-ha1 % vectorAddTime 128

Explore the CUDA kernel execution time based on the block size:

Remember: • CUDA Streaming Multiprocessor executes threads in warps (32 threads)• There is a maximum of 1024 threads per block (for our GPU)• There is a maximum of 1536 threads per multiprocessor (for our GPU)

Dot Product

CUDA: dotProd1.cu

a0

a1

a2

a3

b0

b1

b2

b3

*

*

*

*

+ C

C = A * B = ( a0, a1 , a2 , a3 ) * ( b0, b1 , b2 , b3 ) = a0 * b0 + a1 * b1 + a2 * b2 + a3 * b3

Dot Product

CUDA: dotProd1.cu

A block of threads shares common memory, called shared memory

Shared Memory is extremely fast on-chip memory

To declare shared memory use __shared__ keyword

Shared Memory is not visible to the threads in other blocks

Dot Product

CUDA: dotProd1.cu

#define N 512__global__ voiddot( int*a, int*b, int*c ) {

// Shared memory for results of multiplication __shared__ inttemp[N]; temp[threadIdx.x] = a[threadIdx.x] * b[threadIdx.x];

// Thread 0 sums the pairwise products if( threadIdx.x == 0 ) { int sum = 0; for( int i= 0; i< N; i++ ) sum += temp[i]; *c = sum; }}

What if thread 0 starts to calculate sum before other threads completed their calculations?

Thread Synchronization

CUDA: dotProd1.cu

#define N 512__global__ voiddot( int*a, int*b, int*c ) {

// Shared memory for results of multiplication __shared__ inttemp[N]; temp[threadIdx.x] = a[threadIdx.x] * b[threadIdx.x]; __syncthreads();

// Thread 0 sums the pairwise products if( threadIdx.x == 0 ) { int sum = 0; for( int i= 0; i< N; i++ ) sum += temp[i]; *c = sum; }}

Thread Synchronization

CUDA: dotProd1.cu

int main(void) { . . . // copy input vectors to the device . . .

// Launch CUDA kernel dotProductKernel <<<1, N >>> (dev_A, dev_B, dev_C);

. . . // copy input vectors from the device . . .}

But our vector is limited to the maximum block size. Can we use blocks?

Race Condition

CUDA: dotProd2.cu

a0

a1

a2

a3

b0

b1

b2

b3

*

*

*

*

+ sum

a4

a5

a6

a7

b4

b5

b6

b7

*

*

*

*

+ sum

Block 0

Block 1

C

Race Condition

CUDA: dotProd2.cu#define N (2048*2048)#define THREADS_PER_BLOCK 512__global__ void dotProductKernel( int*a, int*b, int*c ) { __shared__ int temp[THREADS_PER_BLOCK];

int index = threadIdx.x + blockIdx.x * blockDim.x;

temp[threadIdx.x] = a[index] * b[index]; __syncthreads();

if( threadIdx.x == 0) { intsum = 0; for( int i= 0; i< THREADS_PER_BLOCK; i++ )sum += temp[i];

*c += sum; }}Blocks interfere with each other – Race condition

Race Condition

CUDA: dotProd2.cu#define N (2048*2048)#define THREADS_PER_BLOCK 512__global__ void dotProductKernel( int*a, int*b, int*c ) { __shared__ int temp[THREADS_PER_BLOCK];

int index = threadIdx.x + blockIdx.x * blockDim.x;

temp[threadIdx.x] = a[index] * b[index]; __syncthreads();

if( threadIdx.x == 0) { intsum = 0; for( int i= 0; i< THREADS_PER_BLOCK; i++ )sum += temp[i];

atomicAdd(c,sum); }}

Atomic Operations

Race conditions - behavior depends upon relative timing of multiple event sequences.Can occur when an implied read-modify-write is interruptible

Read-Modify-Write uninterruptible – atomic

atomicAdd() atomicInc()atomicSub() atomicDec()atomicMin() atomicExch()atomicMax() atomicCAS()

CUDA Best Practices

NVIDIA’s link: http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html

1. Assess

2. Parallelize

3. Optimize

4. Deploy

Locate part of the slowest part of the codegcc -O2 -g -pg myprog.cgprof ./a.out > profile.txt

Use CUDA to parallelize code;Use optimize cu* libraries if possible;

Overlapping data transfers, fine-tuning operation sequences

Compare the outcome with the original expectations.

CUDA Debugging

CUDA-GDB - GNU Debugger that runs on Linux and Mac: http://developer.nvidia.com/cuda-gdb

The NVIDIA Parallel Nsight debugging and profiling tool for Microsoft Windows Vista and Windows 7 is available as a free plugin for Microsoft Visual Studio: http://developer.nvidia.com/nvidia-parallel-nsight

This tutorial has been made possible by Scientific Computing and Visualization

groupat Boston University.

Katia Oleinikkoleinik@bu.edu

http://www.bu.edu/tech/research/training/tutorials/list/