gpu computing tools
DESCRIPTION
GPU Computing Tools. Varun Sampath University of Pennsylvania CIS 565 - Spring 2012. Agenda. CUDA Toolchain APIs Language bindings Libraries Visual Profiler Parallel Nsight OpenCL C ++ AMP. CUDA Documentation. http:// developer.nvidia.com/nvidia-gpu-computing-documentation - PowerPoint PPT PresentationTRANSCRIPT
GPU Computing Tools
Varun SampathUniversity of Pennsylvania
CIS 565 - Spring 2012
2
Agenda
• CUDA Toolchain– APIs– Language bindings– Libraries– Visual Profiler– Parallel Nsight
• OpenCL• C++ AMP
3
CUDA Documentation
• http://developer.nvidia.com/nvidia-gpu-computing-documentation
• CUDA C Programming Guide• CUDA C Best Practices Guide• CUDA API Reference Manual• Occupancy Calculator• Much more
4
CUDA Organization
• Host code: two layers– Runtime API
• cudart dynamic library• cuda_runtime_api.h
(C)• cuda_runtime.h (C++)
– Driver API• nvcuda dynamic library• cuda.h
• Device code– Kernel PTX (parallel
thread eXecution)
Image from Stack Overflow
5
CUDA API Comparison
CUDA Runtime API// create CUDA device & contextcudaSetDevice( 0 ); // pick first devicekernel_naive_copy<<<cnBlocks, cnBlockSize>>> (i_data, o_data, rows, cols);
CUDA Driver APIcuInit(0);cuDeviceGet(&hContext, 0); // pick first devicecuCtxCreate(&hContext, 0, hDevice));cuModuleLoad(&hModule, “copy_kernel.cubin”);cuModuleGetFunction(&hFunction, hModule, “kernel_naive_copy");…cuLaunchGrid(cuFunction, cnBlocks, 1);
Code from CUDA Best Practices Guide 4.0
Differences?
6
Some CUDA Language Bindings
• Note: the following are not supported by NVIDIA• PyCUDA (Python)– Developed by Andreas Klöckner– Built on top of CUDA Driver API– Also: PyOpenCL
• JCuda (Java)• MATLAB– Parallel Computing Toolbox– AccelerEyes Jacket
12
MATLAB Parallel Computing Toolbox
A = gpuArray(rand(2^16,1)); % copy to GPUB = fft (A); % run FFT (overloaded function)C = gather(B); % copy back to host
• Only differences between this and CPU code are gpuArray()and gather()
• Can also use arrayfun()or your own CUDA kernel• Any performance problems with this approach?
Code from MathWorks
13
CUDA Libraries
• Productivity– Thrust
• Performance– cuBLAS– cuFFT– Plenty more
14
Prelude: C++ Templates Primertemplate <typename T>T sum(const T a, const T b) {
return a + b;}
int main() {cout << sum<int>(1, 2) << endl;cout << sum<float>(1.21, 2.43) << endl;return 0;
}
• Make functions and classes generic• Evaluate at compile-time• Standard Template Library (STL)
– Algorithms, iterators, containersReference: MIT OCW
15
Thrust - “Code at the speed of light”
• Developed by Jared Hoberock and Nathan Bell of NVIDIA Research
• Objectives– Programmer productivity
• Leverage parallel primitives– Encourage generic programming
• E.g. one reduction to rule them all– High performance
• With minimal programmer effort– Interoperability
• Integrates with CUDA C/C++ code
Objectives from Intro to Thrust Slides
16
Thrust - Example#include <thrust/host_vector.h>#include <thrust/device_vector.h>#include <thrust/generate.h>#include <thrust/sort.h>#include <thrust/copy.h>#include <cstdlib>int main(void){
// generate 16M random numbers on the hostthrust::host_vector<int> h_vec(1 << 24);thrust::generate(h_vec.begin(), h_vec.end(), rand);// transfer data to the devicethrust::device_vector<int> d_vec = h_vec;// sort data on the devicethrust::sort(d_vec.begin(), d_vec.end());// transfer data back to hostthrust::copy(d_vec.begin(), d_vec.end(), h_vec.begin());return 0;
}
Code from GPU Computing Gems
17
Thrust Design
• Based off STL ideas– Algorithms, iterators, containers– Generic through C++ templates
• Built on top of CUDA Runtime API– Ships with CUDA 4.0+
• Four fundamental parallel algorithms– for_each– reduce– scan– sort
18
Thrust-CUDA C Interoperabilitysize_t N = 1024;// raw pointer to device memoryint* raw ptr;cudaMalloc(&raw_ptr, N*sizeof(int));// wrap raw pointer with a device ptrdevice_ptr<int> dev_ptr = device_pointer_cast(raw_ptr);// use device ptr in Thrust algorithmssort(dev_ptr, dev_ptr + N);// access device memory through device ptrdev_ptr[0] = 1;// free memorycudaFree(raw ptr);
Code from GPU Computing Gems
19
Thrust with User-Defined Functionsstruct saxpy_functor {
const float a;saxpy_functor(float a) : a( a) {}__host__ __device__float operator()(float x, float y){ return a*x+y; }
};void saxpy(float a, device vector<float>& x, device vector<float>& y) {
// setup functorsaxpy_functor func(a);// call transformtransform(x.begin(), x.end(), y.begin(), y.begin(),
func);}
Code from GPU Computing Gems
20
Thrust Performance
• Templates allow inlining and type analysis– How could knowing types improve global memory
performance?
Image from GPU Computing Gems
21
Thrust Toy-box
• Kernel fusion with transform_iterator and permutation_iterator
• Conversion between arrays of structs (AoS) and structure of arrays (SoA) with zip_iterator
• Implicit ranges
24
CUDA Specialized Libraries
• NVIDIA cuBLAS– Basic Linear Algebra Subprograms (BLAS)
• NVIDIA cuFFT– Compute Fast Fourier Transforms
• NVIDIA NPP– Image and Signal Processing
• See more: http://developer.nvidia.com/gpu-accelerated-libraries
25
CUDA Profiling and Debugging
• Visual Profiler• Parallel Nsight• cuda-gdb
26
Visual Profiler• Graphical profiling application• Collects performance counter data and makes
recommendations– Global memory throughput– IPC– Active warps/cycle– Cache hit rate– Register counts– Bank conflicts– Branch divergence– Many more (Full list in Visual Profiler User Guide)
27
28
Visual ProfilerDoes plots too!
29
Parallel Nsight
• Motivation– Why didn’t breakpoints in Visual Studio work for
debugging CUDA?
30
Parallel Nsight
• Debugger and Profiler for:– CUDA– OpenCL– Direct3D Shaders
• Integrated into Visual Studio 2008/2010• Caveat: requires extra GPU for display while
debugging– Supports NVIDIA Optimus
31
Parallel Nsight, showing breakpoints for different warps, disassembled kernel code, local variables, call stack, and register values per warp
Image from NVIDIA
32CUDA-GDB: No *nix User Left Behind Image from NVIDIA
33
OPENCL
Image from the Khronos Group
34
OpenCL
• Initially developed by Apple with help from AMD, IBM, Intel, and NVIDIA (Wikipedia)
• Specification defined by the Khronos Group
35Slide from the Khronos Group
36
OpenCL Goals
• Parallel Compute Framework for GPUs– And CPUs– And FPGAs– And potentially more
• Some compliant runtimes– AMD APP SDK (for AMD CPUs, GPUs, and APUs)– Intel OpenCL SDK (for Intel CPUs)– NVIDIA OpenCL Runtime (for NVIDIA GPUs)
Do we want CPUs and GPUs executing the same kernels though?
37
OpenCL Host Codesize_t szLocalWorkSize[2];size_t szGlobalWorkSize[2];
szLocalWorkSize[0] = 8;szLocalWorkSize[1] = 8;szGlobalWorkSize[0] = cols;szGlobalWorkSize[1] = rows;
// setup parameter values copyCode = oclLoadProgSource(“copy_kernel.cl", "", ©Len);hCopyProg = clCreateProgramWithSource(t->hContext,1, (const char **)©Code, ©Len, &errcode_ret); clBuildProgram(hCopyProg, 0, NULL, NULL, NULL, NULL);// create kernel t->hCopyKernel = clCreateKernel(hCopyProg, "kernel_naive_copy", &errcode_ret);clSetKernelArg(t->hCopyKernel, 0, sizeof(cl_mem), (void*)&dev_i_data); clSetKernelArg(t->hCopyKernel, 1, sizeof(cl_mem), (void*)&dev_o_data); clSetKernelArg(t->hCopyKernel, 2, sizeof(cl_int), (void*)&rows);clSetKernelArg(t->hCopyKernel, 3, sizeof(cl_int), (void*)&cols);clEnqueueNDRangeKernel(t->hCmdQueue, t->hCopyKernel, 2, NULL, szGlobalWorkSize, szLocalWorkSize, 0, NULL, NULL);
38
OpenCL Host Codesize_t szLocalWorkSize[2];size_t szGlobalWorkSize[2];
szLocalWorkSize[0] = 8;szLocalWorkSize[1] = 8;szGlobalWorkSize[0] = cols;szGlobalWorkSize[1] = rows;
// setup parameter values copyCode = oclLoadProgSource(“copy_kernel.cl", "", ©Len);hCopyProg = clCreateProgramWithSource(t->hContext,1, (const char **)©Code, ©Len, &errcode_ret); clBuildProgram(hCopyProg, 0, NULL, NULL, NULL, NULL);// create kernel t->hCopyKernel = clCreateKernel(hCopyProg, "kernel_naive_copy", &errcode_ret);clSetKernelArg(t->hCopyKernel, 0, sizeof(cl_mem), (void*)&dev_i_data); clSetKernelArg(t->hCopyKernel, 1, sizeof(cl_mem), (void*)&dev_o_data); clSetKernelArg(t->hCopyKernel, 2, sizeof(cl_int), (void*)&rows);clSetKernelArg(t->hCopyKernel, 3, sizeof(cl_int), (void*)&cols);clEnqueueNDRangeKernel(t->hCmdQueue, t->hCopyKernel, 2, NULL, szGlobalWorkSize, szLocalWorkSize, 0, NULL, NULL);
← What are these for?
Look Familiar?
39
__kernel void kernel_naive_copy( __global const float4 * i_data,
__global float4 * o_data, int rows, int cols){ uint x = get_global_id(0); uint y = get_global_id(1); o_data[y*rows + x] = i_data[y*rows + x];}
OpenCL Device Code
See some similarities?
40
OpenCL Code
• Very similar to CUDA Driver API and CUDA C– NVIDIA has a short guide outlining syntax
differences• C-based API– C++ wrappers and bindings to other languages
(e.g. PyOpenCL) available
41
OPENCL OR CUDA?Which should I choose?
42
Compatibility
• CUDA runs only on NVIDIA GPUs– Not necessarily true…
• OpenCL is supported by a lot of vendors
Image from the Khronos Group
43
Doesn’t everyone just want an NVIDIA GPU?
16384 32768 49152 655360
0.00050.001
0.00150.002
0.00250.003
0.00350.004
0.00450.005
Black-Scholes OpenCL Performance with work-group size of 256 and processing of 8 million
options
FermiBarts
Number of Work-Items
Exec
ution
Tim
e (s
)
44
Performance Comparison
on NVIDIA GPUs
256x256
512x512
1024x1024
2048x2048
4096x4096
00.05
0.10.15
0.20.25
SAT OpenCL and CUDA Performance with work-group size of 256
CUDAOpenCL
Problem Size
Exec
ution
Tim
e (s
)
16384 32768 49152 655360
0.0020.0040.0060.008
Black-Scholes OpenCL and CUDA Per-formance with work-group size of 256
and processing of 8 million options
CUDAOpenCL
Number of Work-items/Threads
Exec
ution
Tim
e (s
)
[This was done with the CUDA 3.2 Toolkit. CUDA 4.1 brought a new LLVM compiler to CUDA (OpenCL compiler was already LLVM-based)]
45
Programming Framework Comparison
• CUDA 4.0 brought a lot of advancements– Unified address space– C++ new/delete, virtual functions on device– GPUDirect peer-to-peer GPU communication
• OpenCL does not have these features– And 18-month release cadence is slower than
NVIDIA’s
46
Libraries & Mindshare
• CUDA has a larger ecosystem– Thrust is a particularly important library
• Will OpenCL catch up?– Growing in other ways• OpenCL Embedded Profiles• WebCL
47
C++ AMP
48
C++ AMP (Accelerated Massive Parallelism)
• Announced by Microsoft in June 2011• Targeting “heterogeneous parallel computing”– Multicore CPUs– GPUs– Cloud Infrastructure-as-a-Service (IaaS)
49Slide from Herb Sutter’s AMD Fusion Keynote
50Slide from Herb Sutter’s AMD Fusion Keynote
51
C++ AMP Matrix Multiplyvoid MatrixMult( float* C, const vector<float>& A, const vector<float>& B,int M, int N, int W ){ array_view<const float,2> a(M,W,A), b(W,N,B); array_view<writeonly<float>,2> c(M,N,C); parallel_for_each( c.grid, [=](index<2> idx) restrict(direct3d) { float sum = 0; for(int i = 0; i < a.x; i++) sum += a(idx.y, i) * b(i, idx.x); c[idx] = sum; } );}
52
C++ AMP Matrix Multiplyvoid MatrixMult( float* C, const vector<float>& A, const vector<float>& B,int M, int N, int W ){ array_view<const float,2> a(M,W,A), b(W,N,B); array_view<writeonly<float>,2> c(M,N,C); parallel_for_each( c.grid, [=](index<2> idx) restrict(direct3d) { float sum = 0; for(int i = 0; i < a.x; i++) sum += a(idx.y, i) * b(i, idx.x); c[idx] = sum; } );}• array_view: abstraction for accessing data (like an “iterator range”)• Lambda expressions: like functors of thrust but with less syntactic overhead • restrict: ensure only language capabilities supported by device are used
53Slide from Herb Sutter’s AMD Fusion Keynote
54
C++ AMP
• Only 1 new keyword added to C++– All other functionality in classes and functions
• Released as open specification 2 weeks ago• Debugging and Profiling included in Visual
Studio 11
55
Conclusion: What do you think?
Slide from Herb Sutter’s AMD Fusion Keynote
56
References
• Bell, Nathan and Hoberock, Jared. “Thrust: A Productivity-Oriented Library for CUDA.” GPU Computing Gems: Jade Edition. Link
• Klöckner, Andreas. “PyCUDA: Even Simpler• GPU Programming with Python.” Slides• Reese, Jill and Zaranek, Sarah. “GPU Programming in
MATLAB.” Link• Rosenberg, Ofer. “OpenCL Overview.” Slides• Sutter, Herb. “Heterogeneous Parallelism at Microsoft.”
Link
57
Bibliography
• Klöckner, et al. “PyCUDA and PyOpenCL: A Scripting-Based Approach to GPU Run-Time Code Generation.” arXiv
• Moth, Daniel. “Blazing-fast code using GPUs and more, with C++ AMP.” Link