Welcome

Hi there,

Over the next few months, I'll be reading Programming Massively Parallel Processors by Hwu, Kirk, and Hajj.

I'll be posting my notes here as I go along. I hope it helps anyone who shares my interest in parallel computation!

Happy learning!

- Logan

Running CUDA Code

I'll be using Google Colab to start out.

To run a CUDA program in Colab:

1. Change the runtime to T4 GPU (as of May 2024)
2. Run the following in your notebook to check that NVCC & CUDA are installed. By default, the T4 GPU runtime should come with CUDA 12.

   !nvcc --version
   

3. Copy your .cu files to your Colab environment's file system
4. Add a second code block where you compile your .cu files. For a simple program contained in a single file main.cu, linking a single library (jpeglib, for example), we can use the following:

   !nvcc -o main main.cu -ljpeg
   

5. Add a third code block and run your compiled executable:

   !./main
   

Table of Contents

0. Preface
1. Intro

Part I: Fundamental Concepts

2. Heterogeneous data parallel computing
3. Multidimensional grids and data
4. Compute architecture and scheduling
5. Memory architecture and data locality
6. Performance considerations

Part II: Parallel Patterns

7. Convolution
8. Stencil
9. Parallel histogram
10. Reduction
11. Prefix sum (scan)
12. Merge

Part III: Advanced Patterns and Applications

13. Sorting
14. Sparse matrix computation
15. Graph traversal
16. Deep learning
17. Iterative MRI reconstruction
18. Electrostatic potential map
19. Parallel programming and computational thinking

Part IV: Advanced Practices

20. Programming a heterogeneous computing cluster
21. CUDA dynamic parallelism
22. Advanced practices and future evolution
23. Conclusion

Sample Programs:

• Vector Addition
• RGB to Grayscale
• Thread-Coarsened Tiled Matrix Multiplication