NVIDIA CUDA: General-Purpose GPU Computing

Why it exists

GPUs were originally designed for one job: rendering graphics. This involves performing the same simple calculations across millions of pixels at once. Researchers realized this massively parallel architecture was perfect for scientific computing, but there was no standard way to program it. CUDA was created by NVIDIA to provide a formal platform and API for general-purpose computing on GPUs.

The mental model

Think of a CPU as a few highly skilled chefs who can tackle any complex recipe one after another. A GPU, via CUDA, is like an army of a thousand kitchen assistants who can only do one simple task, like chopping onions, but can all do it at the same time. CUDA is the set of instructions that lets you manage this army for tasks that can be broken down into many identical, parallel steps.

How it works

CUDA is a parallel computing platform and API. It provides a software layer, including a C++-like programming language and libraries, that lets developers write "kernels"—functions that are executed in parallel by thousands of GPU cores. A host program running on the CPU manages the application, but it offloads the computationally intensive, parallelizable parts to the GPU. The CUDA API handles moving data between CPU and GPU memory and scheduling work across the GPU's cores.

When to use it

Use CUDA for problems that are "embarrassingly parallel"—where a large task can be broken into many smaller, independent sub-tasks. This is common in machine learning (training neural networks), scientific simulations (weather modeling), image processing, and cryptography. If your problem involves applying the same mathematical operation to a huge dataset, it's a candidate for CUDA.

When not to use it

Avoid CUDA for tasks that are inherently sequential or require complex branching logic. If each step of your calculation depends on the result of the previous one, a GPU offers no benefit and can even be slower due to the overhead of moving data to and from the GPU. Standard CPU processing is better for general application logic and tasks that aren't easily parallelized.

One canonical example

The canonical example is training a deep learning model. This involves repeatedly performing massive matrix multiplications on large batches of data. Using CUDA, a framework like PyTorch or TensorFlow offloads these matrix operations to the GPU, where thousands of cores process them simultaneously. This drastically reduces training time from weeks on a CPU to hours or days on a GPU.