Google TPU: Built for Matrix Math

WHY IT EXISTS: Machine learning workloads demand enormous computation, yet general-purpose CPUs and GPUs carry hardware for graphics tasks like rasterization and texture mapping that neural networks do not need. In 2013, Google recruited Amir Salek to establish custom silicon development and assigned Norman P. Jouppi as tech lead to build an application-specific integrated circuit that could accelerate neural network math with far greater efficiency.

THE MENTAL MODEL: Think of a GPU as a Swiss Army knife and a TPU as a purpose-built factory line. A GPU can render graphics and train neural networks because it packs flexible parallel units. A TPU rips out the graphics circuitry entirely and replaces it with a systolic array dedicated to low-precision tensor operations. The result is a chip that fits into a data-center hard-drive slot yet delivers 15 to 30 times the inference performance and 30 to 80 times the performance per watt of the CPUs and GPUs it replaced.

HOW IT WORKS: The TPU is an ASIC built around a systolic array, an architecture chosen from three separate accelerator designs explored at Google. It was specifically created for the TensorFlow framework, though it now supports JAX and PyTorch as well. The hardware targets high volumes of low-precision computation, using as little as 8-bit precision, which saves energy compared to the broader floating-point units found in CPUs and GPUs.

WHEN TO USE IT: Reach for a TPU when you are training or inferencing convolutional neural networks at scale through Google Cloud. The chip is designed for the TensorFlow, JAX, and PyTorch frameworks and for workloads that benefit from high-volume, low-precision math.

WHEN NOT TO USE IT: Do not use a TPU for graphics rendering or any workload that relies on rasterization and texture mapping because the chip lacks that hardware entirely. TPUs are well suited for convolutional neural networks, while GPUs have benefits for some fully connected neural networks, and CPUs can have advantages for recurrent neural networks. Forcing the wrong model type onto TPU silicon wastes time and money.

ONE CANONICAL EXAMPLE: In 2017, Google published the seminal paper In-Datacenter Performance Analysis of a Tensor Processing Unit, showing that its first production TPU was designed and deployed in just 15 months and was already powering production services across Google. That chip achieved 15 to 30 times higher performance and 30 to 80 times higher performance per watt than contemporary CPUs and GPUs, proving that a narrow, custom-built accelerator could outperform general-purpose silicon on its home turf.

Source: Wikipedia