Explain BPTT and its computational and memory challenges for long sequences

WHAT THIS TESTS: Whether you understand that training an RNN is mathematically equivalent to training a very deep feedforward network whose depth equals the sequence length, and whether you can articulate the concrete engineering consequences of that equivalence in terms of memory and gradient stability.

A GOOD ANSWER COVERS: First, the conceptual definition of BPTT as unrolling the recurrent model over T time steps and applying ordinary backpropagation to the resulting directed acyclic graph. Second, the memory challenge: because gradients flow backward through every time step, you must retain all intermediate hidden states during the forward pass, so memory cost scales as O(T) with sequence length. Third, the computational challenge: the backward pass also traverses all T steps, so compute is O(T), but the deeper problem is numerical stability. Fourth, the long-sequence pathology: repeated multiplication of Jacobian matrices across many time steps causes gradients to vanish or explode exponentially with T, which is why architectures like LSTM, GRU, and eventually Transformers were developed. Fifth, truncated BPTT as the standard remedy: limit the unrolling to a fixed window so memory and gradient paths are bounded, accepting that credit assignment beyond the truncation point is lost.

COMMON WRONG ANSWERS: Confusing BPTT with real-time recurrent learning, which is an online gradient method that does not unroll the full graph. Claiming that RNN memory usage is constant during training; it is constant per step during inference, but not during training under BPTT. Blaming only the O(T) compute cost while ignoring the exponentially worse vanishing or exploding gradient issue. Suggesting full unrolling for arbitrarily long sequences without mentioning truncation or gradient clipping.

LIKELY FOLLOW-UPS: How does truncated BPTT change the computational graph, and what hyperparameter controls the tradeoff? Why do LSTM and GRU gates help with the long-range gradient problem? How does attention avoid this issue entirely? What is the interaction between batch size and sequence length in GPU memory, and how would you train on documents with millions of tokens?

ONE CONCRETE EXAMPLE: Consider a language model processing a sentence of 1000 tokens. Full BPTT requires storing 1000 hidden vectors of dimension H for the backward pass; if H is 2048 and each activation is 4 bytes, that is roughly 8 MB per layer per sequence, which grows linearly with length. During backprop, the gradient w.r.t. the first word involves multiplying a chain of roughly 1000 Jacobian matrices; if the largest singular value of each Jacobian is even 1.01, the product explodes to roughly 20000, and if it is 0.99, the product vanishes to nearly zero. In practice you would cap the unroll to 32 or 128 steps and clip gradients to keep training stable.