Recurrent Neural Networks: Networks with Memory

WHY IT EXISTS Standard neural networks process inputs independently, which fails for sequential data like text or stock prices where order is everything. A network needs to know what came before to understand what's happening now. RNNs were designed to solve this by explicitly incorporating a notion of memory.

THE MENTAL MODEL Think of an RNN as someone reading a sentence one word at a time. They don't just understand the current word in isolation; they hold the context of the previous words in their short-term memory. An RNN does this with a 'hidden state'—a numerical summary of the sequence seen so far—which it constantly updates as it processes new information.

HOW IT WORKS An RNN contains a loop. For each element in a sequence, the network takes two things as input: the current element and the hidden state from the previous step. It processes these to produce an output for the current step and, crucially, a new hidden state. This updated hidden state is then passed to the next step in the sequence, carrying forward the context. This recurrent feedback loop is what allows the network to learn from ordered data.

WHEN TO USE IT Use an RNN when the order of your data is meaningful and you need to model dependencies over time. This is common in natural language processing (like machine translation or text generation), speech recognition (processing audio waveforms), and time-series forecasting (predicting future stock prices or weather).

WHEN NOT TO USE IT Avoid basic RNNs for sequences with very long-term dependencies. They suffer from the 'vanishing gradient' problem, where their memory of early inputs fades over time, making it hard to connect distant cause and effect. For these cases, more advanced architectures like LSTMs, GRUs, or Transformers are far more effective. Also, do not use them for non-sequential data like static images.

ONE CANONICAL EXAMPLE A character-level language model is a classic RNN task. You feed it a large body of text, one character at a time. At each step, the RNN's job is to predict the next character. After seeing the sequence 'h-e-l-l', the network uses its hidden state (the memory of 'h-e-l-l') and the current input ('l') to predict 'o' with high probability.