RNNs: Neural Networks with Short-Term Memory
A Recurrent Neural Network (RNN) processes sequences by keeping a running memory of what it's seen. It feeds its own output from one step back into the next, like someone reading a sentence one word at a time. This is ideal for sequential data like text or time series where context is key. The main footgun is its notoriously short memory; information from early in a long sequence often gets lost.
### The mental model A Recurrent Neural Network (RNN) processes sequential data not all at once, but step-by-step, maintaining an internal 'state' that acts as a summary of the sequence seen so far. At each step, it combines the current input with the previous step's state to produce an output and update its memory for the next step. This feedback loop is what gives it a form of memory.
### How it works Imagine an RNN processing the word "hello". 1. It takes the first input, 'h', and an initial empty state. It computes an output and a new state that represents 'h'. 2. It takes the second input, 'e', and the state from step 1. It combines them to compute a new output and an updated state that now represents having seen 'he'. 3. This process repeats for each character. The same set of weights is applied at every step, allowing the network to learn a general rule for how the sequence evolves. This shared-weight mechanism is what allows it to handle sequences of variable length.
### When to use it * **Natural Language Processing:** Early models for language modeling, text generation, and machine translation used RNNs to understand sentence structure. * **Time-Series Analysis:** Forecasting stock prices, weather, or sensor data where past values influence future ones. * **Speech Recognition:** Converting audio signals, which are sequences of sound waves, into text.
### When NOT to use it * **Long sequences:** Standard RNNs suffer from the vanishing gradient problem, meaning they can't remember information from many steps ago. For long-range dependencies (e.g., summarizing a book), more advanced architectures like LSTMs, GRUs, or Transformers are necessary. * **Non-sequential data:** If the order of data points doesn't matter (e.g., classifying an image based on its pixels), a feedforward network or a Convolutional Neural Network (CNN) is more appropriate and efficient.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.