Seq2Seq: Turning One Sequence Into Another

Why it exists

Before Seq2Seq, many neural network models required inputs and outputs to have the same, fixed length. This fails for common tasks like machine translation, where a sentence in one language rarely has the same number of words as its translation. Seq2Seq was designed to solve this variable-length sequence problem.

The mental model

Think of a Seq2Seq model as a two-part system: an encoder and a decoder. The encoder reads an entire input sequence, like a person reading a full sentence in French, and compresses its meaning into a single summary of thought—a 'context vector'. The decoder then takes this thought summary and generates the output sequence word by word, like a person speaking the equivalent sentence in English based on their understanding.

How it works

The model uses two Recurrent Neural Networks (RNNs). The encoder RNN processes the input sequence one item at a time, updating its internal state. After the last input item, the encoder's final state becomes the context vector—a single vector of numbers representing the entire input's meaning. This context vector is then fed as the initial state to the decoder RNN. The decoder generates the output sequence one item at a time, using the context and the previously generated item to predict the next, until it outputs a special 'end-of-sequence' token.

When to use it

Seq2Seq is the classic approach for tasks where an input sequence of one length maps to an output sequence of a different length. This includes machine translation (a sentence to its translation), text summarization (a long article to a short paragraph), and conversational AI (a user's question to a chatbot's answer). It's also used for image captioning, where the 'input' is a feature vector from an image and the output is a descriptive sentence.

When not to use it

The primary weakness of basic Seq2Seq is its reliance on a single, fixed-size context vector. For very long inputs, this vector becomes an information bottleneck, as it's difficult to cram all the necessary information into it. This can cause the model to 'forget' details from the beginning of the input. For tasks requiring memory of long-range dependencies, more advanced architectures like the Transformer (which uses an attention mechanism) have largely superseded classic Seq2Seq.

One canonical example

To translate the English sentence "The cat sat on the mat" to French, the encoder processes each word and produces a single context vector. The decoder, given this vector, starts generating the French translation. It might first produce "Le". It then takes "Le" and the context vector to generate "chat", and so on, until it produces the full sequence "Le chat est assis sur le tapis" followed by an end-of-sequence token.