How do BYOL and Barlow Twins avoid representation collapse?

WHAT THIS TESTS This probes deep understanding of self-supervised representation learning beyond the contrastive recipe. The trap is that a naive Siamese network trained only to make two augmented views agree has a trivial solution: output a constant. The question is what stops that.

A GOOD ANSWER COVERS Dimensional collapse is when learned embeddings fill only a low-rank subspace, with most singular values near zero, even if the trivial constant collapse is avoided. Contrastive methods like SimCLR use negatives to repel embeddings and spread them out. BYOL avoids needing negatives via architectural asymmetry: an online network with an extra predictor head learns to predict the output of a momentum-averaged target network, and a stop-gradient prevents the target from being trained directly; the predictor plus stop-gradient combination breaks the symmetry that would allow collapse. Barlow Twins takes a different route, computing the cross-correlation matrix between the two views' embeddings and driving it toward the identity, so diagonal terms encourage invariance while off-diagonal terms penalize redundancy across feature dimensions, directly fighting dimensional collapse.

COMMON WRONG ANSWERS Saying you must have negatives. Claiming BYOL works purely because of the loss, ignoring the predictor, momentum target, and stop-gradient. Confusing the momentum encoder of MoCo with BYOL's mechanism.

LIKELY FOLLOW-UPS Why stop-gradient is essential (SimSiam shows it alone can prevent collapse), the role of batch or feature normalization, and how VICReg uses explicit variance and covariance terms.

ONE CONCRETE EXAMPLE In Barlow Twins, two augmentations of the same image are embedded, the empirical cross-correlation matrix is formed over the batch, and the loss pushes its diagonal to one and off-diagonal to zero, so each feature stays informative and distinct from the others rather than collapsing into a shared direction.