How would you design drift detection for high-dimensional embeddings?

WHAT THIS TESTS: This question tests whether you understand why classical univariate drift detection fails in high-dimensional embedding spaces and whether you can select statistically principled multivariate alternatives. Interviewers want to see that you recognize embeddings live on a nonlinear manifold where per-feature moments are meaningless and where dimensionality destroys the power of traditional tests.

A GOOD ANSWER COVERS: First, mention that standard tests like KS or PSI on individual embedding dimensions fail because dimensions are correlated and univariate shifts can cancel out or hide manifold-level changes. Second, propose kernel-based two-sample tests such as Maximum Mean Discrepancy with an RBF or deep kernel; explain that MMD compares distributions in a reproducing kernel Hilbert space without requiring density estimation. Third, discuss distance-based methods like k-nearest neighbor two-sample tests or relative entropy estimation via average k-NN distances, which work directly in the embedding space and scale with approximate nearest-neighbor indexes. Fourth, describe the domain-classifier or black-box test approach where you train a binary classifier to distinguish reference embeddings from production embeddings and monitor its AUC or accuracy; a score near 0.5 means no drift while 0.8 or above signals severe shift. Fifth, explain operational concerns such as choosing a reference window size, stratifying by model prediction or metadata to localize drift, using Bonferroni correction or false-discovery rate control when monitoring many slices, and setting alert thresholds based on business impact rather than p-value magic cutoffs.

COMMON WRONG ANSWERS: Proposing per-dimension KS tests followed by averaging p-values is a red flag because it ignores covariance structure and loses power in high dimensions. Suggesting PCA projection before testing is risky unless you justify how much variance is retained and prove the projection preserves the drift signal. Recommending simple cosine-distance or centroid-difference thresholds without distributional backing is another weak signal because means can be identical while support changes drastically.

LIKELY FOLLOW-UPS: How do you handle computational cost at millions of embeddings per hour? How do you choose the kernel bandwidth or k in k-NN tests? What do you do when drift is detected only in a specific subpopulation or class? How would you distinguish data drift from label drift when labels are sparse?

ONE CONCRETE EXAMPLE: Suppose you run a visual search model with 512-dimensional image embeddings. You store a reference set of 100,000 embeddings from the previous month. Each hour you sample 5,000 production embeddings and compute MMD with a deep kernel trained on your domain; you also maintain an approximate nearest-neighbor index to run a k-NN two-sample test in under 30 seconds. You additionally train a lightweight domain classifier each day and track its AUC; if AUC exceeds 0.75 for two consecutive windows you trigger a model retraining pipeline and notify the team via severity-2 alert.