Measuring fairness in an embedding-based text classifier

WHAT THIS TESTS Whether you can select fairness metrics that capture distinct definitions, reason about their tradeoffs, and confront the messiness of measuring groups in unstructured text.

A GOOD ANSWER COVERS Define the protected groups and the favorable outcome first. Then pick metrics that disagree by design. Demographic parity asks whether the positive-prediction rate is equal across groups, ignoring true labels. Equalized odds asks whether true positive and false positive rates are equal across groups, conditioning on the ground truth. Equal opportunity is the weaker variant requiring only equal true positive rates. These can be mathematically incompatible, so you must state which harm you prioritize. For embeddings specifically, note that the representation may encode demographic proxies even when the raw text does not mention them, so a classifier can discriminate via correlated features. Report metrics with confidence intervals because group sample sizes are often small.

COMMON WRONG ANSWERS Naming only accuracy gap. Citing one metric and treating it as universal fairness. Assuming clean protected-attribute labels are available. Ignoring that demographic parity and equalized odds generally cannot both hold when base rates differ.

LIKELY FOLLOW-UPS Which metric do you choose when base rates differ across groups and why? How do you obtain group labels without collecting sensitive data? How do embeddings leak protected attributes? How would you mitigate once you measure a gap?

ONE CONCRETE EXAMPLE A resume-screening classifier predicts interview yes or no from embedded text. You infer candidate groups from a labeled audit set. Demographic parity shows group A gets positive predictions at 40 percent versus group B at 25 percent. But equalized odds reveals group B also has a lower true positive rate, meaning qualified B candidates are rejected. You report both, flag that they conflict because base rates differ, and recommend equal opportunity as the operative target, plus an embedding audit to find the leaking proxy features.