OpenTelemetry agent and gateway architecture

WHAT THIS TESTS: Whether you can design a telemetry pipeline that scales across a large fleet and correctly partition processing between local agents and a central gateway, understanding which decisions require global context.

A GOOD ANSWER COVERS: The standard pattern is two tiers of OpenTelemetry Collectors. Agents run close to the workload, one per node as a daemonset or as sidecars, scraping and receiving local metrics, logs, and traces. They should stay lightweight and do only local, context-free work: receiving data, adding host and Kubernetes resource metadata (node, pod, namespace), basic batching, and offloading quickly so they do not consume application resources. Gateways are a separate, horizontally scalable pool of collectors that all agents forward to. Because the gateway sees aggregated traffic, it is the right place for processing that needs broader context or central control: tail-based sampling (which requires all spans of a trace, impossible on a single node that only sees part of it), organization-wide filtering and dropping of noisy data, PII redaction and policy enforcement, cardinality control, batching for efficiency, and fan-out routing to multiple backends. Keeping global logic central also means you update sampling and filtering rules in one place rather than redeploying every agent.

COMMON WRONG ANSWERS: Putting tail-based sampling on agents, which cannot work because an agent only sees the spans on its node, not the full trace. Doing heavy enrichment or filtering on agents and bloating per-node overhead, or skipping the gateway and pointing every agent directly at the backend, losing central control and overwhelming the backend.

LIKELY FOLLOW-UPS: How do you scale and load-balance the gateway pool? How does tail-based sampling buffer traces? How do you handle agent backpressure when the gateway is down?

ONE CONCRETE EXAMPLE: Agents on each Kubernetes node attach pod and namespace labels and ship spans to a gateway pool behind a load balancer. The gateway buffers complete traces, applies tail-based sampling to keep all error and slow traces, scrubs PII, and routes metrics to one backend and traces to another, all configured centrally.