Compare ephemeral container agents versus persistent build agents

WHAT THIS TESTS: This question tests whether you can reason about build infrastructure tradeoffs beyond personal preference. The interviewer wants to see if you understand reproducibility, caching strategies, operational overhead, and security boundaries in CI/CD systems. They are listening for evidence that you have scaled build environments and felt the pain of both models.

A GOOD ANSWER COVERS: A strong answer walks through four dimensions in order. First, reproducibility and hermetic builds: ephemeral containers start from a known image every time, eliminating works-on-my-machine drift and snowflake agents. Second, performance and caching: persistent agents keep incremental compiler caches, dependency downloads, and Docker layer caches warm, which matters when builds run every few minutes. Third, maintenance and security: ephemeral agents reduce patching burden because the host is short-lived, while persistent agents require OS updates, toolchain upgrades, and cleanup scripts to prevent disk exhaustion. Fourth, cost and scalability: ephemeral agents fit serverless or Kubernetes-based autoscaling but may pull large images repeatedly; persistent agents sit idle between jobs unless you manage pool sizing carefully. The best candidates mention hybrid patterns such as VM scale sets that recreate agents periodically or cached sidecars that inject dependencies without full image rebuilds.

COMMON WRONG ANSWERS: The biggest red flag is treating one model as universally correct. Another mistake is ignoring startup latency entirely; candidates who claim ephemeral is always faster forget that a two-gigabyte container image pull can take thirty to sixty seconds. Conversely, claiming persistent agents are cheaper without factoring in maintenance labor and security patching is a junior-level answer. A subtle trap is conflating containerized builds with ephemeral hosts; you can run persistent containers or ephemeral VMs, so the answer should separate the host lifecycle from the build sandbox.

LIKELY FOLLOW-UPS: The interviewer may ask how you would cache dependencies in an ephemeral world, how you would detect configuration drift on persistent agents, or what you would do if a build modifies the local toolchain. They might also probe cost optimization, asking when to use spot instances for ephemeral agents or how to right-size persistent pools.

ONE CONCRETE EXAMPLE: Consider a monorepo with two-hundred engineers running five-hundred builds per day. Purely ephemeral agents would spend significant time reinstalling a three-gigabyte Node_modules tree and compiler toolchain on every job. A persistent agent pool with nightly image refreshes and automated disk cleanup keeps builds under ninety seconds while avoiding week-old state. Alternatively, ephemeral agents with a mounted read-only cache volume or a pre-baked image rebuilt hourly can capture eighty percent of the speed benefit without the drift risk.

Source: learn.microsoft.com