Advanced everything in DevOps & Cloud, page 10
Game Days: Practice Breaking Your System Before It Breaks Itself
A Game Day is a live fire drill for your systems. You intentionally inject failure—like shutting down a service—to see how your team and automation respond, finding weaknesses before a real outage does.
Query Federation: Combining Prometheus Servers
Query federation lets one Prometheus server scrape metrics from another, creating a meta-monitor. Use it to build a global view from local servers or to combine application and infrastructure metrics for richer alerts.
Inverted Index: The Engine of Fast Log Search
An inverted index makes log search fast by mapping terms to the logs containing them, like a book's index. It powers platforms like Splunk or Elasticsearch, enabling instant searches across terabytes of data. The footgun is indexing high-cardinality fields.
Loki: The Log System That Indexes Labels, Not Text
Loki is a log system that indexes only metadata labels, not the full log content. This makes it cheaper and simpler to run than full-text indexing systems, storing compressed logs in object storage.
Prometheus Exemplars: Link Your Metrics to Traces
Exemplars are like footnotes for your metrics, linking a data point like a latency spike directly to a specific trace ID. This lets you jump from a 'what' on a dashboard to the 'why' in your tracing system.
Telemetry Processors: The Middle of the OTel Pipeline
A Telemetry Processor is a configurable stage in an OpenTelemetry Collector pipeline, sitting between data reception and export. You configure them in config.yaml to act on telemetry data.
Context Propagation: Stitching Microservices Together
Context propagation stitches a user request's journey across microservices by passing a shared ID. It's essential for distributed tracing, letting you see one request flow through many APIs.
Sampling: Tracing Everything Without Storing Everything
Sampling makes high-volume observability affordable by deciding which traces to keep and which to discard. It's essential in distributed systems where capturing every request is too costly.
Cardinality: The Hidden Cost of Time-Series Metrics
Cardinality is the number of unique label combinations in your metrics. High cardinality, from labels like user IDs, is the silent killer of monitoring systems like Prometheus, exploding memory and cost. The footgun is adding a label with unbounded values.
SRE Engagement Models: From Gatekeeper to Platform Builder
SRE engagement models define how reliability experts help product teams, evolving from gatekeeping existing services to providing reliable platforms. This applies when scaling an SRE team's impact.
Service Level Agreement (SLA): The Contract Behind Uptime
An SLA is a business contract, not a technical target. It defines the minimum service quality a provider promises a customer, with financial penalties for failure. You see them in every cloud provider contract.
Egress Gateway: Control Your Mesh's Outbound Traffic
An Egress Gateway is a monitored exit door for all outbound traffic from your service mesh. Use it to enforce security on external calls, like restricting domains or originating mTLS.
Traffic Mirroring: Test in Production, Safely
Traffic mirroring copies live production requests to a new service without affecting the user's response. It's used to test new code with real traffic before a full rollout. The main footgun is accidentally duplicating writes or other stateful actions.
Kubernetes API Aggregation Layer: Extending the API Server
The API Aggregation Layer bolts custom API servers onto the main Kubernetes API, with kube-apiserver acting as a proxy. This powers features like the metrics server (kubectl top) and enables complex extensions.
controller-runtime: The Engine for Kubernetes Operators
Think of controller-runtime as the standard library for writing Kubernetes controllers. It handles the boilerplate of watching resources and reconciling state, forming the foundation for tools like Kubebuilder and Operator SDK.
Helm Post-Rendering: Customize Charts Without Forks
Helm post-rendering lets you modify a chart's Kubernetes manifests just before deployment. It's ideal for applying kustomize patches or injecting sidecars without forking a public chart.
Flux Image Update Automation: Closing the GitOps Loop
Flux's image update automation acts like a bot that watches your container registry. It finds new image tags that match your policies (like semver) and automatically commits the change back to your Git repository, triggering a deployment.
Argo CD Sync Phases and Waves: Ordering Your Deployments
Argo CD Sync Phases and Waves are a recipe for ordering deployments. Use them for complex apps where a database migration must run pre-sync. The footgun: a single failed resource in a wave halts the entire sync process, making it brittle if overused.
Helm Hooks: Running Operations During a Release
Helm Hooks are Kubernetes resources that run at specific moments in a release lifecycle. Use them to run a database migration before an app upgrade or to back up data before a deletion. The footgun: a failing hook Job will block and fail the entire.
Helm Subcharts: Composing Applications from Dependencies
Helm subcharts are nested components a parent chart can configure, like a docker-compose for Kubernetes manifests. Use them to bundle dependencies like a database with your app. The footgun: subcharts are sandboxed and cannot access parent values directly.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles