Everything in DevOps & Cloud, page 13
Idempotency in data ingestion pipelines
Idempotency means re-running a step yields the same result with no duplicates; it matters because retries and at-least-once delivery are inevitable; achieve it with deduplication keys or upserts.
Handling late data in streaming windows
Use event-time windows with watermarks to bound lateness, allow a grace period before finalizing, and route data later than that to a side output.
Diagnosing and fixing data skew in Spark
This is data skew, caused by uneven key distribution concentrating rows on few partitions; mitigate with salting, broadcast joins, repartitioning, or adaptive execution.
Partitioning order events in a data lake
Partition by the columns queries filter on, typically date hierarchy and category, balancing granularity to avoid too many tiny files.
CSV vs JSON vs Parquet for analytics
CSV and JSON are row-based, human-readable, and bulky; columnar Parquet/ORC compress well and read only needed columns; choose columnar for analytics.
ETL versus ELT in cloud data platforms
ETL transforms before loading into the target; ELT loads raw first then transforms in the warehouse, leveraging cheap storage and elastic compute.
Data lake versus data warehouse
Lakes store raw, schema-on-read data of any type cheaply; warehouses store curated, schema-on-write structured data for fast SQL; choose a lake for varied raw data and ML.
Resilient stateful batch on Spot Instances
Externalize state and checkpoint to durable storage, react to interruption and rebalance notices to drain gracefully, diversify instance pools.
Designing a multi-account cloud chargeback model
Account-per-team or mandatory cost-allocation tags enforced by SCPs and tag policies, plus a pipeline over the cost and usage report grouped by tag/account.
Savings Plans vs Reserved Instances for mixed compute
Compute Savings Plans cover EC2, Fargate, and Lambda flexibly; EC2 Instance Plans and RIs trade flexibility for slightly deeper discounts.
Reducing cross-region data transfer costs
Identify cross-region, cross-AZ, and internet egress; co-locate chatty components; add VPC endpoints, CDN caching, and compression.
Lifecycle storage tiering for compliance logs
Hot tier for 7-day query window, lifecycle rules transitioning to infrequent-access then archive, expiration at one year.
Rightsizing an underutilized VM fleet safely
Gather multi-week percentile metrics across CPU, memory, network and disk; pick smaller or right-family types; roll out gradually with monitoring.
On-Demand vs Reserved vs Spot pricing models
On-Demand is flexible but priciest, Reserved trades a 1-3 year commitment for discounts, Spot is cheapest but interruptible.
Attribute cloud costs to teams
Tag resources with team and project metadata, activate them as cost-allocation tags, group the cost report by that tag, and enforce tagging with policy.
Design automated cloud cost optimization
Target idle resources, oversized instances, orphaned storage, and commitment gaps; act via rightsizing and cleanup; safeguard with tagging, scoping, and approvals.
Monitoring with SLOs and error budgets
Define SLIs from the user's view, set SLO targets, derive an error budget, and alert on burn rate rather than raw thresholds.
Strategy for large multi-team IaC projects
Versioned reusable modules, state split per environment and component, promotion of identical code via variables, and externalized secrets.
Diagnose 100% CPU on a managed database
Correlate the spike with deploys and traffic, find top queries via the engine's views, inspect plans for missing indexes, then tune before scaling.
Enforce a cloud resource compliance policy
Prevent at creation with org policies or admission checks, detect violations via continuous config scanning, and auto-remediate by stripping the IP or alerting owners.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles