More in DevOps & Cloud — page 13
Handling late data in streaming windows
WHAT IT TESTS: event-time stream processing. OUTLINE: use event-time windows with watermarks to bound lateness, allow a grace period before finalizing, and route data later than that to a side output.
Diagnosing and fixing data skew in Spark
WHAT IT TESTS: distributed processing skew. OUTLINE: this is data skew, caused by uneven key distribution concentrating rows on few partitions; mitigate with salting, broadcast joins, repartitioning, or adaptive execution. RED FLAG: just adding more executors.
Partitioning order events in a data lake
WHAT IT TESTS: partition design for query pruning. OUTLINE: partition by the columns queries filter on, typically date hierarchy and category, balancing granularity to avoid too many tiny files. RED FLAG: partitioning on high-cardinality keys like order ID.
CSV vs JSON vs Parquet for analytics
WHAT IT TESTS: file format tradeoffs. OUTLINE: CSV and JSON are row-based, human-readable, and bulky; columnar Parquet/ORC compress well and read only needed columns; choose columnar for analytics. RED FLAG: defaulting to CSV for large analytical workloads.
ETL versus ELT in cloud data platforms
WHAT IT TESTS: data pipeline pattern fluency. OUTLINE: ETL transforms before loading into the target; ELT loads raw first then transforms in the warehouse, leveraging cheap storage and elastic compute.
Data lake versus data warehouse
WHAT IT TESTS: storage architecture fundamentals. OUTLINE: lakes store raw, schema-on-read data of any type cheaply; warehouses store curated, schema-on-write structured data for fast SQL; choose a lake for varied raw data and ML.
Resilient stateful batch on Spot Instances
WHAT IT TESTS: fault tolerance on interruptible compute. OUTLINE: externalize state and checkpoint to durable storage, react to interruption and rebalance notices to drain gracefully, diversify instance pools.
Designing a multi-account cloud chargeback model
WHAT IT TESTS: cost allocation architecture. OUTLINE: account-per-team or mandatory cost-allocation tags enforced by SCPs and tag policies, plus a pipeline over the cost and usage report grouped by tag/account. RED FLAG: relying on voluntary tagging.
Savings Plans vs Reserved Instances for mixed compute
WHAT IT TESTS: commitment-discount strategy. OUTLINE: Compute Savings Plans cover EC2, Fargate, and Lambda flexibly; EC2 Instance Plans and RIs trade flexibility for slightly deeper discounts. RED FLAG: over-committing or ignoring usage variability.
Reducing cross-region data transfer costs
WHAT IT TESTS: data egress cost awareness. OUTLINE: identify cross-region, cross-AZ, and internet egress; co-locate chatty components; add VPC endpoints, CDN caching, and compression. RED FLAG: assuming all transfer is free or ignoring NAT gateway egress.
Lifecycle storage tiering for compliance logs
WHAT IT TESTS: object storage lifecycle design. OUTLINE: hot tier for 7-day query window, lifecycle rules transitioning to infrequent-access then archive, expiration at one year.
Rightsizing an underutilized VM fleet safely
WHAT IT TESTS: data-driven cost optimization. OUTLINE: gather multi-week percentile metrics across CPU, memory, network and disk; pick smaller or right-family types; roll out gradually with monitoring. RED FLAG: resizing on average CPU alone.
On-Demand vs Reserved vs Spot pricing models
WHAT IT TESTS: cloud cost fluency. OUTLINE: On-Demand is flexible but priciest, Reserved trades a 1-3 year commitment for discounts, Spot is cheapest but interruptible. RED FLAG: putting stateful or latency-critical work on Spot.
Attribute cloud costs to teams
WHAT IT TESTS: cost allocation fundamentals. OUTLINE: tag resources with team and project metadata, activate them as cost-allocation tags, group the cost report by that tag, and enforce tagging with policy. RED FLAG: attribution with no tagging.
Design automated cloud cost optimization
WHAT IT TESTS: FinOps automation with guardrails. OUTLINE: target idle resources, oversized instances, orphaned storage, and commitment gaps; act via rightsizing and cleanup; safeguard with tagging, scoping, and approvals.
Monitoring with SLOs and error budgets
WHAT IT TESTS: SRE reliability targets. OUTLINE: define SLIs from the user's view, set SLO targets, derive an error budget, and alert on burn rate rather than raw thresholds. RED FLAG: paging on every CPU blip with no link to user impact.
Strategy for large multi-team IaC projects
WHAT IT TESTS: scaling IaC organizationally. OUTLINE: versioned reusable modules, state split per environment and component, promotion of identical code via variables, and externalized secrets. RED FLAG: one giant shared state or secrets in code.
Diagnose 100% CPU on a managed database
WHAT IT TESTS: structured DB triage. OUTLINE: correlate the spike with deploys and traffic, find top queries via the engine's views, inspect plans for missing indexes, then tune before scaling. RED FLAG: scaling up without finding the offending query.
Enforce a cloud resource compliance policy
WHAT IT TESTS: policy-as-code guardrails. OUTLINE: prevent at creation with org policies or admission checks, detect violations via continuous config scanning, and auto-remediate by stripping the IP or alerting owners. RED FLAG: manual periodic audits only.
State drift in Terraform
WHAT IT TESTS: IaC state versus reality. OUTLINE: drift is when live infrastructure diverges from recorded state, usually via manual console changes; detect with plan or refresh, remediate by re-applying or importing. RED FLAG: hand-editing state.