More in Backend Dev — page 49
Cache Eviction: Deciding What to Forget
A cache eviction policy is the rule for discarding data when fast-access memory is full. This is crucial for databases and CDNs. The common mistake is assuming one policy, like LRU, fits all workloads, which can cripple performance on certain access patterns.
Full-Text Search: Beyond Simple String Matching
Full-text search isn't just string matching; it's a search engine for your data that understands language. Use it for e-commerce search or log analysis. The footgun is thinking a simple `LIKE` query is a substitute for a real search engine like Elasticsearch.
Data Retention Policy: Your Schedule for Deleting Data
A data retention policy is your company's official schedule for deleting data, not a plan to keep it forever. It's essential for legal compliance (like GDPR) and managing storage costs.
Vector Embeddings: Turning Meaning into Math
Vector embeddings turn complex data like words or images into lists of numbers (vectors). This lets computers measure "similarity" by calculating the distance between these vectors, powering search and recommendations.
Cache-Aside Pattern: Your App Owns the Cache
The Cache-Aside pattern makes your application the gatekeeper for the cache. On a read, your code checks the cache first; on a miss, it fetches from the database and writes to the cache. This speeds up read-heavy apps. The key footgun is stale data.
Inverted Index: How Search Engines Find Your Keywords
An inverted index is like a book's index: it maps keywords to the documents containing them. This is the core of full-text search in search engines and databases, allowing instant lookups.
Windowing: Taming Infinite Data Streams
Windowing chops infinite data streams into finite chunks for aggregation, like counting clicks per minute. It's essential for real-time dashboards, fraud detection, and IoT sensor analysis. The main footgun is mishandling late data by confusing event time vs.
Delta Lake: Database Reliability for Your Data Lake
Delta Lake adds a transaction log to your data lake, giving you database-like reliability over raw files. This enables ACID transactions, schema enforcement, and unified batch/streaming pipelines.
Lambda Architecture: Batch + Stream for Big Data
Lambda Architecture handles massive datasets by combining slow, accurate batch processing with fast, real-time stream processing. It's used for analytics needing both historical and live views.
Schema Evolution: Changing a Live Database Without Outages
Schema evolution is like renovating a house while you live in it: you must change your database's structure without breaking the live application. This is critical when adding or renaming columns.
Message Queues: Decoupling Your Services
Think of a message queue as a digital post office for your services. It lets one part of your system drop off a task for another to handle later, decoupling them so they don't have to run in lock-step.
Data Pipelines: From Raw Data to Actionable Insights
A data pipeline is the plumbing for your data, moving it from raw sources to a refined state for analysis. It feeds dashboards and ML models by cleaning data from APIs and databases. The key footgun is choosing batch processing for real-time needs.
CockroachDB: A SQL Database That Survives Disasters
CockroachDB is a distributed SQL database designed to be unkillable. Use it for global apps needing strong consistency and high availability, like financial ledgers or identity systems. The footgun: ignoring network latency between nodes can kill performance.
Google Cloud Spanner: A Globally Distributed SQL Database
Spanner is a globally distributed SQL database that scales like NoSQL but keeps the strong consistency of a relational database. Use it for global applications like financial ledgers that need ACID transactions across continents.
NewSQL: SQL Scalability Without Sacrificing ACID
NewSQL databases aim for NoSQL's horizontal scaling with the ACID guarantees of a traditional relational database. They suit high-throughput OLTP systems, like e-commerce, that must scale out. The footgun is assuming they are a simple drop-in replacement.
AWS DMS: Your Managed Database Migration Engine
AWS DMS is a managed service for migrating databases. It acts like a replication server you point at a source and target, handling the data transfer. It's used for one-time migrations to AWS or for continuous replication.
Amazon DynamoDB: Scalable NoSQL as a Service
Think of DynamoDB as a database where you trade complex queries for near-infinite, hands-off scaling. It's a managed NoSQL service from AWS for key-value and document data, built for high-performance applications.
Amazon Aurora: AWS's Proprietary Relational Database
Amazon Aurora is a proprietary relational database from AWS, offered as part of the Amazon Relational Database Service (RDS). It provides a managed database solution within the AWS cloud ecosystem, available since October 2014.
Compute & Storage Separation: Scale One Without the Other
This architecture treats your data warehouse (cheap storage) and query engine (expensive compute) as separate services. You can scale compute for peak demand without overprovisioning storage.
Amazon RDS: Managed Relational Databases in the Cloud
Amazon RDS is like hiring a DBA to manage your database's plumbing. It's for when you need a SQL database like PostgreSQL or MySQL without the hassle of patching and backups. The footgun is assuming it's 'serverless'—you still manage cost and performance.