Data Lake: A Single Repository for Raw Data
A data lake is a central repository that stores vast amounts of raw data in its native format. It acts as a single source for analytics and machine learning, but without proper management it can become a useless "data swamp".
WHY IT EXISTS: Traditional systems struggled to store and process the growing volume and variety of data. A centralized place was needed to hold everything—structured, unstructured, raw—without expensive upfront transformation, enabling new kinds of analysis and machine learning.
THE MENTAL MODEL: Think of a data lake as a literal lake. It collects water (data) from many sources (rivers, streams, rain) in its natural, unfiltered state. You can then draw from this large body of water for various purposes, like reporting, analytics, or machine learning. The key is that all data, regardless of origin or format, flows into one place.
HOW IT WORKS: A data lake stores data as files or object blobs, often in cloud storage. It ingests raw copies of data from source systems, like sensor feeds and application logs, alongside structured data from relational databases. It doesn't require a predefined schema before storing data. This "schema-on-read" approach provides flexibility to handle structured, semi-structured, unstructured, and binary data in one system.
WHEN TO USE IT: Use a data lake when you need a single, scalable store for diverse data types for future analysis. It's ideal for exploratory analytics, data science, and machine learning workloads that benefit from access to raw, granular data. It's also used as the primary repository for feeding transformed data into downstream systems.
WHEN NOT TO USE IT: A data lake is not a replacement for a database for transactional workloads. Its schema-on-read approach and focus on batch processing make it unsuitable for applications requiring fast, indexed queries on structured data with strong consistency guarantees. If your data is purely structured and query patterns are well-defined, a data warehouse might be a simpler choice.
ONE CANONICAL EXAMPLE: A company collects user clickstream data (semi-structured JSON), customer records from a SQL database (structured), and support call transcripts (unstructured text). All this data is ingested into a cloud-based data lake. Data scientists can then query this single repository to build a model that predicts customer churn, combining behavioral, transactional, and sentiment data.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.