Data Catalog: The Library Card for Your Data
A data catalog is like a library card catalog for your company's data, telling you what exists, where it lives, and what it means. It helps analysts find trustworthy datasets and engineers trace the impact of schema changes. The footgun is letting it go stale.
THE MENTAL MODEL: A data catalog is a searchable inventory of metadata about all your organization's data. It doesn't hold the data itself, but rather acts like a library card catalog, telling you everything about the data: its location, schema, owner, quality, and lineage. It's the single source of truth for discovering and understanding data assets.
HOW IT WORKS: A cataloging tool connects to various data sources like databases, data warehouses, and data lakes. It automatically ingests technical metadata, such as table definitions, views, columns, and indexes. Humans then enrich this with business context, like plain-language descriptions, ownership details, and quality ratings. Users can then search this repository to find relevant data without needing direct database access or tribal knowledge.
WHEN TO USE IT: Use a data catalog when your organization's data landscape becomes too large for any single person to know. It's essential for enabling self-service analytics, where analysts need to find and trust data independently. It is also critical for data governance programs, compliance audits (like GDPR), and for performing impact analysis before changing a database schema.
WHEN NOT TO USE IT: For a very small team with a single, simple database, a dedicated catalog is often overkill. The maintenance overhead outweighs the benefits. In these cases, a well-maintained wiki page or shared document can serve the same purpose. A catalog's value emerges at scale, when the cost of not finding or misunderstanding data becomes high.
ONE CANONICAL EXAMPLE: An analyst needs to report on customer churn. Instead of asking in a public chat, she searches the data catalog for "customer churn". The results point to a certified table named analytics.mart_customer_churn. The catalog entry shows the table schema with descriptions for each column, lists the 'Customer Success Team' as the owner, and displays a data quality score of 98%. This gives her the confidence to use the dataset immediately.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.