Intermediate concepts in Data Science & Analytics, page 3
Topic Modeling: Finding Themes in Unstructured Text
Topic modeling automatically finds themes in text by grouping words that often appear together. It's used to analyze customer feedback or organize large document sets.
Named Entity Recognition: Finding the 'Who, What, Where' in Text
Named Entity Recognition (NER) is a smart highlighter for text, automatically finding and tagging nouns like people, places, and organizations. It powers search and extracts structured data from news or support tickets.
Q-Learning: Teaching an Agent by Trial and Error
Q-Learning teaches an agent the 'quality' of an action in a given state through trial and error, like training a pet with treats. It's used in robotics for navigation or in games where an AI learns optimal moves.
Markov Decision Process: A Map for Sequential Decisions
A Markov Decision Process models sequential choices with uncertain outcomes. Think of it as a game with states, actions, and rewards, but where your next move is probabilistic.
YARN: Hadoop's Cluster Operating System
YARN acts as the operating system for a Hadoop cluster, separating resource management from job processing. This allows multiple engines like Spark or MapReduce to run on the same hardware.
Spark RDDs: Immutable, Distributed Data Collections
An RDD is Spark's core abstraction: an immutable, partitioned collection of items processed in parallel. It's the go-to for low-level, unstructured data tasks. The main footgun is using RDDs when higher-level DataFrames offer better performance.
CAP Theorem: Pick Two of Three Guarantees
The CAP Theorem states a distributed system can only have two of three guarantees: Consistency, Availability, or Partition Tolerance. When the network fails (a partition), you must choose: stop responding to ensure data is consistent (CP) or keep responding…
Apache Hive: SQL on Hadoop
Apache Hive translates SQL-like queries into Hadoop MapReduce jobs, letting you analyze huge datasets with familiar syntax. Don't mistake it for a real-time database; its power is in high-throughput batch processing, not low-latency queries.
Columnar Storage: Read Less Data, Analyze Faster
Columnar formats like Parquet store data by column, not by row. This makes analytical queries that select a few columns from a wide table incredibly fast by minimizing disk I/O. It's a poor fit for transactional workloads that need entire rows at once.
Pre-attentive Attributes: Your Brain on Autopilot
Pre-attentive attributes are visual properties your brain processes in milliseconds, before you consciously focus. They're the workhorses of data visualization, making outliers and patterns pop out of a dashboard. The biggest mistake is using too many at once.
ML Pipeline: Systematic Model Delivery
A machine learning pipeline is the systematic workflow that carries models from data labeling through deployment inside MLOps. It keeps the AI lifecycle repeatable rather than ad hoc. The footgun is treating a one-off notebook as a production pipeline.
Model Monitoring: A Health Check for Production AI
Model monitoring is a smoke detector for your AI, alerting you when its performance degrades. It compares live data to training data to catch data drift or shifts in user behavior. The footgun is assuming a model, once deployed, performs well forever.
Data Drift: Why Good Models Go Bad
Data drift is when a model's accuracy decays because the real-world data it was trained on has changed. This happens in fraud detection as scams evolve or in e-commerce as trends shift.
IaC for MLOps: Your ML Factory's Blueprint
Treat ML infrastructure—training clusters, model registries, serving endpoints—as code in version-controlled files. This ensures reproducible experiments and consistent deployments across environments.
AI Transparency: Explaining the Black Box's 'Why'
AI transparency means seeing the 'why' behind an algorithm's decision, not just its code. It's vital for high-stakes systems like credit scoring or news feeds. The footgun is thinking open-sourcing the model is enough; true transparency explains the logic.
AI Accountability: Who's Responsible When AI Fails?
AI accountability means someone is answerable for an AI's actions. It requires organizations to manage risks and trace decisions throughout the AI's lifecycle, ensuring systems function properly and align with human-centric values.
Fairness Metrics: Auditing Your AI for Bias
Fairness metrics are statistical checks to see if your model's decisions are biased against certain groups. They're crucial for automated systems in hiring or loan approvals.
Datasheets for Datasets: The Nutrition Label for Data
A datasheet is like a nutrition label for a dataset, documenting its origins, contents, and intended use. This is crucial for high-stakes ML systems where hidden biases could cause harm.
AI Safety: Preventing Unintended Consequences
AI Safety is the engineering discipline for preventing intelligent systems from causing harm, by accident or misuse. It's crucial for autonomous systems like self-driving cars or large models that can amplify bias.
Proxy Metrics: Estimate Long-Term Impact Now
A proxy metric uses a model to estimate a slow, long-term outcome, like annual revenue. It lets you quickly judge an A/B test's impact without waiting months for the true result. The footgun is trusting a biased model or ignoring its error, giving you false.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles