Everything in Data Science & Analytics, page 3
Diagnosing Spark executor OutOfMemoryError
Check the Spark UI for skew and spills, inspect executor memory and partition count, find culprits like wide collect, huge shuffles, or skewed keys, and fix via more partitions, memory tuning, or…
Spark RDDs, DataFrames, and Datasets
RDDs are low-level typed object collections with no built-in optimization; DataFrames are named columns optimized by Catalyst and Tungsten; Datasets add compile-time type safety in…
Data skew in Spark and salting
Data skew is uneven key distribution sending most rows to one partition and straggler task; salting appends a random suffix to hot keys to spread them across partitions, joining in two…
Spark broadcast join versus shuffle join
A broadcast join sends the small table to every executor so the large table joins locally with no shuffle of its rows; the default sort-merge join shuffles both tables across the network, which is costly.
The MapReduce paradigm explained
Map applies a function to each input record emitting key-value pairs in parallel; a shuffle groups values by key; reduce aggregates each key's values into a result.
HDFS purpose and fault tolerance
HDFS stores huge files across many commodity machines as large blocks, replicating each block across nodes for fault tolerance; unlike NTFS or ext4 it is distributed, write-once, and optimized for…
Spark transformations versus actions
Transformations like map and filter are lazy and build a lineage DAG returning a new RDD; actions like count or collect trigger execution and return a value to the driver.
Mode collapse in GANs and how to fix it
Mode collapse is the generator producing few similar outputs, missing data diversity to fool the discriminator; fixes include Wasserstein loss, minibatch discrimination, unrolled GANs, and feature…
Why Transformers use multi-head attention
A single head averages into one representation subspace; multiple heads attend in parallel to different subspaces, letting the model capture diverse relations like syntax and coreference at once, then…
Exploration versus exploitation: epsilon-greedy and UCB
Exploit current best to earn reward, explore to discover better options; epsilon-greedy explores randomly with probability epsilon; UCB explores by an uncertainty bonus…
RL components and how Q-learning works
Agent acts on the environment, observes state and reward, seeking to maximize cumulative discounted reward; Q-learning iteratively updates Q(s,a) toward reward plus discounted best…
Analyzing skewed revenue-per-user experiments
Heavy tails inflate variance and slow significance, and the mean is dominated by whales; mitigate via winsorization or capping, log transforms, CUPED variance reduction, or bootstrap and rank tests.
Interpreting a black-box gradient boosting model
Global tools like permutation importance or aggregated SHAP rank overall feature influence; local tools like per-instance SHAP or LIME explain one prediction; SHAP unifies both via additive…
Random Forest versus Gradient Boosting
Random Forest trains deep trees in parallel and averages to cut variance; boosting builds shallow trees sequentially, each correcting prior errors to cut bias, often higher accuracy but…
Stemming versus lemmatization in text preprocessing
Stemming chops affixes fast but crudely, yielding non-words; lemmatization maps to real dictionary base forms using POS, slower but accurate; skip both for embedding or transformer models.
Handling missing numerical values
Dropping rows is simple but loses data and can bias if missingness is non-random; mean or median imputation keeps rows but shrinks variance and ignores correlations; model-based imputation is…
Pandas loc versus iloc indexing
Loc selects by label and is inclusive of both endpoints; iloc selects by integer position and is exclusive of the stop; passing a string label to iloc fails.
MLE versus MAP estimation and the role of priors
MLE maximizes likelihood alone; MAP maximizes likelihood times a prior, acting as regularization that shrinks toward prior beliefs; with abundant data they converge.
Eigenvalues, eigenvectors, and their role in PCA
An eigenvector keeps direction under a matrix, its eigenvalue scales it; PCA finds eigenvectors of the covariance matrix as principal axes.
How gradient descent and the learning rate work
Gradient descent steps downhill along the negative gradient to minimize cost; the learning rate sets step size; too high diverges or oscillates, too low converges painfully slowly.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles