pandas .apply() versus vectorized operations
pandas performance literacy.
apply runs a Python function per row or column, flexible but slow due to per-element looping; prefer vectorized ops; use apply only for custom logic with no vectorized equivalent.
What's really being asked
This checks whether you understand the performance model of pandas: that operations pushed down to vectorized, C-implemented routines vastly outperform calling a Python function repeatedly, and whether you know when each is appropriate.
The full answer
The .apply() method runs a user-supplied Python function across an axis of a DataFrame, once per row or per column, or per element for a Series. Its strength is flexibility: you can express arbitrary custom logic that has no built-in equivalent. Its weakness is performance. Under the hood .apply() is essentially an interpreted Python loop, so the Python-level function-call overhead is paid for every row, and it cannot exploit the columnar, C-level optimizations that vectorized operations use. Vectorized operations, such as arithmetic on whole columns, boolean comparisons, string accessors, or np.where, operate on entire arrays at once in compiled code, often one to two orders of magnitude faster. The rule of thumb is to reach for vectorized expressions first and treat .apply() as a fallback. Note that .apply() with axis=1 over rows is especially slow; df.apply over columns or built-in aggregations are faster, and itertuples beats iterrows when iteration is truly needed.
The mistakes people make
Believing .apply() is itself vectorized or fast. Saying it should always be avoided, when sometimes there is no vectorized alternative. Confusing applymap, apply, and map. Recommending iterrows, which is even slower.
What usually comes next
How do you vectorize a conditional that you wrote with apply? When is a lookup table or merge better than apply? What does the GIL have to do with this?
A concrete example
Necessary case: you must call an external geocoding library on each address string; there is no array operation for that, so df['addr'].apply(geocode) is the right tool. Avoid case: to compute a discounted price you might write df.apply(lambda r: r.price 0.9, axis=1), but the vectorized df['price'] 0.9 produces the identical result far faster because it runs in compiled code over the whole column rather than looping in Python.
Interview question
You compute df['price'] * 0.9 instead of df.apply(lambda r: r.price * 0.9, axis=1). Why is the first far faster?
- a.The two produce different results, so speed is irrelevant
- b.Column arithmetic runs as a compiled vectorized op over the whole array, avoiding a per-row Python loopCorrect
- c.apply parallelizes across cores while arithmetic does not
- d.apply compiles to C while arithmetic stays in Python
Why? this is the answer
Vectorized column arithmetic executes in compiled C over the entire array, while apply with axis=1 invokes a Python function per row, paying interpreter overhead each time. apply is not parallel or compiled, and both give the same result here.
Just read this? Test yourself on what you have been reading.
Read the original → datacamp.com
- #pandas
- #apply
- #vectorization
- #performance
- #numpy
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles