Scraping Dynamic Sites: Find the API, Not Just Render

WHY IT EXISTS Web pages increasingly load a minimal HTML skeleton and then use JavaScript to fetch and display the actual content. A simple HTTP request from a scraper like Scrapy or cURL only gets the initial skeleton, not the final data you see in a browser. This creates a mismatch between what the scraper sees and what the user sees.

THE MENTAL MODEL Act like a detective, not a photographer. Instead of capturing the final rendered page (the "picture"), investigate how the page was assembled. Your goal is to find the source of the data—the hidden API call—and interact with it directly. The rendered page is just a side effect of the JavaScript fetching and displaying that data.

HOW IT WORKS The process involves three main steps. First, open the target webpage in your browser and open the Developer Tools, specifically the Network tab. Second, interact with the page to trigger the data loading, and watch for XHR (XMLHttpRequest) or Fetch requests. These are the JavaScript-initiated API calls. Look for one that returns your target data, often in a structured format like JSON. Third, replicate that request in your scraper. You can right-click the request in most browsers and copy it as a cURL command, then use a tool or library to convert that into code for your scraper, making sure to copy the URL, headers, and any request body.

WHEN TO USE IT Use this technique whenever you inspect a page's source code (e.g., with curl or Scrapy's fetch command) and find that the data you see in your browser is missing. This is the primary method for scraping single-page applications (SPAs) and sites with infinite scroll or other dynamic content loading mechanisms.

WHEN NOT TO USE IT If the data is present in the initial static HTML response, this method is overkill; simple selectors are sufficient. Also, if finding and replicating the API call is excessively complex (e.g., due to complex authentication or obfuscated JavaScript), it may be more time-efficient to use a headless browser like Playwright or Selenium as a last resort, accepting the performance penalty.

ONE CANONICAL EXAMPLE You want to scrape product prices from an e-commerce category page that uses infinite scroll. Your scraper only gets the first 20 products. Using the Network tab, you discover that scrolling triggers a GET request to api.example.com/products?page=2. The response is a JSON object containing the next 20 products. Instead of trying to simulate scrolling, your scraper can now simply loop through page numbers and call this API directly to get all product data quickly and reliably.