Build a competitor sitemap scraper for H1 and word count

WHAT THIS TESTS: Your ability to design a distributed data-collection system that is technically robust, legally defensible, and respectful of target infrastructure. Interviewers want to see you treat scraping as an engineering problem with operational and ethical constraints, not just a quick Python script. They are listening for signals that you have run production crawlers before and understand the difference between a proof of concept and a maintainable system.

A GOOD ANSWER COVERS: First, sitemap.xml parsing using an XML parser with namespace handling and URL filtering to isolate blog paths, plus handling sitemap index files that point to child sitemaps. Second, an async HTTP client with configurable concurrency, exponential backoff, and retry logic for 5xx errors, 429 rate-limit responses, and timeouts. Third, HTML parsing with a lenient parser to extract the H1 tag and word count while handling missing elements gracefully and avoiding regex on markup. Fourth, rate limiting set explicitly below typical thresholds, say one request every two to five seconds, plus honoring robots.txt directives and using a transparent user-agent string. Fifth, data storage and deduplication using checksums or ETags to avoid reprocessing unchanged URLs. Sixth, ethical and legal guardrails including checking Terms of Service, respecting noindex or nocache tags, and consulting legal counsel before using proprietary content commercially.

COMMON WRONG ANSWERS: Writing a naive synchronous loop with no delays, which looks like a denial-of-service attack and will get the company blocked. Ignoring robots.txt or claiming it is not legally binding without acknowledging its role in establishing authorized access under computer fraud statutes. Failing to mention that raw article text may be copyrighted even if metadata is not. Using regex to parse HTML instead of a proper DOM parser, which breaks on minor markup changes. Not handling sitemap indexes or assuming every sitemap is a single flat file.

LIKELY FOLLOW-UPS: How would you detect and adapt if the site blocks your IP or serves a captcha? What if the sitemap is paginated or uses a sitemap index? How do you calculate word count accurately when JavaScript renders the content? Would you use a headless browser, and at what operational cost? How do you store and version this data over time?

ONE CONCRETE EXAMPLE: A senior candidate might say they would use Python with aiohttp for concurrent fetches capped at five workers, feed parsed URLs through a Redis-backed queue for resilience and crash recovery, parse HTML with BeautifulSoup, enforce a two-second delay between requests to the same domain, log every request for audit purposes, and run the final dataset past legal before any public use.