What is robots.txt, why respect it, and consequences of ignoring it?

What's really being asked

This question checks whether a candidate sees web scraping as more than a technical exercise. Interviewers want to know if you understand the Robots Exclusion Protocol, server etiquette, and the legal and ethical boundaries around automated data collection. At the senior level, they expect you to discuss liability, operational impact, and how to build scrapers that respect external systems.

The full answer

First, define robots.txt as a file that uses the Robots Exclusion Protocol to tell crawlers which portions of a website they are allowed to visit. Second, explain that respecting it matters for three reasons in order: it prevents unnecessary server load and bandwidth costs for the target site; it reduces legal risk because ignoring directives can violate laws like the Computer Fraud and Abuse Act or terms of service; and it preserves professional trust and your own infrastructure since violators often get IP blocked. Third, mention concrete consequences such as rate limiting, permanent bans, cease-and-desist letters, or litigation. Fourth, note that robots.txt is a courtesy standard and not a hard security boundary, so a well-designed scraper should also check response codes and implement polite rate limiting regardless of what the file says.

The mistakes people make

A major red flag is claiming that robots.txt is irrelevant because the data is public or because the file lacks legal enforcement everywhere. Another weak response is saying the file is a security mechanism that hides sensitive data; it does not hide anything, it only requests polite behavior. Candidates who admit to ignoring it in past projects without discussing risk mitigation also signal poor judgment. Finally, confusing robots.txt with an allowlist rather than a disallow list shows a lack of familiarity with the standard.

What usually comes next

The interviewer may ask how you would handle a site that has no robots.txt file, how you implement rate limiting in practice, or whether you have ever faced a legal challenge related to scraping. They might also ask how you would scrape ethically if robots.txt blocks the exact data you need for a business-critical project, probing your ability to negotiate API access or find alternative data sources.

A concrete example

Suppose you are building a price-monitoring scraper for an e-commerce site. You check robots.txt and see that the Disallow directive covers the product detail path. A good answer means you stop crawling that path, reduce your request rate to one page every few seconds on allowed paths, and reach out to the site for a data partnership or API key. Ignoring the directive would likely trigger a Cloudflare block within hours, cost the site money in compute and bandwidth, and expose your company to a CFAA lawsuit or a cease-and-desist letter.