HDFS purpose and fault tolerance

WHAT THIS TESTS This checks whether you understand the design goals of a distributed file system and can contrast it meaningfully with a single-machine file system.

A GOOD ANSWER COVERS HDFS exists to store and stream very large datasets, far bigger than one machine's disk, reliably across a cluster of inexpensive commodity servers. A file is split into large blocks, commonly 128 megabytes, which are distributed across DataNodes; a central NameNode holds the metadata mapping files to blocks and their locations. Fault tolerance is achieved by replication: each block is stored on multiple DataNodes, with a default replication factor of three placed on different nodes and racks. If a DataNode fails, the data still exists on its replicas, and the NameNode detects the loss and re-replicates the under-replicated blocks elsewhere. Compared with NTFS or ext4, which manage one disk on one machine optimized for low-latency random reads and writes, HDFS is distributed across many machines, follows a write-once-read-many model with append rather than arbitrary in-place edits, uses much larger block sizes, and is tuned for high aggregate streaming throughput to support batch analytics.

COMMON WRONG ANSWERS Describing HDFS as just a faster local file system. Saying fault tolerance comes from RAID rather than block replication. Forgetting the NameNode and DataNode roles. Claiming it supports fast random in-place writes like a database.

LIKELY FOLLOW-UPS What happens if the NameNode fails and how is that mitigated. Why such large block sizes. How does data locality help MapReduce or Spark jobs.

ONE CONCRETE EXAMPLE Store a one-terabyte log file on a ten-node cluster. HDFS splits it into roughly eight thousand 128-megabyte blocks spread across the DataNodes, each block replicated three times. If one server's disk dies, no data is lost because every block it held survives on two other nodes, and HDFS automatically copies those blocks to restore three-way replication, all transparent to the application reading the file.