HDFS

The Hadoop Distributed File System, or HDFS, is a distributed storage platform at the core of the Hadoop ecosystem. It provides a storage foundation for large-scale data processing that can grow from gigabytes to petabytes or beyond. Its design priorities are high throughput, scalability, and fault tolerance rather than low-latency random access.

This article takes an overview-oriented view. It discusses design motivations, fundamental principles, comparisons with other storage systems, and common use cases. It does not go deep into NameNode internals, high availability elections, or federation.

Why Distributed Storage Matters for Big Data

Modern data platforms often begin with single-node storage. As data grows from gigabytes to terabytes, then to petabytes or exabytes, local disk capacity and input/output operations become bottlenecks. Workloads frequently require high-throughput sequential reads and writes, not frequent random writes with low latency.

Large clusters are often made of inexpensive servers, so hardware failures are treated as normal events. Redundancy is therefore essential. HDFS addresses these challenges with a write-once, read-many model and a distributed copy mechanism.

Core Design Principles

HDFS is guided by several principles that shape its behavior.

• Block storage: A file is divided into fixed-size blocks. Blocks serve as the basic unit of storage, replication, and scheduling. Larger blocks support efficient sequential access and reduce seek overhead.
• Multi-replica mechanism: Each block is typically stored on multiple nodes, often across different racks. This improves fault tolerance and availability.
• Metadata and data separation: The NameNode manages namespace, directory structure, permissions, and block mappings. DataNodes handle actual data storage and access. This separation makes it easier to scale the data plane.
• Move computation to data: Processing frameworks try to run tasks where the data resides. This reduces network movement and improves throughput.
• Throughput first: HDFS favors aggregate throughput for batch data over single-access latency.

These choices create clear boundaries. HDFS works well for large files, sequential reads, batch writes, and low-frequency updates. It is less suited to small files, random writes, low-latency access, and frequent modifications.

Architecture at a Glance

HDFS uses a master-worker architecture. The main components include:

• NameNode: Manages the filesystem namespace and metadata, such as directory structure, permissions, and block locations.
• DataNode: Stores, reads, and writes actual data blocks, then reports status to the NameNode.
• Client: Accesses HDFS through APIs, command-line tools, or compute engines.
• Standby NameNode and JournalNode: Support high availability and metadata synchronization, though these are advanced topics.

A typical read begins when a client asks the NameNode for file metadata and block locations. The client then connects to the relevant DataNodes and reads the data. A write follows a similar path: the client requests file creation, the NameNode assigns block locations, and the client writes to replica locations.

HDFS Compared with Other Storage