Detailed Explanation of Sequential and Random I/O

In the realm of storage performance, it is crucial to distinguish between the capabilities of a device and its access pa

In the realm of storage performance, it is crucial to distinguish between the capabilities of a device and its access patterns. Sequential and random IO are not fixed attributes of a hardware device; rather, they describe the way data is requested from a block device.

Sequential IO occurs when the requested Logical Block Addresses (LBAs) are contiguous or nearly contiguous. For example, a process reading 1MiB, then 2MiB, then 3MiB in a continuous stream is performing sequential access. Because the data is laid out in a predictable order, the storage system can often anticipate the next request.

Random IO, conversely, involves requests that "jump" across the logical address space. An example would be reading 4KiB at offset 0, followed by 4KiB at offset 1MiB, and then writing 4KiB at offset 512KiB. These disjointed requests prevent the system from leveraging predictive optimizations.

The size of the IO request also plays a decisive role in performance characteristics:

  • Sequential IO is typically paired with large transfer sizes (e.g., 128KiB or 1MiB) to maximize throughput.
  • Random IO is usually associated with small block accesses (e.g., 4KiB or 8KiB), where the primary goals are maximizing IOPS (Input/Output Operations Per Second) and minimizing latency.
  • While large-block random IO can approach the throughput of sequential IO, the addresses remain non-contiguous, preventing certain hardware optimizations.
  • Small-block sequential IO is more efficient than small-block random IO, but it is still limited by the per-request overhead of the system.

The Root Causes of Performance Disparity

The performance gap between sequential and random access varies significantly depending on the underlying storage technology.

Hard Disk Drives (HDD)

On mechanical HDDs, the advantage of sequential IO is massive. Accessing data on an HDD requires physical movement: the actuator arm must seek the correct track, and the platter must rotate to the correct sector.

In a sequential read, the head moves to the starting position once and then reads a continuous stream of data as the disk spins. In a random read, however, every single request may require a new seek and a wait for rotational latency. This mechanical overhead can push latencies into the millisecond range, often limiting random IOPS to just a few hundred, even while sequential throughput reaches hundreds of MB/s.

Solid State Drives (SSD)

SSDs eliminate mechanical seek time and rotational latency, which drastically improves random performance. However, they are not immune to the complexities of NAND flash memory. Several factors keep random IO from being "free":

  • NAND Characteristics: NAND flash is read and written in pages but must be erased in larger blocks.
  • FTL (Flash Translation Layer): The SSD must maintain a complex mapping table to translate logical addresses to physical NAND locations. Random reads require frequent lookups in this mapping table.
  • Write Amplification and Garbage Collection: Random writes often trigger "out-of-place" updates. This forces the controller to perform Garbage Collection to reclaim space, leading to Write Amplification, where more data is physically written to the NAND than requested by the host.
  • Parallelism: While high queue depths and NVMe multi-queue architectures allow SSDs to handle many requests in parallel, small random writes remain significantly more "expensive" in terms of internal management than sequential writes.

Consequently, while SSDs have narrowed the gap, the performance penalty for small-block random writes remains a critical bottleneck in high-performance computing.

Key Performance Metrics

To accurately evaluate storage performance, one must look beyond a single number and understand the relationship between several key metrics:

  • IOPS (Input/Output Operations Per Second): The number of completed IO requests per second. This is the gold standard for measuring small-block, random workloads.
  • Throughput (Bandwidth): The amount of data transferred per second (e.g., MB/s or GB/s). This is the primary metric for large-block, sequential workloads.
  • Latency: The time elapsed from the moment an IO is submitted to the moment it is completed.
  • Queue Depth (QD): The number of outstanding (uncompleted) IO requests. Higher queue depths can help saturate the device's parallelism.
  • Read/Write Ratio: The proportion of read vs. write operations, as random writes are generally more taxing on the hardware than random reads.

The relationship between these metrics can be expressed by the fundamental formula:

Throughput = IOPS × Block Size

For instance, a drive achieving 100,000 IOPS with a 4KiB block size results in a throughput of approximately 400MB/s. Conversely, a drive achieving 1,000 IOPS with a 1MiB block size results in a throughput of approximately 1GB/s. When benchmarking, it is vital to keep the block size, read/write ratio, and queue depth constant to ensure a fair comparison.

Typical Use Case Scenarios

Sequential IO Scenarios

  • Logging: Database Write-Ahead Logs (WAL) or binlogs that append data to the end of a file.
  • Data Movement: Large file copies, system backups, and long-term archiving.
  • Media Processing: Video streaming, playback, and high-bitrate transcoding.
  • Analytics: Full table scans in data warehouses.
  • LSM Trees: The flushing of memtables to disk and the subsequent compaction processes.

Random IO Scenarios

  • OLTP Databases: Transactional workloads using B+ tree indexes to look up specific records.
  • Virtualization: Multiple virtual machines accessing different virtual disks simultaneously on a single physical host.
  • File Systems: Frequent updates to file system metadata and small-file read/writes.
  • Key-Value Stores: Point lookups and individual key updates in NoSQL databases.
  • Cache Misses: When a system must fetch scattered data from disk because it is not present in the RAM cache.

In practice, most real-world workloads are mixed. For example, an LSM-tree-based storage engine performs mostly sequential writes (appends), but its read pattern may be highly random, and its compaction process generates massive amounts of both sequential and random IO.

Benchmarking with fio

The fio (Flexible I/O Tester) tool is the industry standard for measuring these patterns. Below are examples of how to test different patterns on a file (always use a test file rather than a raw production device to prevent data loss).

# Sequential Read, 1MiB block size
fio --name=seqread --rw=read --bs=1M --size=1G \
    --filename=/tmp/fio.test --direct=1 --iodepth=16 \
    --runtime=60 --time_based

# Random Read, 4KiB block size
fio --name=randread --rw=randread --bs=4k --size=1G \
    --filename=/tmp/fio.test --direct=1 --iodepth=32 \
    --runtime=60 --time_based

# Random Write, 4KiB block size
fio --name=randwrite --rw=randwrite --bs=4k --size=1G \
    --filename=/tmp/fio.test --direct=1 --iodepth=32 \
    --runtime=60 --time_based

When analyzing the output, pay close attention to iops, bw (bandwidth), and lat (latency).

Optimization Strategies

Application Layer

  • IO Merging: Batch small IO requests into larger ones before submission.
  • Log-Structured Storage: Use append-only patterns to transform random updates into sequential writes.
  • Alignment: Ensure IO requests are aligned to 4KiB, page boundaries, or RAID stripe sizes to prevent "split IOs."
  • Asynchronous IO: Utilize io_uring or AIO to increase concurrency and hide latency.

File System and Storage Layer

  • Read-Ahead Tuning: Adjust the kernel's read-ahead parameters to improve sequential read efficiency.
  • Mount Options: Use noatime to reduce unnecessary metadata writes.
  • Fragmentation Management: Periodically defragment or rebuild file systems to maintain sequentiality.
  • SSD Over-provisioning: Leave unallocated space on SSDs to reduce write amplification and improve garbage collection efficiency.
  • I/O Schedulers: For NVMe devices, use none or mq-deadline to leverage the device's native multi-queue capabilities.

Database Layer

  • Workload Isolation: Place WAL files on devices optimized for sequential writes.
  • Buffer Pool Management: Increase the memory buffer pool to cache more data and reduce the frequency of random disk reads.
  • Direct IO: Use O_DIRECT when the database manages its own cache to avoid double-buffering and unpredictable OS-level caching.
  • Compaction Monitoring: Closely monitor how background processes like compaction impact foreground transaction latency.

Summary

Mastering the distinction between sequential and random IO is fundamental to high-performance system design. While HDDs suffer heavily from the mechanical penalties of random access, SSDs have significantly mitigated this issue through parallelism. However, the inherent nature of NAND flash means that small random writes remain a costly operation. The ultimate goal in storage engineering is to design systems that either transform random access into sequential access or use caching, batching, and high concurrency to mask the inherent costs of random IO.