Spark Architecture and Process

Apache Spark is a distributed computing engine built for processing massive datasets. Its design goal is to deliver fast

Apache Spark is a distributed computing engine built for processing massive datasets. Its design goal is to deliver fast, general-purpose, and scalable in-memory computation. In many data engineering and machine learning pipelines, Spark becomes the central tool because it can support batch, streaming, and graph workloads through its APIs.

The framework follows a master-worker pattern. A single Driver manages the application, while Executors run on cluster nodes and carry out the actual tasks. A Cluster Manager allocates resources and oversees the cluster. Together, these components form the foundation of Spark's runtime.

The Driver

The Driver is the process that runs the user's main function. It creates a SparkContext or SparkSession, converts the user program into an execution plan, and builds a DAG. The Driver also splits the DAG into stages, creates tasks, schedules them, and monitors their status.

This central role means the Driver is critical for coordination. It knows the lineage of the data and decides how to break a long computation into smaller stages.

Executors

Executors are processes running on worker nodes. They execute tasks, store cached RDD partitions when needed, and send status updates to the Driver. An Executor can host multiple threads, allowing it to run more than one task at a time.

When an Executor finishes a task, it returns the result to the Driver. If an Executor fails, the Driver can request replacement resources and rerun unfinished tasks.

Cluster Manager and ApplicationMaster

The Cluster Manager handles cluster-wide resource management. Popular choices include Standalone, YARN, and Kubernetes. In YARN deployments, the ResourceManager allocates resources, NodeManagers manage node-level resources, and ApplicationMaster requests containers for the application.

In some setups, the ApplicationMaster works with the Cluster Manager to obtain resources and launch Executors. In YARN cluster mode, the Driver may run inside the ApplicationMaster.

Core Concepts That Drive Spark

To understand Spark, several abstractions matter.

  • RDD: The Resilient Distributed Dataset is Spark's earliest core abstraction. It is partitioned, immutable, supports parallel operations, and tracks lineage for fault tolerance.
  • DataFrame and Dataset: These are higher-level abstractions built on RDDs. They include schema information and can benefit from the Catalyst optimizer and Tungsten execution engine.
  • DAG: A directed acyclic graph shows how RDDs depend on one another. When an Action occurs, Spark generates a DAG from the RDD lineage.
  • Job: A Job is triggered by an Action, such as count, collect, or saveAsTextFile.
  • Stage: The DAGScheduler divides a DAG into stages based on wide dependencies. Stages include ShuffleMapStage and ResultStage.
  • Task: A Task is the smallest unit of work. The number of tasks in a stage usually equals the number of partitions.
  • Narrow and wide dependencies: A narrow dependency means a parent RDD partition is used by at most one child partition, common with map and filter. A wide dependency means multiple parent partitions may be used by multiple child partitions, common with reduceByKey and groupByKey. Wide dependencies often introduce shuffles.
  • Shuffle: A shuffle redistributes data by key, moving data from map-side tasks to reduce-side tasks. It can be expensive in terms of disk and network usage.

Execution Flow in Spark

A typical Spark application follows a clear sequence.

  1. A user submits the application with spark-submit, providing the main class, resource settings, and dependencies.
  2. The Cluster Manager allocates resources and starts the Driver.
  3. The Driver creates the SparkContext, initializes the DAGScheduler, TaskScheduler, and SchedulerBackend.
  4. The Driver requests Executor resources from the Cluster Manager.
  5. The Cluster Manager launches Executors on worker nodes.
  6. Executors register with the Driver and report available capacity.
  7. The user code reaches an Action, which triggers a Job.
  8. The DAGScheduler builds the DAG and splits it into stages.
  9. Each stage creates tasks, and the TaskScheduler assigns tasks to Executors.
  10. Executors run tasks. Shuffle-related work may involve writing map-side shuffle files and reading them on the reduce side.
  11. Task results return to the Driver. When all tasks finish successfully, the Job completes.
  12. After the application ends, the Driver releases resources and Executors shut down.

A WordCount Example

Consider a typical WordCount example. The application reads text, splits it into words, creates key-value pairs, and reduces them into counts.

The process begins with reading data and applying transformations. Operations such as flatMap and map are narrow dependencies and may not trigger immediate computation. A wide transformation such as reduceByKey can trigger a shuffle.

When the final Action appears, Spark submits a Job. The DAGScheduler divides the work into stages. The first stage may perform local aggregation and write shuffle data. The second stage may read that data and complete the final aggregation.

The number of Tasks depends on the number of partitions. If an input has ten blocks, the first stage might create ten Tasks.

Scheduling and Fault Tolerance

Spark relies on several mechanisms for scheduling and fault tolerance.

  • DAGScheduler: manages stage boundaries, dependencies, and recovery.
  • TaskScheduler: assigns tasks, considers data locality, and retries failed tasks.
  • Data locality: tasks run on or near the node holding the data to reduce network movement.
  • Lineage: RDDs keep dependency information, allowing recovery if partitions are lost.
  • Checkpoint: truncates long RDD lineages to reduce recovery cost.
  • Speculative execution: launches backup tasks for slow-running tasks and uses the first successful result.
  • Executor fault tolerance: if an Executor fails, the Driver can request new resources and reschedule pending tasks.

Conclusion

Spark's architecture combines a Driver, Executors, and a Cluster Manager to support distributed computation. Its abstractions, including RDDs, DataFrames, DAGs, Jobs, Stages, and Tasks, help organize complex workflows. The execution flow shows how user code becomes stages and tasks, while scheduling and fault tolerance keep the system resilient.

Understanding Spark's architecture and execution process is useful for performance tuning, troubleshooting, and resource planning.