Apache Spark and PySpark

Apache Spark is “a unified analytics engine for large-scale data processing” (GitHub description). It is an Apache Software Foundation project under the Apache License 2.0, with over 2,000 contributors (Spark site). PySpark is the Python API for Spark.

Repository snapshot (GitHub API, 2026-11-19)

ItemDetail
Repoapache/spark, Apache-2.0
Created2014-02-25
Activitylast push 2026-11-19
Size44,583 stars, 29,598 forks
VersionThe PySpark docs cover version 4.3.0 (page dated 2026-10-27). GitHub’s latest-release lookup returned 404, so no Spark release date is confirmed.

Components (spark.apache.org and PySpark docs)

  • Spark SQL and DataFrames, with Adaptive Query Execution and ANSI SQL support.
  • Pandas API on Spark (see pandas) and MLlib for machine learning.
  • Structured Streaming, a fault-tolerant stream engine on the SQL engine; the older Spark Streaming is “no longer being updated” and GraphX is marked deprecated.
  • Spark Connect, Spark Declarative Pipelines and Spark Core with RDDs (a low-level API the docs call difficult to use).
  • Install for Python: pip install pyspark, or the spark:python3 Docker image.

Where it fits

Spark is the engine under Databricks, and the target for Spark-scale quality tools such as deequ and databricks-dqx. delta-lake and apache-iceberg are common table formats on it; apache-airflow and dagster schedule its jobs.

Sources