PySpark Masterclass — 100 Hours of Apache Spark Internals, Tuning and Streaming

• Cluster managers: standalone, YARN, Kubernetes, Databricks
• Driver/executor memory model, cores, slots, dynamic allocation
• RDD lineage, DAG, jobs/stages/tasks, narrow vs wide transformations
• Catalyst optimiser, Tungsten, whole-stage codegen, AQE
• Lab: run the same job on 1, 4 and 16 cores and explain the curve

• Schemas: StructType, nested structs, arrays, maps, explode/flatten
• All join types and strategies: broadcast hash, sort-merge, shuffle hash, bucketed
• Window functions, ranking, running totals, sessionisation
• UDFs vs pandas UDFs vs Spark built-ins — measured performance comparison
• Handling nulls, dedup, data-quality checks, quarantine patterns
• Lab: flatten a deeply nested JSON API payload into a star schema

• Parquet internals: row groups, column chunks, statistics, predicate pushdown
• Avro, ORC, JSON, Delta, Iceberg — when each wins
• Partitioning vs bucketing; the small-file problem and how to fix it
• Compression codecs: snappy vs zstd vs gzip benchmarks
• Lab: cut a table's scan time 6x by re-laying-out the files

• Micro-batch vs continuous; sources and sinks; foreachBatch
• Checkpointing, offsets, idempotent writes, exactly-once semantics
• Event time, watermarks, windowing (tumbling, sliding, session)
• Stream-stream and stream-static joins; state store internals and RocksDB
• Kafka integration: consumer groups, maxOffsetsPerTrigger, back-pressure
• Lab: streaming sessionisation with late-arriving events

• Reading the Spark UI end to end: stages, tasks, SQL tab, executor tab
• Diagnosing skew and fixing it: salting, AQE, iterative broadcast
• Spill, GC pressure, memory fractions, off-heap, serialization (Kryo)
• Shuffle partitions: the maths behind spark.sql.shuffle.partitions
• Caching and persistence levels — when caching makes things worse
• Cost-based optimisation, statistics, ANALYZE TABLE
• Lab: three broken jobs, three root-cause reports, three fixes

• pytest fixtures for SparkSession; chispa DataFrame assertions
• Property-based testing, fixtures from JSON, golden datasets
• Project layout, config with pydantic/YAML, dependency injection
• Linting and typing: ruff, mypy; pre-commit hooks
• Lab: retrofit tests onto an untested legacy pipeline

• spark-submit anatomy: deploy modes, --packages, --conf, --files
• Running on AWS EMR (EMR Serverless too), GCP Dataproc, Kubernetes (Spark Operator)
• Airflow DAGs for Spark: SparkSubmitOperator, EmrOperators, sensors, retries
• CI/CD: build, test, publish artefact, deploy, smoke test
• Lab: promote a job through dev → staging → prod with GitHub Actions

• Spark Connect and Spark 4.0 changes; PySpark on pandas API
• Arrow, vectorisation, Photon and native engines (Comet, Gluten, Velox)
• Graph processing with GraphFrames; ML pipelines with MLlib overview
• Iceberg and Hudi with Spark: table formats compared
• Common anti-patterns: collect(), toPandas(), row-by-row loops

• Capstone build and architecture review
• 80+ Spark interview questions with model answers, from junior to architect level
• Live coding drill: 12 timed PySpark problems
• Databricks Certified Associate Developer for Apache Spark exam walkthrough
• Resume writing for Spark roles and how to describe your projects

This course is about Spark itself — internals, tuning, streaming and testing — and runs on Databricks, EMR, Dataproc and local Docker. The Databricks course is about the platform: Unity Catalog, DLT, Workflows, SQL warehouses and governance. Many students take both; there is a bundle discount.

Spark 3.5 as the baseline with a dedicated session on Spark 4.0 changes (Spark Connect, ANSI mode by default, VARIANT type). Deprecated APIs are called out explicitly as we go.

Not for the first half — labs run in Docker locally. From Module 8 you'll use free-tier AWS/GCP credits or Databricks Community Edition; we walk through setup and cost guardrails.

Only for reading. The course is Python-first, but you'll learn to read Scala Spark code because much of the ecosystem's source and Stack Overflow answers are in Scala.