Databricks Masterclass — 100-Hour Advanced Lakehouse & Data Engineering Program

• Driver, executors, jobs → stages → tasks; lazy evaluation and the Catalyst optimiser
• Adaptive Query Execution, dynamic partition pruning, whole-stage codegen, Photon vectorisation
• DataFrame API: select, filter, withColumn, joins, aggregations, UDFs vs pandas UDFs vs built-ins
• Spark SQL, temp views, catalogs; reading JSON, CSV, Parquet, Avro, XML, fixed-width
• Reading the Spark UI: skew, spill, shuffle read/write, GC time
• Lab: rewrite a slow 12-minute join to under 90 seconds

• Why Delta: ACID on object storage, the _delta_log, checkpoints and protocol versions
• MERGE, UPDATE, DELETE; SCD Type 1 and Type 2 patterns; deletion vectors
• Time travel, RESTORE, CLONE (shallow/deep), CDF (Change Data Feed)
• OPTIMIZE, Z-ORDER, liquid clustering, VACUUM, predictive optimisation, file sizing
• Schema enforcement vs schema evolution; constraints and generated columns
• Delta Sharing and open-format interop (Iceberg/UniForm)
• Lab: build a slowly-changing customer dimension with MERGE + CDF

• Auto Loader (cloudFiles): file notification vs directory listing, schema inference and hints, rescued data
• Structured Streaming: triggers (availableNow, processingTime), checkpoints, watermarks, output modes
• Stateful streaming: joins, dropDuplicatesWithinWatermark, flatMapGroupsWithState
• Kafka + Databricks: offsets, exactly-once semantics, Avro/Protobuf with Schema Registry
• CDC ingestion with Debezium and APPLY CHANGES INTO
• Lab: exactly-once Kafka → Delta pipeline with replay and back-pressure handling

• DLT concepts: streaming tables, materialised views, expectations, pipeline modes
• Development vs production, continuous vs triggered, Enhanced Autoscaling
• Data-quality expectations: expect, expect_or_drop, expect_or_fail; quarantine tables
• APPLY CHANGES INTO for CDC and SCD2 without hand-written MERGE
• Lakeflow Declarative Pipelines: what changed and how to migrate
• Lab: convert a hand-rolled notebook pipeline to DLT and add quality gates

• Metastore, catalog, schema, table, volume, model; three-level namespace
• Managed vs external tables, external locations and storage credentials
• Grants, dynamic views, row filters and column masks for PII
• Lineage, audit logs, system tables, tags and Data Classification
• Delta Sharing to partners; catalog binding and workspace isolation
• Lab: implement GDPR-safe access for a customer table across three roles

• SQL warehouses: classic vs pro vs serverless, scaling, query queuing, result cache
• Query profile analysis, materialised views, caching strategies
• Dashboards, alerts, parameters; Power BI / Tableau / Looker connectivity
• Modelling the gold layer: star schema, surrogate keys, aggregate tables
• Lab: sub-second executive dashboard on a 500M-row fact table

• Databricks Workflows: tasks, dependencies, job clusters, retries, repair-run, job parameters
• Apache Airflow with the Databricks provider; when to use Airflow vs Workflows
• Databricks Asset Bundles (DABs): bundle.yml, targets, dev/stage/prod promotion
• CI/CD with GitHub Actions / Azure DevOps: unit tests with pytest + chispa, integration tests
• Terraform provider for workspaces, clusters, policies and permissions
• Lab: one-command deploy of a pipeline from dev to prod via a bundle

• Diagnosing skew: salting, AQE skew join, broadcast thresholds
• Partitioning vs liquid clustering vs Z-ORDER — what to use when
• Caching, disk cache, Photon eligibility, serverless compute trade-offs
• Cluster policies, spot/pre-emptible fleets, autotermination, instance pools
• System tables for DBU chargeback; tagging strategy and budget alerts
• Lab: reduce a nightly job from 55 minutes / $180 to 14 minutes / $38

• Medallion architecture in practice; data mesh and domain ownership on Unity Catalog
• Migration playbooks: Hadoop/Hive, Teradata, Informatica and Synapse → Databricks
• MLflow 3, Feature Store, Model Serving and Vector Search — what a data engineer must know
• Databricks Apps, Genie and AI/BI: serving analytics to business users
• Capstone build, code review, architecture defence and demo day
• Certification drill: 120 practice questions for Associate + Professional

No. Modules 1–2 build Python and Spark fundamentals from scratch. If you already know Spark you can skip ahead — recordings are released from day one.

You can follow along on AWS, Azure or GCP. Labs are written to be cloud-neutral, with cloud-specific notes for storage and identity. Databricks Community Edition plus free cloud tiers cover most exercises.

Yes. The course maps to the Databricks Certified Data Engineer Associate and Professional exam guides, and includes 120 practice questions plus two timed mock exams. The exam fee itself is paid directly to Databricks.

Every session is recorded and posted within a few hours. You also get lifetime access to recordings and can rejoin any future batch at no cost.

Yes — resume review, LinkedIn optimisation, a mock interview and an interview question bank are included. We do not guarantee placement, and we will never sit in an interview on your behalf.