GCP Data Engineering Masterclass — 100 Hours across BigQuery, Dataflow, Dataproc and Composer

• Architecture: Dremel, Colossus, slots, shuffle; storage vs compute pricing
• Datasets, tables, views, external tables, BigLake, BigQuery Omni overview
• Partitioning (time/integer/ingestion) and clustering — with measured impact
• Loading data: batch load, Storage Write API, streaming inserts, transfers
• Standard SQL depth: window functions, arrays, structs, UNNEST, JSON, geography
• Lab: reduce a report's bytes scanned by 94%

• Query plan explanation, slot contention, reservations vs on-demand, editions
• Materialised views, BI Engine, caching, table snapshots and clones
• Scripting, stored procedures, UDFs, remote functions, BigQuery ML basics
• Time travel, fail-safe, data retention; row/column-level security and policy tags
• Cost guardrails: custom quotas, maximum bytes billed, INFORMATION_SCHEMA monitoring
• Lab: build a slot-usage and cost dashboard from INFORMATION_SCHEMA

• Beam model: PCollection, PTransform, ParDo, GroupByKey, Combine, side inputs
• Batch vs streaming; event time, watermarks, windows, triggers, accumulation modes
• Python and Java SDK basics; runners; Dataflow Prime and streaming engine
• Templates (classic and Flex), autoscaling, shuffle service, fusion and its pitfalls
• Error handling: dead-letter patterns, retries, update/drain of streaming jobs
• Lab: Project 2 streaming pipeline with late data and a DLQ

• Pub/Sub: topics, subscriptions (pull/push), ordering keys, exactly-once delivery, dead letters
• Pub/Sub Lite, BigQuery subscriptions, Cloud Storage subscriptions
• Kafka fundamentals: partitions, consumer groups, offsets, compaction, schema registry
• Managed Service for Apache Kafka on GCP; Confluent on GCP; when to prefer Kafka over Pub/Sub
• Datastream for CDC from MySQL/Postgres/Oracle
• Lab: CDC pipeline with schema-change handling

• Dataproc clusters vs Dataproc Serverless vs Dataproc on GKE
• Sizing, preemptible/spot workers, autoscaling policies, initialization actions
• Running PySpark jobs against GCS; connector tuning; Iceberg and Delta on GCS
• Dataproc Metastore, Hive/Spark SQL, migration from on-prem Hadoop
• Lab: run a Spark job on Dataproc Serverless and compare cost vs Dataflow

• Airflow core: DAGs, TaskFlow API, operators, sensors, XCom, pools, SLAs, backfills
• Composer 2/3: environment sizing, autoscaling, plugins, private IP
• GCP operators: BigQuery, Dataflow, Dataproc, GCS, Kubernetes
• Deferrable operators, dynamic task mapping, datasets/data-aware scheduling
• CI/CD for DAGs; testing DAGs with pytest
• Lab: orchestrate Projects 1 and 3 from one Composer environment

• Dimensional modelling in BigQuery; nested/repeated vs flat — the real trade-offs
• dbt on BigQuery: models, refs, tests, snapshots, incremental strategies, macros
• Looker Studio and Looker overview; semantic layer thinking
• Data contracts and freshness SLAs
• Lab: dbt project with tests and docs over the gold layer

• Dataplex: lakes, zones, data quality tasks, automatic discovery, catalog
• IAM deep dive for data: dataset ACLs, authorised views, policy tags, DLP
• Encryption, CMEK, VPC-SC perimeters for exfiltration control
• Monitoring, logging, error reporting; SLOs for pipelines
• Lab: implement PII masking and prove it with an access test

• Vertex AI for data engineers: feature store, pipelines, model endpoints
• BigQuery ML and Gemini in BigQuery: SQL-native ML and AI functions
• Vector search in BigQuery for RAG use cases
• Capstone presentation and architecture review
• Professional Data Engineer exam guide, 120 practice questions, two mocks
• GCP interview questions and resume framing

Neither blindly. You build the same pipeline both ways in Modules 4 and 6 and produce a cost/latency comparison, so you can defend the choice in an interview or design review.

The $300 free credit covers the entire course comfortably if you follow the teardown checklist at the end of each lab. BigQuery's 1 TB/month free query tier covers most SQL work.

No. We teach Beam in Python. Java examples are shown for reading, because some Dataflow templates and older documentation are Java-only.

It is case-study heavy and tests judgement more than syntax. The course includes two full timed mocks and a decision-framework session specifically for the scenario questions.