Free guide · No signup required
How to become a data engineer in 2026: the honest roadmap
Eight stages, roughly 8–12 months part-time, written by someone who has trained 1,200+ people into these roles. It tells you what to skip as well as what to learn — including where our own courses are not the answer.
Written by Venu Katragadda · 14+ years in data engineering at L&T Infotech, Capgemini, Brillio and Manthan Systems.
Before you start: what the job actually is
A data engineer builds and runs the systems that move data from where it is created to where it is used, reliably and at a cost the business will tolerate. In practice that means ingestion, transformation, storage layout, orchestration, quality checks, monitoring and cost control.
It is a software engineering job with a data specialisation — not an analyst job with extra tools. That single realisation is what separates candidates who get offers from candidates who keep getting rejected in round three. Interviewers are checking whether you can be trusted with a system that runs unattended, not whether you can recite a service list.
The roadmap below is ordered deliberately. Each stage assumes the previous one. Skipping ahead to the cloud services before your SQL and Spark are solid is the single most common reason people stall for a year.
The eight stages
Get SQL genuinely strong — not 'I know SELECT'
3–4 weeks
Every data engineering interview starts here and most candidates are weaker than they think. You need window functions, CTEs, correlated subqueries, grouping sets, and the ability to reason about a query plan. If you cannot write a running-total-by-customer query without looking it up, you are not ready to move on.
- Window functions: ROW_NUMBER, RANK, LAG/LEAD, running aggregates
- CTEs and recursive CTEs; when a CTE hurts performance
- Joins: inner/outer/semi/anti, and what a cartesian blow-up looks like
- Reading an EXPLAIN plan and spotting a full scan
- Practice: 60+ problems on StrataScratch, DataLemur or LeetCode SQL
Python for data engineering, not for data science
3–4 weeks
You are not building models. You are building software that runs unattended at 2 a.m. That means functions, error handling, logging, config, packaging and tests — the parts most self-taught candidates skip and every interviewer probes.
- Core: collections, comprehensions, generators, decorators, typing
- Files and formats: JSON, CSV, Parquet; pandas basics and its limits
- Packaging, virtualenv/uv, dependency pinning
- Logging, retries, idempotency, exception design
- pytest — writing a test for a transformation, not just asserting True
Learn how distributed processing actually works
6–8 weeks
This is the step that separates a data engineer from an ETL operator. Spark is the standard, and the interview questions are about internals: partitions, shuffles, skew, spill, joins. You can learn the DataFrame API in a weekend; understanding why a job is slow takes months of real data.
- Spark architecture: driver, executors, jobs → stages → tasks
- Narrow vs wide transformations; why shuffles are expensive
- Join strategies: broadcast hash, sort-merge, bucketed
- Skew, spill and the Spark UI — diagnosing without guessing
- File formats and layout: Parquet internals, partitioning, the small-file problem
- Practice on a dataset big enough to hurt — 100M+ rows, not a 5,000-row CSV
Pick one cloud and go deep before you go wide
8–10 weeks
Job descriptions say 'AWS or Azure or GCP', but interviews test depth in one. Pick the cloud your target employers use — in India that is most often Azure or AWS — and learn its storage, compute, warehouse and orchestration services properly. The second cloud takes a quarter of the time.
- Storage and identity: S3/ADLS/GCS, IAM roles, encryption, private networking
- Batch compute: Glue / Databricks / Dataflow / Dataproc
- Warehouse: Redshift / Synapse / Fabric / BigQuery — and how to model for it
- Streaming: Kinesis or MSK / Event Hubs / Pub/Sub
- Cost: this is what makes you senior. Know what your pipeline costs per run
Learn the Lakehouse — Databricks and Delta Lake
6–8 weeks
Databricks appears in a large and growing share of Indian data engineering job posts, and Delta Lake concepts (ACID on object storage, MERGE, time travel, schema evolution) now show up in interviews even at companies that do not use Databricks.
- Delta Lake: the transaction log, MERGE, time travel, OPTIMIZE, Z-ORDER, liquid clustering
- Medallion architecture — bronze, silver, gold, and why the layers exist
- Auto Loader and Structured Streaming for incremental ingestion
- Unity Catalog: governance, lineage, row and column security
- Delta Live Tables / declarative pipelines and Workflows
- Target the Databricks Certified Data Engineer Associate exam as a forcing function
Orchestration, CDC and streaming
4–5 weeks
Nobody runs pipelines by hand. Airflow is still the default orchestrator in most Indian shops, and Kafka knowledge separates candidates for the better-paying real-time roles.
- Airflow: DAGs, TaskFlow, sensors, XComs, backfills, retries, SLAs, idempotent tasks
- Kafka fundamentals: topics, partitions, consumer groups, offsets, compaction
- Exactly-once semantics — and why at-least-once plus idempotency is usually the real answer
- CDC with Debezium / DMS / Datastream; handling schema drift
- Data quality gates: Great Expectations, dbt tests, DLT expectations
Build three portfolio projects that survive questioning
4–6 weeks
A GitHub repo with a notebook that reads a CSV impresses nobody. Build things with the messy parts left in: late-arriving data, a schema change, a skewed key, a cost problem you fixed. Then write the README as if a hiring manager is the reader.
- Batch: multi-source ingestion → medallion lakehouse → BI layer, orchestrated by Airflow
- Streaming: Kafka → processing → sink, with replay, watermarks and a dead-letter path
- Optimisation: take a slow, expensive job and document the before/after with numbers
- For each: an architecture diagram, the trade-offs you rejected, and the cost per run
- Deploy it. A project that has never run on a schedule is a demo, not a project
Prepare the interview itself
2–3 weeks, ongoing
Data engineering interviews in India are typically four rounds: SQL, Python/Spark coding, a system/pipeline design round, and a project deep-dive. Most rejections happen in the design round and the project deep-dive, not the coding round.
- SQL round: 2–3 hard queries, live, with someone watching
- Coding round: PySpark transformations, sometimes plain Python data structures
- Design round: 'design a pipeline for X' — practise saying the trade-offs out loud
- Project deep-dive: expect 'why did you choose that?' five levels deep
- Behavioural: a failure story with a real root cause and what you changed afterwards
- Rewrite your resume around outcomes and numbers, not tool lists
Six mistakes that cost people a year
Collecting certificates instead of building things
A certification proves you studied. A project that survives a deep-dive proves you can do the job. Get one certification as a forcing function, then spend the rest of your time building.
Learning three clouds at 20% depth
Interviewers test depth in the cloud on their job description. Breadth is a second-year problem.
Practising on toy data
Skew, spill, small files and shuffle pain do not exist at 10,000 rows. Everything you would actually be hired to fix only appears at scale.
Ignoring cost
In every senior interview someone asks what your pipeline costs. Candidates who can answer with a number stand out immediately.
Skipping software engineering fundamentals
Version control, tests, code review, CI/CD. Data teams are being held to the same standard as application teams now, and the gap shows fast.
Applying before the portfolio exists
Applying with nothing to show burns the referral network you will want in three months.
Where our courses fit — and where they do not
You can do all eight stages above for free. The material exists: vendor documentation, the Spark source, public datasets, YouTube. What you cannot get for free is someone experienced telling you your design is wrong before you spend three weeks building it, and reading your code afterwards.
That is what our programmes sell. If you are disciplined and have time, self-study works — plenty of good engineers got there that way. If you have a job, a family and eight hours a week, a structured live cohort with code review usually gets you there faster.
Stages 1–5 · Foundation
databrickstraining.in
70–80 hour courses from ₹13,000. Best if you are early in the roadmap or changing careers.
Stages 4–8 · Masterclass
This site
100-hour advanced programmes from ₹24,000, with internals, tuning, cost engineering and three code-reviewed portfolio projects.
Next: 38 data engineer interview questions with answers →
Roadmap FAQs
Realistically 8–12 months of consistent part-time study if you already work in IT with SQL exposure, and 15–24 months if you are starting from outside the industry. Anyone promising 3 months is selling something.
No. Hiring managers care about whether you can write correct SQL and Python, reason about distributed systems, and show work you have actually built. Degrees help get past some HR filters, particularly at large service companies, but they are not the constraint.
Look at the job posts you actually want. Across Indian data engineering roles, Azure and AWS dominate, with GCP concentrated in product companies and analytics-heavy firms. Learn one to depth; the second takes about a quarter of the time.
Yes. Spark teaches you the engine; Databricks adds Delta Lake, Unity Catalog, DLT and Workflows, and it appears by name in a large share of Indian data engineering job descriptions. Spark knowledge transfers everywhere, so learn it first, then the platform.
It is the fastest-appreciating skill on the list right now, because RAG systems are data pipelines and teams are staffing them with data engineers. But do it after your core data engineering is solid — GenAI on a weak foundation does not survive an interview.