Duration
8.0 hr(s) · 1 Day Intensive
Course Overview
Data engineering looks familiar from the outside - it's code, storage and pipelines - right up until a backfill silently doubles a table, or a schema change three systems upstream breaks a dashboard nobody knew depended on it. This course is about the parts that don't transfer from application development.
In a single hands-on day you'll build a working data lake on Databricks with Delta Lake, moving data through bronze, silver and gold layers. The discipline you already have - version control, testing, modular code - is an advantage here, and you'll apply it directly to pipelines. No prior data engineering experience is assumed.
Learning Objectives
1
Lakes & Warehouses
Explain why data lakes exist, where warehouses still win, and what keeps a lake from becoming a swamp.
2
Delta Lake Basics
Build Delta tables with ACID writes and schema enforcement, and recover from bad writes with time travel.
3
Medallion Pipelines
Move data through bronze, silver and gold layers with idempotent, re-runnable transformations.
4
Production Discipline
Apply version control, configuration and testing to pipelines, and recognise the failure modes that bite first.
Course Outline
1
From Applications to Data Lakes
- Compare warehouse and lake approaches, and identify which problem each one solves
- Apply object storage mechanics - file layout, consistency and cost - and see why raw files alone fail
- Introduce the medallion architecture and define what belongs in bronze, silver and gold
- Set up the Databricks workspace and run a first job end to end
2
Delta Lake Fundamentals
- Build Delta tables and observe what ACID writes actually guarantee
- Apply schema enforcement and schema evolution against a changing source
- Use time travel and rollback to recover from a bad write
3
Building the Pipeline: Bronze to Silver
- Ingest batch and incremental data into bronze while preserving raw fidelity
- Handle late-arriving and duplicate records without corrupting downstream layers
- Write idempotent, re-runnable transformations and execute a controlled backfill
- Structure the pipeline as a versioned, tested repository rather than loose scripts
4
Gold and Going to Production
- Model gold tables around the questions consumers actually ask
- Add data quality and freshness checks that fail loudly before consumers see bad data
- Recognise the small-file problem and apply basic compaction
- Avoid the failure modes that catch newcomers - over-partitioning, unbounded retention, unmonitored jobs
Certificate Obtained and Conferred by
Certificate of Completion from Zenika
Upon meeting at least 75% attendance and passing the assessment(s), participants will receive a Certificate of Completion from Zenika.
Your Trainer
Michael Isvy
Head of Engineering
Michael is Head of Engineering with broad experience leading teams that build large-scale software systems and modern engineering platforms. He is especially interested in how data teams keep pipelines maintainable as they grow - applying the same structure, testing and review discipline to data code that software teams take for granted.
Areas of focus
- Shares practical perspectives on modern data platform engineering at scale
- Interested in data engineering, code quality and architectural integrity
- Helps teams apply modern engineering practices to real delivery environments
What to Prepare
A short technical setup is required before the session. Detailed instructions will be sent to confirmed participants in advance - no preparation is needed at the point of registration. The hands-on labs run in a provided Databricks and Delta Lake environment.
Fees
| Participant | Standard Fee | Special Rate |
|---|---|---|
| Per participant (per pax) | $1,250 | $1,000 |
Team & private cohorts. The special rate applies to confirmed registrations. Group bookings and private in-house cohorts can be arranged — get in touch to discuss dates, numbers, and on-site delivery.