Databricks Migration Workshop: From Stored Procedures to Lakehouse (5-Day Intensive) Training Course

Introduction, Learning Objectives, and Migration Strategy

• Course goals, alignment with participant profiles, and definitions of success criteria
• Overview of high-level migration approaches and associated risk factors
• Configuration of workspaces, repositories, and lab datasets

Day 1 — Migration Fundamentals and Architecture

• Lakehouse principles, Delta Lake introduction, and Databricks structural components
• Distinguishing between SMP and MPP architectures and their impact on migration
• Medallion (Bronze, Silver, Gold) design patterns and an introduction to Unity Catalog

Day 1 Lab — Converting Stored Procedures

• Practical exercise to migrate a sample stored procedure into a notebook format
• Converting temporary tables and cursors into DataFrame transformations
• Verifying accuracy by comparing results with the original output

Day 2 — Advanced Delta Lake and Incremental Loading

• Understanding ACID transactions, commit logs, versioning, and time travel features
• Utilizing Auto Loader, MERGE INTO patterns, upserts, and handling schema evolution
• Optimization techniques including OPTIMIZE, VACUUM, Z-ORDER, partitioning, and storage tuning

Day 2 Lab — Incremental Ingestion and Optimization

• Deploying Auto Loader for ingestion and establishing MERGE workflows
• Applying OPTIMIZE, Z-ORDER, and VACUUM commands while verifying outcomes
• Evaluating enhancements in read and write performance

Day 3 — SQL in Databricks, Performance Analysis, and Debugging

• Exploring analytical SQL capabilities: window functions, higher-order functions, and JSON/array manipulation
• Interpreting Spark UI to analyze DAGs, shuffles, stages, tasks, and identify bottlenecks
• Query optimization techniques such as broadcast joins, hints, caching, and reducing spill

Day 3 Lab — SQL Refactoring and Performance Tuning

• Transforming resource-intensive SQL processes into optimized Spark SQL queries
• Leveraging Spark UI traces to detect and resolve data skew and shuffle issues
• Conducting before-and-after benchmarks and documenting tuning procedures

Day 4 — Tactical PySpark: Replacing Procedural Logic

• Spark execution mechanics: driver, executors, lazy evaluation, and partitioning methods
• Converting loops and cursors into efficient vectorized DataFrame operations
• Implementing modularization, UDFs/pandas UDFs, widgets, and reusable libraries

Day 4 Lab — Refactoring Procedural Scripts

• Converting a procedural ETL script into structured PySpark notebooks
• Incorporating parametrization, unit-style testing, and reusable functional blocks
• Applying code reviews and best-practice checklists

Day 5 — Orchestration, End-to-End Pipeline, and Best Practices

• Databricks Workflows: job architecture, task dependencies, triggers, and error management
• Creating incremental Medallion pipelines with quality rules and schema validation
• Integrating with Git (GitHub/Azure DevOps), CI processes, and testing strategies for PySpark logic

Day 5 Lab — Constructing a Complete End-to-End Pipeline

• Building a Bronze, Silver, Gold pipeline managed via Workflows
• Implementing logging, auditing, retry mechanisms, and automated checks
• Executing the full pipeline, verifying outputs, and drafting deployment documentation

Operationalization, Governance, and Production Readiness

• Best practices for Unity Catalog governance, lineage tracking, and access control
• Managing costs, cluster sizing, autoscaling, and job concurrency patterns
• Creating deployment checklists, rollback plans, and operational runbooks

Final Review, Knowledge Transfer, and Next Steps

• Participant presentations on migration efforts and key takeaways
• Gap analysis, suggested follow-up actions, and handover of training resources
• Curated references, advanced learning paths, and support options