Start here in Data Engineering
The most useful guides in this topic, not just the most recent.
Data Engineering Guides & Tutorials
Build the data backbone — ETL/ELT, warehouses, lakes, Spark, Kafka, Airflow, Databricks and the medallion architecture.
Data Engineering articles
Schema Evolution: Handling Changing Data Structures
How to handle changing data structures, using a real public dataset whose schema changed three ways in three years: detecting an added column, a renamed column and five type changes, why a strict merge refuses and a permissive one keeps both spellings, reading by name, compatibility rules, and safe practices.
Slowly Changing Dimensions (SCD) Types Explained
The slowly changing dimension types every data engineer uses: Type 0, 1, 2 and 3 and when each fits, a working example on the real Chinook database where a support-rep reassignment makes Type 1 misattribute revenue while Type 2 keeps it right, how to load Type 2 rows, and the mistakes that corrupt history.
Snowflake Explained: The Cloud Data Warehouse
Snowflake explained for data engineers: the three layers of storage, compute and cloud services, why separate virtual warehouses isolate workloads, how credits and per-second billing work with a worked example, micro-partitions and pruning, Time Travel and cloning, and habits that keep performance up and costs down.
Window Functions in SQL for Data Engineers
The window function patterns data engineers use every week, each run on the real Chinook database: ROW_NUMBER for the latest row per key, RANK and DENSE_RANK on ties, LAG for gaps between events, frames for running totals and moving averages, and the default RANGE frame that silently changes a running total.
Apache Kafka Explained: Streaming Data Basics
Apache Kafka explained from the ground up: events, topics and partitions, why keys decide ordering, how consumer groups track offsets and why data is not deleted when read, each shown by producing 1,000 real taxi events to a running broker and reading them back, plus replication and delivery guarantees.
Batch vs Streaming Data Processing: A Clear Comparison
Batch and streaming data processing compared honestly: what each is, a measurement on a real day of 77,033 taxi trips showing what a nightly, hourly, five-minute and per-event pipeline would each know at 12:30, where micro-batching fits, what streaming costs in complexity, and how to choose.