Databricks Interview Questions and Answers

Databricks interview questions cover the platform, such as clusters, jobs and Unity Catalog, but most of the technical depth is in Delta Lake, the table format underneath almost everything on Databricks. Interviewers ask what a write really does, how to fix bad data, and why a table slows down over months. The answers below come from operating a real Delta table on daily bike-share data, using the open-source delta-rs library, which implements the same Delta protocol outside Databricks.

The data is October 2012 from the UCI Bike Sharing set: hourly rentals in Washington, D.C. (UCI: Bike Sharing), loaded one day at a time, the way a daily job would.

import os
import shutil

import pandas as pd
import pyarrow as pa
from deltalake import DeltaTable, write_deltalake

PATH = "bike_hours"                                             # a Delta table on local disk
shutil.rmtree(PATH, ignore_errors=True)
hours = pd.read_csv("hour.csv")                                 # UCI Bike Sharing, hourly rentals 2011-2012
hours = hours[["dteday", "hr", "weathersit", "cnt"]].rename(columns={"dteday": "day"})
october = hours[hours.day.between("2012-10-01", "2012-10-31")]

print("D1 every write is a new version in the transaction log")
for day, rows in october.groupby("day"):                        # one small append per day, like a daily job
    write_deltalake(PATH, rows, mode="append")
dt = DeltaTable(PATH)
log = sorted(f for f in os.listdir(os.path.join(PATH, "_delta_log")) if f.endswith(".json"))
print(f"   version {dt.version()}, {len(log)} commit files in _delta_log, {len(dt.file_uris())} data files, {len(dt.to_pandas()):,} rows")

print("\nD2 time travel: read an older version")
print(f"   rows at version 9: {len(DeltaTable(PATH, version=9).to_pandas()):,} | now: {len(dt.to_pandas()):,}")

print("\nD3 MERGE: corrected counts arrive for 30 October, plus one missing hour (12:00)")
fix = october[october.day == "2012-10-30"].copy()
fix["cnt"] = fix["cnt"] + 1


def upsert(source):
    return (DeltaTable(PATH).merge(pa.Table.from_pandas(source, preserve_index=False),
                                   predicate="t.day = s.day AND t.hr = s.hr", source_alias="s", target_alias="t")
            .when_matched_update_all().when_not_matched_insert_all().execute())


try:                                                            # a source with two rows for the same key
    upsert(pd.concat([fix, fix.tail(1)]))
except Exception as error:
    print("   duplicate keys in the source ->", "rejected:", "multiple source rows" in str(error))
result = upsert(pd.concat([fix, pd.DataFrame([{"day": "2012-10-30", "hr": 12, "weathersit": 3, "cnt": 5}])]))
print(f"   updated {result['num_target_rows_updated']}, inserted {result['num_target_rows_inserted']}, version now {DeltaTable(PATH).version()}")

print("\nD4 schema enforcement: an append with an extra column")
extra = october.head(2).assign(temperature=0.5)
try:
    write_deltalake(PATH, extra, mode="append")
except Exception as error:
    print("   rejected:", type(error).__name__)
write_deltalake(PATH, extra, mode="append", schema_mode="merge")
print("   with schema_mode='merge': columns are now", DeltaTable(PATH).schema().to_arrow().names)

print("\nD5 compaction of small files")
dt = DeltaTable(PATH)
before = len(dt.file_uris())
metrics = dt.optimize.compact()
print(f"   files before {before}, after {len(DeltaTable(PATH).file_uris())} (files added {metrics['numFilesAdded']}, removed {metrics['numFilesRemoved']})")

print("\nD6 VACUUM: old files stay until the retention period passes")
dt = DeltaTable(PATH)
print(f"   files VACUUM would delete with the default retention: {len(dt.vacuum(dry_run=True))}")
print(f"   with retention 0 hours: {len(dt.vacuum(retention_hours=0, enforce_retention_duration=False, dry_run=True))}")
shutil.rmtree(PATH)

D1 every write is a new version in the transaction log
   version 30, 31 commit files in _delta_log, 31 data files, 708 rows

D2 time travel: read an older version
   rows at version 9: 240 | now: 708

D3 MERGE: corrected counts arrive for 30 October, plus one missing hour (12:00)
   duplicate keys in the source -> rejected: True
   updated 11, inserted 1, version now 31

D4 schema enforcement: an append with an extra column
   rejected: SchemaMismatchError
   with schema_mode='merge': columns are now ['day', 'hr', 'weathersit', 'cnt', 'temperature']

D5 compaction of small files
   files before 32, after 1 (files added 1, removed 32)

D6 VACUUM: old files stay until the retention period passes
   files VACUUM would delete with the default retention: 0
   with retention 0 hours: 33

D1. What happens when you write to a Delta table?

Each write adds Parquet data files and a new commit file to the _delta_log folder; the log, not the folder listing, defines which files make up the table. Thirty-one daily appends produced version 30, 31 commit files and 31 data files for 708 hourly rows. That log is what gives Delta atomic commits: a reader sees a write completely or not at all. It also explains the small-files problem in D5, since each daily append added another small file.

D2. What is time travel?

Reading the table as of an earlier version or timestamp. Version 9 held 240 rows, the first ten days, while the current version held 708 (Delta Lake: batch reads). Interview uses: reproducing a report as it was, auditing a change, and rolling back a bad load. Time travel only reaches as far back as the retention settings and VACUUM allow.

D3. How do you upsert with MERGE, and what can make it fail?

MERGE matches source rows to target rows on a condition, updates the matches and inserts the rest. Corrected counts for 30 October updated 11 rows, and a missing 12:00 hour was inserted. But when the source contained the same key twice, the merge was rejected: Delta refuses because it cannot know which source row should win (Delta Lake: merge). Deduplicate the source on the merge key before merging. Interviewers like this question because the failure appears only on real, messy data, usually when a source system resends records or a join upstream has duplicated them.

D4. What is schema enforcement?

A write whose columns do not match the table is rejected. Appending rows with an extra temperature column raised a SchemaMismatchError. Allowing evolution explicitly, with schema merging enabled on the write, added the column to the table. Enforcement by default protects downstream readers; evolution is a deliberate choice, made per write.

D5. Why do Delta tables slow down, and what does OPTIMIZE do?

Many small files: each needs listing, opening and a task. After the daily appends, the merge and the extra append, the table had 32 files; compaction rewrote them into 1. On Databricks, OPTIMIZE performs this bin-packing and is idempotent, so a second run changes nothing (Databricks: OPTIMIZE). Z-ordering or liquid clustering additionally co-locates rows by chosen columns for better skipping.

D6. What does VACUUM do, and why is it careful?

It deletes files the table no longer references, but only once they are older than the retention threshold, 7 days by default, and it does not run automatically (Delta Lake: utility commands). Right after compaction, a dry run with the default retention would delete nothing; with retention forced to zero hours, all 33 superseded files would go. Removing them immediately frees storage, but also ends time travel to those versions and can break readers still using them, which is why the default protects them.

Platform questions to expect

• Job clusters or all-purpose clusters? Job clusters start for a job and stop after it, which suits scheduled production work; all-purpose clusters are for interactive development.
• What is Unity Catalog? Databricks' governance layer for data and AI assets: one place for permissions, lineage and discovery across workspaces.
• What is Auto Loader? Incremental ingestion of new files from cloud storage, tracking which files have already been processed.
• What is the medallion architecture? Layers of tables by quality: bronze for raw ingested data, silver for cleaned and conformed data, gold for business-level aggregates. Each layer is a Delta table, so every step can be rerun, audited and time-travelled.
• Batch or streaming on Databricks? The same Delta tables serve both; Structured Streaming can read a Delta table as a stream of new commits, which is how many pipelines move from bronze to silver incrementally.

How to answer Databricks questions

• Explain operations through the transaction log.
• Deduplicate before MERGE, and say why.
• Plan maintenance: compaction, clustering and VACUUM with a retention that matches your time-travel needs.

Delta concepts are explained in Delta Lake explained and compared in Iceberg vs Delta Lake vs Hudi. The platform is introduced in the Databricks tutorial, and Spark questions are in Apache Spark interview questions.

Databricks and Delta Lake are part of the Data Engineering course.

Sources

• Table batch reads and writes (Delta Lake documentation) — Delta Lake project
• Table deletes, updates and merges (Delta Lake documentation) — Delta Lake project
• Table utility commands (Delta Lake documentation) — Delta Lake project
• Optimize data file layout (Databricks documentation) — Databricks
• Bike Sharing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository