Databricks Tutorial: A Beginner's Guide

Databricks is where many teams run Spark in production. Its documentation describes it as a unified, open analytics platform for data, analytics and AI, built on the lakehouse idea, Delta Lake and Apache Spark, with Unity Catalog for governance (Databricks docs: introduction). For a beginner, it removes most of the work of running Spark: no clusters to install, notebooks in the browser, and tables that are governed and shareable. This tutorial walks through a first session, from account to scheduled job.

What Databricks adds to Spark

Open source Spark gives you an engine. Databricks wraps it in a platform:

• Managed compute: serverless or configured clusters that start on demand.
• Notebooks for Python, SQL, Scala and R, with results, charts and collaboration.
• Delta Lake tables by default, with the transaction log, merges and time travel described in Delta Lake explained.
• Unity Catalog for permissions, lineage and discovery across tables, files and models.
• Lakeflow Jobs for scheduling and orchestrating work.

Step 1: get a workspace

The simplest start is Databricks Free Edition, a no-cost version for students, educators, hobbyists and learners that replaced the old Community Edition, retired in 2025 (Databricks docs: Free Edition). Sign up on the Free Edition page and Databricks provisions a workspace. Know its limits before building on it (Databricks docs: Free Edition limitations):

• Not for commercial use.
• Serverless compute only, with one small SQL warehouse and no custom clusters.
• At most five concurrent job tasks, one workspace and one metastore.
• No R or Scala, no SLA, and inactive accounts may be deleted.

For company work, your organisation's own workspace on AWS, Azure or Google Cloud is the right place.

Step 2: a notebook and the sample data

Create a notebook from the workspace sidebar and attach it to serverless compute. Each notebook has a default language, and any cell can switch with a magic command: %python, %sql, %scala or %r, plus %md for text (Databricks docs: notebooks). Variables do not cross languages, so a DataFrame defined in Python is not visible to a SQL cell unless you save it as a table or view.

Databricks ships sample data in a samples catalog, including samples.nyctaxi.trips, New York City taxi trip records (Databricks docs: sample datasets). In a Python cell, a SparkSession named spark is ready to use:

trips = spark.read.table("samples.nyctaxi.trips")
trips.printSchema()
display(trips.limit(10))

display renders an interactive table with built-in charting, which is the everyday way to look at results in Databricks. Check the schema before writing queries; sample tables can differ from the raw public files.

Step 3: query with SQL

A SQL cell queries the same table. This summarises trips per day:

%sql
SELECT to_date(tpep_pickup_datetime) AS pickup_date,
       COUNT(*)                      AS trips,
       ROUND(AVG(fare_amount), 2)    AS avg_fare
FROM samples.nyctaxi.trips
GROUP BY 1
ORDER BY 1

Use the chart option on the result to plot trips by day. Mixing SQL for exploration and Python for logic is normal in Databricks; both run on the same Spark engine, as the PySpark tutorial explains.

Step 4: save a governed table

Results worth keeping become tables in Unity Catalog, which organises data in a three-level namespace, catalog, schema and table, and controls who can use them (Databricks docs: Unity Catalog):

from pyspark.sql import functions as F

daily = (trips.groupBy(F.to_date("tpep_pickup_datetime").alias("pickup_date"))
              .agg(F.count("*").alias("trips"), F.round(F.avg("fare_amount"), 2).alias("avg_fare")))
spark.sql("CREATE SCHEMA IF NOT EXISTS my_catalog.taxi")
daily.write.mode("overwrite").saveAsTable("my_catalog.taxi.daily_trips")

Replace my_catalog with a catalog you can write to; Catalog Explorer in the sidebar lists yours. The table is stored in Delta format, so DESCRIBE HISTORY my_catalog.taxi.daily_trips in a SQL cell shows its versions, and other users can query it once you grant them access.

Step 5: schedule it

A notebook that builds a table is a pipeline once it runs on its own. Lakeflow Jobs orchestrates notebooks, pipelines, Python scripts and other tasks, triggered on a schedule, such as daily at 2 AM, or by events such as new files arriving in cloud storage, with notifications by email, Slack or webhooks (Databricks docs: Lakeflow Jobs). From the notebook, create a job, add the notebook as a task, set a daily schedule and an alert on failure. Before scheduling, make the notebook safe to rerun: overwriting the whole table, as above, is idempotent; appending is not.

Teams that orchestrate across many systems, not just Databricks, often use an external orchestrator instead; the Airflow tutorial covers that approach.

Mistakes beginners make

• Relying on hidden notebook state. Cells run in whatever order you click, so a notebook can work interactively and fail as a job. Before scheduling, clear state and run it top to bottom.
• Pulling big tables into pandas with toPandas(), which moves everything to one machine; aggregate in Spark first.
• Hard-coding dates and names that should be job parameters.
• Leaving paid compute running in company workspaces after the work is done.

Habits worth forming early

• Keep notebooks short and parameterised, and move reusable logic into Python files in a Git folder.
• Write to tables, not to ad hoc files, so data is governed and discoverable.
• Watch compute costs: stop what you do not need, and prefer serverless for bursty work.
• Learn the engine underneath. Databricks runs Spark, so Apache Spark explained still applies to every slow job.

Azure Databricks is part of the Microsoft Azure data engineering module of the Data Engineering course, alongside Apache Spark and PySpark.

Sources

• What is Databricks? (Databricks documentation) — Databricks
• Databricks Free Edition (Databricks documentation) — Databricks
• Free Edition limitations (Databricks documentation) — Databricks
• Sample datasets (Databricks documentation) — Databricks
• Develop code in Databricks notebooks (Databricks documentation) — Databricks
• What is Unity Catalog? (Databricks documentation) — Databricks
• Lakeflow Jobs (Databricks documentation) — Databricks