AWS Glue Studio: Visual ETL for Beginners

Writing a Spark job is a barrier for many people who understand their data perfectly well. AWS Glue Studio lowers it. AWS describes it as a graphical interface for creating, running and monitoring data integration jobs, in which you visually compose a transformation workflow and run it on Glue's Spark-based serverless engine (Glue docs: what is AWS Glue).

The important word is "Spark-based". A visual job is not a different, lighter kind of job. It is an ordinary Glue Spark job whose script was written by the tool instead of by you. Knowing that explains both what Glue Studio is good for and where it stops.

This guide follows the documentation. The steps were not carried out in an AWS account for this article.

Three ways to author a job

Glue Studio offers three starting points (Glue docs: starting visual ETL jobs):

All three produce the same thing: a Glue job listed on the same Jobs page, run and scheduled the same way.

The canvas: sources, transforms, targets

A visual job is a diagram with three kinds of node (Glue docs: job editor features):

• Data source nodes read data, typically a table from the Glue Data Catalog or a location in S3.
• Transform nodes modify it.
• Data target nodes write it.

You add a node, connect it to its parent, and fill in its properties in a side panel. Each node shows its output schema, so you can see what the data looks like at every step.

The built-in transforms cover everyday ETL (Glue docs: managed transforms):

The SQL node deserves a mention. Anyone who can write a SELECT can express joins, aggregations and derived columns there without learning the DataFrame API. For many analysts it is the bridge between a visual job and real transformation logic.

A first job, step by step

A typical first pipeline converts raw CSV into cleaned Parquet:

1. Create the job. In Glue Studio choose ETL jobs, then Visual ETL.
2. Add a source. Choose a Data Catalog table that describes the raw files, or point at an S3 location and format. The Glue Data Catalog explains where such tables come from.
3. Add ChangeSchema. Rename columns to lower case with underscores, and set types: dates as dates, amounts as decimals.
4. Add a Filter. Remove rows that fail a rule, such as a negative amount.
5. Add a target. Choose S3, format Parquet, a compression type, and partition keys. Optionally have the job create or update the catalog table for the output.
6. Set job details. Choose an IAM role, the Glue version, and the worker type and number.
7. Save and run. Watch progress on the Runs tab, where logs are linked.

Previews: seeing data before running

Two features let you check work without running the whole job.

Schema preview shows the output schema of any node. Data preview goes further and shows sample rows after each node, 200 by default. The guide lists the uses: testing that the IAM role can reach the data, and checking that a transform is doing what you intended, for example that a filter selects the right subset.

Previews are useful and not free. They run on a real interactive session:

• They cost money. The pricing page states that each data preview session uses 2 DPUs, runs for 30 minutes and stops automatically (AWS: Glue pricing).
• They need a role. The role chosen for the preview is also used for the job.
• Custom SQL and code nodes run in full. The guide warns that a preview executes a custom SQL or code node as-is for the entire dataset, not for a sample.
• Large data is slow. The guide says data under 1 GB can take up to a minute to appear, and that a query taking more than 15 minutes times out. Preview against a small partition.

End the session when you have finished looking.

The script underneath

When you build a job visually, Glue Studio generates a complete job script and saves it in S3. The Script tab shows it. Reading that script is one of the best ways to learn Glue: every node you drew appears as a few lines of PySpark, using the names you gave the nodes.

The script of a visual job is locked. To edit it by hand, you convert the job to script-only mode; treat that as a permanent move from diagram to code (Glue docs: editing a job script). That is the natural path for a job that outgrows the canvas: start visually, then take over the generated script. AWS Glue jobs with PySpark covers working at that level.

The other tabs complete the picture: Job details for the role, version and workers; Runs for history and logs; Schedules; Data quality, for rules checked during the job; and Version Control, to connect a Git service.

When visual, when code

Visual jobs have real limits. A diagram with forty nodes is harder to follow than forty lines of code. Generated scripts are awkward to compare between versions. Logic inside node settings cannot be tested in isolation. None of this argues against Glue Studio; it argues for using it where pipelines are simple and for moving to code as they grow.

Costs and habits for beginners

• Check the worker count. A Spark job is allocated 10 DPUs by default and billed by the second. A small job needs far fewer; set the number in Job details.
• Close preview sessions.
• Name nodes meaningfully. The names appear in the generated script and the logs.
• Write Parquet, partitioned, from the first job.
• Keep one job to one purpose. Several small jobs are easier to rerun than one sprawling diagram.

For Glue's other parts and how jobs are triggered, see the AWS Glue tutorial.

AWS Glue and Apache Spark are covered in the Data Engineering course.

Sources

• What is AWS Glue? (AWS Glue User Guide) — AWS Documentation
• Starting visual ETL jobs in AWS Glue Studio (AWS Glue User Guide) — AWS Documentation
• Job editor features (AWS Glue User Guide) — AWS Documentation
• Transform data with AWS Glue managed transforms (AWS Glue User Guide) — AWS Documentation
• Editing or uploading a job script (AWS Glue User Guide) — AWS Documentation
• AWS Glue pricing (Amazon Web Services) — Amazon Web Services