AWS Lambda Explained: Serverless for Beginners

Most code runs on a server that is always on, waiting. AWS Lambda turns that around: your code exists as a function, and it runs only when something happens. AWS describes Lambda as a serverless compute service that runs code without provisioning or managing servers, taking care of capacity, scaling and patching (Lambda docs: what is AWS Lambda).

"Serverless" does not mean there are no servers. It means they are not yours to think about. What you do have to think about is the model Lambda imposes, because it is different from ordinary programs in a few important ways.

The model: function, event, trigger

Three ideas cover most of it (Lambda docs: how Lambda works):

• A function is a piece of code with one job, packaged as a zip archive or a container image.
• The handler is the entry point. Lambda calls it with two arguments: the event, a document describing what happened, and a context object with information about the invocation.
• A trigger connects an event source to the function: a file landing in S3, a message on a queue, a schedule, an HTTP request through API Gateway.

Each invocation is independent. The guide puts it as running independently with no shared state. A function receives an event, does its work, returns, and should not assume anything is remembered for next time.

A handler, run locally

The handler below validates a batch of payment records. Because a handler is only a function, it can be called on your own machine with a sample event, which is how Lambda code should be tested before it is deployed. That is what was done here; nothing ran on AWS.

import json

# --- Initialisation: runs once per execution environment, not once per event ---
print("init: loading configuration")
RULES = {"max_amount": 10_000, "currencies": {"USD", "EUR", "GBP"}}
invocations = 0


def handler(event, context):
    """Validate a batch of payment records and report what was accepted."""
    global invocations
    invocations += 1
    accepted, rejected = [], []
    for record in event["records"]:
        ok = record["currency"] in RULES["currencies"] and 0 < record["amount"] <= RULES["max_amount"]
        (accepted if ok else rejected).append(record["id"])
    return {"statusCode": 200,
            "body": json.dumps({"accepted": accepted, "rejected": rejected,
                                "invocation_in_this_environment": invocations})}


# --- Local invocation: call the handler the way Lambda would, with an event and a context ---
class Context:                      # the two attributes this example needs
    function_name = "validate-payments"
    aws_request_id = "local-test"


events = [
    {"records": [{"id": "p1", "amount": 120.0, "currency": "USD"},
                 {"id": "p2", "amount": -5.0, "currency": "USD"}]},
    {"records": [{"id": "p3", "amount": 99.0, "currency": "INR"}]},
    {"records": [{"id": "p4", "amount": 10_000, "currency": "GBP"}]},
]
for event in events:
    print(handler(event, Context())["body"])

init: loading configuration
{"accepted": ["p1"], "rejected": ["p2"], "invocation_in_this_environment": 1}
{"accepted": [], "rejected": ["p3"], "invocation_in_this_environment": 2}
{"accepted": ["p4"], "rejected": [], "invocation_in_this_environment": 3}

Two things are worth noticing.

"init" prints once, the handler runs three times. Code outside the handler runs when the module is loaded. Code inside runs for every event. The counter reaching 3 shows that module-level state survived between calls.

That is how Lambda behaves, sometimes. When a function is first invoked, Lambda creates an execution environment and runs the initialisation code. Afterwards it does not discard the environment straight away; if the function is invoked again, the environment can be reused, and only the handler runs. Whether a given event gets a fresh or a reused environment is not under your control.

Cold starts and what follows from them

An invocation that needs a new environment is a cold start: the code must be loaded and initialised before the handler can run, which adds latency. A reused environment is "warm".

The practical rules:

• Do expensive setup outside the handler: creating SDK clients, loading configuration, opening connections. Warm invocations then skip it.
• Never rely on state being there. The counter above is a demonstration, not a design. The next event may go to a different environment, or to several at once.
• Keep packages small. Less code to load means a shorter cold start.

When demand rises, Lambda does not queue events behind one copy of your function. It runs more environments in parallel. That is how it scales, and it is also how a busy function can overwhelm a database that accepts only a limited number of connections.

The limits that define Lambda

Lambda's quotas say plainly what it is for (Lambda docs: quotas):

Fifteen minutes and ten gigabytes of memory rule out heavy data processing. Lambda is for short, bounded tasks. The memory setting is also the performance setting: more memory brings more CPU, so a function can finish faster, and sometimes cost less, with a higher allocation.

How events arrive, and what happens on failure

Functions are invoked in three ways, and the difference matters when something goes wrong:

• Synchronously. The caller waits for the result, as with an API request. Errors go back to the caller.
• Asynchronously. The event is queued and the caller moves on, as with S3 notifications. If the function returns an error, Lambda by default tries two more times, then discards the event unless you have configured a dead-letter queue or an on-failure destination (Lambda docs: asynchronous error handling).
• By polling. For queues and streams such as SQS and Kinesis, an event source mapping reads the source, batches records and invokes the function with the batch.

Two consequences for anyone writing data code. Events can be delivered more than once, so a handler must be safe to run twice on the same event. And a failed asynchronous event disappears silently unless you capture it. Configure the failure destination before you need it.

Permissions

A function runs with an execution role, an IAM role that defines what it may do: read one bucket, write to one table, write logs. Separately, a policy on the function controls who may invoke it. Give each function its own role with the least it needs; a shared, powerful role is the common shortcut that turns a small bug into a large incident.

What it costs

Functions are priced on two things: the number of requests, and duration measured in GB-seconds, which is run time multiplied by the memory allocated. The free tier includes one million requests and 400,000 GB-seconds per month (AWS: Lambda pricing). An idle function costs nothing, which is the appeal for workloads that run occasionally. A function invoked constantly at high memory is a different calculation, and a container or server can be cheaper.

Lambda in data work

In pipelines, Lambda is usually the trigger and the glue, not the engine. A file lands in S3, a function checks it and starts a Glue job or a Step Functions workflow, and the heavy work happens elsewhere. Event-driven pipelines with Lambda and S3 builds that pattern; AWS Step Functions covers chaining functions into workflows; and Glue vs EMR vs Lambda for ETL compares it with the heavier engines.

Beginner mistakes

• Putting a long batch job in a function and discovering the timeout.
• Relying on local files or global variables to carry state between invocations.
• Ignoring duplicates. Write handlers so the same event twice does no harm.
• No failure destination for asynchronous events.
• One broad IAM role shared by every function.
• Recursive triggers: a function that writes to the bucket that triggers it, invoking itself without end.

For where Lambda sits among the services a data engineer should learn, see the AWS roadmap for data engineers.

Cloud data services on AWS and pipeline design are covered in the Data Engineering course.

Sources

• What is AWS Lambda? (AWS Lambda Developer Guide) — AWS Documentation
• How Lambda works (AWS Lambda Developer Guide) — AWS Documentation
• Lambda quotas (AWS Lambda Developer Guide) — AWS Documentation
• How Lambda handles errors and retries with asynchronous invocation (AWS Lambda Developer Guide) — AWS Documentation
• AWS Lambda pricing (Amazon Web Services) — Amazon Web Services