Product Analytics 101: Events, Funnels and Retention

Product analytics studies what people do inside a product: which features they use, where they get stuck, and whether they come back. Web analytics tells you how visitors arrive and marketing analytics what they cost; product analytics explains what happens after that, and it is where many questions about growth are answered. Its raw material is event data, and its main tools are funnels, retention, segmentation and experiments.

The event data model

Product analytics tools store behaviour as events. Mixpanel's documentation puts the minimum clearly: every event needs an event name, a timestamp, and an ID that ties it to a user; optional properties describe the event in more detail, such as the items in an order (Mixpanel: events and properties). A single table of rows like that can answer a remarkable range of questions.

The same structure can be built from almost any transactional data. Here, a UK online gift retailer's order lines (UCI: Online Retail) become one "Order Completed" event per order, with the number of distinct products and the order value as properties:

import io
import urllib.request
import zipfile
import pandas as pd

url = "https://archive.ics.uci.edu/static/public/352/online+retail.zip"
with zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(url).read())) as archive:
    raw = pd.read_excel(archive.open("Online Retail.xlsx"), dtype={"InvoiceNo": str, "StockCode": str})
services = {"POST", "DOT", "M", "m", "C2", "D", "S", "B", "BANK CHARGES", "AMAZONFEE", "CRUK"}
sales = raw[~raw["InvoiceNo"].str.startswith("C") & ~raw["StockCode"].isin(services)
            & ~raw["StockCode"].str.startswith("gift_") & (raw["Quantity"] > 0) & (raw["UnitPrice"] > 0)
            & raw["CustomerID"].notna()]
events = (sales.assign(value=sales["Quantity"] * sales["UnitPrice"])
          .groupby("InvoiceNo")
          .agg(user_id=("CustomerID", "first"), timestamp=("InvoiceDate", "min"),
               products=("StockCode", "nunique"), value=("value", "sum"))
          .reset_index().assign(event="Order Completed").sort_values("timestamp")
          .astype({"user_id": "int64"}).round({"value": 2}))
print(events[["event", "timestamp", "user_id", "products", "value"]].head(3).to_string(index=False))
print(f"{len(events)} events from {events['user_id'].nunique()} users")

          event           timestamp  user_id  products  value
Order Completed 2010-12-01 08:26:00    17850         7 139.12
Order Completed 2010-12-01 08:28:00    17850         2  22.20
Order Completed 2010-12-01 08:34:00    13047        12 278.73
18402 events from 4334 users

A real product would record many event types, such as sign-ups, searches, views and feature use, but the shape is the same: who did what, when, with what details.

Write a tracking plan before you track

Most event data problems are created at collection time. A tracking plan is a shared document listing each event, when it fires, and its properties:

Good plans use one naming convention, such as object then action in the past tense, track outcomes on the server where possible so ad blockers and failed page loads do not drop them, and include only events that answer a known question. Tracking everything produces data nobody trusts.

Retention: do users come back?

Retention is the test of whether a product delivers lasting value. For customers first seen from January 2011 onwards, and observed for at least 90 days:

events["order_number"] = events.groupby("user_id").cumcount() + 1
users = events[events["order_number"] == 1].set_index("user_id")[["timestamp", "products"]]
users["second_order"] = events[events["order_number"] == 2].set_index("user_id")["timestamp"]
end = events["timestamp"].max()
new = users[(users["timestamp"] >= "2011-01-01") & (users["timestamp"] <= end - pd.Timedelta(days=90))].copy()
new["days_to_second"] = (new["second_order"] - new["timestamp"]).dt.days
new["returned_90d"] = new["days_to_second"] <= 90
print(f"new users observed for 90 days: {len(new)}")
print(f"returned within 90 days: {new['returned_90d'].mean():.1%}")
print(f"median days to second order, among those who returned at all: {new['days_to_second'].median():.0f}")

new users observed for 90 days: 2484
returned within 90 days: 43.3%
median days to second order, among those who returned at all: 67

Two choices keep these numbers honest. Customers first seen in December 2010 are excluded, because the data starts that month and many of them are long-standing customers, not new ones. And only customers with at least 90 days of history are counted, so recent sign-ups do not look like failures simply because they have not had time to return. Retention by sign-up month, the cohort view, is covered step by step in cohort analysis explained, and retention reports in product tools follow the same logic of a starting event and a returning event (Mixpanel: retention).

Activation: early behaviour that predicts return

Product teams look for early behaviour associated with users sticking around, often called activation. Here, the breadth of the first order is a candidate:

new["first_basket"] = pd.cut(new["products"], [0, 5, 15, 30, 1000], labels=["1-5 products", "6-15", "16-30", "31+"])
print(new.groupby("first_basket", observed=True)["returned_90d"].agg(users="size", returned_90d="mean").round(3).to_string())

              users  returned_90d
first_basket                     
1-5 products    363         0.375
6-15            773         0.384
16-30           793         0.463
31+             555         0.497

Customers whose first order included more than 30 products returned within 90 days about half the time; those who bought five or fewer, just over a third. That is a correlation, not a cause: shops buying stock for resale naturally order broadly and reorder often. The analytical next step is an experiment, such as a first-order offer that encourages broader baskets, to see whether changing the behaviour changes retention.

Funnels and segments

A funnel measures how many users complete each step of a sequence, such as viewing a product, adding it to the basket and paying, and where they drop off (Mixpanel: funnels); funnel analysis builds one on real sessions. Segmentation splits any of these measures by user type, since averages hide very different groups; customer segmentation for analysts shows RFM segments. Comparing periods fairly, especially with seasonal products, is covered in time intelligence.

Experiments: from correlation to cause

Funnels, retention and segments describe behaviour; they cannot prove that a change caused an improvement, because many things change at once. An experiment, or A/B test, randomly splits users between the current experience and a new one, so the only systematic difference between the groups is the change itself. Decide the success metric and the sample size before starting, run the test for full weeks so weekday patterns are balanced, and check a guardrail metric such as refunds or cancellations alongside the main one. A/B testing for data scientists covers the statistics.

Pitfalls that corrupt event data

• Identity: the same person as an anonymous visitor and as a logged-in user counts as two users unless the IDs are merged.
• Duplicate events from retries or double clicks inflate counts; deduplicate on an event ID.
• Changed definitions: an event renamed or moved mid-year breaks every trend that crosses the change. Version the tracking plan.
• Time zones: users in one zone, servers in another, and a dashboard in a third put events on different days.
• Vanity totals: cumulative sign-ups always rise; vanity vs actionable metrics explains what to report instead.

Customer segmentation is part of the business analytics concepts module of the Data Analytics course.

Sources

• Events and properties (Mixpanel documentation) — Mixpanel
• Funnels overview (Mixpanel documentation) — Mixpanel
• Retention (Mixpanel documentation) — Mixpanel
• Online Retail (UCI Machine Learning Repository) — UCI Machine Learning Repository