Kaggle for Beginners: How to Start and Win Medals

Kaggle is the best-known platform for machine learning competitions, and much more than that: it hosts public datasets, shared code notebooks and discussion forums, and Kaggle Notebooks, a free browser-based environment for running code on CPUs, GPUs or TPUs. Founded in 2010 and acquired by Google in 2017, it has run many competitions for companies and researchers (Wikipedia: Kaggle). For a beginner, it offers something hard to find elsewhere: real problems, a fixed metric, and thousands of other people's solutions to learn from.

This guide explains how Kaggle works, how to spend your first month, the workflow that separates improving competitors from stuck ones, and a simulation of the most common beginner mistake: trusting small leaderboard gains.

How Kaggle works

• Competitions. A host provides training data, a test set without answers and a metric. You submit predictions for the test set and receive a score. Some are introductory, with no prizes and no deadline pressure; others run for months with prize money.
• The two leaderboards. During a competition, your submissions are scored on part of the test set, and the results form the public leaderboard. The final ranking uses a separate part that stays hidden until the end, often called the private leaderboard. Only that final ranking counts.
• Notebooks, datasets and discussions. Participants share code, publish datasets and discuss approaches. Many winners publish their solutions after a competition closes, which makes finished competitions an excellent library.
• Progression. Kaggle's progression system has five tiers, Novice, Contributor, Expert, Master and Grandmaster, earned separately through competitions, datasets, notebooks and discussions, largely by winning medals. The exact medal cut-offs depend on the number of teams and are listed on Kaggle's progression page; check it for the current rules rather than trusting old blog posts.

Your first month

Week 1: one complete submission. Join an introductory competition, such as Titanic or House Prices. Read the overview, the data description and, carefully, the metric. Make a simple submission on the first day: a baseline model, even a mean prediction. The goal is to have the whole loop working, from data to submission file to score.

Week 2: build a validation you trust. Set up cross-validation that mirrors how the test set was split. If the test data comes from a later time period, validate on later data; if rows are grouped by customer or patient, keep each group in one fold. Everything you do later depends on this. Cross-validation in machine learning covers the options.

Week 3: learn from others. Read the most-upvoted public notebooks and discussion threads. Reproduce one notebook yourself, then change one thing and measure the effect with your own validation.

Week 4: improve and write up. Try feature ideas and a second model type, keep what improves validation, and write a short summary of what worked and what did not. That write-up is the start of a portfolio piece.

A workflow that keeps improving

1. Understand the metric. Optimising the wrong target is the fastest way to waste a month. If the metric is log loss, calibrated probabilities matter; if it is ranking-based, only the order of predictions does.
2. Start simple and log everything. Record each experiment's validation score, settings and public score in a table.
3. Spend most of your time on data and features. On tabular problems, feature engineering usually moves the score more than switching algorithms.
4. Use strong defaults. Gradient-boosted trees for tabular data, pre-trained networks for images and text.
5. Blend at the end. Averaging diverse, well-validated models is a standard final step in competitions.
6. Choose final submissions by validation. Where a competition lets you select which submissions count for the final ranking, choose the ones your validation supports, not the ones that happen to top the public board.

Why small score gaps mislead: a simulation

Here is a competition in miniature. The phoneme dataset (OpenML: phoneme) is split into training data and a hidden test set of 2,162 rows. Two gradient-boosting models, A and B, differ only in their learning rate:

import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, train_test_split

data = fetch_openml(data_id=1489, as_frame=True)
X, y = data.data, (data.target == "2").astype(int)
X_train, X_hidden, y_train, y_hidden = train_test_split(X, y, test_size=0.4, stratify=y, random_state=0)

models = {"A": HistGradientBoostingClassifier(random_state=0),
          "B": HistGradientBoostingClassifier(learning_rate=0.05, random_state=0)}
correct = {}
for name, model in models.items():
    cv = cross_val_score(model, X_train, y_train, cv=5).mean()
    correct[name] = (model.fit(X_train, y_train).predict(X_hidden) == y_hidden).to_numpy()
    print(f"model {name}: local CV {cv:.4f} | whole hidden test {correct[name].mean():.4f}")

model A: local CV 0.8843 | whole hidden test 0.8760
model B: local CV 0.8782 | whole hidden test 0.8802

On the whole hidden test set, B is better by 0.4 percentage points, yet local cross-validation ranked A first. Now simulate the public leaderboard: score both models on a random 30% of the hidden set, 1,000 times over:

rng = np.random.default_rng(0)
n = len(y_hidden)
public_a, a_ahead = [], 0
for _ in range(1000):
    public = rng.permutation(n)[: int(0.3 * n)]          # a random 30% is scored publicly
    public_a.append(correct["A"][public].mean())
    a_ahead += correct["A"][public].mean() > correct["B"][public].mean()
print(f"model A's public score across splits: {min(public_a):.4f} to {max(public_a):.4f}")
print(f"A ranked above B on the public board in {a_ahead} of 1000 splits")

model A's public score across splits: 0.8441 to 0.9074
A ranked above B on the public board in 187 of 1000 splits

The same predictions from model A score anywhere from 84.4% to 90.7% depending only on which rows happen to be public. And the worse model ranks higher on the public board in almost one split in five. The lesson is not that the public board is useless: across more than a hundred Kaggle competitions, researchers found little evidence that participants substantially overfitted the public leaderboard overall (Roelofs et al., NeurIPS 2019). The lesson is that a gap of a few tenths of a point, whether on the public board or in one cross-validation run, is not evidence. Treat such models as tied, reduce the noise with repeated cross-validation, or blend them.

Downloading data and submitting from the command line

The official Kaggle command-line tool downloads competition data and submits files, after you create an API token on your account settings page (Kaggle CLI: competitions):

pip install kaggle
kaggle competitions download titanic
kaggle competitions submit titanic -f submission.csv -m "baseline: logistic regression"

Writing a clear message with every submission turns your submission history into an experiment log. Read each competition's rules on its website before you start; they cover team sizes, daily submission limits and whether external data is allowed.

Realistic expectations about medals

Medals in popular competitions go to a small share of teams, and many strong data scientists take several competitions to earn their first. Improve your odds by choosing competitions that suit your skills, joining late-stage teams where your validation or feature work adds something, and studying the published solutions of past winners before starting. Treat each finished competition as a course: compare your approach with the top solutions and write down the one idea you missed.

For your career, a medal is a signal, not the goal. A clear write-up of your approach, your validation strategy and what you learned is often more convincing to an employer than the rank itself; how to build a data science portfolio covers presenting it.

Competitions reward the same skills as real projects; an end-to-end machine learning project shows the full process. Model evaluation, cross-validation and hyperparameter tuning are covered in the Data Science course.

Sources

• Kaggle (Wikipedia) — Wikipedia
• Kaggle CLI: competitions commands — Kaggle, GitHub
• A Meta-Analysis of Overfitting in Machine Learning — NeurIPS 2019
• phoneme (OpenML dataset 1489) — www.openml.org