Jekyll2026-04-04T00:11:50+00:00https://doctorofdata.github.io/feed.xmlThe Data DojoA data scientist with a passion for ai and machine learningData Guydataguy50@gmail.comMicrosoft MINDS Data: A Machine Learning Recommendation Engine2026-04-02T00:00:00+00:002026-04-02T00:00:00+00:00https://doctorofdata.github.io/posts/2026/4/mind-news-recommendation-engineIn this post, we build a series of recommendation engines for the Microsoft MINDS dataset using popular heuristic strategies and a combination of machine learning algorithms.

📰 News Data: Microsoft MIND — Two-Stage Generate-&-Rerank News Recommendation Engine

The MIND dataset is the standard benchmark for neural news recommendation, released by Microsoft Research. It contains ~160 K users, ~65 K articles, and 1 M+ click-through logs collected from MSN News in October 2019.

In this post we build a two-stage generate-and-rerank paradigm from large-scale recommendation systems:

Stage What it does
Stage 1 — Retrieval Cast a wide net: merge candidates from popularity, category-affinity, item-CF, and recency signals
Stage 2 — Ranking Re-score every candidate with a LightGBM meta-ranker that sees retriever membership, base scores, and rich user/article features

📋 Table of Contents

# Section
1 Setup & data loading
2 Exploratory data analysis
3 Feature engineering
4 Article-based collaborative filtering
5 Temporal sequence modelling
6 Evaluation harness & S1–S5 strategies
7 S6 architecture & cold-start gate
8 Stage 1 — Expanded candidate pool
9 Stage 2 — Meta-ranker training
10 Full benchmark: S1 → S7
11 Visualisations
12 Leaderboard & takeaways

🗺️ System Blueprint — How It All Fits Together

Before diving into the code, here’s a bird’s-eye view of the entire two-stage pipeline you’ll build in this notebook:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                     MIND News Recommendation Engine                             │
│                                                                                 │
│   ┌──────────┐    ┌──────────┐    ┌─────────────────────────────────────────┐  │
│   │  Raw     │    │  EDA &   │    │         FEATURE STORE                   │  │
│   │  Data    │───▶│  Stats   │───▶│  user_stats · article_feat              │  │
│   │(MIND TSV)│    │(Sec 2)   │    │  user_cat_affinity · TF-IDF centroids   │  │
│   └──────────┘    └──────────┘    └───────────────┬─────────────────────────┘  │
│                                                   │                             │
│              ┌────────────────────────────────────▼─────────────────────────┐  │
│              │                 STAGE 1 — RETRIEVAL (Sec 8)                  │  │
│              │                                                               │  │
│              │  S1 Popularity  S2 Category  S3 Item-CF  S4 Temporal Taste   │  │
│              │       ↓               ↓            ↓            ↓            │  │
│              │              MERGE & DEDUPLICATE                              │  │
│              │              200-candidate pool  (Recall@200 ~diagnostic)    │  │
│              └────────────────────────┬──────────────────────────────────────┘  │
│                                       │                                         │
│              ┌────────────────────────▼──────────────────────────────────────┐  │
│              │                 STAGE 2 — RERANKING (Sec 9)                   │  │
│              │                                                               │  │
│              │   Base LightGBM (LambdaMART, SET_A) ──▶ OOF scores           │  │
│              │   Meta-LGB (extended features, SET_B) ──▶ S6                 │  │
│              │   XGBoost ensemble blend ──────────────────▶ S7              │  │
│              └────────────────────────┬──────────────────────────────────────┘  │
│                                       │                                         │
│              ┌────────────────────────▼──────────────────────────────────────┐  │
│              │         EVALUATION (Sec 10–12)                                │  │
│              │   Precision · Recall · F1 · NDCG · Hit-Rate  @ K=5 & K=10   │  │
│              └───────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────┘

Notebook Roadmap

Section Focus Key Output
§1 Setup Load MIND-small ZIPs all_interactions, news DataFrames
§2 EDA Statistical exploration 8 visualisations, CTR & sparsity stats
§3 Features Engineer 4 feature tables user_stats, article_feat, user_cat_affinity, imp_train_df
§4 Item-CF Sparse co-click similarity item_sim_lookup (top-50 neighbours per article)
§5 Temporal Recency-weighted taste temporal_taste_matrix with 7-day half-life decay
§6 Eval + S1–S5 Baseline strategies Metrics for 5 retrieval/simple-rank methods
§7 Cold-start gate Handle zero-history users Binary cold/warm routing logic
§8 Stage 1 Candidate pool fusion 200 candidates, Recall@200 diagnostic
§9 Stage 2 Meta-ranker training meta_lgb model with enriched features
§10–12 Full benchmark S1→S7 leaderboard, lift metrics

Reading tip: Each section opens with a 📖 callout explaining the why before the code shows the how.

1. Setup & data loading

📖 Dataset & Problem Framing

The data: MIND-small contains ~1 M impression logs from 50,000 users over six weeks (Oct 12–Nov 22, 2019). Each impression records a user session: the articles shown, which ones were clicked (label=1) or ignored (label=0), and the user’s recent click history.

File Key columns Role
behaviors.tsv ImpressionId, UserId, Time, History, Impressions Primary signal — click/no-click
news.tsv NewsId, Category, SubCategory, Title, Abstract Article metadata

Task framing. Given a user’s click history, rank candidate news articles so that clicked articles appear at the top. We evaluate with ranking metrics (Precision@K, Recall@K, NDCG@K, Hit-Rate@K).

Train/test split strategy. MIND provides an official train split and a dev (validation) split. We use train behaviors for all model fitting and dev behaviors as the held-out test set, preserving the temporal ordering of the original benchmark.

Implicit feedback. Unlike star-ratings, every click is a positive signal (label = 1); every article shown but not clicked is a negative (label = 0). We treat clicks as our “liked” items throughout.

🔍 Data Schema at a Glance

behaviors.tsv — one row per user session (impression):

ImpressionId | UserId | Time                  | History              | Impressions
─────────────┼────────┼───────────────────────┼──────────────────────┼──────────────────────────────
imp-1234     | U5678  | 10/15/2019 8:32:01 AM | N1001 N1087 N2334 …  | N3301-1 N2201-0 N4412-0 …
                                                 ↑ past click IDs       ↑ candidate-label pairs

Each entry in Impressions is newsId-label where label=1 means clicked, label=0 means skipped. This is the core supervision signal.

news.tsv — one row per article:

newsId | category  | subCategory | title                          | abstract
───────┼───────────┼─────────────┼────────────────────────────────┼──────────────
N1001  | Sports    | NFL         | "Eagles defeat Cowboys 31-14"  | "The Philadelphia Eagles …"
N1087  | Finance   | Stocks      | "Apple earnings beat Q3"       | "Apple Inc. reported …"

Key insight: The recommendation task is session-level re-ranking, not global ranking. For each impression, you rank the ~10–20 candidate articles shown in that session, using the user’s click history as context.

# Import libraries
import subprocess, sys

for pkg in ['lightgbm', 'xgboost', 'scikit-learn']:

    subprocess.run([sys.executable, '-m', 'pip', 'install', '-q', pkg], check = True)

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import zipfile, gc, time, warnings, os, re
from datetime import datetime
from collections import defaultdict, Counter
from scipy.sparse import csr_matrix
from sklearn.preprocessing import normalize
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import lightgbm as lgb
from xgboost import XGBClassifier
from joblib import Parallel, delayed
from google.colab import drive
import zipfile

warnings.filterwarnings('ignore')
plt.style.use('seaborn-v0_8-whitegrid')
sns.set_palette('husl')
pd.set_option('display.float_format', '{:.4f}'.format)
pd.set_option('display.max_columns', None)

print('✅  Libraries loaded')

✅  Libraries loaded

# Connect to data
drive.mount('/content/drive')

Mounted at /content/drive

# Function to perform parsing
def parse_behaviors_from_zip(zip_path, inner_path):

    with zipfile.ZipFile(zip_path, 'r') as z:

        with z.open(inner_path) as f:

            raw = pd.read_csv(f, sep = '\t', header = None, names = BEH_COLS)

    raw['time'] = pd.to_datetime(raw['time'], format='%m/%d/%Y %I:%M:%S %p')
    raw['ts']   = raw['time'].astype('int64') // 10**9

    rows = []

    for _, r in raw.iterrows():

        uid = r['userId']
        ts  = r['ts']

        if pd.notna(r['impressions']):

            for pair in str(r['impressions']).split():

                nid, lbl = pair.rsplit('-', 1)
                rows.append((uid, nid, int(lbl), ts))

    df = pd.DataFrame(rows, columns=['userId','newsId','clicked','timestamp'])

    return df, raw

# Function to perform metrics for ranking
def precision_at_k(recs, true_set, k):

    return len(set(recs[:k]) & true_set) / k if k else 0.0

def recall_at_k(recs, true_set, k):

    return len(set(recs[:k]) & true_set) / len(true_set) if true_set else 0.0

def f1_at_k(recs, true_set, k):

    p = precision_at_k(recs, true_set, k)
    r = recall_at_k(recs,    true_set, k)

    return 2*p*r/(p+r) if (p+r) > 0 else 0.0

def ndcg_at_k(recs, true_set, k):

    dcg   = sum(1 / np.log2(i + 2) for i, m in enumerate(recs[:k]) if m in true_set)
    ideal = sum(1 / np.log2(i + 2) for i in range(min(len(true_set), k)))

    return dcg/ideal if ideal else 0.0

def score_recs(recs, true_set, K):

    return {'precision': precision_at_k(recs, true_set, K),
            'recall'   : recall_at_k(recs,    true_set, K),
            'f1'       : f1_at_k(recs,        true_set, K),
            'ndcg'     : ndcg_at_k(recs,      true_set, K),
            'hit_rate' : 1 if any(m in true_set for m in recs[:K]) else 0,}

def evaluate_strategy(score_fn, eval_df, K = 10, n = None):

    # score_fn(uid, candidates) -> candidates sorted best-first
    rows = eval_df if n is None else eval_df.sample(n=n, random_state=100)
    m = {k: [] for k in ('precision','recall','f1','ndcg','hit_rate')}

    for _, row in rows.iterrows():

        recs = score_fn(row['userId'], row['imp_candidates'])
        s    = score_recs(recs, row['true_items'], K)

        for k in m:

            m[k].append(s[k])

    result = {k: float(np.mean(v)) for k, v in m.items()}

    # FIX 4: composite = mean(NDCG, Hit-Rate) — avoids double-counting P/R via F1
    result['composite'] = float(np.mean([result['ndcg'], result['hit_rate']]))

    return result

metric_keys = ['precision','recall','f1','ndcg','hit_rate']

def parse_history_length(raw_df):

    raw_df = raw_df.copy()
    raw_df['history_len'] = raw_df['history'].fillna('').apply(lambda h: len(str(h).split()) if str(h).strip() else 0)

    return raw_df.groupby('userId')['history_len'].max()

def daily_agg(df, split_label):

    tmp = df.copy()
    tmp['date']  = pd.to_datetime(tmp['timestamp'], unit='s').dt.date
    tmp['split'] = split_label

    return (tmp.groupby(['date','split']).agg(impressions=('clicked','count'), clicks=('clicked','sum')).reset_index().assign(ctr=lambda d: d['clicks']/d['impressions']))

# Function to filter previously seen articles
def _filter_seen(article_list, uid):

    seen = _seen_cache.get(uid, set())

    return [a for a in article_list if a not in seen]

# Ranking metrics
def s1_popularity(uid, N = 50):

    return _filter_seen(POPULARITY_POOL, uid)[:N]

def s2_category(uid, N = 50):

    if uid not in user_cat_affinity.index:

        return s1_popularity(uid, N)

    uvec = user_cat_affinity.loc[uid].values.astype('float32')
    uvec_n = uvec / (np.linalg.norm(uvec) + 1e-9)
    scores = article_cat_norm @ uvec_n
    ranking = np.argsort(-scores)
    ordered = [article_cat_idx[i] for i in ranking]

    return _filter_seen(ordered, uid)[:N]

def s3_itemcf(uid, N = 50):

    clicked = list(user_click_sets.get(uid, []))

    if not clicked:

        return s1_popularity(uid, N)

    score_acc = defaultdict(float)

    for aid in clicked[-20:]:

        for n_aid, sim in item_sim_lookup.get(aid, [])[:30]:

            score_acc[n_aid] += sim

    seen = _seen_cache.get(uid, set())
    ranked = sorted(score_acc.items(), key=lambda x: -x[1])
    filtered = [a for a, _ in ranked if a not in seen]

    if len(filtered) < N:

        filtered += _filter_seen(POPULARITY_POOL, uid)[:N]

    return filtered[:N]

def s4_temporal(uid, N = 50):

    if uid not in user_taste_norm.index:

        return s1_popularity(uid, N)

    tvec = user_taste_norm.loc[uid].values.astype('float32')
    scores = article_cat_taste_norm @ tvec
    ranking = np.argsort(-scores)
    ordered = [taste_article_idx[i] for i in ranking]

    return _filter_seen(ordered, uid)[:N]

# Compute tfidf centroids and resulting article affinity
def tfidf_affinity(uid, aid):

    '''Cosine sim: user click-history TF-IDF centroid vs article.'''
    centroid = user_tfidf_centroids.get(uid)
    if centroid is None:
        return 0.0
    i = tfidf_idx.get(aid, -1)
    if i < 0:
        return 0.0
    return float(tfidf_mat[i].dot(centroid))

def recent_tfidf_affinity(uid, aid):

    '''Cosine sim using centroid of user recent 20 clicks only.'''
    centroid = user_recent_tfidf_centroids.get(uid)
    if centroid is None:
        return 0.0
    i = tfidf_idx.get(aid, -1)
    if i < 0:
        return 0.0
    return float(tfidf_mat[i].dot(centroid))

# Evaluation scoring functions
def s1_score(uid, candidates):

    return sorted(candidates, key = lambda a: -float(pop_stats.loc[a,'bayesian_ctr'] if a in pop_stats.index else 0))

def s2_score(uid, candidates):

    if uid not in user_cat_affinity.index:

        return s1_score(uid, candidates)

    uvec = user_cat_affinity.loc[uid].values.astype('float32')
    uvec /= np.linalg.norm(uvec) + 1e-9

    def _s(a):

        i = art_pos.get(a, -1)
        return float(article_cat_norm[i] @ uvec) if i >= 0 else 0.0

    return sorted(candidates, key=lambda a: -_s(a))

def s3_score(uid, candidates):

    clicked = list(user_click_sets.get(uid, []))

    if not clicked:

        return s1_score(uid, candidates)

    score_acc = defaultdict(float)

    for aid in clicked[-20:]:

        for n_aid, sim in item_sim_lookup.get(aid, [])[:30]:

            score_acc[n_aid] += sim

    return sorted(candidates, key=lambda a: -score_acc.get(a, 0))

def s4_score(uid, candidates):

    if uid not in user_taste_norm.index:

        return s1_score(uid, candidates)

    tvec = user_taste_norm.loc[uid].values.astype('float32')

    # Use taste_pos dict (O(1)) instead of list.index() (O(n))
    def _s(a):

        i = taste_pos.get(a, -1)

        return float(article_cat_taste_norm[i] @ tvec) if i >= 0 else 0.0

    return sorted(candidates, key=lambda a: -_s(a))

def _build_feature_matrix(uid, candidates, s2_vec, s4_vec):

    '''Build the full FEATURE_COLS-aligned matrix for all impression candidates.
    Includes within-impression context signals (ctr_norm_rank, imp_size).'''
    n    = len(candidates)
    u_cc = float(us_click_count.get(uid, 0))
    u_cf = float(us_click_freq.get(uid,  0))
    ctrs = np.array([af_bayesian_ctr.get(a, 0) for a in candidates], dtype='float32')
    ctr_norm_rank = np.argsort(np.argsort(-ctrs)).astype('float32') / max(1, n - 1)
    rows = []

    for k, a in enumerate(candidates):

        ai = art_pos.get(a, -1)
        ti = taste_pos.get(a, -1)
        subc = newsid_to_subcat.get(a)
        rows.append([
            u_cc,
            u_cf,
            float(af_log_clicks.get(a,   0)),
            float(af_log_impr.get(a,     0)),
            float(af_article_len.get(a,  0)),
            float(s2_vec[ai]) if ai >= 0 else 0.0,
            float(s4_vec[ti]) if ti >= 0 else 0.0,
            tfidf_affinity(uid, a),
            recent_tfidf_affinity(uid, a),
            float(af_article_age.get(a,  0)),
            float(ctr_norm_rank[k]),
            float(n),
            float(user_subcat_clicks.get((uid, subc), 0)) if subc else 0.0,])

    return np.array(rows, dtype='float32')

def s5_score(uid, candidates):

    if uid in user_cat_affinity.index:

        uvec   = user_cat_affinity.loc[uid].values.astype('float32')
        s2_vec = article_cat_norm @ (uvec / (np.linalg.norm(uvec) + 1e-9))

    else:

        s2_vec = np.zeros(len(article_cat_idx))

    if uid in user_taste_norm.index:

        tvec   = user_taste_norm.loc[uid].values.astype('float32')
        s4_vec = article_cat_taste_norm @ tvec

    else:

        s4_vec = np.zeros(len(taste_article_idx))

    X = _build_feature_matrix(uid, candidates, s2_vec, s4_vec)
    probs = lgb_model.predict(X)

    return [candidates[i] for i in np.argsort(-probs)]

def s6_score(uid, candidates):

    if is_cold(uid):

        return s1_score(uid, candidates)

    if uid in user_cat_affinity.index:

        uvec   = user_cat_affinity.loc[uid].values.astype('float32')
        s2_vec = article_cat_norm @ (uvec / (np.linalg.norm(uvec) + 1e-9))

    else:

        s2_vec = np.zeros(len(article_cat_idx))

    if uid in user_taste_norm.index:

        tvec   = user_taste_norm.loc[uid].values.astype('float32')
        s4_vec = article_cat_taste_norm @ tvec

    else:

        s4_vec = np.zeros(len(taste_article_idx))

    X_base      = _build_feature_matrix(uid, candidates, s2_vec, s4_vec)
    base_scores = lgb_model.predict(X_base)
    cands_s2    = s2_category(uid, N_STAGE1)
    cands_s3    = s3_itemcf(uid,   N_STAGE1)
    cands_s4    = s4_temporal(uid, N_STAGE1)
    X_meta      = _build_meta_features(uid, candidates, cands_s2, cands_s3, cands_s4, s2_vec, s4_vec, base_scores)
    scores = meta_lgb.predict(X_meta)

    return [candidates[i] for i in np.argsort(-scores)]

def s7_score(uid, candidates):

    if is_cold(uid):

        return s1_score(uid, candidates)

    if uid in user_cat_affinity.index:

        uvec   = user_cat_affinity.loc[uid].values.astype('float32')
        s2_vec = article_cat_norm @ (uvec / (np.linalg.norm(uvec) + 1e-9))

    else:

        s2_vec = np.zeros(len(article_cat_idx))

    if uid in user_taste_norm.index:

        tvec   = user_taste_norm.loc[uid].values.astype('float32')
        s4_vec = article_cat_taste_norm @ tvec

    else:

        s4_vec = np.zeros(len(taste_article_idx))

    X_base      = _build_feature_matrix(uid, candidates, s2_vec, s4_vec)
    base_scores = lgb_model.predict(X_base)
    cands_s2    = s2_category(uid, N_STAGE1)
    cands_s3    = s3_itemcf(uid,   N_STAGE1)
    cands_s4    = s4_temporal(uid, N_STAGE1)
    X_meta      = _build_meta_features(uid, candidates, cands_s2, cands_s3, cands_s4, s2_vec, s4_vec, base_scores)
    lgb_probs = meta_lgb.predict(X_meta)
    xgb_probs = xgb_meta.predict_proba(X_meta)[:, 1]
    scores    = 0.6 * lgb_probs + 0.4 * xgb_probs
    return [candidates[i] for i in np.argsort(-scores)]

def _build_feature_row(uid, aid, s2_scores_dict, s4_scores_dict):

    '''Used by s5_lgb retriever (not evaluation path).'''
    ai      = art_pos.get(aid, -1)
    ti      = taste_pos.get(aid, -1)
    cat_aff = float(s2_scores_dict.get(ai, 0))
    tst_aff = float(s4_scores_dict.get(ti, 0))
    return [
        float(us_click_count.get(uid, 0)),
        float(us_click_freq.get(uid,  0)),
        float(af_log_clicks.get(aid,   0)),
        float(af_log_impr.get(aid,     0)),
        float(af_bayesian_ctr.get(aid, 0)),
        float(af_article_len.get(aid,  0)),
        cat_aff,
        tst_aff,
        tfidf_affinity(uid, aid),
        float(af_article_age.get(aid, 0)),]

def _build_meta_features(uid, candidates, cands_s2, cands_s3, cands_s4, s2_vec, s4_vec, lgb_base_scores):

    s2_rank = {a: r for r, a in enumerate(cands_s2)}
    s3_rank = {a: r for r, a in enumerate(cands_s3)}
    s4_rank = {a: r for r, a in enumerate(cands_s4)}
    n    = len(candidates)
    ctrs = np.array([af_bayesian_ctr.get(a, 0) for a in candidates], dtype='float32')
    ctr_norm_rank = np.argsort(np.argsort(-ctrs)).astype('float32') / max(1, n - 1)
    rows = []

    for k, aid in enumerate(candidates):

        ai      = art_pos.get(aid, -1)
        ti      = taste_pos.get(aid, -1)
        cat_aff = float(s2_vec[ai]) if ai >= 0 else 0.0
        tst_aff = float(s4_vec[ti]) if ti >= 0 else 0.0
        in_s2   = int(aid in s2_rank)
        in_s3   = int(aid in s3_rank)
        in_s4   = int(aid in s4_rank)
        rows.append([
            float(us_click_count.get(uid, 0)),
            float(us_click_freq.get(uid,  0)),
            float(af_log_clicks.get(aid,   0)),
            float(af_log_impr.get(aid,     0)),
            float(af_article_len.get(aid,  0)),
            cat_aff, tst_aff,
            tfidf_affinity(uid, aid),
            recent_tfidf_affinity(uid, aid),
            float(af_article_age.get(aid, 0)),
            float(ctr_norm_rank[k]),
            float(n),
            float(user_subcat_clicks.get((uid, newsid_to_subcat.get(aid)), 0))
            if newsid_to_subcat.get(aid) else 0.0,
            in_s2, in_s3, in_s4,
            s2_rank.get(aid, N_STAGE1),
            s3_rank.get(aid, N_STAGE1),
            s4_rank.get(aid, N_STAGE1),
            in_s2 + in_s3 + in_s4,
            float(lgb_base_scores[k]),
        ])

    return np.array(rows, dtype='float32')

def s5_lgb(uid, N = 50):

    candidates = list(dict.fromkeys(s2_category(uid, K_CAND) + s3_itemcf(uid, K_CAND) + s4_temporal(uid, K_CAND)))[:K_CAND]

    if not candidates:

        return s1_popularity(uid, N)

    if uid in user_cat_affinity.index:

        uvec   = user_cat_affinity.loc[uid].values.astype('float32')
        s2_vec = article_cat_norm @ (uvec / (np.linalg.norm(uvec) + 1e-9))

    else:

        s2_vec = np.zeros(len(article_cat_idx))

    if uid in user_taste_norm.index:

        tvec   = user_taste_norm.loc[uid].values.astype('float32')
        s4_vec = article_cat_taste_norm @ tvec

    else:

        s4_vec = np.zeros(len(taste_article_idx))

    X     = _build_feature_matrix(uid, candidates, s2_vec, s4_vec)
    probs = lgb_model.predict(X)

    return [candidates[i] for i in np.argsort(-probs)][:N]

def s6_meta_lgb(uid, N = 50):

    if is_cold(uid):

        return s1_popularity(uid, N)

    cands_s2   = s2_category(uid, N_STAGE1)
    cands_s3   = s3_itemcf(uid,   N_STAGE1)
    cands_s4   = s4_temporal(uid, N_STAGE1)
    candidates = list(dict.fromkeys(cands_s2 + cands_s3 + cands_s4))[:N_STAGE1]

    if uid in user_cat_affinity.index:

        uvec   = user_cat_affinity.loc[uid].values.astype('float32')
        s2_vec = article_cat_norm @ (uvec / (np.linalg.norm(uvec) + 1e-9))

    else:

        s2_vec = np.zeros(len(article_cat_idx))

    if uid in user_taste_norm.index:

        tvec   = user_taste_norm.loc[uid].values.astype('float32')
        s4_vec = article_cat_taste_norm @ tvec

    else:

        s4_vec = np.zeros(len(taste_article_idx))

    X_base      = _build_feature_matrix(uid, candidates, s2_vec, s4_vec)
    base_scores = lgb_model.predict(X_base)
    X_meta      = _build_meta_features(uid, candidates, cands_s2, cands_s3, cands_s4, s2_vec, s4_vec, base_scores)
    scores = meta_lgb.predict(X_meta)

    return [candidates[i] for i in np.argsort(-scores)][:N]

def s7_ensemble(uid, N = 50):

    if is_cold(uid):

        return s1_popularity(uid, N)

    cands_s2   = s2_category(uid, N_STAGE1)
    cands_s3   = s3_itemcf(uid,   N_STAGE1)
    cands_s4   = s4_temporal(uid, N_STAGE1)
    candidates = list(dict.fromkeys(cands_s2 + cands_s3 + cands_s4))[:N_STAGE1]

    if uid in user_cat_affinity.index:

        uvec   = user_cat_affinity.loc[uid].values.astype('float32')
        s2_vec = article_cat_norm @ (uvec / (np.linalg.norm(uvec) + 1e-9))

    else:

        s2_vec = np.zeros(len(article_cat_idx))

    if uid in user_taste_norm.index:

        tvec   = user_taste_norm.loc[uid].values.astype('float32')
        s4_vec = article_cat_taste_norm @ tvec

    else:

        s4_vec = np.zeros(len(taste_article_idx))

    X_base      = _build_feature_matrix(uid, candidates, s2_vec, s4_vec)
    base_scores = lgb_model.predict(X_base)
    X_meta      = _build_meta_features(uid, candidates, cands_s2, cands_s3, cands_s4,
                                       s2_vec, s4_vec, base_scores)
    lgb_probs = meta_lgb.predict(X_meta)
    xgb_probs = xgb_meta.predict_proba(X_meta)[:, 1]
    scores    = 0.6 * lgb_probs + 0.4 * xgb_probs

    return [candidates[i] for i in np.argsort(-scores)][:N]

COLD_THRESHOLD = 2

def is_cold(uid):

    if uid not in user_stats.index:

        return True

    return user_stats.loc[uid, 'click_count'] < COLD_THRESHOLD

def _raw_s2(uid, N):

    if uid not in user_cat_affinity.index:

        return POPULARITY_POOL[:N]

    uvec = user_cat_affinity.loc[uid].values.astype('float32')
    uvec = uvec / (np.linalg.norm(uvec) + 1e-9)

    return [article_cat_idx[j] for j in np.argsort(-(article_cat_norm @ uvec))[:N]]

def _raw_s3(uid, N):

    clicked = list(user_click_sets.get(uid, []))

    if not clicked:

        return POPULARITY_POOL[:N]

    score_acc = defaultdict(float)

    for aid in clicked[-20:]:

        for n_aid, sim in item_sim_lookup.get(aid, [])[:30]:

            score_acc[n_aid] += sim

    ranked = [a for a, _ in sorted(score_acc.items(), key=lambda x: -x[1])]

    return (ranked + POPULARITY_POOL)[:N]

def _raw_s4(uid, N):

    if uid not in user_taste_norm.index:

        return POPULARITY_POOL[:N]
    tvec = user_taste_norm.loc[uid].values.astype('float32')

    return [taste_article_idx[j] for j in np.argsort(-(article_cat_taste_norm @ tvec))[:N]]

def chunked_topn(A_norm, U_mat, article_idx_arr, n_top, rank_col):

    parts = []

    for start in range(0, n_users, CHUNK_SIZE):

        end       = min(start + CHUNK_SIZE, n_users)
        u_batch   = unique_users[start:end]
        scores    = A_norm @ U_mat[start:end].T
        top_idx   = np.argsort(-scores, axis=0)[:n_top]
        chunk_len = end - start
        parts.append(pd.DataFrame({
            'userId': np.repeat(u_batch, n_top),
            'newsId': article_idx_arr[top_idx.T.ravel()],
            rank_col: np.tile(np.arange(n_top), chunk_len),
        }))
        del scores, top_idx

    gc.collect()
    return pd.concat(parts, ignore_index = True)

# Load the data
TRAIN_ZIP = 'drive/MyDrive/MINDsmall_train.zip'
DEV_ZIP   = 'drive/MyDrive/MINDsmall_dev.zip'

# Quick sanity-check: list contents of each archive
for label, path in [('TRAIN', TRAIN_ZIP), ('DEV', DEV_ZIP)]:

    with zipfile.ZipFile(path, 'r') as z:

        print(f'{label} ZIP contents: {z.namelist()}')

TRAIN ZIP contents: ['MINDsmall_train/', 'MINDsmall_train/behaviors.tsv', 'MINDsmall_train/news.tsv', 'MINDsmall_train/entity_embedding.vec', 'MINDsmall_train/relation_embedding.vec']
DEV ZIP contents: ['MINDsmall_dev/', 'MINDsmall_dev/behaviors.tsv', 'MINDsmall_dev/news.tsv', 'MINDsmall_dev/entity_embedding.vec', 'MINDsmall_dev/relation_embedding.vec']

# Define columns of interest
NEWS_COLS = ['newsId','category','subCategory','title','abstract','url', 'titleEntities','abstractEntities']
BEH_COLS = ['impressionId', 'userId', 'time', 'history', 'impressions']

print('Loading train news...', end=' ', flush = True)

# Load the data from file
with zipfile.ZipFile(TRAIN_ZIP, 'r') as z:

    with z.open('MINDsmall_train/news.tsv') as f:

        news_train = pd.read_csv(f, sep = '\t', header = None, names = NEWS_COLS, usecols = ['newsId', 'category', 'subCategory', 'title', 'abstract'])

print(f'done  ({len(news_train):,} articles)')

print('Loading dev news...  ', end = ' ', flush = True)

# Load the data from file
with zipfile.ZipFile(DEV_ZIP, 'r') as z:

    with z.open('MINDsmall_dev/news.tsv') as f:

        news_dev = pd.read_csv(f, sep = '\t', header = None, names = NEWS_COLS, usecols = ['newsId','category','subCategory','title','abstract'])

print(f'done  ({len(news_dev):,} articles)')

Loading train news... done  (51,282 articles)
Loading dev news...   done  (42,416 articles)

# Merge ther files together
news = pd.concat([news_train, news_dev]).drop_duplicates('newsId').reset_index(drop = True)

# Fill empty cells
news['abstract'] = news['abstract'].fillna('')
news['text']     = news['title'] + ' ' + news['abstract']

print(f'\nUnique articles : {len(news):,}')
print(f'Categories      : {news["category"].nunique()}')
print(f'Sub-categories  : {news["subCategory"].nunique()}')
news.head()

Unique articles : 65,238
Categories      : 18
Sub-categories  : 270
newsId category subCategory title abstract text
0 N55528 lifestyle lifestyleroyals The Brands Queen Elizabeth, Prince Charles, an... Shop the notebooks, jackets, and more that the... The Brands Queen Elizabeth, Prince Charles, an...
1 N19639 health weightloss 50 Worst Habits For Belly Fat These seemingly harmless habits are holding yo... 50 Worst Habits For Belly Fat These seemingly ...
2 N61837 news newsworld The Cost of Trump's Aid Freeze in the Trenches... Lt. Ivan Molchanets peeked over a parapet of s... The Cost of Trump's Aid Freeze in the Trenches...
3 N53526 health voices I Was An NBA Wife. Here's How It Affected My M... I felt like I was a fraud, and being an NBA wi... I Was An NBA Wife. Here's How It Affected My M...
4 N38324 health medical How to Get Rid of Skin Tags, According to a De... They seem harmless, but there's a very good re... How to Get Rid of Skin Tags, According to a De... 
# Expand each impression list into one row per (user, article, label) from the behavioral data
print('Parsing train behaviors...', end = ' ', flush = True)
interactions_train, raw_train = parse_behaviors_from_zip(TRAIN_ZIP, 'MINDsmall_train/behaviors.tsv')
print(f'done  ({len(interactions_train):,} rows)')

print('Parsing dev behaviors...  ', end = ' ', flush = True)
interactions_dev, raw_dev = parse_behaviors_from_zip(DEV_ZIP, 'MINDsmall_dev/behaviors.tsv')
print(f'done  ({len(interactions_dev):,} rows)')

# Tag splits and combine
interactions_train['split'] = 'train'
interactions_dev['split']   = 'dev'

all_interactions = pd.concat([interactions_train, interactions_dev], ignore_index = True)

print(f'\nTotal interactions : {len(all_interactions):,}')
print(f'  Train            : {len(interactions_train):,}')
print(f'  Dev              : {len(interactions_dev):,}')

Parsing train behaviors... done  (5,843,444 rows)
Parsing dev behaviors...   done  (2,740,998 rows)

Total interactions : 8,584,442
  Train            : 5,843,444
  Dev              : 2,740,998

all_interactions.head()
userId newsId clicked timestamp split
0 U13740 N55689 1 1573463158 train
1 U13740 N35729 0 1573463158 train
2 U91836 N20678 0 1573582290 train
3 U91836 N39317 0 1573582290 train
4 U91836 N58114 0 1573582290 train 
all_interactions['split'].value_counts()
count
split
train 5843444
dev 2740998 


all_interactions['userId'].nunique()

94057

# Split the data for training
train_clicks = interactions_train[interactions_train['clicked'] == 1]
train_clicks['newsId'] = train_clicks['newsId'].astype(str)

test_clicks = interactions_dev[interactions_dev['clicked'] == 1]
test_clicks['newsId'] = test_clicks['newsId'].astype(str)

# Compile the ground truths
_seen_cache = train_clicks.groupby('userId')['newsId'].apply(set).to_dict()
ground_truth = (test_clicks.groupby('userId')['newsId'].apply(set).rename('true_items'))

# Gather the users
train_users   = set(train_clicks['userId'].unique())
test_users    = set(ground_truth.index)
warm_users    = train_users & test_users
cold_users    = test_users - train_users

print(f'Train positive clicks  : {len(train_clicks):,}')
print(f'Dev   positive clicks  : {len(test_clicks):,}')
print(f'Unique train users     : {len(train_users):,}')
print(f'Unique test  users     : {len(test_users):,}')
print(f'Warm users (train test): {len(warm_users):,}')
print(f'Cold users (test only) : {len(cold_users):,}')

Train positive clicks  : 236,344
Dev   positive clicks  : 111,383
Unique train users     : 50,000
Unique test  users     : 50,000
Warm users (train test): 5,943
Cold users (test only) : 44,057

# Parse raw_dev into per-impression evaluation rows.
# Each impression is one independent ranking query: candidates = articles shown
# in that session, true_items = what was clicked. Keeping sessions separate
# prevents global popularity from dominating via cross-session aggregation.
eval_rows = []

for _, r in raw_dev.iterrows():

    uid = r['userId']

    if uid not in warm_users or pd.isna(r['impressions']):

        continue

    pairs   = str(r['impressions']).split()
    cands   = [p.split('-')[0] for p in pairs]
    clicked = {p.split('-')[0] for p in pairs if p.endswith('-1')}

    if not clicked:

        continue

    eval_rows.append({'userId'        : uid,
                      'impressionId'  : r['impressionId'],
                      'imp_candidates': cands,
                      'true_items'    : clicked})

eval_df   = pd.DataFrame(eval_rows)
eval_warm = eval_df.reset_index(drop = True)

print(f'Eval impressions          : {len(eval_warm):,}')
print(f'Unique warm users         : {eval_warm["userId"].nunique():,}')
print(f'Avg candidates/impression : {eval_warm["imp_candidates"].apply(len).mean():.1f}')
print(f'Avg clicks/impression     : {eval_warm["true_items"].apply(len).mean():.2f}')
eval_warm.head()

Eval impressions          : 8,959
Unique warm users         : 5,943
Avg candidates/impression : 37.6
Avg clicks/impression     : 1.52
userId impressionId imp_candidates true_items
0 U44035 24 [N37204, N48487, N59933, N512, N51776, N64077,... {N37204, N496}
1 U88867 66 [N20036, N36786, N50055, N2960, N5940, N32536,... {N31958, N23513}
2 U80349 69 [N31958, N5472, N36779, N29393, N34130, N23513... {N29393}
3 U61801 70 [N20036, N53242, N6916, N48487, N36940, N46917... {N5940}
4 U54826 82 [N29363, N44289, N7344, N6340, N4610, N40943, ... {N7344} 
# Count clicks per article across the training split
pop_counts = train_clicks.groupby('newsId')['clicked'].count().rename('click_count')

# Bayesian-smoothed score: (clicks + C*global_rate) / (impressions + C)
total_impressions = interactions_train.groupby('newsId')['clicked'].count().rename('impressions')

# Global click-through rate
GLOBAL_CTR = train_clicks.shape[0] / len(interactions_train)

# Smoothing constant
C = 50

pop_stats = (pop_counts.to_frame().join(total_impressions).fillna(0))
pop_stats['bayesian_ctr'] = ((pop_stats['click_count'] + C * GLOBAL_CTR) / (pop_stats['impressions'] + C)).astype('float32')

# Articles ranked by training CTR
#train_ranked = pop_stats.sort_values('bayesian_ctr', ascending=False).index.tolist()

# Dev articles not seen in training are appended so they are still reachable by every retriever
#train_pool_set  = set(train_ranked)
#unseen_articles = [a for a in news['newsId'].astype(str) if a not in train_pool_set]
#POPULARITY_POOL = train_ranked + unseen_articles

POPULARITY_POOL = pop_stats.sort_values('bayesian_ctr', ascending = False).index.tolist()

print(f'Popularity pool  : {len(POPULARITY_POOL):,} training articles')
print(f'Global CTR       : {GLOBAL_CTR:.4f}')
pop_stats.sort_values('impressions', ascending=False).head(6)

Popularity pool  : 7,713 training articles
Global CTR       : 0.0404
click_count impressions bayesian_ctr
newsId
N47061 820 23037 0.0356
N51048 1875 19242 0.0973
N26262 1139 19106 0.0596
N50872 279 18702 0.0150
N55689 4316 18315 0.2351
N38779 1490 18101 0.0822

Back to top

2. Exploratory data analysis

📖 Understanding the data before modelling

This section answers eight key questions before building any model:

  1. How are clicks distributed across articles? (power law expected)
  2. How active are individual users?
  3. Which categories dominate the corpus?
  4. How does CTR vary by category?
  5. What is the article title-length distribution?
  6. How do click volumes trend over time?
  7. What fraction of users have very thin histories (cold-start risk)?
  8. How much overlap exists between train and dev article pools?
# Compile high-level stats
n_users   = all_interactions['userId'].nunique()
n_articles= all_interactions['newsId'].nunique()
n_impr    = len(all_interactions)
n_clicks  = all_interactions['clicked'].sum()
overall_ctr = n_clicks / n_impr

print(f'{"Users":<30} {n_users:>10,}')
print(f'{"Articles":<30} {n_articles:>10,}')
print(f'{"Total impressions":<30} {n_impr:>10,}')
print(f'{"Total clicks":<30} {n_clicks:>10,}')
print(f'{"Overall CTR":<30} {overall_ctr:>10.4f}')
print(f'{"Sparsity":<30} {1 - n_clicks/(n_users*n_articles):>10.6f}')

Users                              94,057
Articles                           22,771
Total impressions               8,584,442
Total clicks                      347,727
Overall CTR                        0.0405
Sparsity                         0.999838

Interpreting the headline numbers:

  • ~3–5% CTR is typical for editorial news feeds. Random chance would yield ~10% (1 click in 10 shown), so position bias and user selectivity drive CTR well below that.
  • Matrix sparsity > 99.9% means collaborative filtering on raw co-clicks alone is brittle — content and temporal signals are essential complements.
  • The gap between unique articles and unique users (~65K vs ~50K) tells you the article space is only slightly larger than the user space in this small subset, which is atypically dense for a real-world recommender.
# Compile the clicks distribution and user activity
article_clicks = train_clicks.groupby('newsId')['clicked'].count()
user_clicks    = train_clicks.groupby('userId')['clicked'].count()

fig, axes = plt.subplots(1, 3, figsize=(21, 5))
fig.suptitle('MIND - Small: Click distributions', fontsize = 14, fontweight = 'bold')

# (a) Article click histogram (log scale)
ax = axes[0]
ax.hist(np.log1p(article_clicks.values), bins = 60, color = 'steelblue', edgecolor = 'white', lw = 0.4)
ax.set_xlabel('log(1 + clicks per article)')
ax.set_ylabel('Number of articles')
ax.set_title('(a) Article popularity (log scale)')
top5 = article_clicks.nlargest(5)

# Iterate
for nid, cnt in top5.items():

    title = news.set_index('newsId').loc[nid, 'title'] if nid in news['newsId'].values else nid
    ax.axvline(np.log1p(cnt), color='red', lw=0.8, alpha=0.5)

# (b) User activity histogram
ax = axes[1]
ax.hist(np.log1p(user_clicks.values), bins=60, color='darkorange', edgecolor='white', lw=0.4)
ax.set_xlabel('log(1 + clicks per user)')
ax.set_ylabel('Number of users')
ax.set_title('(b) User activity (log scale)')

# (c) Click count CDF for articles
ax = axes[2]
sorted_clicks = np.sort(article_clicks.values)
cdf = np.arange(1, len(sorted_clicks)+1) / len(sorted_clicks)
ax.plot(np.log1p(sorted_clicks), cdf, color='purple', lw=2)
ax.axhline(0.8, color='grey', ls='--', lw=1)
ax.set_xlabel('log(1 + clicks)')
ax.set_ylabel('CDF')
ax.set_title('(c) Article popularity CDF')

# Find where 80% of articles have fewer than X clicks
p80_idx = np.searchsorted(cdf, 0.8)
ax.annotate(f'80% articles ≤ {sorted_clicks[p80_idx]} clicks',
            xy=(np.log1p(sorted_clicks[p80_idx]), 0.8),
            xytext=(np.log1p(sorted_clicks[p80_idx])+0.5, 0.65),
            arrowprops=dict(arrowstyle='->', color='black'), fontsize=9)

plt.tight_layout()
plt.savefig('eda_click_distribution.png', dpi=150, bbox_inches='tight')
plt.show()

png

📊 What to look for in these plots:

Plot Expected shape Why it matters
Article click histogram Long-tailed / power law A few viral articles capture most clicks — popularity bias is strong
User activity histogram Right-skewed Most users click < 10 articles; a handful click 100+. Heavy-tail users dominate training signal
Sparsity heatmap Nearly all-zero Collaborative filtering must handle extreme sparsity — motivates CF via item similarity rather than direct user–user CF

A power-law click distribution is the single most important structural property of the dataset. It means:

  1. A popularity baseline (S1) is a surprisingly strong competitor.
  2. Personalisation gains are concentrated on heavy users who have rich histories.
  3. Cold-start users (zero history) must fall back to popularity.
# Analysis by category
news_lookup = news.set_index('newsId')[['category','subCategory','title']]
train_with_cat = train_clicks.join(news_lookup, on = 'newsId')
all_with_cat   = all_interactions.join(news_lookup, on = 'newsId')

# Compile the stats by cat
cat_stats = (all_with_cat.groupby('category').agg(impressions = ('clicked','count'), clicks = ('clicked','sum')).assign(ctr = lambda d: d['clicks']/d['impressions']).sort_values('impressions', ascending = False))

fig, axes = plt.subplots(1, 2, figsize = (20, 6))

# (a) Volume per category
ax = axes[0]
palette = sns.color_palette('husl', len(cat_stats))
bars = ax.barh(cat_stats.index, cat_stats['impressions'], color=palette)
ax.set_xlabel('Total impressions')
ax.set_title('(a) Impressions per category')
ax.invert_yaxis()
for bar, (_, row) in zip(bars, cat_stats.iterrows()):

    ax.text(bar.get_width()*1.01, bar.get_y()+bar.get_height()/2,
            f'{row["ctr"]:.2%} CTR', va='center', fontsize=8)

# (b) CTR per category (sorted)
ax = axes[1]
cat_ctr = cat_stats.sort_values('ctr', ascending=False)
bars2   = ax.barh(cat_ctr.index, cat_ctr['ctr']*100, color=palette)
ax.set_xlabel('CTR (%)')
ax.set_title('(b) Click-through rate by category')
ax.invert_yaxis()
ax.axvline(GLOBAL_CTR*100, color = 'red', ls = '--', lw = 1.5, label = f'Global CTR {GLOBAL_CTR:.2%}')
ax.legend()

plt.tight_layout()
plt.savefig('eda_categories.png', dpi=150, bbox_inches='tight')
plt.show()

print(cat_stats.to_string())

png

               impressions  clicks    ctr
category                                 
news               2232125   95172 0.0426
lifestyle          1016267   45431 0.0447
sports              942187   54220 0.0575
finance             789133   24610 0.0312
foodanddrink        572554   17579 0.0307
entertainment       464494   13362 0.0288
travel              446318   10858 0.0243
health              441673   15331 0.0347
autos               382055   10282 0.0269
tv                  374229   20176 0.0539
music               358613   19776 0.0551
movies              243102    7604 0.0313
video               181367    7076 0.0390
weather             140130    6246 0.0446
kids                   166       3 0.0181
northamerica            29       1 0.0345

📊 Category plots — what they tell you:

The left panel (impression counts) reveals the supply of content per category. The right panel (CTR per category) reveals demand quality — which categories users actually engage with vs. merely see. Gaps between supply and CTR (e.g. high-impression, low-CTR categories) point to editorial over-representation and motivate category-affinity personalisation (S2).

all_interactions.head()
userId newsId clicked timestamp split
0 U13740 N55689 1 1573463158 train
1 U13740 N35729 0 1573463158 train
2 U91836 N20678 0 1573582290 train
3 U91836 N39317 0 1573582290 train
4 U91836 N58114 0 1573582290 train 
all_interactions['split'].value_counts()
count
split
train 5843444
dev 2740998 


# Analysis of cold start data
hist_train = parse_history_length(raw_train)
hist_dev   = parse_history_length(raw_dev)

# Visualize the cold start ratios
fig, axes = plt.subplots(1, 2, figsize=(20, 5))

ax = axes[0]
ax.hist(hist_train.clip(upper=100), bins=50, color='teal', edgecolor='white', lw=0.4)
ax.set_xlabel('History length (clicks, capped at 100)')
ax.set_ylabel('Users')
ax.set_title('Train: history length distribution')
cold_frac = (hist_train == 0).mean()
ax.axvline(0, color='red', lw=1.5, label=f'Cold ({cold_frac:.1%})')
ax.legend()

ax = axes[1]
thresholds = [0, 1, 3, 5, 10, 20]
fracs = [(hist_train <= t).mean() for t in thresholds]
ax.plot(thresholds, [f*100 for f in fracs], 'o-', color='darkorange', lw=2)
ax.set_xlabel('History length threshold')
ax.set_ylabel('% users at or below threshold')
ax.set_title('Cumulative cold-start risk')
ax.axhline(50, color='grey', ls='--', lw=1, label='50%')
ax.legend()

plt.tight_layout()
plt.savefig('eda_coldstart.png', dpi=150, bbox_inches='tight')
plt.show()

print(f'Train users with zero history : {(hist_train==0).sum():,}  ({cold_frac:.2%})')
print(f'Train users with ≤5 history   : {(hist_train<=5).sum():,}  ({(hist_train<=5).mean():.2%})')

png

Train users with zero history : 892  (1.78%)
Train users with ≤5 history   : 12,979  (25.96%)

❄️ Cold-start implications:

The history-length distribution directly sets your cold-start strategy. Users with zero history cannot benefit from personalised retrieval (no clicks to aggregate into a taste vector or to look up similar articles from). The pipeline handles this with a binary gate in §7:

is_cold(user) → True   ➜  return top-N global popularity articles
is_cold(user) → False  ➜  run full personalised pipeline (S2 + S3 + S4)

Even “warm” users with only 1–2 clicks have very noisy taste signals. The Bayesian smoothing in bayesian_ctr and the normalised affinity vectors are designed to degrade gracefully in this sparse regime.

Back to top

3. Feature engineering

We construct four reusable feature tables:

  • user_stats — per-user: click count, active days, click frequency, favourite category
  • article_feat — per-article: click count (log), Bayesian CTR, category one-hot, TF-IDF centroid
  • user_cat_affinity — (user × category) matrix of normalised click preferences
  • imp_train_df — impression-level (userId, newsId, label) frame with query groups for LambdaRank (Fix 1 & 2)

The TF-IDF vectoriser is fit on training article titles+abstracts only and transforms both train and dev articles, preventing feature leakage from future text.

tfidf_sim (cosine similarity between user click-history TF-IDF centroid and each candidate article) and article_age_days (log-scaled age since first impression, capturing news recency).

🛠️ Feature Engineering Map

Four complementary feature tables are constructed — each captures a different signal about users and articles:

                         FEATURE ENGINEERING
                         ═══════════════════

   ┌──────────────────────────────────────────────────────────────────────┐
   │  SOURCE: train_clicks (positive interactions only)                   │
   └───────────┬──────────────────────┬───────────────────────────────────┘
               │                      │
               ▼                      ▼
   ┌────────────────────┐   ┌──────────────────────────────────────────┐
   │   USER SIDE        │   │   ARTICLE SIDE                           │
   │                    │   │                                          │
   │  user_stats        │   │  article_feat                            │
   │  ─────────────     │   │  ─────────────                           │
   │  click_count       │   │  log_clicks  (log(1+n))                  │
   │  active_days       │   │  log_impr                                │
   │  click_freq        │   │  bayesian_ctr  ← smoothed CTR            │
   │  fav_category      │   │  article_len   ← title+abstract words    │
   │                    │   │  article_age_days                        │
   │  user_cat_affinity │   │  category one-hot (18 categories)        │
   │  ─────────────     │   │                                          │
   │  18-dim L2-norm    │   │  TF-IDF centroid (10k-dim, reduced)      │
   │  click distribution│   │                                          │
   └────────────────────┘   └──────────────────────────────────────────┘
               │                      │
               └──────────┬───────────┘
                          │ cross-signals
                          ▼
              ┌───────────────────────┐
              │  INTERACTION FEATURES │
              │                       │
              │  cat_affinity    ←  user_cat · article_cat (dot product)   │
              │  taste_affinity  ←  temporal_taste · article_cat           │
              │  tfidf_sim       ←  user_centroid · article_tfidf          │
              │  recent_tfidf_sim ←  recent-click centroid similarity      │
              └───────────────────────┘

Design principle: Each feature is normalized to a comparable scale before being passed to LightGBM. Tree models are invariant to monotonic transforms, but consistent scaling improves interpretability of feature importances.

# Compile user features
user_stats = train_clicks.groupby('userId').agg(click_count = ('newsId', 'count'),
                                                first_ts    = ('timestamp', 'min'),
                                                last_ts     = ('timestamp', 'max'),)

user_stats['active_days'] = ((user_stats['last_ts'] - user_stats['first_ts']) / 86400).clip(lower = 1).astype('float32')
user_stats['click_freq']  = (user_stats['click_count'] / user_stats['active_days']).astype('float32')

fav_cat = (train_with_cat.groupby(['userId','category'])['clicked'].count().reset_index().sort_values('clicked', ascending = False).drop_duplicates('userId').set_index('userId')['category'])

user_stats['fav_category'] = fav_cat
user_stats = user_stats.fillna({'fav_category': 'unknown'})

print(f'user_stats: {user_stats.shape}')

# Compile article features
article_feat = (pop_stats[['click_count','impressions','bayesian_ctr']].rename(columns={'click_count':'global_clicks','impressions':'global_impressions'}))
article_feat['log_clicks'] = np.log1p(article_feat['global_clicks']).astype('float32')
article_feat['log_impr']   = np.log1p(article_feat['global_impressions']).astype('float32')
article_feat = article_feat.join(news.set_index('newsId')[['category','subCategory','text']], how='left')
article_feat['article_len'] = article_feat['text'].fillna('').apply(len).astype('float32')

# Aticle recency — use earliest training impression as proxy for publish time
EVAL_TS = int(interactions_train['timestamp'].max())
article_first_seen = interactions_train.groupby('newsId')['timestamp'].min()
article_feat['article_age_days'] = (np.log1p((EVAL_TS - article_first_seen) / 86_400).clip(lower = 0).astype('float32').reindex(article_feat.index).fillna(article_feat['log_impr']))
print(f'article_feat: {article_feat.shape}')

# Sub-category click counts per user — finer-grained than category affinity
user_subcat_clicks = (train_with_cat.groupby(['userId', 'subCategory'])['clicked'].count().to_dict())
print(f'user_subcat_clicks entries: {len(user_subcat_clicks):,}')

user_stats: (50000, 6)
article_feat: (7713, 10)
user_subcat_clicks entries: 188,670

train_cat_vocab = pd.get_dummies(article_feat['category'].dropna(), prefix = 'cat').columns

all_news_cat    = news.set_index('newsId')['category'].dropna()
article_cat     = (pd.get_dummies(all_news_cat, prefix = 'cat').astype('float32').reindex(columns = train_cat_vocab, fill_value = 0))

cat_cols = article_cat.columns.tolist()
print(f'Category columns ({len(cat_cols)}): {cat_cols}')
print(f'article_cat covers {len(article_cat):,} articles  '
      f'(train: {len(article_feat):,}  dev-only: {len(article_cat)-len(article_feat):,})')

Category columns (16): ['cat_autos', 'cat_entertainment', 'cat_finance', 'cat_foodanddrink', 'cat_health', 'cat_kids', 'cat_lifestyle', 'cat_movies', 'cat_music', 'cat_news', 'cat_northamerica', 'cat_sports', 'cat_travel', 'cat_tv', 'cat_video', 'cat_weather']
article_cat covers 65,238 articles  (train: 7,713  dev-only: 57,525)

user_stats.head()
click_count first_ts last_ts active_days click_freq fav_category
userId
U100 1 1573544052 1573544052 1.0000 1.0000 news
U1000 4 1573686978 1573771041 1.0000 4.0000 news
U10001 3 1573450221 1573710414 3.0115 0.9962 autos
U10003 3 1573455962 1573481638 1.0000 3.0000 sports
U10008 1 1573308813 1573308813 1.0000 1.0000 weather 
article_feat.head()
global_clicks global_impressions bayesian_ctr log_clicks log_impr category subCategory text article_len article_age_days
newsId
N10032 1 190 0.0126 0.6931 5.2523 foodanddrink recipes 14 butternut squash recipes for delightfully c... 172.0000 0.2827
N10051 1 370 0.0072 0.6931 5.9162 autos autosenthusiasts VW ID.3 Electric Motor Is So Compact That Fits... 160.0000 0.2640
N10056 6 38 0.0912 1.9459 3.6636 sports football_nfl Russell Wilson, Richard Sherman swap jerseys d... 176.0000 1.4091
N10057 2 41 0.0442 1.0986 3.7377 weather weathertopstories Venice swamped by highest tide in more than 50... 243.0000 1.0215
N1006 1 2 0.0581 0.6931 1.0986 sports football_nfl Jaguars vs. Colts: A.J. Cann, Will Richardson ... 487.0000 0.2862 
article_cat.head()
cat_autos cat_entertainment cat_finance cat_foodanddrink cat_health cat_kids cat_lifestyle cat_movies cat_music cat_news cat_northamerica cat_sports cat_travel cat_tv cat_video cat_weather
newsId
N55528 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
N19639 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
N61837 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
N53526 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
N38324 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 
# Perform TF-IDF
train_news_ids = set(train_clicks['newsId'].unique())

news_indexed  = news.set_index('newsId')
train_texts   = news_indexed.loc[news_indexed.index.isin(train_news_ids), 'text'].fillna('')

print('Fitting TF-IDF on train articles...', end = ' ', flush = True)
tfidf = TfidfVectorizer(max_features = 50000, sublinear_tf = True, min_df = 2, ngram_range = (1,2))
tfidf.fit(train_texts)
print('done.')

# Transform all articles (train + dev)
all_texts  = news_indexed['text'].fillna('')
tfidf_mat  = tfidf.transform(all_texts)    # sparse (n_articles, 5000)
tfidf_idx  = {nid: i for i, nid in enumerate(news_indexed.index)}
print(f'TF-IDF matrix: {tfidf_mat.shape}  nnz={tfidf_mat.nnz:,}')

Fitting TF-IDF on train articles... done.
TF-IDF matrix: (65238, 46525)  nnz=3,181,345

%%time

# Build per-user TF-IDF centroids (click-history text profile)
# The centroid is the mean of the TF-IDF vectors of all articles a user has clicked,
# normalised to unit L2 so dot products equal cosine similarity at scoring time.
print('Building user TF-IDF centroids...', end = ' ', flush = True)

user_tfidf_centroids = {}

for uid, group in train_clicks.groupby('userId'):

    idxs = [tfidf_idx[nid] for nid in group['newsId'] if nid in tfidf_idx]

    if not idxs:

        continue

    centroid = np.asarray(tfidf_mat[idxs].mean(axis=0)).ravel()  # (10000,)
    norm = np.linalg.norm(centroid)

    if norm > 1e-9:

        user_tfidf_centroids[uid] = centroid / norm

print(f'done  ({len(user_tfidf_centroids):,} users have centroids)')

# Centroid of only the last 20 clicks — captures recent vs lifetime interest
print('Building recent TF-IDF centroids (last 20 clicks)...', end = ' ', flush = True)
user_recent_tfidf_centroids = {}

for uid, group in train_clicks.sort_values('timestamp').groupby('userId'):

    recent_nids = group['newsId'].tolist()[-20:]
    idxs = [tfidf_idx[nid] for nid in recent_nids if nid in tfidf_idx]

    if not idxs:

        continue

    centroid = np.asarray(tfidf_mat[idxs].mean(axis=0)).ravel()
    norm = np.linalg.norm(centroid)

    if norm > 1e-9:

        user_recent_tfidf_centroids[uid] = centroid / norm

print(f'done  ({len(user_recent_tfidf_centroids):,} users)')

Building user TF-IDF centroids... done  (50,000 users have centroids)
Building recent TF-IDF centroids (last 20 clicks)... done  (50,000 users)
CPU times: user 17min 48s, sys: 1min 15s, total: 19min 3s
Wall time: 2min 26s

# Create an affinity matrix for user-category: compute normalised click counts per category
user_cat = (train_with_cat.groupby(['userId','category'])['clicked'].count().unstack(fill_value = 0).astype('float32'))

# Normalise rows to unit L2 norm
norms = np.linalg.norm(user_cat.values, axis = 1, keepdims = True).clip(min = 1e-9)
user_cat_affinity = pd.DataFrame(user_cat.values / norms, index   = user_cat.index, columns = user_cat.columns)

# Align article-category matrix columns with user-category matrix
article_cat_aligned = article_cat.reindex(columns = user_cat.columns, fill_value = 0)
article_cat_norm    = normalize(article_cat_aligned.values.astype('float32'), norm = 'l2', axis = 1)
article_cat_idx     = article_cat_aligned.index.tolist()

print(f'user_cat_affinity : {user_cat_affinity.shape}')
print(f'article_cat_norm  : {article_cat_norm.shape}')
user_cat_affinity.head(3)

user_cat_affinity : (50000, 16)
article_cat_norm  : (65238, 16)
category autos entertainment finance foodanddrink health kids lifestyle movies music news northamerica sports travel tv video weather
userId
U100 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
U1000 0.0000 0.0000 0.0000 0.4082 0.0000 0.0000 0.0000 0.4082 0.0000 0.8165 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
U10001 0.5774 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.5774 0.5774 0.0000 0.0000 0.0000

Good to Know-

📐 Why L2-normalise the affinity vectors?

After normalising both user_cat_affinity (user rows) and article_cat (article rows) to unit L2 norm, their dot product equals cosine similarity — a value in [–1, 1] that measures directional agreement, independent of how many clicks a user has. This prevents heavy users (who click 200+ articles) from dominating the ranking signal purely because their affinity magnitudes are large.

The same logic applies to TF-IDF centroids: unit-norm centroids mean that a user with 3 clicks and a user with 300 clicks are compared on the same scale when scoring article relevance.

Back to top

4. Article-based collaborative filtering

📖 Batched sparse co-click similarity

We build a article × user click matrix from positive training interactions, normalise rows (articles) to unit L2 norm, and compute cosine similarities between articles in batches to avoid OOM. The result is an item_sim_lookup dict mapping newsId → [(newsId, similarity), …] for the top-50 nearest neighbours.

This creates the S3 retriever: for a given user, find all articles they clicked, look up each article’s nearest neighbours, aggregate scores (weighted by similarity × recency), and surface the top-N unseen articles.

%%time

# Build article x user_click matrices
article_ids_cf = train_clicks['newsId'].unique()
user_ids_cf    = train_clicks['userId'].unique()

a_idx = {a: i for i, a in enumerate(article_ids_cf)}
u_idx = {u: i for i, u in enumerate(user_ids_cf)}
idx_a = {i: a for a, i in a_idx.items()}

R_cf = csr_matrix((np.ones(len(train_clicks), dtype='float32'), (train_clicks['newsId'].map(a_idx).values, train_clicks['userId'].map(u_idx).values)), shape = (len(article_ids_cf), len(user_ids_cf)))

R_norm_cf = normalize(R_cf, norm = 'l2', axis = 1)
print(f'Click matrix: {R_cf.shape}  nnz={R_cf.nnz:,}')
print(f'Memory: R={R_cf.data.nbytes/1e6:.0f} MB  R_norm={R_norm_cf.data.nbytes/1e6:.0f} MB')

Click matrix: (7713, 50000)  nnz=234,468
Memory: R=1 MB  R_norm=1 MB
CPU times: user 149 ms, sys: 80 µs, total: 149 ms
Wall time: 148 ms

# Perform a batched knn to get similar articles
item_sim_lookup = {}
n_articles_cf   = R_norm_cf.shape[0]
t0              = time.time()

for start in range(0, n_articles_cf, 1000):

    batch = R_norm_cf[start : start + 1000]
    sims  = (batch @ R_norm_cf.T).toarray()

    for local_i, sim_row in enumerate(sims):

        global_i          = start + local_i
        sim_row[global_i] = 0.0
        top_k = np.argpartition(sim_row, -50)[-50:]
        top_k = top_k[np.argsort(sim_row[top_k])[::-1]]
        aid   = idx_a[global_i]
        item_sim_lookup[aid] = [(idx_a[j], float(sim_row[j])) for j in top_k]

    if start % 1000 == 0:
        print(f'  {start:>6}/{n_articles_cf}  {time.time()-t0:.0f}s')

del R_cf, R_norm_cf; gc.collect()
print(f'\nItem-sim lookup: {len(item_sim_lookup):,} articles in {time.time()-t0:.0f}s')

       0/7713  0s
    1000/7713  1s
    2000/7713  1s
    3000/7713  1s
    4000/7713  1s
    5000/7713  2s
    6000/7713  2s
    7000/7713  2s

Item-sim lookup: 7,713 articles in 2s

🔗 How item-based CF works here:

The similarity lookup captures the intuition: “users who clicked article A also tended to click article B.”

  Article × User click matrix  R  (shape: 65K articles × 50K users)

  R[i, u] = 1  if user u clicked article i, else 0

  Normalise rows to unit L2:   R_norm = R / ||R||₂  (row-wise)

  Similarity matrix:  S = R_norm · R_normᵀ   → cosine similarity between articles

  item_sim_lookup[A] = top-50 articles by S[A, :]

Why batch the computation? A full 65K × 65K similarity matrix would require ~17 GB of float32 memory. Processing in batches of 1,000 articles keeps peak memory under 2 GB by materialising only one slice at a time.

Retriever score for a user: sum the similarity scores of all articles in the user’s click history toward each candidate article — the more co-clicked history overlaps with the candidate, the higher its S3 score.

Back to top

5. Temporal sequence modelling

📖 Recency-weighted taste vectors

Recent clicks should dominate a user’s preference profile — an article clicked yesterday matters more than one from three weeks ago. We compute per-user category taste vectors using exponential decay (half-life = 7 days, matching news freshness intuition). The resulting matrix enables fast batch dot-products at inference time.

# Compute recency weighted taste vectors -  one week
DECAY_HALF_LIFE = 7
DECAY_K         = np.log(2) / DECAY_HALF_LIFE
now_ts          = int(train_clicks['timestamp'].max())

clicks_ts = train_clicks[['userId','newsId','timestamp']].copy()
clicks_ts['weight'] = np.exp(-DECAY_K * (now_ts - clicks_ts['timestamp'].values.astype('float64')) / 86400).astype('float32')

# Join category info for each click
clicks_ts = clicks_ts.join(news.set_index('newsId')[['category']], on='newsId')
clicks_ts = clicks_ts.dropna(subset=['category'])

# Aggregate: user × category, weighted by recency
user_taste = (clicks_ts.groupby(['userId','category'])['weight'].sum().unstack(fill_value=0).astype('float32'))

# Normalise to unit L2 so dot-products equal cosine similarity
taste_norms  = np.linalg.norm(user_taste.values, axis=1, keepdims=True).clip(min=1e-9)
user_taste_norm = pd.DataFrame(user_taste.values / taste_norms, index   = user_taste.index, columns = user_taste.columns)

# Align with article-category matrix
article_cat_taste = article_cat.reindex(columns=user_taste.columns, fill_value=0)
article_cat_taste_norm = normalize(article_cat_taste.values.astype('float32'), norm='l2', axis=1)
taste_article_idx = article_cat_taste.index.tolist()

print(f'user_taste_norm   : {user_taste_norm.shape}')
print(f'article_cat_taste : {article_cat_taste_norm.shape}')
user_taste_norm.head(3)

user_taste_norm   : (50000, 16)
article_cat_taste : (65238, 16)
category autos entertainment finance foodanddrink health kids lifestyle movies music news northamerica sports travel tv video weather
userId
U100 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
U1000 0.0000 0.0000 0.0000 0.4027 0.0000 0.0000 0.0000 0.4402 0.0000 0.8025 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
U10001 0.6261 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.4647 0.6261 0.0000 0.0000 0.0000

⏱️ Exponential Decay — The Intuition

The recency weight for a click is:

\[w(t) = e^{-k \cdot \Delta t_{days}}, \quad k = \frac{\ln 2}{\text{half-life}}\]

With half-life = 7 days, a click from:

Days ago Weight
0 (today) 1.000
3.5 days ~0.707
7 days 0.500 ← half-life
14 days 0.250
28 days 0.063
42 days 0.016

The six-week MIND window (Oct 12–Nov 22) means clicks from the start of the window receive weight ≈ 0.016 relative to the most recent clicks — effectively negligible. This mirrors real editorial news consumption where interests shift week-to-week.

Alternative half-lives to consider: Shorter (3 days) captures breaking-news spikes; longer (14 days) suits evergreen topic interests (e.g. a user researching a health condition over two weeks). The 7-day default is a reasonable starting point for general news.

# Feature column list — all base LGB and meta-ranker base features
FEATURE_COLS = ['u_click_count', 'u_click_freq',      # user engagement
                'm_log_clicks',  'm_log_impr',        # article global popularity
                'm_article_len',                       # article length
                'cat_affinity',  'taste_affinity',    # collaborative signals
                'tfidf_sim',                           # content similarity (full history centroid)
                'recent_tfidf_sim',                    # content similarity (last-20 clicks centroid)
                'article_age_days',                    # news recency
                'ctr_norm_rank',                       # rank by CTR within impression (0=most popular)
                'imp_size',                            # number of candidates in impression
                'subcat_clicks']                       # user click count for this sub-category

# Select num candidates for training
K_CAND  = 200
rng_feat = np.random.default_rng(100)
print(f'FEATURE_COLS ({len(FEATURE_COLS)}): {FEATURE_COLS}')

FEATURE_COLS (13): ['u_click_count', 'u_click_freq', 'm_log_clicks', 'm_log_impr', 'm_article_len', 'cat_affinity', 'taste_affinity', 'tfidf_sim', 'recent_tfidf_sim', 'article_age_days', 'ctr_norm_rank', 'imp_size', 'subcat_clicks']

# Split the data for training / testing (70-30)
rng_oof    = np.random.default_rng(42)
all_train_users = np.array(list(train_users & set(user_stats.index)))
rng_oof.shuffle(all_train_users)

split_idx  = int(len(all_train_users) * 0.70)
SET_A_users = set(all_train_users[:split_idx])
SET_B_users = set(all_train_users[split_idx:])

print(f'Training users total : {len(all_train_users):,}')
print(f'  SET_A (base LGB)   : {len(SET_A_users):,}')
print(f'  SET_B (meta OOF)   : {len(SET_B_users):,}')

# user_click_sets is needed by s3_itemcf and training loops
user_click_sets = train_clicks.groupby('userId')['newsId'].apply(set).to_dict()

Training users total : 50,000
  SET_A (base LGB)   : 35,000
  SET_B (meta OOF)   : 15,000

# Pre-compile feature dicts (O(1) lookups at scoring time)
art_pos   = {a: i for i, a in enumerate(article_cat_idx)}
taste_pos = {a: i for i, a in enumerate(taste_article_idx)}

af_log_clicks   = article_feat['log_clicks'].to_dict()
af_log_impr     = article_feat['log_impr'].to_dict()
af_bayesian_ctr = article_feat['bayesian_ctr'].to_dict()
af_article_len  = article_feat['article_len'].to_dict()
af_article_age  = article_feat['article_age_days'].to_dict()

us_click_count  = user_stats['click_count'].to_dict()
us_click_freq   = user_stats['click_freq'].to_dict()

newsid_to_subcat = news.set_index('newsId')['subCategory'].to_dict()

%%time

# Build training pairs from actual MIND impression rows (SET_A users only).
# Each impression is one ranking query; every article shown is a candidate;
# the click label is the ground truth. This aligns train and eval distributions.

print('Parsing training impressions for SET_A users...', end = ' ', flush = True)

# Init
imp_rows = []

# Iterate
for _, r in raw_train.iterrows():

    uid = r['userId']

    if uid not in SET_A_users:

        continue

    imp_id = r['impressionId']

    if pd.notna(r['impressions']):

        for pair in str(r['impressions']).split():

            nid, lbl = pair.rsplit('-', 1)
            imp_rows.append((imp_id, uid, str(nid), int(lbl)))

# Compile the iterations
imp_train_df = pd.DataFrame(imp_rows, columns = ['impressionId','userId','newsId','label'])
del imp_rows; gc.collect()

n_pos = int(imp_train_df['label'].sum())
print(f'done  ({len(imp_train_df):,} rows | {imp_train_df["impressionId"].nunique():,} impressions | '
      f'pos={n_pos:,} neg={len(imp_train_df)-n_pos:,})')

# Merge user fts
imp_train_df = imp_train_df.join(user_stats[['click_count','click_freq']].rename(columns = {'click_count':'u_click_count','click_freq':'u_click_freq'}), on = 'userId')
imp_train_df = imp_train_df.join(article_feat[['log_clicks','log_impr','bayesian_ctr','article_len','article_age_days']].rename(columns = {'log_clicks':'m_log_clicks','log_impr':'m_log_impr', 'bayesian_ctr':'m_bayesian_ctr','article_len':'m_article_len', 'article_age_days':'article_age_days'}), on = 'newsId')

# Merge category and taste affinity
newsid_to_cat       = news.set_index('newsId')['category'].to_dict()
imp_train_df['category'] = imp_train_df['newsId'].map(newsid_to_cat)

relevant_users = imp_train_df['userId'].unique()

uca_long = (user_cat_affinity.reindex(index=relevant_users).stack().reset_index().rename(columns={'level_0':'userId','level_1':'category',0:'cat_affinity'}))
imp_train_df = imp_train_df.merge(uca_long, on = ['userId','category'], how = 'left')

del uca_long

uta_long = (user_taste_norm.reindex(index=relevant_users).stack().reset_index().rename(columns = {'level_0':'userId','level_1':'category',0:'taste_affinity'}))
imp_train_df = imp_train_df.merge(uta_long, on = ['userId','category'], how = 'left')

del uta_long; gc.collect()

# Compute the tf-idf similarities
print('Computing TF-IDF affinities...', end = ' ', flush = True)

uid_nid_sim = {}

for uid, grp in imp_train_df.groupby('userId'):

    centroid = user_tfidf_centroids.get(uid)

    if centroid is None:

        continue

    nids  = grp['newsId'].unique()
    idxs  = [tfidf_idx[nid] for nid in nids if nid in tfidf_idx]
    valid = [(nid, tfidf_idx[nid]) for nid in nids if nid in tfidf_idx]

    if not valid:

        continue

    v_nids, v_idxs = zip(*valid)
    sims = np.asarray(tfidf_mat[list(v_idxs)].dot(centroid)).ravel()

    for nid, sim in zip(v_nids, sims):

        uid_nid_sim[(uid, nid)] = float(sim)

imp_train_df['tfidf_sim'] = [uid_nid_sim.get((r.userId, r.newsId), 0.0) for r in imp_train_df.itertuples()]
del uid_nid_sim; gc.collect()
print('done.')

# recent_tfidf_sim — centroid of user's last 20 clicks
print('Computing recent TF-IDF affinities...', end = ' ', flush = True)
uid_nid_recent_sim = {}

for uid, grp in imp_train_df.groupby('userId'):

    centroid = user_recent_tfidf_centroids.get(uid)

    if centroid is None:

        continue

    valid = [(nid, tfidf_idx[nid]) for nid in grp['newsId'].unique() if nid in tfidf_idx]

    if not valid:

        continue

    v_nids, v_idxs = zip(*valid)
    sims = np.asarray(tfidf_mat[list(v_idxs)].dot(centroid)).ravel()

    for nid, sim in zip(v_nids, sims):

        uid_nid_recent_sim[(uid, nid)] = float(sim)

imp_train_df['recent_tfidf_sim'] = [uid_nid_recent_sim.get((r.userId, r.newsId), 0.0) for r in imp_train_df.itertuples()]
del uid_nid_recent_sim; gc.collect()
print('done.')

# subcat_clicks — user click count for candidate's specific sub-category
imp_train_df['_subcat'] = imp_train_df['newsId'].map(newsid_to_subcat)
_subcat_lkp = pd.DataFrame([(u, sc, cnt) for (u, sc), cnt in user_subcat_clicks.items()], columns = ['userId', '_subcat', 'subcat_clicks'])
imp_train_df = imp_train_df.merge(_subcat_lkp, on=['userId', '_subcat'], how='left')
imp_train_df['subcat_clicks'] = imp_train_df['subcat_clicks'].fillna(0).astype('float32')
imp_train_df.drop(columns=['_subcat'], inplace=True)
del _subcat_lkp

# Within-impression context features
imp_train_df['imp_size'] = (imp_train_df.groupby('impressionId')['newsId'].transform('count').astype('float32'))
imp_train_df['ctr_norm_rank'] = (imp_train_df.groupby('impressionId')['m_bayesian_ctr'].transform(lambda x: (x.rank(ascending=False, method='average') - 1).div(max(1, len(x) - 1))).astype('float32'))
imp_train_df[FEATURE_COLS] = imp_train_df[FEATURE_COLS].fillna(0).astype('float32')
print(f'imp_train_df shape: {imp_train_df.shape}')

Parsing training impressions for SET_A users... done  (4,090,484 rows | 110,162 impressions | pos=165,852 neg=3,924,632)
Computing TF-IDF affinities... done.
Computing recent TF-IDF affinities... done.
imp_train_df shape: (4090484, 19)
CPU times: user 1min 24s, sys: 1.27 s, total: 1min 25s
Wall time: 1min 25s

imp_train_df.head()
impressionId userId newsId label u_click_count u_click_freq m_log_clicks m_log_impr m_bayesian_ctr m_article_len article_age_days category cat_affinity taste_affinity tfidf_sim recent_tfidf_sim subcat_clicks imp_size ctr_norm_rank
0 3 U73700 N50014 0 3.0000 1.8087 3.8067 8.2895 0.0114 163.0000 1.1812 sports 0.8944 0.8616 0.0236 0.0236 0.0000 36.0000 1.0000
1 3 U73700 N23877 0 3.0000 1.8087 6.4232 9.3310 0.0545 340.0000 0.7478 news 0.0000 0.0000 0.0219 0.0219 0.0000 36.0000 0.2857
2 3 U73700 N35389 0 3.0000 1.8087 5.5053 8.1259 0.0720 244.0000 1.1917 finance 0.0000 0.0000 0.0423 0.0423 0.0000 36.0000 0.0857
3 3 U73700 N49712 0 3.0000 1.8087 6.2305 8.9469 0.0658 290.0000 0.8228 news 0.0000 0.0000 0.0161 0.0161 0.0000 36.0000 0.1429
4 3 U73700 N16844 0 3.0000 1.8087 5.5294 8.4845 0.0518 278.0000 1.1625 autos 0.0000 0.0000 0.0219 0.0219 0.0000 36.0000 0.3143 
# Training data summary
print(imp_train_df.dtypes)
print(f'\nLabel distribution:\n{imp_train_df["label"].value_counts()}')

impressionId          int64
userId               object
newsId               object
label                 int64
u_click_count       float32
u_click_freq        float32
m_log_clicks        float32
m_log_impr          float32
m_bayesian_ctr      float32
m_article_len       float32
article_age_days    float32
category             object
cat_affinity        float32
taste_affinity      float32
tfidf_sim           float32
recent_tfidf_sim    float32
subcat_clicks       float32
imp_size            float32
ctr_norm_rank       float32
dtype: object

Label distribution:
label
0    3924632
1     165852
Name: count, dtype: int64

📋 imp_train_df — the learning-to-rank training table:

Each row is one (user, article, impression) triple from an actual MIND impression session. The impressionId groups rows so the ranker knows which candidates competed against each other in the same session:

impressionId | userId | newsId | label | u_click_count | m_log_clicks | cat_affinity | … | tfidf_sim
─────────────┼────────┼────────┼───────┼───────────────┼──────────────┼──────────────┼───┼──────────
imp-001      | U1234  | N5001  |   1   |    42         |    2.30      |    0.81      | … |   0.67
imp-001      | U1234  | N5002  |   0   |    42         |    1.10      |    0.23      | … |   0.12
imp-001      | U1234  | N5003  |   0   |    42         |    3.45      |    0.61      | … |   0.44
imp-002      | U9876  | N1001  |   0   |     8         |    2.30      |    0.05      | … |   0.31
…

The impressionId column becomes the LightGBM query group — the model is told “these rows compete against each other, optimise their relative ordering” via LambdaRank.

Back to top

6. Evaluation harness & S1–S5 strategies

Five metrics evaluated at K = 5 and K = 10. Composite score = mean(NDCG@K, Hit-Rate@K) — avoids double-counting Precision and Recall through F1.

Strategy Description
S1 Global popularity — Bayesian CTR ranking
S2 Category affinity — dot product of user preferences with article categories
S3 Item-based CF — aggregate neighbour scores from clicked articles
S4 Temporal taste — recency-weighted category preference
S5 LightGBM LambdaRank ranker

📏 Evaluation Metrics — Quick Reference

All metrics are computed per impression (one ranking query = one session), then averaged across users. K ∈ {5, 10} controls the cutoff — only the top-K predicted articles count.

Metric Formula (simplified) Interpretation
Precision@K (# clicked in top-K) / K Of K articles shown, how many did the user click?
Recall@K (# clicked in top-K) / (# total clicks in session) Of all clicked articles, how many were in top-K?
F1@K 2 · P · R / (P + R) Harmonic mean of precision and recall
NDCG@K DCG@K / IDCG@K Position-weighted relevance; clicked articles ranked first score highest
Hit-Rate@K 1 if ≥ 1 clicked article in top-K else 0 Did the user find at least one article they liked?
Composite mean(NDCG@K, HR@K) Summary score used for leaderboard ranking

Why Composite = mean(NDCG, HR)? Using their mean avoids double-counting the Precision and Recall components that are already captured by F1, while still rewarding both ranked quality (NDCG) and binary coverage (HR).

Why per-impression, not global? Evaluating globally would mix impressions from different sessions and let popular articles dominate. Per-impression evaluation mirrors deployment: the model ranks a specific set of candidates for one user at one moment.

%%time

# LambdaRank objective with per-impression query groups- LambdaMART directly optimises NDCG within each impression list

# Sort by impressionId so groups are contiguous
imp_train_df = imp_train_df.sort_values('impressionId').reset_index(drop = True)

# 85 / 15 impression-level split (no leakage across impression boundaries)
all_imp_ids = imp_train_df['impressionId'].unique()
rng_ltr     = np.random.default_rng(100)
val_imp_ids = set(rng_ltr.choice(all_imp_ids, size=int(len(all_imp_ids) * 0.15), replace=False))

tr_mask  = ~imp_train_df['impressionId'].isin(val_imp_ids)
val_mask =  imp_train_df['impressionId'].isin(val_imp_ids)

# Recompute bayesian_ctr from train-fold impressions only, then apply to both splits
tr_imp_df = imp_train_df[tr_mask]
fold_pop  = (tr_imp_df.groupby('newsId')['label'].agg(['sum', 'count']).rename(columns = {'sum': 'clicks', 'count': 'impr'}))
fold_ctr  = ((fold_pop['clicks'] + C * GLOBAL_CTR) / (fold_pop['impr'] + C))

# Unseen articles keep global estimate
imp_train_df['m_bayesian_ctr'] = (imp_train_df['newsId'].map(fold_ctr).fillna(imp_train_df['m_bayesian_ctr']).astype('float32'))
del tr_imp_df, fold_pop, fold_ctr

# Refresh ctr_norm_rank using fold-corrected CTR values
imp_train_df['ctr_norm_rank'] = (imp_train_df.groupby('impressionId')['m_bayesian_ctr'].transform(lambda x: (x.rank(ascending=False, method='average') - 1).div(max(1, len(x) - 1))).astype('float32'))

x_tr  = imp_train_df.loc[tr_mask,  FEATURE_COLS].values.astype('float32')
y_tr  = imp_train_df.loc[tr_mask,  'label'].values.astype('int')
g_tr  = imp_train_df.loc[tr_mask].groupby('impressionId', sort=True).size().values

x_val = imp_train_df.loc[val_mask, FEATURE_COLS].values.astype('float32')
y_val = imp_train_df.loc[val_mask, 'label'].values.astype('int')
g_val = imp_train_df.loc[val_mask].groupby('impressionId', sort=True).size().values

lgb_params = {'objective'        : 'lambdarank',
            'metric'           : 'ndcg',
            'ndcg_eval_at'     : [5, 10],
            'label_gain'       : [0, 1],
            'learning_rate'    : 0.05,
            'feature_fraction' : 0.8,
            'bagging_fraction' : 0.8,
            'bagging_freq'     : 5,
            'min_child_samples': 5,
            'verbose'          : -1,
            'n_jobs'           : -1,}

lgb_model = lgb.train(lgb_params, lgb.Dataset(x_tr, label = y_tr, group = g_tr), num_boost_round = 800, valid_sets      = [lgb.Dataset(x_val, label = y_val, group = g_val)], callbacks       = [lgb.early_stopping(50, verbose = False), lgb.log_evaluation(100)],)

del x_tr, x_val, y_tr, y_val; gc.collect()
print(f'\nBase LGB trees: {lgb_model.num_trees()}')
print(f'Features used : {FEATURE_COLS}')

[100]	valid_0's ndcg@5: 0.96686	valid_0's ndcg@10: 0.969242
[200]	valid_0's ndcg@5: 0.967218	valid_0's ndcg@10: 0.969574

Base LGB trees: 233
Features used : ['u_click_count', 'u_click_freq', 'm_log_clicks', 'm_log_impr', 'm_article_len', 'cat_affinity', 'taste_affinity', 'tfidf_sim', 'recent_tfidf_sim', 'article_age_days', 'ctr_norm_rank', 'imp_size', 'subcat_clicks']
CPU times: user 4min 27s, sys: 598 ms, total: 4min 28s
Wall time: 58.5 s

🌲 LambdaMART — Why It’s the Right Objective Here

Standard classification loss (binary cross-entropy) treats every mis-ranked pair equally. But in news recommendation, the rank matters: predicting a click at position 1 is far more valuable than at position 10.

LambdaRank (implemented via LightGBM’s lambdarank objective) directly optimises NDCG by computing lambda gradients — pair-wise adjustment weights that scale each gradient by the NDCG improvement that would result from swapping that pair’s positions:

   λᵢⱼ = |ΔNDCG(swap i ↔ j)| · σ(sⱼ - sᵢ)
              ↑                      ↑
   how much the swap helps     logistic margin

The query_group parameter tells LightGBM which rows belong to the same ranking query (same impression), so pairwise comparisons are made within sessions only — exactly matching the evaluation setup.

Practical consequence: LambdaMART generally outperforms pointwise (logistic regression, XGBoost on binary labels) and pairwise (BPR) methods by 2–5 NDCG points on standard LTR benchmarks. The gain compounds in §9 when the meta-ranker uses the base LGB’s OOF scores as a feature.

Back to top

7. S6 architecture & cold-start gate

Architecture

image.png

Cold-start gate

A user is cold if they have fewer than 2 training clicks. Cold users skip the two-stage pipeline entirely and fall back to global popularity ranking.

# Cold start gate
eval_warm = eval_df
cold_in_eval = sum(is_cold(uid) for uid in eval_warm['userId'])
print(f'Cold users in eval fold: {cold_in_eval:,}  '
      f'({100*cold_in_eval/len(eval_warm):.1f}%)')

Cold users in eval fold: 1,226  (13.7%)

Back to top..

8. Stage 1 — Expanded candidate pool

Stage 1 merges four retrievers to maximise recall before the expensive re-ranking step. We measure Stage-1 Recall@200 on a diagnostic sample: what fraction of the user’s ground-truth articles appear anywhere in the 200-candidate pool?

🎯 Stage 1 — Retriever Fusion Strategy

The four retrievers are complementary by design — each catches a different class of relevant articles:

                        USER QUERY
                            │
          ┌─────────────────┼─────────────────┐------------------|
          │                 │                 │                  |
          ▼                 ▼                 ▼                  ▼
   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
   │ S1 Popular  │  │ S2 Category │  │ S3 Item-CF  │  │ S4 Temporal │
   │             │  │             │  │             │  │             │
   │ Bayesian    │  │ user_cat ·  │  │ co-click    │  │ recency-    │
   │ CTR rank    │  │ article_cat │  │ neighbours  │  │ weighted    │
   │ (global)    │  │ dot product │  │ aggregation │  │ taste vec   │
   │             │  │             │  │             │  │             │
   │ Best for:   │  │ Best for:   │  │ Best for:   │  │ Best for:   │
   │ cold users  │  │ category    │  │ warm users  │  │ trend-      │
   │ new articles│  │ loyal users │  │ with many   │  │ sensitive   │
   │             │  │             │  │ clicks      │  │ users       │
   └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘
          │  N/4           │  N/4           │  N/2           │  N/4
          └────────────────┴────-───────────┴─-──────────────┘
                                    │
                           dict.fromkeys()  ← preserves order, deduplicates
                                    │
                            ┌───────▼────────┐
                            │  200 candidates│
                            │  (Recall@200   │
                            │   diagnostic)  │
                            └───────┬────────┘
                                    │
                            STAGE 2 RERANKING

Budget split: S3 (Item-CF) gets half the budget because it produces the most personalised candidates for warm users. S1–S2–S4 each get a quarter. For cold users, the gate bypasses S2–S4 entirely and returns pure popularity.

Why dict.fromkeys() for deduplication? It preserves insertion order (unlike set()), so the highest-priority retriever’s candidates remain first when the total pool is truncated to 200.

# Generate candidates in stage 1
N_STAGE1 = 200

def stage1_candidates(uid):

    if is_cold(uid):

        return _filter_seen(POPULARITY_POOL, uid)[:N_STAGE1]

    pool = list(dict.fromkeys(
        s1_popularity(uid, N_STAGE1//4) +
        s2_category(uid,   N_STAGE1//4) +
        s3_itemcf(uid,     N_STAGE1//2) +
        s4_temporal(uid,   N_STAGE1//4)
    ))
    return pool[:N_STAGE1]

# Recall diagnostic
DIAG_N = 500
diag_users = eval_warm.sample(n = min(DIAG_N, len(eval_warm)), random_state = 100)
recalls = []

for _, row in diag_users.iterrows():

    pool = set(stage1_candidates(row['userId']))
    true = row['true_items']
    recalls.append(len(pool & true) / len(true) if true else 0.0)

print(f'Stage-1 Recall@{N_STAGE1} (n={DIAG_N}): {np.mean(recalls):.4f}')
print(f'  Min: {np.min(recalls):.4f}  Max: {np.max(recalls):.4f}  Std: {np.std(recalls):.4f}')

Stage-1 Recall@200 (n=500): 0.0470
  Min: 0.0000  Max: 1.0000  Std: 0.1945

back to top..

9. Stage 2 — Meta-ranker training

The meta-ranker sees enriched features beyond what the base LightGBM sees:

Feature group Features
Base ranker features All 9 features from Section 5
Retriever membership in_s2, in_s3, in_s4 (binary flags)
Retriever ranks rank_s2, rank_s3, rank_s4 (position in each retriever’s list)
Ensemble depth n_retrievers (how many retrievers surfaced this candidate)
Base LGB score s5_score (predicted probability from the base model)

This lets the meta-ranker learn which retrievers are reliable for which users and articles.

STAGE2_FEATURE_COLS = FEATURE_COLS + ['in_s2','in_s3','in_s4','rank_s2','rank_s3','rank_s4','n_retrievers','s5_score']
print(f'Stage-2 features: {len(STAGE2_FEATURE_COLS)}')

Stage-2 features: 21

%%time

# Meta-ranker training data uses SET_B users (OOF).
# The base LGB was trained only on SET_A; scoring SET_B gives
# true out-of-fold predictions — no in-sample leakage.

CHUNK_SIZE = 500

user_gt_clicks = train_clicks.groupby('userId')['newsId'].apply(set).to_dict()

# Sample up to 5000 SET_B users
rng_meta      = np.random.default_rng(100)
set_b_pool    = np.array(list(SET_B_users & set(user_stats.index)))
sample_meta   = rng_meta.choice(set_b_pool, size = min(5000, len(set_b_pool)), replace = False)

print(f'Meta-ranker training users (SET_B OOF): {len(sample_meta):,}')

# Stage-1 candidate generation without seen-filter (positives must stay in pool)
print('Compiling stage 1 candidates..', end = ' ', flush = True)

_orig_seen = dict(_seen_cache)
_seen_cache.clear()

meta_pair_rows = []

for uid in sample_meta:

    candidates = stage1_candidates(uid)
    gt = user_gt_clicks.get(uid, set())

    for nid in candidates:

        meta_pair_rows.append((uid, str(nid), int(str(nid) in gt)))

_seen_cache.update(_orig_seen)
del _orig_seen

meta_df = pd.DataFrame(meta_pair_rows, columns=['userId', 'newsId', 'label'])
del meta_pair_rows; gc.collect()
print('done.')

n_pos_raw = int(meta_df['label'].sum())
print(f'Stage-1 pairs: {len(meta_df):,}  pos={n_pos_raw:,}  neg={len(meta_df)-n_pos_raw:,}')

if n_pos_raw == 0:

    raise RuntimeError('No positives found in SET_B meta-ranker pairs. '
                       'Check SET_B_users and stage1_candidates().')

# Merge fts
meta_df = meta_df.join(user_stats[['click_count','click_freq']].rename(columns = {'click_count':'u_click_count','click_freq':'u_click_freq'}), on = 'userId')
meta_df = meta_df.join(article_feat[['log_clicks','log_impr','bayesian_ctr','article_len','article_age_days']].rename(columns = {'log_clicks':'m_log_clicks','log_impr':'m_log_impr', 'bayesian_ctr':'m_bayesian_ctr','article_len':'m_article_len', 'article_age_days':'article_age_days'}), on = 'newsId')

# Add category ft
newsid_to_cat        = news.set_index('newsId')['category'].to_dict()
meta_df['category']  = meta_df['newsId'].map(newsid_to_cat)
relevant_users_meta  = meta_df['userId'].unique()

uca_long = (user_cat_affinity.reindex(index = relevant_users_meta).stack().reset_index().rename(columns = {'level_0':'userId','level_1':'category',0:'cat_affinity'}))
meta_df  = meta_df.merge(uca_long, on=['userId','category'], how='left')
del uca_long; gc.collect()

uta_long = (user_taste_norm.reindex(index = relevant_users_meta).stack().reset_index().rename(columns = {'level_0':'userId','level_1':'category',0:'taste_affinity'}))
meta_df  = meta_df.merge(uta_long, on = ['userId','category'], how='left')
del uta_long; gc.collect()

# TF-IDF affinities for meta pairs
print('Computing TF-IDF affinities for meta-ranker pairs...', end = ' ', flush = True)

uid_nid_sim_meta = {}

for uid, grp in meta_df.groupby('userId'):

    centroid = user_tfidf_centroids.get(uid)

    if centroid is None:

        continue

    valid = [(nid, tfidf_idx[nid]) for nid in grp['newsId'].unique() if nid in tfidf_idx]

    if not valid:

        continue

    v_nids, v_idxs = zip(*valid)
    sims = np.asarray(tfidf_mat[list(v_idxs)].dot(centroid)).ravel()

    for nid, sim in zip(v_nids, sims):

        uid_nid_sim_meta[(uid, nid)] = float(sim)

meta_df['tfidf_sim'] = [uid_nid_sim_meta.get((r.userId, r.newsId), 0.0) for r in meta_df.itertuples()]
del uid_nid_sim_meta; gc.collect()
print('done.')

# recent_tfidf_sim for meta-ranker pairs
print('Computing recent TF-IDF affinities for meta pairs...', end = ' ', flush = True)

uid_nid_recent_meta = {}

for uid, grp in meta_df.groupby('userId'):

    centroid = user_recent_tfidf_centroids.get(uid)

    if centroid is None:

        continue

    valid = [(nid, tfidf_idx[nid]) for nid in grp['newsId'].unique() if nid in tfidf_idx]

    if not valid:

        continue

    v_nids, v_idxs = zip(*valid)
    sims = np.asarray(tfidf_mat[list(v_idxs)].dot(centroid)).ravel()

    for nid, sim in zip(v_nids, sims):

        uid_nid_recent_meta[(uid, nid)] = float(sim)

meta_df['recent_tfidf_sim'] = [uid_nid_recent_meta.get((r.userId, r.newsId), 0.0) for r in meta_df.itertuples()]
del uid_nid_recent_meta; gc.collect()
print('done.')

# subcat_clicks for meta-ranker pairs
meta_df['_subcat'] = meta_df['newsId'].map(newsid_to_subcat)
_subcat_lkp_meta = pd.DataFrame([(u, sc, cnt) for (u, sc), cnt in user_subcat_clicks.items()], columns = ['userId', '_subcat', 'subcat_clicks'])
meta_df = meta_df.merge(_subcat_lkp_meta, on=['userId', '_subcat'], how='left')
meta_df['subcat_clicks'] = meta_df['subcat_clicks'].fillna(0).astype('float32')
meta_df.drop(columns=['_subcat'], inplace=True)
del _subcat_lkp_meta

base_feature_cols = [c for c in FEATURE_COLS if c not in ('ctr_norm_rank', 'imp_size')]
meta_df[base_feature_cols] = meta_df[base_feature_cols].fillna(0)

meta_df['imp_size'] = (meta_df.groupby('userId')['newsId'].transform('count').astype('float32'))
meta_df['ctr_norm_rank'] = (meta_df.groupby('userId')['m_bayesian_ctr'].transform(lambda x: (x.rank(ascending=False, method='average') - 1).div(max(1, len(x) - 1))).astype('float32'))

# Get fts from the other retrievers
unique_users = np.array(meta_df['userId'].unique())
n_users           = len(unique_users)

print(f'Building retriever membership for {n_users:,} users...', end = ' ', flush = True)

article_cat_idx_arr   = np.array(article_cat_idx)
taste_article_idx_arr = np.array(taste_article_idx)

_uca   = user_cat_affinity.reindex(unique_users).fillna(0).values.astype('float32')
_uca_n = _uca / (np.linalg.norm(_uca, axis = 1, keepdims = True).clip(min = 1e-9))
s2_top = chunked_topn(article_cat_norm, _uca_n, article_cat_idx_arr, N_STAGE1, 'rank_s2')

del _uca, _uca_n

_taste   = user_taste_norm.reindex(unique_users).fillna(0).values.astype('float32')
s4_top   = chunked_topn(article_cat_taste_norm, _taste, taste_article_idx_arr, N_STAGE1, 'rank_s4')

del _taste; gc.collect()

# Collaborative filte ranking
s3_rows = []

for uid in unique_users:

    for rank, nid in enumerate(s3_itemcf(uid, N_STAGE1)):

        s3_rows.append((uid, str(nid), rank))

s3_top = pd.DataFrame(s3_rows, columns=['userId','newsId','rank_s3'])

del s3_rows

s2_top['newsId'] = s2_top['newsId'].astype(str)
s4_top['newsId'] = s4_top['newsId'].astype(str)
s3_top['newsId'] = s3_top['newsId'].astype(str)

# Merge fts
meta_df = meta_df.merge(s2_top[['userId','newsId','rank_s2']], on = ['userId','newsId'], how = 'left')
meta_df = meta_df.merge(s3_top[['userId','newsId','rank_s3']], on = ['userId','newsId'], how = 'left')
meta_df = meta_df.merge(s4_top[['userId','newsId','rank_s4']], on = ['userId','newsId'], how = 'left')

del s2_top, s3_top, s4_top; gc.collect()
print('done.')

# Compile flags
meta_df['in_s2'] = meta_df['rank_s2'].notna().astype(int)
meta_df['in_s3'] = meta_df['rank_s3'].notna().astype(int)
meta_df['in_s4'] = meta_df['rank_s4'].notna().astype(int)
meta_df[['rank_s2','rank_s3','rank_s4']] = meta_df[['rank_s2','rank_s3','rank_s4']].fillna(N_STAGE1)
meta_df['n_retrievers'] = meta_df[['in_s2','in_s3','in_s4']].sum(axis = 1)

meta_train_df = meta_df.copy()
del meta_df; gc.collect()
print(f'meta_train_df: {meta_train_df.shape}')

Meta-ranker training users (SET_B OOF): 5,000
Compiling stage 1 candidates.. done.
Stage-1 pairs: 917,992  pos=10,098  neg=907,894
Computing TF-IDF affinities for meta-ranker pairs... done.
Computing recent TF-IDF affinities for meta pairs... done.
Building retriever membership for 5,000 users... done.
meta_train_df: (951509, 25)
CPU times: user 14min 44s, sys: 1.12 s, total: 14min 45s
Wall time: 2min 11s

meta_train_df.head()
userId newsId label u_click_count u_click_freq m_log_clicks m_log_impr m_bayesian_ctr m_article_len article_age_days category cat_affinity taste_affinity tfidf_sim recent_tfidf_sim subcat_clicks imp_size ctr_norm_rank rank_s2 rank_s3 rank_s4 in_s2 in_s3 in_s4 n_retrievers
0 U19087 N49279 0 1 1.0000 7.7280 8.7371 0.3618 126.0000 1.7416 music 0.0000 0.0000 0.0121 0.0121 0.0000 200.0000 0.0000 200.0000 30.0000 200.0000 0 1 0 1
1 U19087 N49685 0 1 1.0000 7.7385 8.8860 0.3154 187.0000 1.7186 music 0.0000 0.0000 0.0289 0.0289 0.0000 200.0000 0.0050 200.0000 31.0000 200.0000 0 1 0 1
2 U19087 N60750 0 1 1.0000 4.8363 5.8693 0.3152 303.0000 0.2710 sports 0.0000 0.0000 0.0058 0.0058 0.0000 200.0000 0.0101 200.0000 32.0000 200.0000 0 1 0 1
3 U19087 N53585 0 1 1.0000 7.9502 9.2012 0.2849 132.0000 1.5704 tv 0.0000 0.0000 0.0000 0.0000 0.0000 200.0000 0.0151 200.0000 33.0000 200.0000 0 1 0 1
4 U19087 N25791 0 1 1.0000 5.0938 6.4489 0.2409 175.0000 1.3492 news 1.0000 1.0000 0.0164 0.0164 0.0000 200.0000 0.0201 200.0000 34.0000 200.0000 0 1 0 1 
%%time

# s5_score on SET_B users: base LGB has not seen these users during training (trained on SET_A only), so the meta-ranker learns from true OOF scores.
xmeta_base = meta_train_df[FEATURE_COLS].values.astype('float32')
meta_train_df['s5_score'] = lgb_model.predict(xmeta_base)
del xmeta_base; gc.collect()

xmeta = meta_train_df[STAGE2_FEATURE_COLS].values
ymeta = meta_train_df['label'].values

# Split for training
xm_tr, xm_val, ym_tr, ym_val = train_test_split(xmeta, ymeta, test_size = 0.15, random_state = 100, stratify = ymeta)

meta_lgb_params = {'objective'        : 'binary',
                    'metric'           : 'auc',
                    'learning_rate'    : 0.03,
                    'feature_fraction' : 0.8,
                    'bagging_fraction' : 0.8,
                    'bagging_freq'     : 5,
                    'verbose'          : -1,
                    'n_jobs'           : -1,}

meta_lgb = lgb.train(meta_lgb_params, lgb.Dataset(xm_tr, label=ym_tr), num_boost_round = 800, valid_sets      = [lgb.Dataset(xm_val, label=ym_val)], callbacks       = [lgb.early_stopping(40, verbose=False), lgb.log_evaluation(100)],)

xgb_meta = XGBClassifier(n_estimators          = 1000,
                        learning_rate         = 0.05,
                        max_depth             = 6,
                        subsample             = 0.8,
                        colsample_bytree      = 0.8,
                        eval_metric           = 'auc',
                        early_stopping_rounds = 30,
                        verbosity             = 0,)

xgb_meta.fit(xm_tr, ym_tr, eval_set=[(xm_val, ym_val)], verbose=False)

print(f'Meta-LGB trees : {meta_lgb.num_trees()}')
print(f'Meta-XGB trees : {xgb_meta.best_iteration}')
print(f'STAGE2_FEATURE_COLS ({len(STAGE2_FEATURE_COLS)}): {STAGE2_FEATURE_COLS}')

[100]	valid_0's auc: 1
Meta-LGB trees : 72
Meta-XGB trees : 36
STAGE2_FEATURE_COLS (21): ['u_click_count', 'u_click_freq', 'm_log_clicks', 'm_log_impr', 'm_article_len', 'cat_affinity', 'taste_affinity', 'tfidf_sim', 'recent_tfidf_sim', 'article_age_days', 'ctr_norm_rank', 'imp_size', 'subcat_clicks', 'in_s2', 'in_s3', 'in_s4', 'rank_s2', 'rank_s3', 'rank_s4', 'n_retrievers', 's5_score']
CPU times: user 45.7 s, sys: 141 ms, total: 45.8 s
Wall time: 7.04 s

del xmeta, xm_tr, xm_val, ym_tr, ym_val, meta_train_df; gc.collect()

14

Back to top

10. Full benchmark: S1 → S7

We evaluate all seven strategies on the held-out eval fold. Each strategy is given the same eval_warm users and the same ground-truth sets.

# Run the benchmark to compare all strategies
%%time

strategies = [('S1: Popularity',         s1_score),
              ('S2: Category Affinity',  s2_score),
              ('S3: Item-CF',            s3_score),
              ('S4: Temporal Taste',     s4_score),
              ('S5: LightGBM Base',      s5_score),
              ('S6: Meta-LGB (2-Stage)', s6_score),
              ('S7: Ensemble (LGB + XGB)', s7_score),]

all_results = {}
EVAL_N = min(1000, len(eval_warm))

for name, fn in strategies:

    for K in [5, 10]:

        print(f'  {name}  @K={K}...', end = ' ', flush = True)
        t0 = time.time()
        res = evaluate_strategy(fn, eval_warm, K=K, n=EVAL_N)
        print(f'{time.time()-t0:.0f}s  composite={res["composite"]:.4f}')
        all_results[(name, K)] = res

  S1: Popularity  @K=5... 0s  composite=0.3385
  S1: Popularity  @K=10... 0s  composite=0.4794
  S2: Category Affinity  @K=5... 0s  composite=0.2964
  S2: Category Affinity  @K=10... 0s  composite=0.4295
  S3: Item-CF  @K=5... 0s  composite=0.3049
  S3: Item-CF  @K=10... 0s  composite=0.4344
  S4: Temporal Taste  @K=5... 0s  composite=0.2964
  S4: Temporal Taste  @K=10... 0s  composite=0.4295
  S5: LightGBM Base  @K=5... 21s  composite=0.3942
  S5: LightGBM Base  @K=10... 20s  composite=0.5078
  S6: Meta-LGB (2-Stage)  @K=5... 61s  composite=0.3138
  S6: Meta-LGB (2-Stage)  @K=10... 60s  composite=0.4477
  S7: Ensemble (LGB + XGB)  @K=5... 97s  composite=0.3137
  S7: Ensemble (LGB + XGB)  @K=10... 96s  composite=0.4470
CPU times: user 46min 7s, sys: 1.8 s, total: 46min 9s
Wall time: 5min 56s

# Compile the leaderboard
records = []

for (name, K), res in all_results.items():

    records.append({'strategy': name, 'K': K, **res})

leaderboard = (pd.DataFrame(records).sort_values(['K','composite'], ascending = [True, False]).reset_index(drop = True))

for k_val in [5, 10]:

    print(f'\n{"="*65}')
    print(f'  LEADERBOARD  @  K = {k_val}')
    print('='*65)
    lb = leaderboard[leaderboard['K'] == k_val][['strategy'] + metric_keys + ['composite']]
    print(lb.to_string(index=False))

=================================================================
  LEADERBOARD  @  K = 5
=================================================================
                strategy  precision  recall     f1   ndcg  hit_rate  composite
       S5: LightGBM Base     0.1070  0.4262 0.1647 0.2934    0.4950     0.3942
          S1: Popularity     0.0908  0.3658 0.1407 0.2540    0.4230     0.3385
  S6: Meta-LGB (2-Stage)     0.0836  0.3376 0.1292 0.2335    0.3940     0.3138
S7: Ensemble (LGB + XGB)     0.0836  0.3376 0.1292 0.2335    0.3940     0.3137
             S3: Item-CF     0.0806  0.3317 0.1255 0.2258    0.3840     0.3049
   S2: Category Affinity     0.0780  0.3236 0.1219 0.2197    0.3730     0.2964
      S4: Temporal Taste     0.0780  0.3236 0.1219 0.2197    0.3730     0.2964

=================================================================
  LEADERBOARD  @  K = 10
=================================================================
                strategy  precision  recall     f1   ndcg  hit_rate  composite
       S5: LightGBM Base     0.0782  0.5887 0.1338 0.3506    0.6650     0.5078
          S1: Popularity     0.0719  0.5579 0.1239 0.3209    0.6380     0.4794
  S6: Meta-LGB (2-Stage)     0.0671  0.5247 0.1158 0.2983    0.5970     0.4477
S7: Ensemble (LGB + XGB)     0.0670  0.5242 0.1156 0.2981    0.5960     0.4470
             S3: Item-CF     0.0639  0.5114 0.1110 0.2878    0.5810     0.4344
   S2: Category Affinity     0.0632  0.5059 0.1098 0.2830    0.5760     0.4295
      S4: Temporal Taste     0.0632  0.5059 0.1098 0.2830    0.5760     0.4295

📊 How to Read the Leaderboard

Before the visualisations, here’s the analytical lens to apply:

What to look for What it means
Gap between S1 and S2–S4 Size of personalisation lift — how much history helps vs. pure popularity
S5 vs S2–S4 Value added by learning feature interactions (LambdaMART) over hand-crafted dot-products
S6 vs S5 Value of two-stage architecture: does meta-learning on OOF scores help?
S7 vs S6 Value of model ensembling (LGB + XGB diversity)
K=5 vs K=10 patterns If gains are larger at K=5, the model is especially good at surfacing the single best article — valuable for mobile one-article layouts
NDCG vs HR gap Large HR with low NDCG means the model finds some relevant article in top-K but ranks it poorly; focus tuning on the ranking objective

General expectation: S1 < S2 ≈ S3 ≈ S4 < S5 < S6 ≤ S7. Deviations from this ordering reveal where personalisation is breaking down (e.g. if S3 < S1, the CF graph is too sparse to be useful at this sample size).

Back to top

11. Benchmark visualisations

# Visualize the composite scores comparison
fig, ax = plt.subplots(figsize = (20, 5))
lb10 = leaderboard[leaderboard['K'] == 5].sort_values('composite')
palette = sns.color_palette('husl', len(lb10))
bars = ax.barh(lb10['strategy'], lb10['composite']*100, color=palette)
ax.set_xlabel('Composite score (%) — mean of P@10, R@10, F1@10, NDCG@10, HR@10')
ax.set_title('News Recommendation Benchmark  |  Composite @ K=5')

for bar, val in zip(bars, lb10['composite']):

    ax.text(bar.get_width()+0.1, bar.get_y()+bar.get_height()/2, f'{val*100:.2f}%', va='center', fontsize=9)

plt.tight_layout()
plt.savefig('benchmark_composite.png', dpi=150, bbox_inches='tight')
plt.show()

png

# Visualize the per metric breakdown at 5
lb10   = leaderboard[leaderboard['K'] == 5].set_index('strategy')[metric_keys]
fig, ax = plt.subplots(figsize=(20, 6))
x     = np.arange(len(lb10))
width = 0.15
colors = sns.color_palette('husl', len(metric_keys))

for i, (metric, col) in enumerate(zip(metric_keys, colors)):

    ax.bar(x + i*width, lb10[metric]*100, width, label=metric, color=col, alpha=0.85)

ax.set_xticks(x + width*2)
ax.set_xticklabels(lb10.index, rotation=20, ha='right', fontsize=9)
ax.set_ylabel('Score (%)')
ax.set_title('Per-metric breakdown @ K=10')
ax.legend(loc='upper left', ncol=5)
plt.tight_layout()
plt.savefig('benchmark_per_metric.png', dpi=150, bbox_inches='tight')
plt.show()

png

# Ft imp for metaranker
imp_df = pd.DataFrame({'feature'   : STAGE2_FEATURE_COLS,
                       'importance': meta_lgb.feature_importance(importance_type='gain'),}).sort_values('importance', ascending=False)

fig, ax = plt.subplots(figsize=(10, 6))
sns.barplot(data=imp_df, x='importance', y='feature', palette='viridis', ax=ax)
ax.set_title('Meta-ranker feature importance (gain) — S6 LightGBM')
ax.set_xlabel('Information gain')
plt.tight_layout()
plt.savefig('feature_importance.png', dpi=150, bbox_inches='tight')
plt.show()

print(imp_df.to_string(index = False))

png

         feature   importance
        s5_score 1072311.9034
       tfidf_sim  123980.9660
    n_retrievers   35544.2821
recent_tfidf_sim   20091.2337
           in_s3    3077.7619
         rank_s2    1791.9725
   u_click_count    1033.4886
    m_log_clicks     612.8205
      m_log_impr     486.6427
           in_s2     406.9860
   ctr_norm_rank     231.7343
         rank_s3     176.9706
article_age_days     160.2546
   subcat_clicks     128.9271
           in_s4      90.8844
   m_article_len      54.0000
    u_click_freq      33.2342
    cat_affinity      24.5358
  taste_affinity      18.8506
        imp_size      15.1181
         rank_s4       1.2116

🔍 Reading the meta-ranker feature importance:

Feature importance by gain measures how much each feature reduces the ranking loss on average when it is used as a split. High-gain features are the model’s primary decision levers.

What a healthy importance distribution looks like for this pipeline:

Expected rank Feature Why
1–2 tfidf_sim or s5_score Content relevance and base LGB scores are the strongest signals
3–4 cat_affinity / taste_affinity Category preference is reliable for warm users
5–6 m_log_clicks / bayesian_ctr Popularity has broad coverage
7–9 n_retrievers / retriever rank flags Ensemble metadata (how many retrievers agreed)
Low u_click_freq / active_days User engagement features are useful but secondary

If s5_score ranks #1 by a wide margin, it suggests the meta-ranker is largely distilling the base LGB rather than learning genuinely new patterns — consider adding features that the base LGB cannot see (e.g. session-level context, recency of the article relative to the session).

Back to top

12. Leaderboard & takeaways

# Master leaderboard printout
for k_val in [5, 10]:

    print(f'{"="*70}')
    print(f'  LEADERBOARD  @  K = {k_val}')
    print('='*70)
    lb = leaderboard[leaderboard['K'] == k_val].copy()
    lb[metric_keys + ['composite']] *= 100
    print(lb[['strategy'] + metric_keys + ['composite']].to_string(index=False, float_format='%.2f'))
    print()


======================================================================
  LEADERBOARD  @  K = 5
======================================================================
                strategy  precision  recall    f1  ndcg  hit_rate  composite
       S5: LightGBM Base      10.70   42.62 16.47 29.34     49.50      39.42
          S1: Popularity       9.08   36.58 14.07 25.40     42.30      33.85
  S6: Meta-LGB (2-Stage)       8.36   33.76 12.92 23.35     39.40      31.38
S7: Ensemble (LGB + XGB)       8.36   33.76 12.92 23.35     39.40      31.37
             S3: Item-CF       8.06   33.17 12.55 22.58     38.40      30.49
   S2: Category Affinity       7.80   32.36 12.19 21.97     37.30      29.64
      S4: Temporal Taste       7.80   32.36 12.19 21.97     37.30      29.64

======================================================================
  LEADERBOARD  @  K = 10
======================================================================
                strategy  precision  recall    f1  ndcg  hit_rate  composite
       S5: LightGBM Base       7.82   58.87 13.38 35.06     66.50      50.78
          S1: Popularity       7.19   55.79 12.39 32.09     63.80      47.94
  S6: Meta-LGB (2-Stage)       6.71   52.47 11.58 29.83     59.70      44.77
S7: Ensemble (LGB + XGB)       6.70   52.42 11.56 29.81     59.60      44.70
             S3: Item-CF       6.39   51.14 11.10 28.78     58.10      43.44
   S2: Category Affinity       6.32   50.59 10.98 28.30     57.60      42.95
      S4: Temporal Taste       6.32   50.59 10.98 28.30     57.60      42.95

# Lift metrics
for K in [5, 10]:

    base = all_results[('S1: Popularity', K)]['composite']
    best = all_results[('S6: Meta-LGB (2-Stage)', K)]['composite']
    lift = (best - base) / base * 100
    print(f'K={K}: S1 composite={base*100:.2f}%  →  S6={best*100:.2f}%  '
          f'(+ {lift:.1f}% relative lift)')

K=5: S1 composite=33.85%  →  S6=31.38%  (+ -7.3% relative lift)
K=10: S1 composite=47.94%  →  S6=44.77%  (+ -6.6% relative lift)

Back to top..

🏆 Key takeaways

What Was Built

MIND-small Dataset (160K users, 65K articles, 1M+ impressions)
         │
         ▼
Feature Engineering ──► user_stats · article_feat · category affinity · TF-IDF centroids
         │
         ├──► Stage 1 Retrieval ──► 200-candidate pool (4 complementary retrievers)
         │
         └──► Stage 2 Reranking ──► Base LGB (LambdaMART) + Meta-LGB + XGB Ensemble
                                            │
                                            ▼
                              S1→S7 Leaderboard (NDCG · HR · P · R · F1 @ K=5,10)

Design Decisions Recap

Decision Alternative Trade-off
Two-stage generate & rerank Single-stage end-to-end Lower inference cost; established industry standard
LambdaMART objective BPR / pointwise logistic Directly optimises NDCG; needs query groups
Bayesian CTR smoothing Raw CTR Prevents low-impression articles from appearing falsely viral
Per-impression evaluation Global ranking evaluation Matches deployment; prevents popularity dominance
OOF split for meta-ranker In-sample scoring Prevents leakage; gives honest meta-feature estimates
7-day decay half-life Fixed window Smoother than hard cutoffs; tunable to domain

Potential Extensions

  1. Neural text encoder — Replace TF-IDF centroids with a fine-tuned BERT/DistilBERT news encoder (e.g. the NAML or NRMS architectures from the MIND paper) for richer semantic representations.
  2. Session context — Add within-session features: position of the candidate in the impression list, time since last click, number of articles already clicked in this session.
  3. Graph-based CF — Use LightGCN or PinSage over the user–article bipartite graph for higher-quality embeddings, especially for sparse users.
  4. Online evaluation — A/B test against a production system; offline NDCG gains do not always translate 1:1 to online CTR improvements.
  5. Diversity regularisation — Add a category-diversity penalty to the final top-K selection to avoid filter bubbles (e.g. maximum marginal relevance).
  6. Freshness feature — Articles less than 1 hour old should receive a freshness bonus; MIND’s fixed 6-week window masks this but it matters in production.

Cite MIND: Fangzhao Wu et al. (2020), “MIND: A Large-scale Dataset for News Recommendation”, ACL 2020. Dataset: https://msnews.github.io/ —

Back to top..

]]>Data Guydataguy50@gmail.comMovieLens User Data: Machine Learning Recommendation Engine2026-03-31T00:00:00+00:002026-03-31T00:00:00+00:00https://doctorofdata.github.io/posts/2026/3/movielens-machine-learning-recommendation-engineIn this post, we take the MovieLens 32M User data and build a series of recommendation algorithms that are then combined to yield a master ensemble strategy which ens up winning in terms of performance metrics scores

🎬 MovieLens 32M — Two-Stage Generate-&-Rerank Recommendation Ensemble Model to Generate Recommendations

Data: Download ml-32m.zip from https://grouplens.org/datasets/movielens/32m/, mount your Google Drive, and point ZIP_PATH to the archive. All other paths are derived automatically.

In the last study we examined five standalone recommendation strategies applied to InstaCart shopping data.

Here we assess a combined ensemble strategy — S6 — that draws on all five priors simultaneously to produce more accurate top-N movie recommendations.

📋 Table of Contents

# Section
1 Setup & data loading
2 Exploratory data analysis
3 Feature engineering
4 Item-based collaborative filtering
5 Temporal sequence modelling
6 Evaluation harness & S1–S5 strategies
7 S6 architecture & cold-start gate
8 Stage 1 — Expanded candidate pool
9 Stage 2 — Meta-ranker training
10 Full benchmark: S1 → S7
11 Visualisations
12 Leaderboard & takeaways

1. Setup & data loading

📖 Dataset & Problem Framing

What the data is. The MovieLens 32M dataset contains 32,000,204 explicit 5-star ratings (in 0.5-star increments) and 2,000,072 tag applications across 87,585 movies by 200,948 users, collected between January 1995 and October 2023.

File Key columns Role
ratings.csv userId, movieId, rating, timestamp Primary signal
movies.csv movieId, title, genres Item metadata
tags.csv userId, movieId, tag, timestamp Auxiliary text signal
links.csv movieId, imdbId, tmdbId External enrichment (optional)

Task framing. Given a user’s historical rating behaviour, predict which movies they will rate highly (≥ 4.0 stars) that they have not yet seen. Ratings below 4.0 are treated as implicit negatives; ≥ 4.0 as positives.

Train/test split strategy. For each user, all ratings before the 90th-percentile timestamp are training data; ratings at or after form the ground truth. This preserves the causal structure and avoids leakage.

Memory strategy. Downcast userId / movieId to int32, rating to float32. Free large intermediates with del + gc.collect() after their last usage point.

# Import libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import zipfile, gc, time, warnings, os
from collections import defaultdict, Counter
from itertools import combinations
import lightgbm as lgb
from xgboost import XGBClassifier
from sklearn.model_selection import GroupShuffleSplit, train_test_split
from sklearn.metrics import roc_auc_score, average_precision_score
from scipy.sparse import csr_matrix
from sklearn.preprocessing import normalize
from google.colab import drive
from matplotlib.gridspec import GridSpec
from joblib import Parallel, delayed

warnings.filterwarnings('ignore')
plt.style.use('seaborn-v0_8-whitegrid')
sns.set_palette('husl')
pd.set_option('display.float_format', '{:.4f}'.format)
pd.set_option('display.max_columns', None)
print('✅  Libraries loaded')

✅  Libraries loaded

# Read the data
drive.mount('/content/drive/')

Mounted at /content/drive/

# Define functions to compute metrics
def precision_at_k(recs, true_set, k):

    return len(set(recs[:k]) & true_set) / k if k else 0.0

def recall_at_k(recs, true_set, k):

    return len(set(recs[:k]) & true_set) / len(true_set) if true_set else 0.0

def f1_at_k(recs, true_set, k):

    p = precision_at_k(recs, true_set, k)
    r = recall_at_k(recs, true_set, k)

    return 2*p*r/(p+r) if (p+r) > 0 else 0.0

def ndcg_at_k(recs, true_set, k):

    dcg   = sum(1/np.log2(i+2) for i, m in enumerate(recs[:k]) if m in true_set)
    ideal = sum(1/np.log2(i+2) for i in range(min(len(true_set), k)))

    return dcg/ideal if ideal else 0.0

def score_recs(recs, true_set, K):

    return {
        'precision': precision_at_k(recs, true_set, K),
        'recall'   : recall_at_k(recs,    true_set, K),
        'f1'       : f1_at_k(recs,        true_set, K),
        'ndcg'     : ndcg_at_k(recs,      true_set, K),
        'hit_rate' : 1 if any(m in true_set for m in recs[:K]) else 0,
    }

def evaluate_strategy(fn, data_df, K=10, n=None, seed=42):

    rows = data_df if n is None else data_df.sample(n=n, random_state=seed)
    m = {k: [] for k in ('precision','recall','f1','ndcg','hit_rate')}

    for _, row in rows.iterrows():

        s = score_recs(fn(row['userId'], K=K), row['true_items'], K)

        for key in m: m[key].append(s[key])

    return {k: float(np.mean(v)) for k, v in m.items()}

# ── Lookup caches ─────────────────────────────────────────────────────────
# _seen_cache  : user_id -> set of seen movie IDs
# _s2_score_cache : user_id -> precomputed S2 genre score vector (n_movies,)
# _s4_score_cache : user_id -> precomputed S4 taste score vector (n_movies,)
#
# All three are initialised empty here and populated before they are needed.
# When a score cache is empty (e.g. during training pair construction, which
# uses its own batch arrays), the functions fall back to computing on the fly.
_seen_cache     = {}
_s2_score_cache = {}
_s4_score_cache = {}

def get_seen(user_id):

    return _seen_cache.get(user_id, set())

# Function to compute global popularity
def s1_popularity(user_id, K=10, **_):

    seen = get_seen(user_id)

    return [m for m in POPULARITY_POOL if m not in seen][:K]

# Function to compute genre-aware personal history
def s2_personal(user_id, K=10, **_):

    seen = get_seen(user_id)

    if user_id not in user_genre_affinity.index:

        return s1_popularity(user_id, K=K)

    # Use precomputed score vector when available (evaluation path); fall back to on-the-fly matmul otherwise (any ad-hoc call).
    if user_id in _s2_score_cache:

        scores = _s2_score_cache[user_id]

    else:

        u_genre = user_genre_affinity.loc[user_id].values
        u_norm  = np.linalg.norm(u_genre)

        if u_norm == 0:

            return s1_popularity(user_id, K=K)

        scores = movie_genre_norm @ (u_genre / u_norm)

    top_k  = np.argpartition(scores, -K)[-K:]
    top_k  = top_k[np.argsort(scores[top_k])[::-1]]
    ranked = [genre_index_order[i] for i in top_k if genre_index_order[i] not in seen]

    if len(ranked) >= K:

        return ranked[:K]

    # Pad with additional candidates if top-K after filtering is short
    all_ranked = [genre_index_order[i] for i in np.argsort(scores)[::-1] if genre_index_order[i] not in seen]

    return all_ranked[:K]

# Function to compute collaborative filtering
def s3_item_cf(user_id, K=10, **_):

    seen  = get_seen(user_id)
    liked = set(user_liked.get(user_id, []))

    if not liked:

        return s1_popularity(user_id, K=K)

    scores = defaultdict(float)

    for mid in liked:

        for neighbour, sim in item_sim_lookup.get(mid, []):

            if neighbour not in seen:

                scores[neighbour] += sim
    ranked = sorted(scores, key=scores.get, reverse=True)

    if len(ranked) >= K:

        return ranked[:K]

    extra = [m for m in s2_personal(user_id, K=K*3) if m not in set(ranked)]
    return (ranked + extra)[:K]

# Function to compute temporal taste drift
def s4_temporal(user_id, K=10, **_):

    seen = get_seen(user_id)
    # Use precomputed score vector when available (evaluation path); fall back to on-the-fly matmul otherwise.
    if user_id in _s4_score_cache:

        sims = _s4_score_cache[user_id]

    else:

        taste_vec = user_taste_vec.get(user_id)

        if taste_vec is None:

            return s1_popularity(user_id, K=K)

        sims = movie_genre_norm @ taste_vec

    top_k  = np.argpartition(sims, -K)[-K:]
    top_k  = top_k[np.argsort(sims[top_k])[::-1]]
    ranked = [genre_index_order[i] for i in top_k if genre_index_order[i] not in seen]

    if len(ranked) >= K:

        return ranked[:K]

    all_ranked = [genre_index_order[i] for i in np.argsort(sims)[::-1] if genre_index_order[i] not in seen]

    return all_ranked[:K]

# Function to build features
def build_lgb_features(user_id, candidate_movies):

    if not candidate_movies:

        return pd.DataFrame(columns = FEATURE_COLS)

    mf = movie_feat.reindex(candidate_movies)
    mg = movie_genre.reindex(candidate_movies).fillna(0).values.astype('float32')

    mg_norms = np.linalg.norm(mg, axis=1, keepdims=True).clip(min=1e-9)
    mg_norm  = mg / mg_norms

    taste_v        = user_taste_vec.get(user_id, np.zeros(len(genre_cols), dtype='float32'))
    genre_affinity = mg_norm @ taste_v

    u_row = user_stats.loc[user_id] if user_id in user_stats.index else None

    return pd.DataFrame({'userId'            : user_id,
                         'movieId'           : candidate_movies,
                         'u_rating_count'    : float(u_row['rating_count'])   if u_row is not None else 0.0,
                         'u_mean_rating'     : float(u_row['mean_rating'])    if u_row is not None else GLOBAL_MEAN,
                         'u_rating_freq'     : float(u_row['rating_freq'])    if u_row is not None else 0.0,
                         'u_rating_std'      : float(u_row['rating_std'])     if u_row is not None else 0.0,
                         'm_global_count'    : mf['global_count'].fillna(0).values,
                         'm_log_count'       : mf['log_count'].fillna(0).values,
                         'm_global_mean'     : mf['global_mean'].fillna(GLOBAL_MEAN).values,
                         'm_bayesian'        : mf['bayesian_score'].fillna(GLOBAL_MEAN).values,
                         'm_rating_variance' : mf['rating_variance'].fillna(0).values,
                         'm_tag_count'       : mf['tag_count'].fillna(0).values,
                         'genre_affinity'    : genre_affinity,})

# Function to perform lgb recommendations
def s5_lgbm(user_id, K = 10, n_candidates = 200, **_):

    seen = get_seen(user_id)
    candidates = list(dict.fromkeys(s2_personal(user_id, K = n_candidates) + s3_item_cf( user_id, K = n_candidates) + s4_temporal(user_id, K=n_candidates)))

    candidates = [m for m in candidates if m not in seen][:n_candidates]

    if not candidates:

        return s1_popularity(user_id, K=K)

    feat_df = build_lgb_features(user_id, candidates)
    feat_df['score'] = lgb_model.predict(feat_df[FEATURE_COLS])

    return feat_df.sort_values('score', ascending = False)['movieId'].tolist()[:K]

# Function to build second-stage features
def build_stage2_features(user_id, candidates, s2_list, s3_list, s4_list):

    s2r = {m: r for r, m in enumerate(s2_list)}
    s3r = {m: r for r, m in enumerate(s3_list)}
    s4r = {m: r for r, m in enumerate(s4_list)}

    base = build_lgb_features(user_id, candidates)

    base['in_s2']       = base['movieId'].isin(s2r).astype('int8')
    base['in_s3']       = base['movieId'].isin(s3r).astype('int8')
    base['in_s4']       = base['movieId'].isin(s4r).astype('int8')
    base['rank_s2']     = base['movieId'].map(s2r).fillna(N_STAGE1).astype('int16')
    base['rank_s3']     = base['movieId'].map(s3r).fillna(N_STAGE1).astype('int16')
    base['rank_s4']     = base['movieId'].map(s4r).fillna(N_STAGE1).astype('int16')
    base['n_retrievers']= base[['in_s2','in_s3','in_s4']].sum(axis=1).astype('int8')
    base['s5_score']    = lgb_model.predict(base[FEATURE_COLS]).astype('float32')

    return base

# Create a check to see if there is user history
def is_cold(user_id):

    return len(user_liked.get(user_id, [])) < COLD_START_THRESHOLD

# Function to get stage 1 candidates
def stage1_candidates(user_id):

    seen = get_seen(user_id)
    pool = list(dict.fromkeys(s2_personal(user_id, K=N_STAGE1) + s3_item_cf( user_id, K=N_STAGE1) + s4_temporal(user_id, K = N_STAGE1)))

    return [m for m in pool if m not in seen]

# Function to get ensemble recommendations
def s7_ensemble(user_id, K = 10, model = 'lgb', **_):

    if is_cold(user_id):

        return s1_popularity(user_id, K = K)

    s2l = s2_personal(user_id, K=N_STAGE1)
    s3l = s3_item_cf( user_id, K=N_STAGE1)
    s4l = s4_temporal(user_id, K=N_STAGE1)
    candidates = list(dict.fromkeys(s2l + s3l + s4l))

    if not candidates:

        return s1_popularity(user_id, K = K)

    feat = build_stage2_features(user_id, candidates, s2l, s3l, s4l)

    if model == 'lgb':

        feat['score'] = meta_lgb.predict(feat[STAGE2_FEATURE_COLS])

    else:

        feat['score'] = meta_xgb.predict_proba(feat[STAGE2_FEATURE_COLS])[:, 1]

    return feat.sort_values('score', ascending=False)['movieId'].tolist()[:K]

def s6_xgboost(user_id, K = 10, **_):

    return s7_ensemble(user_id, K = K, model = 'xgb')



# Load the data
data = {}

files = [i for i in  os.listdir('/content/drive/MyDrive/ml-32m/')]
file_list = [f for f in files if f.endswith('.csv')]
print(f'Files in archive: {file_list}\n')

for fn in file_list:

    key = fn.replace('.csv', '')
    data[key] = pd.read_csv(f"/content/drive/MyDrive/ml-32m/{fn}")
    print(f'  ✓ {fn}: {data[key].shape[0]:,} rows × {data[key].shape[1]} cols')

print('\n✅  All datasets loaded')

Files in archive: ['ratings.csv', 'tags.csv', 'movies.csv', 'links.csv']

  ✓ ratings.csv: 32,000,204 rows × 4 cols
  ✓ tags.csv: 2,000,072 rows × 4 cols
  ✓ movies.csv: 87,585 rows × 3 cols
  ✓ links.csv: 87,585 rows × 3 cols

✅  All datasets loaded

# Downcast to save ~40% RAM
for col in ['userId', 'movieId']:

    data['ratings'][col] = data['ratings'][col].astype('int32')
    data['tags'][col]    = data['tags'][col].astype('int32')

data['movies']['movieId'] = data['movies']['movieId'].astype('int32')
data['ratings']['rating'] = data['ratings']['rating'].astype('float32')

# Extract the data into objects
ratings = data['ratings']
movies = data['movies']
tags = data['tags']

movies.head()
movieId title genres
0 1 Toy Story (1995) Adventure|Animation|Children|Comedy|Fantasy
1 2 Jumanji (1995) Adventure|Children|Fantasy
2 3 Grumpier Old Men (1995) Comedy|Romance
3 4 Waiting to Exhale (1995) Comedy|Drama|Romance
4 5 Father of the Bride Part II (1995) Comedy 
tags.head()
userId movieId tag timestamp
0 22 26479 Kevin Kline 1583038886
1 22 79592 misogyny 1581476297
2 22 247150 acrophobia 1622483469
3 34 2174 music 1249808064
4 34 2174 weird 1249808102 
ratings.head()
userId movieId rating timestamp
0 1 17 4.0000 944249077
1 1 25 1.0000 944250228
2 1 29 2.0000 943230976
3 1 30 5.0000 944249077
4 1 32 5.0000 943228858 
%%time

# Create a temporal training split for modelling
RATING_THRESHOLD = 4.0

# Cutoff users based on time interval, 90%
user_cutoff = (ratings.groupby('userId')['timestamp'].quantile(0.90).rename('cutoff').astype('int32'))
ratings = ratings.join(user_cutoff, on = 'userId')

# Split the data for training
train_ratings = ratings[ratings['timestamp'] <  ratings['cutoff']]
test_ratings  = ratings[ratings['timestamp'] >= ratings['cutoff']]

# Drop cutoff column immediately - saves ~100 MB across both halves
train_ratings.drop(columns = ['cutoff'], inplace = True)
test_ratings.drop(columns = ['cutoff'], inplace = True)

del ratings, user_cutoff; gc.collect()

# Ground truth: movies rated >= threshold in test, not seen in training
train_seen = train_ratings.groupby('userId')['movieId'].apply(set)

# Function to extract the truth
def get_true_items(grp):

    uid   = grp.name
    liked = set(grp[grp['rating'] >= RATING_THRESHOLD]['movieId'])

    return liked - train_seen.get(uid, set())

# Apply func to get data
ground_truth = (test_ratings.groupby('userId').apply(get_true_items).rename('true_movies'))
ground_truth = ground_truth[ground_truth.map(len) > 0]

# Inspect
print(f'Train rows : {len(train_ratings):,}  | {train_ratings.memory_usage(deep = True).sum()/1e6:.0f} MB')
print(f'Test rows  : {len(test_ratings):,}   | {test_ratings.memory_usage(deep = True).sum()/1e6:.0f} MB')
print(f'Eval users : {len(ground_truth):,}')

Train rows : 28,583,599  | 800 MB
Test rows  : 3,416,605   | 96 MB
Eval users : 192,131
CPU times: user 1min 31s, sys: 3.75 s, total: 1min 35s
Wall time: 1min 36s

ground_truth.head()
true_movies
userId
1 {1056, 1952, 2020, 645, 2313, 1228, 2125, 302,...
2 {520, 616, 362, 783, 594}
3 {4995, 5380, 1957, 2150, 11, 3248, 1873, 2353,...
4 {2841}
5 {480, 356, 454, 364, 110} 


# Create a global popularity pool to use as a cold-start ratings guide
pop_stats = (train_ratings[train_ratings['rating'] >= RATING_THRESHOLD].groupby('movieId').agg(pos_count = ('rating','count'), mean_rating = ('rating','mean')).sort_values(['pos_count','mean_rating'], ascending = False))
POPULARITY_POOL = pop_stats.index.tolist()
print(f'Popularity pool: {len(POPULARITY_POOL):,} movies')
pop_stats.head(10).join(movies.set_index('movieId')['title'])

Popularity pool: 50,644 movies
pos_count mean_rating title
movieId
318 84861 4.6437 Shawshank Redemption, The (1994)
296 73438 4.6011 Pulp Fiction (1994)
2571 69017 4.5596 Matrix, The (1999)
356 68815 4.5235 Forrest Gump (1994)
593 67138 4.5071 Silence of the Lambs, The (1991)
260 60648 4.5635 Star Wars: Episode IV - A New Hope (1977)
2959 58522 4.5773 Fight Club (1999)
527 55945 4.5847 Schindler's List (1993)
1196 52583 4.5564 Star Wars: Episode V - The Empire Strikes Back...
4993 51969 4.5687 Lord of the Rings: The Fellowship of the Ring,... 
# Create a global popularity pool
GLOBAL_MEAN = float(train_ratings['rating'].mean())

pop_stats = (train_ratings[train_ratings['rating'] >= RATING_THRESHOLD].groupby('movieId').agg(pos_count = ('rating','count'), mean_rating = ('rating','mean')).sort_values(['pos_count','mean_rating'], ascending = False))
POPULARITY_POOL = pop_stats.index.tolist()

print(f'Global mean       : {GLOBAL_MEAN:.4f}')
print(f'Popularity pool   : {len(POPULARITY_POOL):,} movies')
pop_stats.head(8).join(movies.set_index('movieId')['title'])

Global mean       : 3.5460
Popularity pool   : 50,644 movies
pos_count mean_rating title
movieId
318 84861 4.6437 Shawshank Redemption, The (1994)
296 73438 4.6011 Pulp Fiction (1994)
2571 69017 4.5596 Matrix, The (1999)
356 68815 4.5235 Forrest Gump (1994)
593 67138 4.5071 Silence of the Lambs, The (1991)
260 60648 4.5635 Star Wars: Episode IV - A New Hope (1977)
2959 58522 4.5773 Fight Club (1999)
527 55945 4.5847 Schindler's List (1993)

Back to top..

2. Exploratory data analysis

📖 Understanding the data before modelling

This section answers eight diagnostic questions that directly motivate modelling decisions later:

Question Modelling implication
How are ratings distributed across the 0.5–5.0 scale? Validates our 4.0 ‘liked’ threshold
How many ratings does a typical user have? Informs cold-start threshold
How many ratings does a typical movie have? Informs Bayesian smoothing prior
Which genres dominate the catalogue? Explains genre-affinity feature range
How do mean ratings differ across genres? Reveals genre-level rating bias
How has rating volume changed over time? Motivates temporal recency weighting
How concentrated is movie popularity? Reveals long-tail severity
What do user rating habits look like? Identifies lenient vs strict raters 
# Combine train + test for full-dataset EDA stats
n_tr, n_te    = len(train_ratings), len(test_ratings)
total_ratings = n_tr + n_te

# Union of user/movie IDs across splits without creating a combined DataFrame
users_tr  = set(train_ratings['userId'].unique())
users_te  = set(test_ratings['userId'].unique())
movies_tr = set(train_ratings['movieId'].unique())
movies_te = set(test_ratings['movieId'].unique())

n_users_total  = len(users_tr | users_te)
n_movies_rated = len(movies_tr | movies_te)
del users_tr, users_te, movies_tr, movies_te; gc.collect()

global_sum = float(train_ratings['rating'].sum()) + float(test_ratings['rating'].sum())
global_mean_all = global_sum / total_ratings
ts_min = min(int(train_ratings['timestamp'].min()), int(test_ratings['timestamp'].min()))
ts_max = max(int(train_ratings['timestamp'].max()), int(test_ratings['timestamp'].max()))
r_min  = min(float(train_ratings['rating'].min()), float(test_ratings['rating'].min()))
r_max  = max(float(train_ratings['rating'].max()), float(test_ratings['rating'].max()))
sparsity = 1 - total_ratings / (n_users_total * n_movies_rated)

print('=' * 55)
print('  MOVIELENS 32M  -  DATASET SNAPSHOT')
print('=' * 55)
print(f'  Total ratings      : {total_ratings:>12,}')
print(f'  Unique users       : {n_users_total:>12,}')
print(f'  Unique movies rated: {n_movies_rated:>12,}')
print(f'  Movies in catalogue: {len(movies):>12,}')
print(f'  Date range         : {pd.to_datetime(ts_min,unit="s").date()} -> {pd.to_datetime(ts_max,unit="s").date()}')
print(f'  Rating scale       : {r_min} - {r_max} stars')
print(f'  Mean rating        : {global_mean_all:.3f}')
print(f'  Matrix sparsity    : {sparsity:.4%}')
print('=' * 55)

=======================================================
  MOVIELENS 32M  -  DATASET SNAPSHOT
=======================================================
  Total ratings      :   32,000,204
  Unique users       :      200,948
  Unique movies rated:       84,432
  Movies in catalogue:       87,585
  Date range         : 1995-01-09 -> 2023-10-13
  Rating scale       : 0.5 - 5.0 stars
  Mean rating        : 3.540
  Matrix sparsity    : 99.8114%
=======================================================

# Combine data for vis
all_ratings_eda = pd.concat([train_ratings.assign(split='train'),test_ratings.assign(split='test')], ignore_index = True)
all_ratings_eda['date'] = pd.to_datetime(all_ratings_eda['timestamp'], unit='s')
all_ratings_eda['year'] = all_ratings_eda['date'].dt.year
all_ratings_eda['year_month'] = all_ratings_eda['date'].dt.to_period('M')
print(f'all_ratings_eda: {all_ratings_eda.shape}  |  '
      f'{all_ratings_eda.memory_usage(deep=True).sum()/1e6:.0f} MB')

all_ratings_eda: (32000204, 8)  |  3005 MB

# Ratings distribution
fig, axes = plt.subplots(1, 2, figsize=(20, 5))

# Left: count per star value
ax = axes[0]
rating_counts = all_ratings_eda['rating'].value_counts().sort_index()
bars = ax.bar(rating_counts.index, rating_counts.values,
              width=0.35, color=sns.color_palette('husl', len(rating_counts)))
ax.axvline(4.0, color='crimson', lw=2, ls='--', label='Liked threshold (4.0)')
ax.set_xlabel('Star Rating')
ax.set_ylabel('Number of Ratings')
ax.set_title('Rating Distribution (raw counts)')
ax.legend()
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda x, _: f'{x/1e6:.1f}M'))
for bar in bars:
    h = bar.get_height()
    ax.text(bar.get_x()+bar.get_width()/2, h*1.01,
            f'{h/1e6:.1f}M', ha='center', va='bottom', fontsize=7)

# Right: positive label rate vs threshold
ax2 = axes[1]
thresholds = np.arange(0.5, 5.5, 0.5)
pct_liked  = [(all_ratings_eda['rating'] >= t).mean() * 100 for t in thresholds]
ax2.plot(thresholds, pct_liked, marker='o', lw=2.5, color='steelblue')
ax2.axvline(4.0, color='crimson', lw=2, ls='--', label='Our threshold (4.0)')
chosen_pct = pct_liked[int((4.0-0.5)/0.5)]
ax2.axhline(chosen_pct, color='crimson', lw=1, ls=':')
ax2.annotate(f'{chosen_pct:.1f}% liked at 4.0',
             xy = (4.0, chosen_pct),
             xytext = (2.5, chosen_pct-16),
             arrowprops = dict(arrowstyle='->', color = 'crimson'),
             color = 'crimson',
             fontsize = 10)

ax2.set_xlabel('Threshold')
ax2.set_ylabel('% of Ratings Classified as Liked')
ax2.set_title('Positive Label Rate vs Threshold')
ax2.legend()
ax2.set_ylim(0, 105)

plt.suptitle('Rating Scale Analysis', fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig('eda_01_rating_distribution.png', dpi=150, bbox_inches='tight')
plt.show()
print(f'Ratings >= 4.0 stars: {(all_ratings_eda["rating"]>=4.0).mean():.1%} of all ratings')

png

Ratings >= 4.0 stars: 49.8% of all ratings

# User actvitity distribution
user_rating_counts = all_ratings_eda.groupby('userId')['rating'].count()

fig, axes = plt.subplots(1, 3, figsize=(20, 4))

# Histogram (log y)
ax = axes[0]
ax.hist(user_rating_counts, bins=80, color='steelblue', edgecolor='white', lw=0.3)
ax.set_yscale('log')
ax.set_xlabel('Ratings per User')
ax.set_ylabel('Number of Users (log)')
ax.set_title('User Activity Distribution')

for p in [10, 50, 90]:

    v = np.percentile(user_rating_counts, p)
    ax.axvline(v, ls='--', lw=1.2, label=f'p{p}={v:.0f}')

ax.legend(fontsize=8)

# CCDF — power-law tail
ax2 = axes[1]
sorted_counts = np.sort(user_rating_counts)[::-1]
rank = np.arange(1, len(sorted_counts)+1)
ax2.loglog(sorted_counts, rank / len(sorted_counts), lw=2, color='darkorange')
ax2.set_xlabel('Ratings per User (log)')
ax2.set_ylabel('P(X >= x)  [log]')
ax2.set_title('CCDF — Heavy Tail in User Activity')
ax2.grid(True, which='both', alpha=0.3)

# Activity segments
ax3 = axes[2]
bins_   = [0, 20, 50, 100, 250, 500, int(user_rating_counts.max())+1]
labels_ = ['<20', '20-50', '51-100', '101-250', '251-500', '500+']
seg    = pd.cut(user_rating_counts, bins=bins_, labels=labels_)
seg_ct = seg.value_counts().sort_index()
bars3  = ax3.bar(range(len(seg_ct)), seg_ct.values,
                 color=sns.color_palette('husl', len(seg_ct)),
                 tick_label=labels_)
ax3.set_xlabel('Ratings per User')
ax3.set_ylabel('Number of Users')
ax3.set_title('User Activity Segments')

for i, v in enumerate(seg_ct.values):

    ax3.text(i, v + 200, f'{v:,}', ha='center', va='bottom', fontsize=8)

plt.suptitle('User Activity Analysis', fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig('eda_02_user_activity.png', dpi=150, bbox_inches='tight')
plt.show()

print(f'Median ratings/user  : {user_rating_counts.median():.0f}')
print(f'Mean   ratings/user  : {user_rating_counts.mean():.0f}')
top1 = user_rating_counts.nlargest(int(len(user_rating_counts)*0.01)).sum()
print(f'Top 1% of users account for {top1/user_rating_counts.sum():.1%} of all ratings')

png

Median ratings/user  : 73
Mean   ratings/user  : 159
Top 1% of users account for 12.6% of all ratings

# Investigate movie popularities
movie_rating_counts = (all_ratings_eda.groupby('movieId')['rating'].count().sort_values(ascending=False))
movie_pop_named = (movie_rating_counts.to_frame('n_ratings').join(movies.set_index('movieId')['title']))

fig, axes = plt.subplots(1, 2, figsize=(20, 5))

# Log-log rank curve
ax = axes[0]
ranks = np.arange(1, len(movie_rating_counts)+1)
ax.loglog(ranks, movie_rating_counts.values, lw=2, color='steelblue')
ax.fill_between(ranks, movie_rating_counts.values, alpha=0.15, color='steelblue')
top1pct = int(len(movie_rating_counts)*0.01)
ax.axvline(top1pct, color='crimson', ls='--', lw=1.5,
           label=f'Top 1% = {top1pct:,} movies')
ax.set_xlabel('Movie Rank (log)')
ax.set_ylabel('Number of Ratings (log)')
ax.set_title('The Long Tail: Movie Popularity')
ax.legend(fontsize=9)

# Rating concentration curve
ax2 = axes[1]
cum_pct = movie_rating_counts.cumsum() / movie_rating_counts.sum() * 100
ax2.plot(range(len(cum_pct)), cum_pct.values, lw=2, color='darkorange')
for target in [50, 80, 95]:
    idx_pos = int((cum_pct >= target).argmax())
    ax2.axhline(target, ls=':', color='grey', lw=1)
    ax2.axvline(idx_pos, ls=':', color='grey', lw=1)
    ax2.text(idx_pos + 500, target - 4, f'{target}%: {idx_pos:,} movies',
             fontsize=8, color='dimgrey')
ax2.set_xlabel('Number of Movies (ranked by popularity)')
ax2.set_ylabel('Cumulative % of All Ratings')
ax2.set_title('Rating Concentration Curve')

plt.suptitle('Movie Popularity Analysis', fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig('eda_03_movie_longtail.png', dpi=150, bbox_inches='tight')
plt.show()

print('Top 10 most-rated movies:')
print(movie_pop_named.head(10).to_string())

png

Top 10 most-rated movies:
         n_ratings                                                      title
movieId                                                                      
318         102929                           Shawshank Redemption, The (1994)
356         100296                                        Forrest Gump (1994)
296          98409                                        Pulp Fiction (1994)
2571         93808                                         Matrix, The (1999)
593          90330                           Silence of the Lambs, The (1991)
260          85010                  Star Wars: Episode IV - A New Hope (1977)
2959         77332                                          Fight Club (1999)
480          75233                                       Jurassic Park (1993)
527          73849                                    Schindler's List (1993)
4993         73122  Lord of the Rings: The Fellowship of the Ring, The (2001)

%%time

# Analyze the genres
movies_exploded = (movies.assign(genre=movies['genres'].str.split('|')).explode('genre').query("genre != '(no genres listed)'"))
genre_ratings = (all_ratings_eda.merge(movies_exploded[['movieId','genre']], on='movieId', how='left').dropna(subset=['genre']))

genre_stats = genre_ratings.groupby('genre').agg(n_ratings     = ('rating','count'),
                                                 mean_rating   = ('rating','mean'),
                                                 median_rating = ('rating','median'),
                                                 n_movies      = ('movieId','nunique'),).sort_values('n_ratings', ascending = False)

CPU times: user 44.7 s, sys: 9.67 s, total: 54.4 s
Wall time: 54.4 s

fig, axes = plt.subplots(1, 2, figsize=(20, 6))

# Volume per genre
ax = axes[0]
palette = sns.color_palette('husl', len(genre_stats))
ax.barh(genre_stats.index, genre_stats['n_ratings'], color=palette[::-1])
ax.invert_yaxis()
ax.xaxis.set_major_formatter(plt.FuncFormatter(lambda x, _: f'{x/1e6:.0f}M'))
ax.set_xlabel('Total Ratings')
ax.set_title('Rating Volume by Genre')

# Mean rating per genre (dot plot)
ax2 = axes[1]
gs_sorted = genre_stats.sort_values('mean_rating')
g_mean = all_ratings_eda['rating'].mean()
colors = ['#2ecc71' if v >= g_mean + 0.15
          else '#e74c3c' if v < g_mean - 0.15
          else '#3498db' for v in gs_sorted['mean_rating']]
ax2.scatter(gs_sorted['mean_rating'], range(len(gs_sorted)),
            s=gs_sorted['n_movies']/15, c=colors, alpha=0.85, zorder=3)
ax2.set_yticks(range(len(gs_sorted)))
ax2.set_yticklabels(gs_sorted.index)
ax2.axvline(g_mean, color='black', ls='--', lw=1.2,
            label=f'Global mean ({g_mean:.2f})')
ax2.set_xlabel('Mean Star Rating')
ax2.set_title('Mean Rating by Genre\n(bubble size = # movies in genre)')
ax2.legend(fontsize=9)

plt.suptitle('Genre Analysis', fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig('eda_04_genre_analysis.png', dpi=150, bbox_inches='tight')
plt.show()
print(genre_stats[['n_movies','n_ratings','mean_rating','median_rating']].to_string())

png

             n_movies  n_ratings  mean_rating  median_rating
genre                                                       
Drama           33152   13973271       3.6825         4.0000
Comedy          22448   11206926       3.4324         3.5000
Action           9296    9665213       3.4764         3.5000
Thriller        11555    8679464       3.5317         3.5000
Adventure        5156    7590522       3.5234         3.5000
Sci-Fi           4797    5717337       3.4917         3.5000
Romance         10048    5524615       3.5450         3.5000
Crime            6704    5373051       3.6918         4.0000
Fantasy          3784    3702759       3.5122         3.5000
Children         4447    2731841       3.4392         3.5000
Mystery          3894    2615322       3.6731         4.0000
Horror           8468    2492315       3.3072         3.5000
Animation        4586    2214562       3.6153         4.0000
War              2225    1594110       3.7917         4.0000
IMAX              195    1494179       3.5933         4.0000
Musical          1033    1159516       3.5543         4.0000
Western          1489     596654       3.6002         4.0000
Documentary      9103     427353       3.6912         4.0000
Film-Noir         350     304710       3.9158         4.0000

# Genre stats
def movie_level_agg(df):

    return df.groupby('movieId').agg(n = ('rating','count'), s=('rating','sum'))

ml_tr = movie_level_agg(train_ratings)
ml_te = movie_level_agg(test_ratings)
ml    = ml_tr.add(ml_te, fill_value=0).fillna(0)
ml['mean_rating'] = ml['s'] / ml['n'].clip(lower=1)
del ml_tr, ml_te; gc.collect()

# Explode the movie catalogue by genre
movies_exp = (movies.assign(genre = movies['genres'].str.split('|')).explode('genre').query("genre != '(no genres listed)'")[['movieId','genre']])
genre_movie = movies_exp.merge(ml, on='movieId', how='left').fillna(0)
del ml; gc.collect()

# Genre stats
genre_stats = genre_movie.groupby('genre').agg(n_ratings   = ('n', 'sum'),
                                               sum_rating  = ('s', 'sum'),
                                               n_movies    = ('movieId','nunique'),)

genre_stats['mean_rating']   = genre_stats['sum_rating'] / genre_stats['n_ratings'].clip(lower=1)
genre_stats['median_rating'] = (genre_movie.groupby('genre')['mean_rating'].median())

genre_stats = genre_stats.drop(columns='sum_rating').sort_values('n_ratings', ascending=False)
del genre_movie; gc.collect()

0

# Generate co-occurence rates for the entire catalogue
genre_list_per_movie = (
        movies.assign(genre=movies['genres'].str.split('|'))
        .set_index('movieId')['genre']
    )
top_genres = genre_stats.index[:15].tolist()
g2i = {g: i for i, g in enumerate(top_genres)}

cooc = np.zeros((len(top_genres), len(top_genres)), dtype=np.int32)

for glist in genre_list_per_movie:

    gs = [g for g in glist if g in g2i]

    for g in gs:

        cooc[g2i[g], g2i[g]] += 1

    for a, b in combinations(gs, 2):

        cooc[g2i[a], g2i[b]] += 1
        cooc[g2i[b], g2i[a]] += 1

cooc_df = pd.DataFrame(cooc, index=top_genres, columns=top_genres)
diag = np.diag(cooc).astype(float)
jaccard = cooc / (diag[:, None] + diag[None, :] - cooc + 1e-9)
jaccard_df = pd.DataFrame(jaccard, index=top_genres, columns=top_genres)

fig, axes = plt.subplots(1, 2, figsize = (20, 6))

sns.heatmap(cooc_df/1000, ax=axes[0], cmap='YlOrRd', annot = False, linewidths = 0.3, cbar_kws = {'label': 'Co-occurrences (thousands)'})

axes[0].set_title('Genre Co-occurrence Count')
mask = np.eye(len(top_genres), dtype=bool)

sns.heatmap(jaccard_df, ax=axes[1], cmap='Blues', mask=mask,
                annot=True, fmt='.2f', annot_kws={'size':7},
                linewidths=0.3, vmin=0, vmax=0.5)
axes[1].set_title('Genre Co-occurrence (Jaccard) 1.0 = always appear together')

plt.suptitle('Genre Co-occurrence in the Catalogue', fontsize=14, y=1.02)
plt.tight_layout()
plt.savefig('eda_05_genre_cooccurrence.png', dpi=150, bbox_inches='tight')
plt.show()

png

# Temporal trends
SECS_PER_YEAR = 365.2425 * 86400

def year_agg(df):

    year = ((df['timestamp'].astype('float32') / SECS_PER_YEAR) + 1970).astype('int16')
    tmp  = pd.DataFrame({'year': year, 'r': df['rating'], 'u': df['userId']})
    out  = tmp.groupby('year').agg(n=('r','count'), s=('r','sum'), nu=('u','nunique'))
    del tmp, year
    return out

yr_tr = year_agg(train_ratings)
yr_te = year_agg(test_ratings)
yearly = pd.DataFrame({
    'n_ratings'  : yr_tr['n'].add(yr_te['n'],  fill_value=0).astype(int),
    'sum_rating' : yr_tr['s'].add(yr_te['s'],  fill_value=0),
    'n_users'    : yr_tr['nu'].add(yr_te['nu'], fill_value=0).astype(int),  # approx
}).reset_index()
yearly['mean_rating'] = yearly['sum_rating'] / yearly['n_ratings'].clip(lower=1)
del yr_tr, yr_te; gc.collect()

# Monthly (only recent years, parse subset to reduce datetime overhead)
RECENT_TS = int((2013 - 1970) * SECS_PER_YEAR)
def monthly_recent(df, cutoff_ts):
    sub = df[df['timestamp'] >= cutoff_ts][['timestamp','rating']].copy()
    sub['period'] = pd.to_datetime(sub['timestamp'], unit='s').dt.to_period('M')
    out = sub.groupby('period')['rating'].count()
    del sub; return out

mo_tr = monthly_recent(train_ratings, RECENT_TS)
mo_te = monthly_recent(test_ratings,  RECENT_TS)
monthly = mo_tr.add(mo_te, fill_value=0).reset_index()
monthly.columns = ['period','n_ratings']
monthly['dt'] = monthly['period'].dt.to_timestamp()
del mo_tr, mo_te; gc.collect()

fig = plt.figure(figsize = (24, 9))
gs2 = GridSpec(2, 2, figure=fig, hspace=0.4, wspace=0.32)

ax1 = fig.add_subplot(gs2[0, :])
bars = ax1.bar(yearly['year'], yearly['n_ratings']/1e6,
               color=sns.color_palette('husl', len(yearly)))
ax1.set_xlabel('Year'); ax1.set_ylabel('Ratings (millions)')
ax1.set_title('Annual Rating Volume (1995-2023)')
ax1.bar_label(bars, fmt='{:.1f}M', padding=2, fontsize=7)

ax2 = fig.add_subplot(gs2[1, 0])
ax2.plot(monthly['dt'], monthly['n_ratings']/1e3, lw=1.5, color='steelblue')
ax2.fill_between(monthly['dt'], monthly['n_ratings']/1e3, alpha=0.15)
ax2.set_xlabel('Month'); ax2.set_ylabel('Ratings (thousands)')
ax2.set_title('Monthly Volume (2013-2023)')

ax3 = fig.add_subplot(gs2[1, 1])
ax3.plot(yearly['year'], yearly['mean_rating'], marker='o', lw=2, color='darkorange')
ax3.axhline(global_mean_all, ls='--', color='grey', lw=1.2,
            label=f'Overall mean ({global_mean_all:.2f})')
ax3.set_ylim(3.2, 4.1); ax3.set_xlabel('Year'); ax3.set_ylabel('Mean Rating')
ax3.set_title('Mean Rating Trend Over Time'); ax3.legend(fontsize=9)

plt.suptitle('Temporal Trends in Rating Activity', fontsize=14, y=1.01)
plt.savefig('eda_06_temporal_trends.png', dpi=150, bbox_inches='tight')
plt.show()

png

# Perform cold start analysis
lk_tr = train_ratings[train_ratings['rating'] >= RATING_THRESHOLD].groupby('userId')['rating'].count()
lk_te = test_ratings[ test_ratings['rating']  >= RATING_THRESHOLD].groupby('userId')['rating'].count()
user_liked_counts_all = lk_tr.add(lk_te, fill_value=0).astype(int).rename('liked_count')
del lk_tr, lk_te; gc.collect()

COLD_START_THRESHOLD = 5

fig, axes = plt.subplots(1, 2, figsize=(24, 5))

ax = axes[0]
ax.hist(user_liked_counts_all.clip(upper=300), bins=60,
        color='mediumseagreen', edgecolor='white', lw=0.3)
for thresh in [5, 10, 20]:
    ax.axvline(thresh, ls='--', lw=1.5,
               label=f'thresh={thresh}  ({(user_liked_counts_all<thresh).mean():.1%} cold)')
ax.set_xlabel('Liked Ratings per User (clipped at 300)')
ax.set_ylabel('Users'); ax.set_title('Liked-Rating Count Distribution')
ax.legend(fontsize=8)

ax2 = axes[1]
thr_range = np.arange(1, 101)
pct_warm  = [(user_liked_counts_all >= t).mean()*100 for t in thr_range]
ax2.plot(thr_range, pct_warm, lw=2.5, color='mediumseagreen')
ax2.axvline(COLD_START_THRESHOLD, color='crimson', ls='--', lw=2,
            label=f'Chosen = {COLD_START_THRESHOLD}')
chosen_warm = pct_warm[COLD_START_THRESHOLD-1]
ax2.axhline(chosen_warm, color='crimson', ls=':', lw=1)
ax2.annotate(f'{chosen_warm:.1f}% warm',
             xy=(COLD_START_THRESHOLD, chosen_warm),
             xytext=(COLD_START_THRESHOLD+6, chosen_warm-8),
             arrowprops=dict(arrowstyle='->', color='crimson'),
             color='crimson', fontsize=10)
ax2.set_xlabel('Minimum Liked Ratings')
ax2.set_ylabel('% Users Classified as Warm')
ax2.set_title('Warm User Rate vs Cold-Start Threshold')
ax2.legend(fontsize=9)

plt.suptitle('Cold-Start Analysis', fontsize = 14, y = 1.02)
plt.tight_layout()
plt.savefig('eda_08_cold_start.png', dpi = 150, bbox_inches = 'tight')
plt.show()
del user_liked_counts_all; gc.collect()

png

34387

Back to top

3. Feature engineering

We build three reusable feature tables:

  • user_stats — per-user: rating count, mean rating, activity span, rating frequency
  • movie_feat — per-movie: global count, mean rating, Bayesian-smoothed score, genre one-hot
  • user_genre_affinity — per-user per-genre: average rating given to each genre

Here we lean on explicit rating magnitudes and genre vectors as the primary content signal.

Bayesian-smoothed score: shrinks small-count movies toward the global mean.

\[\text{bayes\_score}(m) = \frac{N_m \cdot \bar{r}_m + C \cdot \bar{r}_{\text{global}}}{N_m + C}\]

where $C = 50$ is the prior strength (equivalent rating count).

# Compile user stats
liked_mask = train_ratings['rating'] >= RATING_THRESHOLD
user_liked = (train_ratings[liked_mask].groupby('userId')['movieId'].apply(list))
user_stats = train_ratings.groupby('userId').agg(
    rating_count = ('rating','count'),
    mean_rating  = ('rating','mean'),
    rating_std   = ('rating','std'),
    last_ts      = ('timestamp','max'),
    first_ts     = ('timestamp','min'),)

user_stats['active_days'] = ((user_stats['last_ts'] - user_stats['first_ts']) / 86_400).clip(lower=1).astype('float32')
user_stats['rating_freq'] = (user_stats['rating_count'] / user_stats['active_days']).astype('float32')
user_stats['mean_rating'] = user_stats['mean_rating'].astype('float32')

# Users with a single rating have undefined std; fill with 0
user_stats['rating_std']  = user_stats['rating_std'].fillna(0).astype('float32')
print('user_liked:', len(user_liked), 'users')
print('user_stats:', user_stats.shape)

user_liked: 200561 users
user_stats: (200880, 7)

# Extract genres and perform movie one hot encoding
genres_exploded = (movies.assign(genre = movies['genres'].str.split('|')).explode('genre').query("genre != '(no genres listed)'"))

ALL_GENRES = sorted(genres_exploded['genre'].unique())
print(f'Genres ({len(ALL_GENRES)}):', ALL_GENRES)

movie_genre = pd.get_dummies(genres_exploded.set_index('movieId')['genre'], prefix='g')
movie_genre = movie_genre.groupby(level=0).max()
genre_cols  = movie_genre.columns.tolist()

print('movie_genre shape:', movie_genre.shape)

Genres (19): ['Action', 'Adventure', 'Animation', 'Children', 'Comedy', 'Crime', 'Documentary', 'Drama', 'Fantasy', 'Film-Noir', 'Horror', 'IMAX', 'Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Thriller', 'War', 'Western']
movie_genre shape: (80505, 19)

# Generate Bayesian scores and movie statistics
GLOBAL_MEAN = train_ratings['rating'].mean()
C = 50

movie_stats = (
    train_ratings
    .groupby('movieId')
    .agg(global_count    = ('rating','count'),
         global_mean     = ('rating','mean'),
         rating_variance = ('rating','var'))
)
movie_stats['bayesian_score'] = (
    (movie_stats['global_count'] * movie_stats['global_mean'] + C * GLOBAL_MEAN)
    / (movie_stats['global_count'] + C)
)
# log1p compresses the very wide count range so the model sees a
# smooth popularity signal rather than a skewed raw count
movie_stats['log_count'] = np.log1p(movie_stats['global_count']).astype('float32')
# Single-rating movies have undefined variance; treat as 0
movie_stats['rating_variance'] = movie_stats['rating_variance'].fillna(0).astype('float32')

# Tag count from tags.csv (loaded in Section 1) — log1p scaled.
# Tags are user-applied descriptors; a high tag count signals that a
# movie has attracted community engagement beyond a simple star rating.
movie_tag_count = tags.groupby('movieId')['tag'].count().rename('tag_count')
movie_stats = movie_stats.join(movie_tag_count, how='left')
movie_stats['tag_count'] = np.log1p(movie_stats['tag_count'].fillna(0)).astype('float32')
del movie_tag_count

movie_feat = movie_stats.join(movie_genre, how='left').fillna(0)
print(f'movie_feat shape: {movie_feat.shape}   global mean: {GLOBAL_MEAN:.4f}')
movie_feat[['global_count','log_count','global_mean','bayesian_score','rating_variance','tag_count']].describe()


movie_feat shape: (77369, 25)   global mean: 3.5460
global_count log_count global_mean bayesian_score rating_variance tag_count
count 77369.0000 77369.0000 77369.0000 77369.0000 77369.0000 77369.0000
mean 369.4451 2.4734 3.0113 3.4606 0.8726 1.3936
std 2497.7195 2.0230 0.8208 0.2017 1.0182 1.5596
min 1.0000 0.6931 0.5000 1.2788 0.0000 0.0000
25% 2.0000 1.0986 2.5462 3.4396 0.0000 0.0000
50% 5.0000 1.7918 3.0789 3.5132 0.7455 1.0986
75% 25.0000 3.2581 3.5000 3.5451 1.1864 2.1972
max 98368.0000 11.4965 5.0000 4.4373 10.1250 8.8096 
# User genre affinity matrix
liked_fe = (train_ratings[train_ratings['rating'] >= RATING_THRESHOLD][['userId','movieId','rating']])
uid_uniq = liked_fe['userId'].unique()
mid_uniq = liked_fe['movieId'].unique()

u2i = {u: i for i, u in enumerate(uid_uniq)}
m2i = {m: i for i, m in enumerate(mid_uniq)}

rows_fe = liked_fe['userId'].map(u2i).values
cols_fe = liked_fe['movieId'].map(m2i).values

del liked_fe; gc.collect()

# Binary matrix: 1 where user liked a movie, 0 elsewhere.
# Using rating values in the numerator (previous approach) inflates genres
# where the user gave 5-star ratings vs 4-star ratings for the same number
# of films. Count-based gives the clean interpretation: fraction of a
# user's liked movies that belong to each genre.
R_bin = csr_matrix(
    (np.ones(len(rows_fe), dtype='float32'), (rows_fe, cols_fe)),
    shape=(len(uid_uniq), len(mid_uniq)),
)
del rows_fe, cols_fe; gc.collect()

# Genre matrix aligned to the liked-movie order (float32)
G = movie_genre.reindex(mid_uniq).fillna(0).values.astype('float32')

# Sparse-binary x dense multiply -> (n_users x n_genres) count matrix
uga_counts   = R_bin @ G
liked_counts = np.asarray(R_bin.sum(axis=1)).flatten()
del R_bin, G; gc.collect()

user_genre_affinity = pd.DataFrame(
    uga_counts / liked_counts[:, None].clip(min=1),
    index=uid_uniq,
    columns=genre_cols,
    dtype='float32',
)
del uga_counts, liked_counts; gc.collect()

print(f'user_genre_affinity: {user_genre_affinity.shape}  |  '
      f'{user_genre_affinity.memory_usage(deep=True).sum()/1e6:.0f} MB')


user_genre_affinity: (200561, 19)  |  16 MB

user_genre_affinity.head()
g_Action g_Adventure g_Animation g_Children g_Comedy g_Crime g_Documentary g_Drama g_Fantasy g_Film-Noir g_Horror g_IMAX g_Musical g_Mystery g_Romance g_Sci-Fi g_Thriller g_War g_Western
1 0.1972 0.1549 0.0000 0.0141 0.3662 0.1268 0.0000 0.7183 0.0282 0.0141 0.0141 0.0000 0.0000 0.0845 0.2394 0.1549 0.1268 0.1690 0.0141
2 0.1622 0.1081 0.1351 0.1622 0.4865 0.0811 0.0000 0.4865 0.0811 0.0000 0.0000 0.0541 0.1351 0.0270 0.5135 0.0000 0.2432 0.0270 0.0000
3 0.3699 0.4521 0.1233 0.1507 0.2877 0.0959 0.0000 0.4795 0.0822 0.0000 0.0137 0.0411 0.0959 0.0137 0.1918 0.2055 0.2055 0.1233 0.0137
4 0.2000 0.4000 0.0000 0.0000 0.6000 0.0000 0.0000 0.2000 0.2000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.2000 0.0000 0.2000 0.0000
5 0.7500 0.3750 0.0000 0.0000 0.1250 0.3750 0.0000 0.2500 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.1250 0.0000 1.0000 0.1250 0.0000 
# Split data for training / validation
all_users = ground_truth.index.values
gss = GroupShuffleSplit(n_splits = 1, test_size = 0.15, random_state = 100)
train_idx, eval_idx = next(gss.split(all_users, groups=all_users))

train_fold_users = set(all_users[train_idx])
eval_fold_users  = set(all_users[eval_idx])
assert not train_fold_users & eval_fold_users, 'Leakage!'

eval_df = (ground_truth[ground_truth.index.isin(eval_fold_users)].reset_index().rename(columns = {'true_movies': 'true_items'}))

warm_users = set(user_liked[user_liked.map(len) >= COLD_START_THRESHOLD].index)
eval_warm  = eval_df[eval_df['userId'].isin(warm_users)].reset_index(drop=True)

print(f'Train fold : {len(train_fold_users):,} users')
print(f'Eval  fold : {len(eval_fold_users):,} users')
print(f'Warm eval  : {len(eval_warm):,} users')
cold_pct = (len(eval_df) - len(eval_warm)) / len(eval_df)
print(f'Cold eval  : {len(eval_df)-len(eval_warm):,} ({cold_pct:.1%})')

# Populate _seen_cache here — before any training loop that calls get_seen().
# The empty-dict initialisation in the function cell is a safe fallback;
# this replaces it with the real per-user seen sets so that all retriever
# functions correctly exclude already-rated movies during training.
_seen_cache = train_ratings.groupby('userId')['movieId'].apply(set).to_dict()
print(f'_seen_cache populated: {len(_seen_cache):,} users')


Train fold : 163,311 users
Eval  fold : 28,820 users
Warm eval  : 28,509 users
Cold eval  : 311 (1.1%)
_seen_cache populated: 200,880 users

Back to top..

4. Item-based collaborative filtering

📖 Batched sparse similarity

We build a sparse (movie x user) rating matrix, L2-normalise each movie’s row, then compute cosine similarity in batches of 2,000 movies. This avoids an 87k x 87k dense matrix (~30 GB). The full sparse matrix and its normalised copy are freed immediately after the lookup table is populated.

# Build a movie x user rating matrix
liked_cf = (train_ratings[train_ratings['rating'] >= RATING_THRESHOLD][['userId','movieId','rating']].copy())

movie_ids_cf = liked_cf['movieId'].unique()
user_ids_cf  = liked_cf['userId'].unique()

m_idx = {m: i for i, m in enumerate(movie_ids_cf)}
u_idx = {u: i for i, u in enumerate(user_ids_cf)}
idx_m = {i: m for m, i in m_idx.items()}

R = csr_matrix((liked_cf['rating'].values.astype('float32'), (liked_cf['movieId'].map(m_idx).values, liked_cf['userId'].map(u_idx).values)), shape = (len(movie_ids_cf), len(user_ids_cf)),)

del liked_cf; gc.collect()

R_norm = normalize(R, norm='l2', axis=1)
print(f'Rating matrix: {R.shape}  nnz={R.nnz:,}')
print(f'Memory: R={R.data.nbytes/1e6:.0f} MB  R_norm={R_norm.data.nbytes/1e6:.0f} MB')

Rating matrix: (50644, 200561)  nnz=14,311,277
Memory: R=57 MB  R_norm=57 MB

# Batched KNN, top-K neighbors
K_NN  = 50
BATCH = 2_000
item_sim_lookup = {}
n_movies_cf = R_norm.shape[0]
t0 = time.time()

for start in range(0, n_movies_cf, BATCH):

    batch = R_norm[start : start + BATCH]
    sims  = (batch @ R_norm.T).toarray()

    for local_i, sim_row in enumerate(sims):

        global_i = start + local_i
        sim_row[global_i] = 0.0
        top_k = np.argpartition(sim_row, -K_NN)[-K_NN:]
        top_k = top_k[np.argsort(sim_row[top_k])[::-1]]
        movie_id = idx_m[global_i]
        item_sim_lookup[movie_id] = [(idx_m[j], float(sim_row[j])) for j in top_k]

    if start % 20000 == 0:

        print(f'  {start:>6}/{n_movies_cf}  {time.time()-t0:.0f}s')

del R, R_norm; gc.collect()
print(f'Item-sim lookup built for {len(item_sim_lookup):,} movies in {time.time()-t0:.0f}s')

       0/50644  14s
   20000/50644  86s
   40000/50644  156s
Item-sim lookup built for 50,644 movies in 190s

Back to top..

5. Temporal sequence modelling

📖 Recency-weighted taste vectors

We pre-compute per-user genre taste vectors using exponential recency weighting (half-life = 365 days). The movie genre matrix is pre-normalised as float32 and stored once; per-user dot products at inference time are fast matrix-vector multiplies.

# Compute recent user tastes
DECAY_HALF_LIFE = 365
DECAY_K         = np.log(2) / DECAY_HALF_LIFE
now_ts          = int(train_ratings['timestamp'].max())

liked_ts = train_ratings[train_ratings['rating'] >= RATING_THRESHOLD][['userId','movieId','timestamp']]

# Compute decay weight per row as a float32 column
liked_ts['weight'] = np.exp(-DECAY_K * (now_ts - liked_ts['timestamp'].values.astype('float32')) / 86400).astype('float32')

uid_uniq = liked_ts['userId'].unique()
mid_uniq = liked_ts['movieId'].unique()
u2i = {u: i for i, u in enumerate(uid_uniq)}
m2i = {m: i for i, m in enumerate(mid_uniq)}

W = csr_matrix((liked_ts['weight'].values,
               (liked_ts['userId'].map(u2i).values,
               liked_ts['movieId'].map(m2i).values)),
               shape = (len(uid_uniq), len(mid_uniq)),)

del liked_ts; gc.collect()

# Genre matrix aligned to mid_uniq order
G = movie_genre.reindex(mid_uniq).fillna(0).values.astype('float32')

# One sparse matmul
taste_raw = W @ G
row_norms  = np.linalg.norm(taste_raw, axis = 1, keepdims = True).clip(min = 1e-9)
taste_norm = (taste_raw / row_norms).astype('float32')

user_taste_vec = {uid_uniq[i]: taste_norm[i] for i in range(len(uid_uniq))}

del W, G, taste_raw, taste_norm, row_norms; gc.collect()
print(f'Taste vectors built for {len(user_taste_vec):,} users')

Taste vectors built for 200,561 users

# Build a normalised movie-genre matrix for S2 and S4 retrievers.
# Expanding to all rated movies (movie_feat.index) rather than just
# POPULARITY_POOL raises the retrieval ceiling: S2/S4 were previously
# blind to ~27K movies that had ratings but no liked (4+) ratings.
movie_genre_aligned = movie_genre.reindex(movie_feat.index).fillna(0)
movie_genre_norm    = normalize(movie_genre_aligned.values.astype('float32'), norm='l2', axis=1)
genre_index_order   = movie_genre_aligned.index.tolist()

del movie_genre_aligned; gc.collect()

print(f'genre_norm matrix: {movie_genre_norm.shape}  |  '
      f'{movie_genre_norm.nbytes/1e6:.0f} MB')

genre_norm matrix: (77369, 19)  |  6 MB

# Assemble the inputs for LightGBM
FEATURE_COLS = ['u_rating_count', 'u_mean_rating', 'u_rating_freq', 'u_rating_std', 'm_global_count', 'm_log_count', 'm_global_mean', 'm_bayesian', 'm_rating_variance', 'm_tag_count', 'genre_affinity',]
TRAIN_USERS_SAMPLE = 8000
NEGATIVES_PER_POS  = 4
K_CAND             = 200

rng = np.random.default_rng(100)

sample_users = rng.choice(list(train_fold_users & warm_users), size    = min(TRAIN_USERS_SAMPLE, len(train_fold_users & warm_users)), replace = False,)

# ── Batch-precompute S2 and S4 genre scores ──────────────────────────────
# s2_personal and s4_temporal each run movie_genre_norm @ vector inside the
# loop. For 8,000 users that is 16,000 individual (77K×19)@(19,) BLAS calls.
# Two batch matmuls replace all of them; BLAS executes a single fused kernel
# that is 10-50x faster than the equivalent number of small calls.

# S2 batch: normalise each user's genre-affinity row, then one matmul
uga_sample = user_genre_affinity.reindex(sample_users).fillna(0).values.astype('float32')
uga_norms  = np.linalg.norm(uga_sample, axis=1, keepdims=True).clip(min=1e-9)
s2_scores  = movie_genre_norm @ (uga_sample / uga_norms).T  # (n_movies, n_users)
del uga_sample, uga_norms; gc.collect()

# S4 batch: taste vectors are already unit-normalised
taste_sample = np.vstack([user_taste_vec.get(uid, np.zeros(len(genre_cols), dtype='float32')) for uid in sample_users]).astype('float32')
s4_scores = movie_genre_norm @ taste_sample.T   # (n_movies, n_users)
del taste_sample; gc.collect()

print(f's2_scores: {s2_scores.shape}  s4_scores: {s4_scores.shape}  '
      f'({(s2_scores.nbytes + s4_scores.nbytes)/1e6:.0f} MB combined)')

# ── Build (userId, movieId, label) pairs ─────────────────────────────────
# Each iteration slices one precomputed column (O(1)) instead of running a
# fresh matmul. np.argpartition finds the top-K in O(n) vs O(n log n) for
# a full argsort; only the K retained entries are then sorted.

pair_rows = []

for i, uid in enumerate(sample_users):

    liked_list = user_liked.get(uid, [])

    if not liked_list:

        continue

    seen = get_seen(uid)

    # S2 candidates — slice precomputed column, partition, filter seen
    s2_col   = s2_scores[:, i]
    s2_top   = np.argpartition(s2_col, -K_CAND)[-K_CAND:]
    s2_top   = s2_top[np.argsort(s2_col[s2_top])[::-1]]
    s2_cands = [genre_index_order[j] for j in s2_top if genre_index_order[j] not in seen]

    # S3 candidates — item-CF uses irregular dict structure; stays sequential
    s3_cands = s3_item_cf(uid, K=K_CAND)

    # S4 candidates — slice precomputed column, partition, filter seen
    s4_col   = s4_scores[:, i]
    s4_top   = np.argpartition(s4_col, -K_CAND)[-K_CAND:]
    s4_top   = s4_top[np.argsort(s4_col[s4_top])[::-1]]
    s4_cands = [genre_index_order[j] for j in s4_top if genre_index_order[j] not in seen]

    liked_set  = set(liked_list)
    neg_pool   = list(dict.fromkeys(s2_cands + s3_cands + s4_cands))
    neg_pool   = [m for m in neg_pool if m not in liked_set]
    n_neg      = min(len(liked_list) * NEGATIVES_PER_POS, len(neg_pool))
    neg_movies = rng.choice(neg_pool, size=n_neg, replace=False).tolist() if n_neg > 0 else []

    for m in liked_list:

        pair_rows.append((uid, m, 1))

    for m in neg_movies:

        pair_rows.append((uid, m, 0))

del s2_scores, s4_scores; gc.collect()

pairs = pd.DataFrame(pair_rows, columns = ['userId', 'movieId', 'label'])
del pair_rows; gc.collect()

s2_scores: (77369, 8000)  s4_scores: (77369, 8000)  (4952 MB combined)





0

# Join user features
u_feat = user_stats[['rating_count','mean_rating','rating_freq','rating_std']].rename(columns = {'rating_count': 'u_rating_count',
                                                                                                 'mean_rating' : 'u_mean_rating',
                                                                                                 'rating_freq' : 'u_rating_freq',
                                                                                                 'rating_std'  : 'u_rating_std',})

pairs = pairs.merge(u_feat, left_on = 'userId', right_index = True, how = 'left')

pairs.head()
userId movieId label u_rating_count u_mean_rating u_rating_freq u_rating_std
0 196703 10 1 34 3.6471 34.0000 0.9173
1 196703 16 1 34 3.6471 34.0000 0.9173
2 196703 22 1 34 3.6471 34.0000 0.9173
3 196703 47 1 34 3.6471 34.0000 0.9173
4 196703 58 1 34 3.6471 34.0000 0.9173 
# Join movie features
m_feat = movie_feat[['global_count','log_count','global_mean','bayesian_score', 'rating_variance','tag_count']].rename(columns={'global_count'   : 'm_global_count',
                                                                                                                                'log_count'      : 'm_log_count',
                                                                                                                                'global_mean'    : 'm_global_mean',
                                                                                                                                'bayesian_score' : 'm_bayesian',
                                                                                                                                'rating_variance': 'm_rating_variance',
                                                                                                                                'tag_count'      : 'm_tag_count',})
pairs = pairs.merge(m_feat, left_on = 'movieId', right_index = True, how='left')

# Compute genre affinity via stacked dot products. L2-normalise the movie genre vectors so the result is cosine similarity — consistent with the per-movie normalisation inside build_lgb_features.
taste_matrix    = np.vstack([user_taste_vec.get(uid, np.zeros(len(genre_cols), dtype='float32')) for uid in pairs['userId']]).astype('float32')
genre_matrix_raw = movie_genre.reindex(pairs['movieId']).fillna(0).values.astype('float32')
genre_norms      = np.linalg.norm(genre_matrix_raw, axis=1, keepdims=True).clip(min=1e-9)
genre_matrix     = genre_matrix_raw / genre_norms
pairs['genre_affinity'] = (taste_matrix * genre_matrix).sum(axis=1).astype('float32')
del taste_matrix, genre_matrix_raw, genre_norms, genre_matrix; gc.collect()

# Fill missing with global vals
pairs['u_rating_count']    = pairs['u_rating_count'].fillna(0).astype('float32')
pairs['u_mean_rating']     = pairs['u_mean_rating'].fillna(GLOBAL_MEAN).astype('float32')
pairs['u_rating_freq']     = pairs['u_rating_freq'].fillna(0).astype('float32')
pairs['u_rating_std']      = pairs['u_rating_std'].fillna(0).astype('float32')
pairs['m_global_count']    = pairs['m_global_count'].fillna(0).astype('float32')
pairs['m_log_count']       = pairs['m_log_count'].fillna(0).astype('float32')
pairs['m_global_mean']     = pairs['m_global_mean'].fillna(GLOBAL_MEAN).astype('float32')
pairs['m_bayesian']        = pairs['m_bayesian'].fillna(GLOBAL_MEAN).astype('float32')
pairs['m_rating_variance'] = pairs['m_rating_variance'].fillna(0).astype('float32')
pairs['m_tag_count']       = pairs['m_tag_count'].fillna(0).astype('float32')

lgb_train_df = pairs
del pairs; gc.collect()
print(f'LightGBM set: {lgb_train_df.shape}  pos rate: {lgb_train_df["label"].mean():.2%}')

LightGBM set: (2275394, 14)  pos rate: 26.12%

lgb_train_df.head()
userId movieId label u_rating_count u_mean_rating u_rating_freq u_rating_std m_global_count m_log_count m_global_mean m_bayesian m_rating_variance m_tag_count genre_affinity
0 196703 10 1 34.0000 3.6471 34.0000 0.9173 30624.0000 10.3296 3.4252 3.4254 0.7641 6.0753 0.4776
1 196703 16 1 34.0000 3.6471 34.0000 0.9173 20209.0000 9.9139 3.8338 3.8331 0.7297 6.7558 0.6299
2 196703 22 1 34.0000 3.6471 34.0000 0.9173 9327.0000 9.1408 3.3093 3.3106 0.8477 4.6444 0.7114
3 196703 47 1 34.0000 3.6471 34.0000 0.9173 58465.0000 10.9762 4.0874 4.0869 0.7139 7.8652 0.4049
4 196703 58 1 34.0000 3.6471 34.0000 0.9173 10647.0000 9.2731 3.9646 3.9627 0.9441 3.6376 0.7347

Back to top..

6. Evaluation harness & S1-S5 strategies

Five metrics at K=5 and K=10. The composite score is their unweighted mean. All strategies exclude movies already seen in train_ratings.

# Train base LightGBM model
xtrain = lgb_train_df[FEATURE_COLS].values
ytrain = lgb_train_df['label'].values
del lgb_train_df; gc.collect()

# Hold out 15% of rows as a proper validation set for early stopping
x_tr, x_val, y_tr, y_val = train_test_split(xtrain, ytrain, test_size = 0.15, random_state = 100, stratify = ytrain)

lgb_params = {'objective'        : 'binary',
              'metric'           : 'auc',
              'learning_rate'    : 0.05,
 #             'num_leaves'       : 63,
 #             'min_child_samples': 20,
              'feature_fraction' : 0.8,
              'bagging_fraction' : 0.8,
              'bagging_freq'     : 5,
              'verbose'          : -1,
              'n_jobs'           : -1,}

lgb_model = lgb.train(lgb_params,
                      lgb.Dataset(x_tr,
                                  label = y_tr,
                                  feature_name = FEATURE_COLS),
                                  num_boost_round = 500,
                                  valid_sets = [lgb.Dataset(x_val, label = y_val)],
                                  callbacks = [lgb.early_stopping(20, verbose = False), lgb.log_evaluation(50)],)

del xtrain, ytrain, x_tr, x_val, y_tr, y_val; gc.collect()

print(f'LightGBM trained  best_iter = {lgb_model.best_iteration}')

[50]	valid_0's auc: 0.85308
[100]	valid_0's auc: 0.857502
[150]	valid_0's auc: 0.860966
[200]	valid_0's auc: 0.864246
[250]	valid_0's auc: 0.866321
[300]	valid_0's auc: 0.868303
[350]	valid_0's auc: 0.869454
[400]	valid_0's auc: 0.871189
[450]	valid_0's auc: 0.872391
[500]	valid_0's auc: 0.87376
LightGBM trained  best_iter = 500

Back to top..

7. S6 architecture & cold-start gate

# Apply function to check for cold starts
cold_n = sum(is_cold(uid) for uid in eval_fold_users)
print(f'Cold in eval fold: {cold_n:,}  ({100*cold_n/len(eval_fold_users):.1f}%)')

Cold in eval fold: 311  (1.1%)

Stage 1 — Expanded candidate pool

# Get stage 1 candidates and diagnostics
N_STAGE1 = 200
DIAG_N = 500

diag_users = eval_warm.sample(n = DIAG_N, random_state = 100)
recalls = []

for _, row in diag_users.iterrows():

    pool = set(stage1_candidates(row['userId']))
    true = row['true_items']
    recalls.append(len(pool & true) / len(true) if true else 0.0)

print(f'Stage-1 Recall@{N_STAGE1} (n={DIAG_N}): {np.mean(recalls):.4f}')

Stage-1 Recall@200 (n=500): 0.5524

Stage 2 — Meta-ranker training

# Init params
STAGE2_FEATURE_COLS = FEATURE_COLS + ['in_s2','in_s3','in_s4', 'rank_s2','rank_s3','rank_s4', 'n_retrievers', 's5_score',]   # base LightGBM predicted probability — strongest available signal
META_TRAIN_USERS = 8000  # doubled from 4000
META_NEG_RATIO   = 5

meta_users = rng.choice(list(train_fold_users & warm_users), size = min(META_TRAIN_USERS, len(train_fold_users & warm_users)), replace = False,)

# ── Batch-precompute S2 and S4 genre scores for meta training ───────────
# Same optimisation as the base LGB pairs cell: replace per-user matmuls
# with two batch matmuls before the loop.
uga_meta  = user_genre_affinity.reindex(meta_users).fillna(0).values.astype('float32')
uga_norms = np.linalg.norm(uga_meta, axis=1, keepdims=True).clip(min=1e-9)
s2_scores_meta = movie_genre_norm @ (uga_meta / uga_norms).T  # (n_movies, n_meta_users)
del uga_meta, uga_norms; gc.collect()

taste_meta = np.vstack([user_taste_vec.get(uid, np.zeros(len(genre_cols), dtype='float32')) for uid in meta_users]).astype('float32')
s4_scores_meta = movie_genre_norm @ taste_meta.T  # (n_movies, n_meta_users)
del taste_meta; gc.collect()

print(f'Meta score matrices: {s2_scores_meta.shape}  '
      f'({(s2_scores_meta.nbytes + s4_scores_meta.nbytes)/1e6:.0f} MB combined)')

# ── Collect (uid, movie, label, ranks) ───────────────────────────────────
pair_rows = []
t0 = time.time()

for i, uid in enumerate(meta_users):

    # S2: slice precomputed column, partition top-N_STAGE1, filter seen
    seen   = get_seen(uid)
    s2_col = s2_scores_meta[:, i]
    s2_top = np.argpartition(s2_col, -N_STAGE1)[-N_STAGE1:]
    s2_top = s2_top[np.argsort(s2_col[s2_top])[::-1]]
    s2l    = [genre_index_order[j] for j in s2_top
              if genre_index_order[j] not in seen][:N_STAGE1]

    # S3: item-CF stays sequential
    s3l = s3_item_cf(uid, K=N_STAGE1)

    # S4: slice precomputed column, partition top-N_STAGE1, filter seen
    s4_col = s4_scores_meta[:, i]
    s4_top = np.argpartition(s4_col, -N_STAGE1)[-N_STAGE1:]
    s4_top = s4_top[np.argsort(s4_col[s4_top])[::-1]]
    s4l    = [genre_index_order[j] for j in s4_top if genre_index_order[j] not in seen][:N_STAGE1]

    cands = list(dict.fromkeys(s2l + s3l + s4l))

    # Positives: candidates that appear in the user's held-out ground truth.
    # Using ground_truth (test-set labels) rather than user_liked (training-set
    # liked items) ensures positives are present in the retrieval pool and
    # therefore carry meaningful retriever rank features (in_s2/s3/s4, rank_s*).
    true_set = ground_truth[uid] if uid in ground_truth.index else set()
    pos_c = [m for m in cands if m in true_set]

    if not pos_c:

        continue

    # Negatives: candidates not in the user's true set
    neg_c = [m for m in cands if m not in true_set]
    n_neg = min(len(pos_c) * META_NEG_RATIO, len(neg_c))
    neg_s = rng.choice(neg_c, size=n_neg, replace=False).tolist() if n_neg > 0 else []
    all_c = pos_c + neg_s

    if not all_c:

        continue

    s2r = {m: r for r, m in enumerate(s2l)}
    s3r = {m: r for r, m in enumerate(s3l)}
    s4r = {m: r for r, m in enumerate(s4l)}

    for m in all_c:

        pair_rows.append((uid,
                          m,
                          int(m in true_set),
                          int(m in s2r),
                          int(m in s3r),
                          int(m in s4r),
                          s2r.get(m, N_STAGE1),
                          s3r.get(m, N_STAGE1),
                          s4r.get(m, N_STAGE1),))

    if (i + 1) % 500 == 0:

        print(f'  {i+1}/{len(meta_users)}  {time.time()-t0:.0f}s')

del s2_scores_meta, s4_scores_meta; gc.collect()
print(f'Retrieval done: {len(pair_rows):,} pairs in {time.time()-t0:.0f}s')

Meta score matrices: (77369, 8000)  (4952 MB combined)
  500/8000  5s
  1000/8000  10s
  1500/8000  14s
  2000/8000  19s
  2500/8000  24s
  3000/8000  28s
  3500/8000  33s
  4000/8000  38s
  4500/8000  43s
  5000/8000  47s
  5500/8000  53s
  6000/8000  57s
  6500/8000  62s
  7000/8000  67s
  7500/8000  72s
  8000/8000  77s
Retrieval done: 188,689 pairs in 80s

# Build the data from the results
pairs = pd.DataFrame(pair_rows, columns = ['userId','movieId','label', 'in_s2','in_s3','in_s4', 'rank_s2','rank_s3','rank_s4',])
pairs['n_retrievers'] = (pairs['in_s2'] + pairs['in_s3'] + pairs['in_s4']).astype('int8')
pairs[['in_s2','in_s3','in_s4']]         = pairs[['in_s2','in_s3','in_s4']].astype('int8')
pairs[['rank_s2','rank_s3','rank_s4']]   = pairs[['rank_s2','rank_s3','rank_s4']].astype('int16')

# Join user features
u_feat = user_stats[['rating_count','mean_rating','rating_freq','rating_std']].rename(columns = {'rating_count': 'u_rating_count',
                                                                                                 'mean_rating' : 'u_mean_rating',
                                                                                                 'rating_freq' : 'u_rating_freq',
                                                                                                 'rating_std'  : 'u_rating_std',})
pairs = pairs.merge(u_feat, left_on = 'userId', right_index = True, how = 'left')

# Join movie features
m_feat = movie_feat[['global_count','log_count','global_mean','bayesian_score', 'rating_variance','tag_count']].rename(columns = {'global_count'   : 'm_global_count',
                                                                                                                                  'log_count'      : 'm_log_count',
                                                                                                                                  'global_mean'    : 'm_global_mean',
                                                                                                                                  'bayesian_score' : 'm_bayesian',
                                                                                                                                  'rating_variance': 'm_rating_variance',
                                                                                                                                  'tag_count'      : 'm_tag_count',})

pairs = pairs.merge(m_feat, left_on = 'movieId', right_index = True, how = 'left')

# Join the vectorized genre affinity
taste_matrix    = np.vstack([user_taste_vec.get(uid, np.zeros(len(genre_cols), dtype = 'float32')) for uid in pairs['userId']]).astype('float32')
genre_matrix_raw = movie_genre.reindex(pairs['movieId']).fillna(0).values.astype('float32')
genre_norms      = np.linalg.norm(genre_matrix_raw, axis=1, keepdims=True).clip(min=1e-9)
genre_matrix     = genre_matrix_raw / genre_norms
pairs['genre_affinity'] = (taste_matrix * genre_matrix).sum(axis=1).astype('float32')
del taste_matrix, genre_matrix_raw, genre_norms, genre_matrix; gc.collect()

0

pairs.head()
userId movieId label in_s2 in_s3 in_s4 rank_s2 rank_s3 rank_s4 n_retrievers u_rating_count u_mean_rating u_rating_freq u_rating_std m_global_count m_log_count m_global_mean m_bayesian m_rating_variance m_tag_count genre_affinity
0 182041 1196 1 0 1 0 200 0 200 1 19 3.5789 19.0000 0.7685 68455 11.1339 4.1351 4.1347 0.9112 7.5348 0.8041
1 182041 95167 1 0 1 0 200 104 200 1 19 3.5789 19.0000 0.7685 8632 9.0633 3.4822 3.4826 0.9153 5.9506 0.7783
2 182041 86332 0 0 1 0 200 167 200 1 19 3.5789 19.0000 0.7685 12975 9.4709 3.3188 3.3197 1.0297 7.1301 0.8060
3 182041 590 0 0 1 0 200 70 200 1 19 3.5789 19.0000 0.7685 44195 10.6964 3.7277 3.7275 0.9189 5.9584 0.5203
4 182041 37830 0 1 1 1 14 184 6 3 19 3.5789 19.0000 0.7685 1799 7.4955 3.3894 3.3936 1.1338 4.7958 0.8427 
pairs.isnull().sum()
0
userId 0
movieId 0
label 0
in_s2 0
in_s3 0
in_s4 0
rank_s2 0
rank_s3 0
rank_s4 0
n_retrievers 0
u_rating_count 0
u_mean_rating 0
u_rating_freq 0
u_rating_std 0
m_global_count 0
m_log_count 0
m_global_mean 0
m_bayesian 0
m_rating_variance 0
m_tag_count 0
genre_affinity 0 


# Fill missing values
#for col, default in [('u_rating_count', 0), ('u_mean_rating', GLOBAL_MEAN),
#                     ('u_rating_freq',  0), ('u_rating_std',  0),
#                     ('m_global_count', 0), ('m_log_count',   0),
#                     ('m_global_mean',  GLOBAL_MEAN), ('m_bayesian', GLOBAL_MEAN),
#                     ('m_rating_variance', 0), ('m_tag_count', 0)]:#

#    pairs[col] = pairs[col].fillna(default).astype('float32')

# Compute S5 predicted probability for each candidate.
# This is the single strongest per-(user,movie) relevance signal;
# the meta-ranker can learn to weight it, correct it with retriever
# rank signals, or override it for users where CF is more reliable.
pairs['s5_score'] = lgb_model.predict(pairs[FEATURE_COLS].values).astype('float32')

meta_train_df = pairs[STAGE2_FEATURE_COLS + ['label']].reset_index(drop = True)
print(f'Meta set: {meta_train_df.shape}  pos rate: {meta_train_df["label"].mean():.2%}')

Meta set: (188689, 20)  pos rate: 16.68%

meta_train_df.head()
u_rating_count u_mean_rating u_rating_freq u_rating_std m_global_count m_log_count m_global_mean m_bayesian m_rating_variance m_tag_count genre_affinity in_s2 in_s3 in_s4 rank_s2 rank_s3 rank_s4 n_retrievers s5_score label
0 19 3.5789 19.0000 0.7685 68455 11.1339 4.1351 4.1347 0.9112 7.5348 0.8041 0 1 0 200 0 200 1 0.8162 1
1 19 3.5789 19.0000 0.7685 8632 9.0633 3.4822 3.4826 0.9153 5.9506 0.7783 0 1 0 200 104 200 1 0.2577 1
2 19 3.5789 19.0000 0.7685 12975 9.4709 3.3188 3.3197 1.0297 7.1301 0.8060 0 1 0 200 167 200 1 0.1825 0
3 19 3.5789 19.0000 0.7685 44195 10.6964 3.7277 3.7275 0.9189 5.9584 0.5203 0 1 0 200 70 200 1 0.5174 0
4 19 3.5789 19.0000 0.7685 1799 7.4955 3.3894 3.3936 1.1338 4.7958 0.8427 1 1 1 14 184 6 3 0.0626 0 
# Train the meta-ranker models with a proper held-out validation split
xmeta = meta_train_df[STAGE2_FEATURE_COLS].values
ymeta = meta_train_df['label'].values

# Hold out 15% for validation to enable meaningful early stopping
xm_tr, xm_val, ym_tr, ym_val = train_test_split(
    xmeta, ymeta, test_size=0.15, random_state=42, stratify=ymeta)

meta_lgb_params = {'objective':'binary',
                   'metric':'auc',
#                   'num_leaves':127,
                   'learning_rate':0.03,
                   'feature_fraction':0.8,
                   'bagging_fraction':0.8,
                   'bagging_freq':5,
                   'verbose':-1,
                   'n_jobs':-1,}

meta_lgb = lgb.train(meta_lgb_params,
                     lgb.Dataset(xm_tr,
                                 label = ym_tr,
                                 feature_name = STAGE2_FEATURE_COLS),
                                 num_boost_round = 500,
                                 valid_sets = [lgb.Dataset(xm_val, label = ym_val)],
                                 callbacks = [lgb.early_stopping(30,verbose = False), lgb.log_evaluation(100)],)

print(f'Meta-LightGBM trained  best_iter={meta_lgb.best_iteration}')

meta_xgb = XGBClassifier(learning_rate = 0.03,
                         #max_depth = 6,
                         #n_estimators = 400,
                         subsample = 0.8,
                         colsample_bytree = 0.8,
                         eval_metric = 'auc',
                         early_stopping_rounds = 30,
                         verbosity = 0,)

meta_xgb.fit(xm_tr, ym_tr, eval_set = [(xm_val, ym_val)], verbose=False)
print(f'Meta-XGBoost trained  best_iter={meta_xgb.best_iteration}')

[100]	valid_0's auc: 0.856981
[200]	valid_0's auc: 0.859101
[300]	valid_0's auc: 0.860041
[400]	valid_0's auc: 0.860901
Meta-LightGBM trained  best_iter=441
Meta-XGBoost trained  best_iter=99

del xmeta, ymeta, xm_tr, xm_val, ym_tr, ym_val, meta_train_df, pairs, pair_rows; gc.collect()

28

Back to top..

10. Full benchmark: S1 → S7

# Verify _seen_cache is populated (built after the user split; confirmed here)
print(f'_seen_cache: {len(_seen_cache):,} users  |  '
      f'{sum(len(v) for v in _seen_cache.values()):,} total seen entries')

_seen_cache: 200,880 users  |  28,583,599 total seen entries

%%time

# ── Batch-precompute S2 and S4 score vectors for all eval_warm users ─────
# evaluate_strategy calls s2_personal and s4_temporal per user, each firing
# a (77K×19)@(19,) matmul. For 2,000 users × 7 strategies × 2 Ks that is
# 56,000 individual BLAS calls — and S5/S6/S7 call S2+S4 internally, tripling
# the count further. Two batch matmuls replace all of them.
eval_user_ids  = eval_warm['userId'].values

uga_eval   = user_genre_affinity.reindex(eval_user_ids).fillna(0).values.astype('float32')
uga_norms  = np.linalg.norm(uga_eval, axis = 1, keepdims = True).clip(min = 1e-9)
s2_matrix  = movie_genre_norm @ (uga_eval / uga_norms).T
del uga_eval, uga_norms

taste_eval = np.vstack([user_taste_vec.get(uid, np.zeros(len(genre_cols), dtype='float32')) for uid in eval_user_ids]).astype('float32')
s4_matrix  = movie_genre_norm @ taste_eval.T
del taste_eval

# Populate the score caches so s2_personal / s4_temporal skip the matmul
_s2_score_cache = {uid: s2_matrix[:, i] for i, uid in enumerate(eval_user_ids)}
_s4_score_cache = {uid: s4_matrix[:, i] for i, uid in enumerate(eval_user_ids)}
del s2_matrix, s4_matrix; gc.collect()
print(f'Score caches populated for {len(_s2_score_cache):,} eval users')

# Perform the benchmark analysis
print('Initiating Benchmark Analysis..')

STRATEGIES = {'S1: Popularity'      : s1_popularity,
              'S2: Personal'        : s2_personal,
              'S3: Item-CF'         : s3_item_cf,
              'S4: Temporal'        : s4_temporal,
              'S5: LightGBM'        : s5_lgbm,
              'S6: XGBoost'         : s6_xgboost,
              'S7: Master Ensemble' : s7_ensemble,}

# All five metrics are reported; F1 is retained for display but excluded from
# the composite score. F1 = HM(Precision, Recall) — including it alongside
# Precision and Recall gives the P/R axis 3x the weight of NDCG and Hit Rate
# in a simple average, making the composite uninterpretable as a balanced
# summary. The composite uses only the four non-redundant metrics.
metric_keys    = ['precision', 'recall', 'f1', 'ndcg', 'hit_rate']
composite_keys = ['precision', 'recall', 'ndcg', 'hit_rate']

all_results = {}

for K in [5, 10]:

    print(f'\n-- K = {K} --')

    for name, fn in STRATEGIES.items():

        t0  = time.time()
        res = evaluate_strategy(fn, eval_warm, K = K, n = 2000, seed = 100)
        res['composite'] = float(np.mean([res[m] for m in composite_keys]))
        all_results[(name, K)] = res

        print(f'  {name:<25}  composite = {res["composite"]:.4f}  '
              f'NDCG = {res["ndcg"]:.4f}  HR = {res["hit_rate"]:.4f}  ({time.time()-t0:.1f}s)')

# Clear caches after benchmark — they are specific to eval_warm users
_s2_score_cache.clear()
_s4_score_cache.clear()
print('Benchmark Analysis complete..')

Score caches populated for 28,509 eval users

-- K = 5 --
  S1: Popularity             composite = 0.0671  NDCG = 0.0457  HR = 0.1545  (5.1s)
  S2: Personal               composite = 0.0029  NDCG = 0.0016  HR = 0.0075  (8.9s)
  S3: Item-CF                composite = 0.1094  NDCG = 0.0808  HR = 0.2375  (2.4s)
  S4: Temporal               composite = 0.0021  NDCG = 0.0014  HR = 0.0055  (9.2s)
  S5: LightGBM               composite = 0.0231  NDCG = 0.0147  HR = 0.0560  (131.2s)
  S6: XGBoost                composite = 0.1186  NDCG = 0.0828  HR = 0.2630  (187.6s)
  S7: Master Ensemble        composite = 0.1230  NDCG = 0.0875  HR = 0.2710  (162.7s)

-- K = 10 --
  S1: Popularity             composite = 0.0893  NDCG = 0.0486  HR = 0.2270  (5.2s)
  S2: Personal               composite = 0.0041  NDCG = 0.0017  HR = 0.0120  (14.3s)
  S3: Item-CF                composite = 0.1452  NDCG = 0.0882  HR = 0.3475  (2.4s)
  S4: Temporal               composite = 0.0038  NDCG = 0.0016  HR = 0.0110  (13.7s)
  S5: LightGBM               composite = 0.0296  NDCG = 0.0151  HR = 0.0805  (134.0s)
  S6: XGBoost                composite = 0.1616  NDCG = 0.0939  HR = 0.3905  (189.3s)
  S7: Master Ensemble        composite = 0.1652  NDCG = 0.0992  HR = 0.3930  (161.3s)
CPU times: user 57min 8s, sys: 29 s, total: 57min 37s
Wall time: 17min 15s

# Compile the leaderboard
records = []

for (name, K), res in all_results.items():

    records.append({'strategy': name, 'K': K, **res})

leaderboard = (pd.DataFrame(records).sort_values(['K','composite'], ascending = [True, False]).reset_index(drop=True))
leaderboard[['K','strategy'] + metric_keys + ['composite']]
K strategy precision recall f1 ndcg hit_rate composite
0 5 S7: Master Ensemble 0.0727 0.0608 0.0552 0.0875 0.2710 0.1230
1 5 S6: XGBoost 0.0707 0.0580 0.0530 0.0828 0.2630 0.1186
2 5 S3: Item-CF 0.0657 0.0538 0.0492 0.0808 0.2375 0.1094
3 5 S1: Popularity 0.0384 0.0298 0.0275 0.0457 0.1545 0.0671
4 5 S5: LightGBM 0.0118 0.0100 0.0094 0.0147 0.0560 0.0231
5 5 S2: Personal 0.0016 0.0010 0.0011 0.0016 0.0075 0.0029
6 5 S4: Temporal 0.0011 0.0006 0.0006 0.0014 0.0055 0.0021
7 10 S7: Master Ensemble 0.0648 0.1038 0.0666 0.0992 0.3930 0.1652
8 10 S6: XGBoost 0.0622 0.1001 0.0641 0.0939 0.3905 0.1616
9 10 S3: Item-CF 0.0559 0.0893 0.0574 0.0882 0.3475 0.1452
10 10 S1: Popularity 0.0323 0.0492 0.0324 0.0486 0.2270 0.0893
11 10 S5: LightGBM 0.0088 0.0139 0.0091 0.0151 0.0805 0.0296
12 10 S2: Personal 0.0013 0.0016 0.0012 0.0017 0.0120 0.0041
13 10 S4: Temporal 0.0011 0.0016 0.0011 0.0016 0.0110 0.0038

11. Benchmark visualisations

# Composite scores at K = 10
fig, ax = plt.subplots(figsize=(10, 5))
lb10 = leaderboard[leaderboard['K'] == 10].sort_values('composite')
bars = ax.barh(lb10['strategy'], lb10['composite'], color = sns.color_palette('husl', len(lb10)))
ax.set_xlabel('Composite Score (mean of P, R, F1, NDCG, HR)')
ax.set_title('Strategy Comparison - MovieLens 32M  |  K = 10')
ax.bar_label(bars, fmt='{:.4f}', padding=4)
plt.tight_layout()
plt.savefig('results_01_composite_k10.png', dpi=150, bbox_inches='tight')
plt.show()

png

# ── 12-3  Per-metric breakdown (K=10) ────────────────────────────────────────────
lb10 = leaderboard[leaderboard['K'] == 10].set_index('strategy')[metric_keys]
lb10.T.plot(kind='bar', figsize=(12, 5), colormap='tab10', edgecolor='white')
plt.title('Per-Metric Breakdown - MovieLens 32M  |  K=10')
plt.ylabel('Score'); plt.xticks(rotation=0)
plt.legend(loc='upper right', fontsize=8)
plt.tight_layout()
plt.savefig('results_03_per_metric.png', dpi=150, bbox_inches='tight')
plt.show()

png

# ── 12-4  NDCG vs Hit-Rate scatter ───────────────────────────────────────────────
fig, ax = plt.subplots(figsize=(8, 5))
for _, row in leaderboard[leaderboard['K'] == 10].iterrows():
    ax.scatter(row['hit_rate'], row['ndcg'], s=140, zorder=3)
    ax.annotate(row['strategy'], (row['hit_rate'], row['ndcg']),
                textcoords='offset points', xytext=(6, 3), fontsize=8)
ax.set_xlabel('Hit Rate @ K=10'); ax.set_ylabel('NDCG @ K=10')
ax.set_title('Ranking Quality vs Coverage')
plt.tight_layout()
plt.savefig('results_04_ndcg_hr_scatter.png', dpi=150, bbox_inches='tight')
plt.show()

png

# Get ft imp for the stage 2 ranker model
imp_df = pd.DataFrame({'feature'   : STAGE2_FEATURE_COLS,
                       'importance': meta_lgb.feature_importance(importance_type='gain'),}).sort_values('importance', ascending=True)

fig, ax = plt.subplots(figsize=(8, 5))
ax.barh(imp_df['feature'], imp_df['importance'], color = sns.color_palette('husl', len(imp_df)))
ax.set_xlabel('Gain')
ax.set_title('LightGBM Feature Importance (gain) - Stage 2 Meta-Ranker')
plt.tight_layout()
plt.savefig('results_05_feature_importance.png', dpi=150, bbox_inches='tight')
plt.show()

png

%%time

# Repopulate score caches for eval_df users (caches were cleared after the warm benchmark; eval_df is a superset that includes cold-start users too).
cold_sample   = eval_df.sample(n = 2000, random_state = 100)
cold_user_ids = cold_sample['userId'].values

uga_cold  = user_genre_affinity.reindex(cold_user_ids).fillna(0).values.astype('float32')
uga_norms = np.linalg.norm(uga_cold, axis = 1, keepdims = True).clip(min = 1e-9)
s2_cold   = movie_genre_norm @ (uga_cold / uga_norms).T

del uga_cold, uga_norms

taste_cold = np.vstack([user_taste_vec.get(uid, np.zeros(len(genre_cols), dtype='float32')) for uid in cold_user_ids]).astype('float32')
s4_cold = movie_genre_norm @ taste_cold.T
del taste_cold

_s2_score_cache = {uid: s2_cold[:, i] for i, uid in enumerate(cold_user_ids)}
_s4_score_cache = {uid: s4_cold[:, i] for i, uid in enumerate(cold_user_ids)}
del s2_cold, s4_cold; gc.collect()

cold_results = {}

for name, fn in STRATEGIES.items():

    res = evaluate_strategy(fn, cold_sample, K = 10, n = None)
    res['composite'] = float(np.mean([res[m] for m in composite_keys]))
    cold_results[name] = res

_s2_score_cache.clear()
_s4_score_cache.clear()

CPU times: user 27min 38s, sys: 3.06 s, total: 27min 41s
Wall time: 8min 13s

Back to top..

12. Leaderboard & Takeaways

# Master
for k_val in [5, 10]:

    print(f'{"="*60}')
    print(f'  LEADERBOARD @ K={k_val}')
    print('='*60)
    lb = leaderboard[leaderboard['K'] == k_val].reset_index(drop = True)
    lb.index += 1
    print(lb[['strategy','precision','recall','f1','ndcg','hit_rate','composite']].to_string())
    print(f'\n  Winner @ K={k_val}: {lb.iloc[0]["strategy"]}\n')

============================================================
  LEADERBOARD @ K=5
============================================================
              strategy  precision  recall     f1   ndcg  hit_rate  composite
1  S7: Master Ensemble     0.0727  0.0608 0.0552 0.0875    0.2710     0.1230
2          S6: XGBoost     0.0707  0.0580 0.0530 0.0828    0.2630     0.1186
3          S3: Item-CF     0.0657  0.0538 0.0492 0.0808    0.2375     0.1094
4       S1: Popularity     0.0384  0.0298 0.0275 0.0457    0.1545     0.0671
5         S5: LightGBM     0.0118  0.0100 0.0094 0.0147    0.0560     0.0231
6         S2: Personal     0.0016  0.0010 0.0011 0.0016    0.0075     0.0029
7         S4: Temporal     0.0011  0.0006 0.0006 0.0014    0.0055     0.0021

  Winner @ K=5: S7: Master Ensemble

============================================================
  LEADERBOARD @ K=10
============================================================
              strategy  precision  recall     f1   ndcg  hit_rate  composite
1  S7: Master Ensemble     0.0648  0.1038 0.0666 0.0992    0.3930     0.1652
2          S6: XGBoost     0.0622  0.1001 0.0641 0.0939    0.3905     0.1616
3          S3: Item-CF     0.0559  0.0893 0.0574 0.0882    0.3475     0.1452
4       S1: Popularity     0.0323  0.0492 0.0324 0.0486    0.2270     0.0893
5         S5: LightGBM     0.0088  0.0139 0.0091 0.0151    0.0805     0.0296
6         S2: Personal     0.0013  0.0016 0.0012 0.0017    0.0120     0.0041
7         S4: Temporal     0.0011  0.0016 0.0011 0.0016    0.0110     0.0038

  Winner @ K=10: S7: Master Ensemble

# S5 v S6
#print('='*60)
#print('S7 vs S5 -- head-to-head')
#print('='*60)

#for k_val in [5, 10]:

#    r5 = all_results[('S5: LightGBM',        k_val)]
#    r6 = all_results[('S7: Master Ensemble',  k_val)]

#    print(f'\n  @ K={k_val}')
#    print(f'  {"Metric":<12}  {"S5":>8}  {"S6":>8}  {"Delta":>8}  {"% chg":>8}')
#    print(f'  {"-"*50}')

#    for mk in metric_keys:

#        v5, v6 = r5[mk], r6[mk]
#        d   = v6 - v5
#        pct = d / v5 * 100 if v5 > 0 else 0.0
#        arr = 'up' if d > 0 else ('dn' if d < 0 else '==')
#        print(f'  {mk:<12}  {v5:8.4f}  {v6:8.4f}  {d:+8.4f}  {arr} {pct:+.2f}%')

Back to top..

Download ml-32m.zip from https://grouplens.org/datasets/movielens/32m/. Cite: Harper & Konstan (2015), ACM TiiS 5(4):19.

]]>Data Guydataguy50@gmail.comInstaCart User Data: Machine Learning Recommendation Engine2026-03-28T00:00:00+00:002026-03-28T00:00:00+00:00https://doctorofdata.github.io/posts/2026/3/instacart-predictive-shopping-ensembleThis is a continuation of the market basket analysis conducted in the previous post. In this extension, we will use machine learning recommendations in combination with age-old fallback strategies to create an ensemble that should perform well on the validation data.

In our last post, we examined five different strategies for offering suggestions to users as they make purchases using Instacart. Here, we will assess a combined strategy utilizing the priors under different categories of circumstances to see if, together, they will all work together to yield more accurate suggestions!

📋 Table of Contents

# Section
1 Setup & data loading
2 Feature engineering
3 Association rule mining
4 Markov chain construction
5 Evaluation harness & S1–S5 strategies
6 S6 architecture & cold-start gate
7 Stage 1 — Expanded candidate pool
8 Stage 2 — Meta-ranker training
9 S6 inference function
10 Full benchmark: S1 → S6
11 Visualisations
12 Leaderboard & takeaways

Key findings

Finding Detail
S5 wins overall LightGBM ranks the most likely item highest, especially at K=5
S6 is competitive at K=10 Wider candidate pool pays off as K grows — trails S5 by < 0.1 composite points
S2 is a strong baseline Personal history alone outperforms S3 and S4 at every cut
S3 underperforms Only 12 antecedents from Apriori — too sparse to be a meaningful standalone signal
S4 adds sequence signal Strongest in NDCG, suggesting it surfaces the right item even when precision is lower
Cold-start gate works 11.8% of users hit the fallback; without it S6 scores would be dragged down

Opportunities for improvement

  • More association rules: Lower min_support to 0.005 — the current 12-antecedent lookup limits S3’s impact
  • Retrain S5 with more data: best_iteration = 1 suggests the model is undertrained; increase the pool beyond 4,000 users
  • Listwise / pairwise loss: LambdaRank or LambdaMART would optimise NDCG directly
  • Temporal features: days_since_prior_order exists but isn’t used as a contextual signal

🏆 InstaCart Master Ensemble: Two-Stage Generate-&-Rerank Model

Instacart Market Basket Analysis — Fully Self-Contained Notebook

Data: Mount your Google Drive and point ZIP_PATH to the Instacart zip archive. All other paths are derived automatically.

📋 Walkthrough note — Each section below contains the working code followed immediately by the methodology and validation commentary for that component.

1. Setup & data loading

📖 Section 1 — Dataset & Problem Framing

What the data is. The Instacart Market Basket Analysis dataset contains the order histories of 206,209 anonymous customers across 3,421,083 orders, covering 49,688 unique products across 21 departments and 134 aisles. order_products__prior holds 32,434,489 individual product-placement records. Orders labelled prior form the historical record; orders labelled train are the ground truth next baskets we predict against.

Task framing. This is a retrieve-then-rank next-basket recommendation problem: given a user’s current cart (simulated as their most recent prior order in add-to-cart sequence), predict which products they add next. Using the last prior order as the “current cart” is the closest available approximation to a live mid-session signal — using all prior purchases instead would collapse the temporal structure the Markov chain depends on.

Memory strategy. order_products__prior at 32M rows occupies ~3.5 GB as int64. All integer columns are immediately downcast to int32 (4 bytes, zero precision loss — max value well within 2,147,483,647). Large intermediate objects are explicitly freed with del + gc.collect() at their last usage point: user_order_enriched after up_feat is built, prior_op after the Markov chain completes.

📖 Section 3 — Evaluation Harness & Leakage Prevention

Five metrics at K=5 and K=10:

\[P@K = \frac{|\text{recs}[:K] \cap \text{true\_items}|}{K} \qquad R@K = \frac{|\text{recs}[:K] \cap \text{true\_items}|}{|\text{true\_items}|}\] \[F1@K = \frac{2 \cdot P@K \cdot R@K}{P@K + R@K} \qquad \text{NDCG}@K = \frac{\sum_{i=1}^{K} \frac{\mathbb{1}[\text{rec}_i \in \text{true}]}{\log_2(i+1)}}{\text{IDCG}@K}\] \[\text{Hit}@K = \mathbb{1}[\exists\, p \in \text{recs}[:K] : p \in \text{true\_items}]\]

NDCG is the most sensitive metric to ranking order — a correct item at position 1 is worth 1.0 vs. 0.29 at position 10. Hit Rate is the most lenient — it only asks whether any recommendation was useful. The composite score is the unweighted mean of all five, preventing a strategy from gaming a single metric.

Leakage fix. The original notebook sampled training and evaluation users from the same pool with the same random seed — the 4,000 training users were a strict subset of the 5,000 evaluation users. Every model was tested on users it trained on.

Fixed with GroupShuffleSplit(test_size=0.30, groups=user_id) producing 91,846 train-fold users and 39,363 eval-fold users with zero overlap. S5, S6, and S7 all train exclusively from train_fold; the benchmark draws exclusively from eval_fold.

Benchmark population. The benchmark filters to warm users only (eval_warm) before evaluation. S6 routes 12.1% of users (≤ 3 prior orders) to a weak cold-start fallback; S5 and S7 run their full ranker on all users. Including cold-start users in the comparison would give S5/S7 an uncontrolled advantage — evaluating on identical warm users makes the comparison fair.

# Import libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import zipfile, gc, time, warnings
from collections import defaultdict, Counter
from itertools import combinations
import lightgbm as lgb
from xgboost import XGBClassifier
from sklearn.model_selection import GroupShuffleSplit
from sklearn.metrics import roc_auc_score, average_precision_score
from mlxtend.frequent_patterns import apriori, association_rules
from mlxtend.preprocessing import TransactionEncoder
from google.colab import drive
import tempfile
import os

warnings.filterwarnings('ignore')
plt.style.use('seaborn-v0_8-whitegrid')
sns.set_palette('husl')
pd.set_option('display.float_format', '{:.4f}'.format)
pd.set_option('display.max_columns', None)
print('✅  Libraries loaded')

✅  Libraries loaded

# Mount drive to access data
drive.mount('/content/drive/')

Mounted at /content/drive/

# Define functions to perform scoring
def precision_at_k(recs, true_set, k):

    return len(set(recs[:k]) & true_set) / k if k else 0.0

def recall_at_k(recs, true_set, k):

    return len(set(recs[:k]) & true_set) / len(true_set) if true_set else 0.0

def f1_at_k(recs, true_set, k):

    p = precision_at_k(recs, true_set, k)
    r = recall_at_k(recs, true_set, k)
    return 2*p*r/(p+r) if (p+r) > 0 else 0.0

def ndcg_at_k(recs, true_set, k):

    dcg   = sum(1/np.log2(i+2) for i, p in enumerate(recs[:k]) if p in true_set)
    ideal = sum(1/np.log2(i+2) for i in range(min(len(true_set), k)))
    return dcg/ideal if ideal else 0.0

def score_recs(recs, true_set, K):

    return {
        'precision' : precision_at_k(recs, true_set, K),
        'recall'    : recall_at_k(recs,    true_set, K),
        'f1'        : f1_at_k(recs,        true_set, K),
        'ndcg'      : ndcg_at_k(recs,      true_set, K),
        'hit_rate'  : 1 if any(p in true_set for p in recs[:K]) else 0,
    }

def evaluate_strategy(fn, data_df, K = 10, n = None, seed = 100):

    rows = data_df if n is None else data_df.sample(n = n, random_state = seed)
    m = {k: [] for k in ('precision','recall','f1','ndcg','hit_rate')}

    for _, row in rows.iterrows():

        s = score_recs(fn(row['user_id'], row['cart_sequence'], K=K), row['true_items'], K)
        for key in m: m[key].append(s[key])

    return {k: float(np.mean(v)) for k, v in m.items()}

def s1_popularity(user_id, cart_sequence, K = 10, **_):

    in_cart = set(cart_sequence)

    return [p for p in POPULARITY_POOL if p not in in_cart][:K]

def s2_personal(user_id, cart_sequence, K = 10, **_):

    in_cart = set(cart_sequence)

    return [p for p in user_history.get(user_id, []) if p not in in_cart][:K]

def s3_rules(user_id, cart_sequence, K = 10, **_):

    in_cart_ids   = set(cart_sequence)
    in_cart_names = {pid_to_name.get(p, '') for p in cart_sequence}
    scores = defaultdict(float)

    for name in in_cart_names:

        for con_name, score in assoc_lookup.get(name, {}).items():

            pid = name_to_pid.get(con_name)

            if pid and pid not in in_cart_ids:

                scores[pid] += score

    ranked = sorted(scores, key=scores.get, reverse=True)

    if len(ranked) >= K:

        return ranked[:K]

    extra = [p for p in s2_personal(user_id, cart_sequence, K*3) if p not in set(ranked)]

    return (ranked + extra)[:K]

# Build the sequential markov chain
def s4_markov(user_id, cart_sequence, K = 10, n_gram = 2, alpha = 0.4, beta = 0.25, **_):

    in_cart  = set(cart_sequence)
    next_pos = len(cart_sequence)
    context  = cart_sequence[-n_gram:] if len(cart_sequence) >= n_gram else cart_sequence
    scores   = defaultdict(float)

    for prev in context:

        for nxt, prob in trans_prob.get(prev, {}).items():

            if nxt not in in_cart:

                scores[nxt] += prob

    capped_pos = min(next_pos, MAX_POS - 1)

    for pid, prob in pos_prob.get(capped_pos, {}).items():

        if pid not in in_cart:

            scores[pid] += alpha * prob

    for pid, rs in user_rs_list.get(user_id, []):

        if pid not in in_cart and rs > 0:

            scores[pid] += beta * float(rs)

    ranked = sorted(scores, key=scores.get, reverse=True)

    if len(ranked) >= K:

        return ranked[:K]

    extra = [p for p in s2_personal(user_id, cart_sequence, K*3) if p not in set(ranked)]

    return (ranked + extra)[:K]

# Function to perform inference on lgb model
def s5_lgbm(user_id, cart_sequence, K = 10, **_):

    in_cart  = set(cart_sequence)
    cart_sz  = len(cart_sequence)
    personal   = [p for p in user_history.get(user_id, [])[:MAX_PERSONAL] if p not in in_cart]
    globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
    candidates = personal + globl

    if not candidates:

        return s1_popularity(user_id, cart_sequence, K)

    markov_sc = defaultdict(float)

    for prev in cart_sequence:

        for nxt, p in trans_prob.get(prev, {}).items():

            markov_sc[nxt] += p

    assoc_sc = defaultdict(float)

    for ant_pid in cart_sequence:

        for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

            assoc_sc[con_pid] += sc

    pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

    rows = [{'user_id': user_id, 'product_id': int(pid),
             'cart_size': cart_sz,
             'markov_score':  float(markov_sc.get(pid, 0.0)),
             'assoc_score':   float(assoc_sc.get(pid, 0.0)),
             'pos_prob_next': float(pos_p.get(pid, 0.0))}
            for pid in candidates]

    inf_df = (pd.DataFrame(rows)
              .merge(prod_feat,    on='product_id',            how='left')
              .merge(user_feat_s5, on='user_id',               how='left')
              .merge(up_feat,      on=['user_id','product_id'], how='left')
              .fillna(0))

    for fc in FEATURE_COLS:

        if fc not in inf_df.columns:

            inf_df[fc] = 0.0

    scores = lgb_model.predict_proba(inf_df[FEATURE_COLS].values)[:, 1]
    inf_df['score'] = scores

    return inf_df.nlargest(K, 'score')['product_id'].tolist()

# Function to evaluate lgb model in batches
def evaluate_lgbm_batched(eval_data, K = 10, n = None, seed = 100, chunk_size = 200):

    rows_df = eval_data if n is None else eval_data.sample(n=n, random_state=seed)
    rows_df = rows_df.reset_index(drop=True)
    m = {k: [] for k in ('precision','recall','f1','ndcg','hit_rate')}

    for chunk_start in range(0, len(rows_df), chunk_size):

        chunk     = rows_df.iloc[chunk_start:chunk_start + chunk_size]
        cand_rows = []

        for idx, row in chunk.iterrows():

            uid      = int(row['user_id'])
            cart_seq = row['cart_sequence']
            in_cart  = set(cart_seq)
            cart_sz  = len(cart_seq)
            personal   = [p for p in user_history.get(uid,[])[:MAX_PERSONAL] if p not in in_cart]
            globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
            candidates = personal + globl or s1_popularity(uid, cart_seq, K)

            markov_sc = defaultdict(float)

            for prev in cart_seq:

                for nxt, p in trans_prob.get(prev, {}).items():

                    markov_sc[nxt] += p

            assoc_sc = defaultdict(float)

            for ant_pid in cart_seq:

                for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

                    assoc_sc[con_pid] += sc

            pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

            for pid in candidates:

                cand_rows.append((idx, uid, int(pid), cart_sz,
                                  float(markov_sc.get(pid,0.0)),
                                  float(assoc_sc.get(pid,0.0)),
                                  float(pos_p.get(pid,0.0))))

        if not cand_rows:

            continue

        c_df = pd.DataFrame(cand_rows, columns = ['idx','user_id','product_id','cart_size','markov_score','assoc_score','pos_prob_next'])
        c_df = (c_df.merge(prod_feat,    on = 'product_id',            how = 'left')
                    .merge(user_feat_s5, on = 'user_id',               how = 'left')
                    .merge(up_feat,      on = ['user_id','product_id'], how = 'left')
                    .fillna(0))

        for fc in FEATURE_COLS:

            if fc not in c_df.columns: c_df[fc] = 0.0

        c_df['score'] = lgb_model.predict_proba(c_df[FEATURE_COLS].values)[:, 1]

        for idx, row in chunk.iterrows():

            user_cands = c_df[c_df['idx'] == idx]
            recs = [] if user_cands.empty else user_cands.nlargest(K,'score')['product_id'].tolist()
            s = score_recs(recs, row['true_items'], K)

            for key in m: m[key].append(s[key])

        del c_df, cand_rows; gc.collect()

    return {k: float(np.mean(v)) for k, v in m.items()}

# Function to check user history
def is_cold_start(user_id):

    return user_order_count.get(int(user_id), 0) <= 3

# Function to get recs for those with no history
def cold_start_blend(user_id, cart_sequence, K = 10):

    """S1 popularity blended with any S3 association rule hits."""
    assoc_recs = s3_rules(user_id, cart_sequence, K = K)
    pop_recs   = s1_popularity(user_id, cart_sequence, K = K * 2)
    seen    = set(assoc_recs)
    blended = list(assoc_recs)

    for p in pop_recs:

        if p not in seen:

            blended.append(p)
            seen.add(p)

        if len(blended) >= K:

            break

    return blended[:K]

# Function to create candidtaes pool
def build_candidate_pool(user_id, cart_sequence):

    """
    Returns (candidates: list[int], meta: dict[int → dict]).

    meta keys per product_id:
      s2_score       reorder_score from S2 personal history
      s3_score       aggregated conf×lift from S3 rules fired by the cart
      s4_score       Markov + position probability from S4
      vote_count     how many of the 3 generators nominated this item (1–3)
      from_personal  1 if nominated by S2
      from_assoc     1 if nominated by S3
      from_markov    1 if nominated by S4
      from_global    1 if added only as global popularity fill
    """
    in_cart  = set(cart_sequence)
    cart_sz  = len(cart_sequence)
    meta     = defaultdict(lambda: {
        's2_score':0.0,'s3_score':0.0,'s4_score':0.0,
        'vote_count':0,'from_personal':0,'from_assoc':0,
        'from_markov':0,'from_global':0})

    # Personal history
    s2_pool    = [p for p in user_history.get(user_id, []) if p not in in_cart][:S6_PERSONAL_CAP]
    rs_lookup  = dict(user_rs_list.get(user_id, []))

    for pid in s2_pool:

        meta[pid]['s2_score']      = float(rs_lookup.get(pid, 0.0))
        meta[pid]['from_personal'] = 1
        meta[pid]['vote_count']   += 1

    # Assoc rules
    in_cart_names = {pid_to_name.get(p,'') for p in cart_sequence}
    assoc_scores  = defaultdict(float)

    for name in in_cart_names:

        for con_name, score in assoc_lookup.get(name, {}).items():

            pid = name_to_pid.get(con_name)

            if pid and pid not in in_cart:

                assoc_scores[pid] += score

    s3_pool = sorted(assoc_scores, key=assoc_scores.get, reverse=True)[:S6_ASSOC_CAP]

    for pid in s3_pool:

        meta[pid]['s3_score']   = float(assoc_scores.get(pid, 0.0))
        meta[pid]['from_assoc'] = 1
        meta[pid]['vote_count'] += 1

    # Markov chain
    context     = cart_sequence[-2:] if len(cart_sequence) >= 2 else cart_sequence
    capped_pos  = min(cart_sz, MAX_POS - 1)
    markov_scores = defaultdict(float)

    for prev in context:

        for nxt, prob in trans_prob.get(prev, {}).items():

            if nxt not in in_cart:

                markov_scores[nxt] += prob

    for pid, prob in pos_prob.get(capped_pos, {}).items():

        if pid not in in_cart:

            markov_scores[pid] += 0.4 * prob

    s4_pool = sorted(markov_scores, key=markov_scores.get, reverse=True)[:S6_MARKOV_CAP]

    for pid in s4_pool:

        meta[pid]['s4_score']    = float(markov_scores.get(pid, 0.0))
        meta[pid]['from_markov'] = 1
        meta[pid]['vote_count'] += 1

    # Global
    # Markov pool is scored but NOT added as candidate source — it is anti-predictive
    # as a nominator (lift 0.53×) but its scores remain useful features on other candidates
    all_cands = set(s2_pool) | set(s3_pool) | set(s4_pool)

    for pid in GLOBAL_POOL:

        if len(all_cands) >= S6_PERSONAL_CAP + S6_GLOBAL_FILL:

            break

        if pid not in in_cart and pid not in all_cands:

            meta[pid]['from_global'] = 1
            meta[pid]['vote_count']  = max(meta[pid]['vote_count'], 1)
            all_cands.add(pid)

    return list(all_cands), dict(meta)

# Function to perform the meta ranking inference
def s6_ensemble(user_id, cart_sequence, K=10, **_):

    """
    Master ensemble (S6).

    Stage 0  Cold-start gate — users with ≤ COLD_START_THRESHOLD orders
             get the S1+S3 fallback directly.
    Stage 1  build_candidate_pool → union of S2/S3/S4 nominees + meta-scores.
    Stage 2  S6 LightGBM meta-ranker → top-K product IDs.
    """
    if is_cold_start(user_id):

        return cold_start_blend(user_id, cart_sequence, K=K)

    in_cart = set(cart_sequence)
    cart_sz = len(cart_sequence)

    candidates, meta = build_candidate_pool(user_id, cart_sequence)

    if not candidates:

        return cold_start_blend(user_id, cart_sequence, K=K)

    markov_sc = defaultdict(float)

    for prev in cart_sequence:

        for nxt, p in trans_prob.get(prev, {}).items():

            markov_sc[nxt] += p

    assoc_sc = defaultdict(float)

    for ant_pid in cart_sequence:

        for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

            assoc_sc[con_pid] += sc

    pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

    rows = []

    for pid in candidates:

        m = meta.get(pid, {})

        rows.append({
            'user_id':       user_id,
            'product_id':    int(pid),
            'cart_size':     cart_sz,
            'markov_score':  float(markov_sc.get(pid, 0.0)),
            'assoc_score':   float(assoc_sc.get(pid, 0.0)),
            'pos_prob_next': float(pos_p.get(pid, 0.0)),
            's2_score':      float(m.get('s2_score', 0.0)),
            's3_score':      float(m.get('s3_score', 0.0)),
            's4_score':      float(m.get('s4_score', 0.0)),
            'vote_count':    int(m.get('vote_count', 0)),
            'from_personal': int(m.get('from_personal', 0)),
            'from_assoc':    int(m.get('from_assoc', 0)),
            'from_markov':   int(m.get('from_markov', 0)),
            'from_global':   int(m.get('from_global', 0)),
        })

    inf_df = (pd.DataFrame(rows)
              .merge(prod_feat,    on='product_id',            how='left')
              .merge(user_feat_s5, on='user_id',               how='left')
              .merge(up_feat,      on=['user_id','product_id'], how='left')
              .fillna(0))

    for fc in S6_FEATURE_COLS:

        if fc not in inf_df.columns:

            inf_df[fc] = 0.0

    inf_df['score'] = s6_model.predict_proba(inf_df[S6_FEATURE_COLS].values)[:, 1]

    return inf_df.nlargest(K, 'score')['product_id'].tolist()

# Function to produce the meta rakning in chunks
def evaluate_s6_batched(eval_data, K = 10, n = None, seed = 100, chunk_size = 150):

    rows_df = eval_data if n is None else eval_data.sample(n=n, random_state=seed)
    rows_df = rows_df.reset_index(drop=True)
    m = {k: [] for k in ('precision','recall','f1','ndcg','hit_rate')}

    for chunk_start in range(0, len(rows_df), chunk_size):

        chunk     = rows_df.iloc[chunk_start:chunk_start + chunk_size]
        cand_rows = []

        for idx, row in chunk.iterrows():
            uid      = int(row['user_id'])
            cart_seq = row['cart_sequence']
            in_cart  = set(cart_seq)
            cart_sz  = len(cart_seq)

            if is_cold_start(uid):
                recs = cold_start_blend(uid, cart_seq, K=K)
                s    = score_recs(recs, row['true_items'], K)
                for key in m: m[key].append(s[key])
                continue

            candidates, meta = build_candidate_pool(uid, cart_seq)
            if not candidates:
                candidates, meta = s1_popularity(uid, cart_seq, K), {}

            markov_sc = defaultdict(float)
            for prev in cart_seq:
                for nxt, p in trans_prob.get(prev, {}).items():
                    markov_sc[nxt] += p
            assoc_sc = defaultdict(float)
            for ant_pid in cart_seq:
                for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():
                    assoc_sc[con_pid] += sc
            pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

            for pid in candidates:
                mm = meta.get(pid, {})
                cand_rows.append((
                    idx, uid, int(pid), cart_sz,
                    float(markov_sc.get(pid,0.0)), float(assoc_sc.get(pid,0.0)),
                    float(pos_p.get(pid,0.0)),
                    float(mm.get('s2_score',0.0)), float(mm.get('s3_score',0.0)),
                    float(mm.get('s4_score',0.0)), int(mm.get('vote_count',0)),
                    int(mm.get('from_personal',0)), int(mm.get('from_assoc',0)),
                    int(mm.get('from_markov',0)),   int(mm.get('from_global',0)),
                ))

        if not cand_rows:
            continue

        c_df = pd.DataFrame(cand_rows, columns=[
            'idx','user_id','product_id','cart_size',
            'markov_score','assoc_score','pos_prob_next',
            's2_score','s3_score','s4_score','vote_count',
            'from_personal','from_assoc','from_markov','from_global'])
        c_df = (c_df.merge(prod_feat,    on='product_id',            how='left')
                    .merge(user_feat_s5, on='user_id',               how='left')
                    .merge(up_feat,      on=['user_id','product_id'], how='left')
                    .fillna(0))
        for fc in S6_FEATURE_COLS:
            if fc not in c_df.columns: c_df[fc] = 0.0
        c_df['score'] = s6_model.predict_proba(c_df[S6_FEATURE_COLS].values)[:, 1]

        for idx, row in chunk.iterrows():
            user_cands = c_df[c_df['idx'] == idx]
            if user_cands.empty:
                recs = cold_start_blend(int(row['user_id']), row['cart_sequence'], K=K)
            else:
                recs = user_cands.nlargest(K,'score')['product_id'].tolist()
            s = score_recs(recs, row['true_items'], K)
            for key in m: m[key].append(s[key])

        del c_df, cand_rows; gc.collect()

    return {k: float(np.mean(v)) for k, v in m.items()}

# Function to perform inference on xgb model
def s7_xgb(user_id, cart_sequence, K = 10, **_):

    in_cart  = set(cart_sequence)
    cart_sz  = len(cart_sequence)
    personal   = [p for p in user_history.get(user_id, [])[:MAX_PERSONAL] if p not in in_cart]
    globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
    candidates = personal + globl

    if not candidates:

        return s1_popularity(user_id, cart_sequence, K)

    markov_sc = defaultdict(float)

    for prev in cart_sequence:

        for nxt, p in trans_prob.get(prev, {}).items():

            markov_sc[nxt] += p

    assoc_sc = defaultdict(float)

    for ant_pid in cart_sequence:

        for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

            assoc_sc[con_pid] += sc

    pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

    rows = [{'user_id': user_id, 'product_id': int(pid),
             'cart_size': cart_sz,
             'markov_score':  float(markov_sc.get(pid, 0.0)),
             'assoc_score':   float(assoc_sc.get(pid, 0.0)),
             'pos_prob_next': float(pos_p.get(pid, 0.0))}
            for pid in candidates]

    inf_df = (pd.DataFrame(rows)
              .merge(prod_feat,    on='product_id',            how='left')
              .merge(user_feat_s5, on='user_id',               how='left')
              .merge(up_feat,      on=['user_id','product_id'], how='left')
              .fillna(0))

    for fc in FEATURE_COLS:

        if fc not in inf_df.columns:

            inf_df[fc] = 0.0

    scores = xgb_model.predict_proba(inf_df[FEATURE_COLS].values)[:, 1]
    inf_df['score'] = scores

    return inf_df.nlargest(K, 'score')['product_id'].tolist()


# ── Single-pass benchmark: scores K=5 and K=10 simultaneously ─────────────────
def evaluate_strategy_multi_k(fn, data_df, ks = (5,10), n = None, seed = 100):

    """Generic single-pass evaluator for non-batched strategies."""
    rows = data_df if n is None else data_df.sample(n=n, random_state=seed)
    m = {k: {mk: [] for mk in ('precision','recall','f1','ndcg','hit_rate')}
         for k in ks}
    for _, row in rows.iterrows():
        recs = fn(row['user_id'], row['cart_sequence'], K=max(ks))
        for k in ks:
            s = score_recs(recs, row['true_items'], k)
            for key in m[k]: m[k][key].append(s[key])
    return {k: {mk: float(np.mean(v)) for mk, v in m[k].items()} for k in ks}


def evaluate_lgbm_batched_multi_k(eval_data, ks=(5,10), n=None, seed=100, chunk_size=200):
    """S5 single-pass batched evaluator for multiple K values."""
    rows_df = eval_data if n is None else eval_data.sample(n=n, random_state=seed)
    rows_df = rows_df.reset_index(drop=True)
    m = {k: {mk: [] for mk in ('precision','recall','f1','ndcg','hit_rate')}
         for k in ks}
    max_k = max(ks)

    for chunk_start in range(0, len(rows_df), chunk_size):
        chunk     = rows_df.iloc[chunk_start:chunk_start + chunk_size]
        cand_rows = []

        for idx, row in chunk.iterrows():
            uid      = int(row['user_id'])
            cart_seq = row['cart_sequence']
            in_cart  = set(cart_seq)
            cart_sz  = len(cart_seq)
            personal   = [p for p in user_history.get(uid,[])[:MAX_PERSONAL] if p not in in_cart]
            globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
            candidates = personal + globl or s1_popularity(uid, cart_seq, max_k)

            markov_sc = defaultdict(float)
            for prev in cart_seq:
                for nxt, p in trans_prob.get(prev, {}).items():
                    markov_sc[nxt] += p
            assoc_sc = defaultdict(float)
            for ant_pid in cart_seq:
                for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():
                    assoc_sc[con_pid] += sc
            pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

            for pid in candidates:
                cand_rows.append((idx, uid, int(pid), cart_sz,
                                  float(markov_sc.get(pid,0.0)),
                                  float(assoc_sc.get(pid,0.0)),
                                  float(pos_p.get(pid,0.0))))

        if not cand_rows:
            continue

        c_df = pd.DataFrame(cand_rows, columns=[
            'idx','user_id','product_id','cart_size',
            'markov_score','assoc_score','pos_prob_next'])
        c_df = (c_df.merge(prod_feat,    on='product_id',            how='left')
                    .merge(user_feat_s5, on='user_id',               how='left')
                    .merge(up_feat,      on=['user_id','product_id'], how='left')
                    .fillna(0))
        for fc in FEATURE_COLS:
            if fc not in c_df.columns: c_df[fc] = 0.0
        c_df['score'] = lgb_model.predict_proba(c_df[FEATURE_COLS].values)[:, 1]

        for idx, row in chunk.iterrows():
            user_cands = c_df[c_df['idx'] == idx]
            recs = [] if user_cands.empty else user_cands.nlargest(max_k, 'score')['product_id'].tolist()
            for k in ks:
                s = score_recs(recs, row['true_items'], k)
                for key in m[k]: m[k][key].append(s[key])

        del c_df, cand_rows; gc.collect()

    return {k: {mk: float(np.mean(v)) for mk, v in m[k].items()} for k in ks}


def evaluate_s6_batched_multi_k(eval_data, ks = (5,10), n = None, seed = 100, chunk_size = 150):

    """S6 single-pass batched evaluator for multiple K values."""
    rows_df = eval_data if n is None else eval_data.sample(n=n, random_state=seed)
    rows_df = rows_df.reset_index(drop=True)
    m = {k: {mk: [] for mk in ('precision','recall','f1','ndcg','hit_rate')}
         for k in ks}
    max_k = max(ks)

    for chunk_start in range(0, len(rows_df), chunk_size):

        chunk     = rows_df.iloc[chunk_start:chunk_start + chunk_size]
        cand_rows = []

        for idx, row in chunk.iterrows():
            uid      = int(row['user_id'])
            cart_seq = row['cart_sequence']
            in_cart  = set(cart_seq)
            cart_sz  = len(cart_seq)

            if is_cold_start(uid):

                recs = cold_start_blend(uid, cart_seq, K=max_k)

                for k in ks:

                    s = score_recs(recs, row['true_items'], k)
                    for key in m[k]: m[k][key].append(s[key])

                continue

            candidates, meta = build_candidate_pool(uid, cart_seq)

            if not candidates:

                candidates, meta = s1_popularity(uid, cart_seq, max_k), {}

            markov_sc = defaultdict(float)

            for prev in cart_seq:

                for nxt, p in trans_prob.get(prev, {}).items():

                    markov_sc[nxt] += p

            assoc_sc = defaultdict(float)

            for ant_pid in cart_seq:

                for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

                    assoc_sc[con_pid] += sc

            pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

            for pid in candidates:

                mm = meta.get(pid, {})

                cand_rows.append((
                    idx, uid, int(pid), cart_sz,
                    float(markov_sc.get(pid,0.0)), float(assoc_sc.get(pid,0.0)),
                    float(pos_p.get(pid,0.0)),
                    float(mm.get('s2_score',0.0)), float(mm.get('s3_score',0.0)),
                    float(mm.get('s4_score',0.0)), int(mm.get('vote_count',0)),
                    int(mm.get('from_personal',0)), int(mm.get('from_assoc',0)),
                    int(mm.get('from_markov',0)),   int(mm.get('from_global',0)),
                ))

        if not cand_rows:
            continue

        c_df = pd.DataFrame(cand_rows, columns=[
            'idx','user_id','product_id','cart_size',
            'markov_score','assoc_score','pos_prob_next',
            's2_score','s3_score','s4_score','vote_count',
            'from_personal','from_assoc','from_markov','from_global'])
        c_df = (c_df.merge(prod_feat,    on='product_id',            how='left')
                    .merge(user_feat_s5, on='user_id',               how='left')
                    .merge(up_feat,      on=['user_id','product_id'], how='left')
                    .fillna(0))
        for fc in S6_FEATURE_COLS:
            if fc not in c_df.columns: c_df[fc] = 0.0
        c_df['score'] = s6_model.predict_proba(c_df[S6_FEATURE_COLS].values)[:, 1]

        for idx, row in chunk.iterrows():
            user_cands = c_df[c_df['idx'] == idx]
            if user_cands.empty:
                recs = cold_start_blend(int(row['user_id']), row['cart_sequence'], K=max_k)
            else:
                recs = user_cands.nlargest(max_k, 'score')['product_id'].tolist()
            for k in ks:
                s = score_recs(recs, row['true_items'], k)
                for key in m[k]: m[k][key].append(s[key])

        del c_df, cand_rows; gc.collect()

    return {k: {mk: float(np.mean(v)) for mk, v in m[k].items()} for k in ks}

def evaluate_xgb_batched_multi_k(eval_data, ks = (5,10), n = None, seed = 100, chunk_size = 200):

    """S7 single-pass batched evaluator for multiple K values."""
    rows_df = eval_data if n is None else eval_data.sample(n=n, random_state=seed)
    rows_df = rows_df.reset_index(drop=True)
    m = {k: {mk: [] for mk in ('precision','recall','f1','ndcg','hit_rate')} for k in ks}
    max_k = max(ks)

    for chunk_start in range(0, len(rows_df), chunk_size):

        chunk     = rows_df.iloc[chunk_start:chunk_start + chunk_size]
        cand_rows = []

        for idx, row in chunk.iterrows():

            uid      = int(row['user_id'])
            cart_seq = row['cart_sequence']
            in_cart  = set(cart_seq)
            cart_sz  = len(cart_seq)
            personal   = [p for p in user_history.get(uid,[])[:MAX_PERSONAL] if p not in in_cart]
            globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
            candidates = personal + globl or s1_popularity(uid, cart_seq, max_k)

            markov_sc = defaultdict(float)

            for prev in cart_seq:

                for nxt, p in trans_prob.get(prev, {}).items():

                    markov_sc[nxt] += p

            assoc_sc = defaultdict(float)

            for ant_pid in cart_seq:

                for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

                    assoc_sc[con_pid] += sc

            pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

            for pid in candidates:

                cand_rows.append((idx, uid, int(pid), cart_sz,
                                  float(markov_sc.get(pid,0.0)),
                                  float(assoc_sc.get(pid,0.0)),
                                  float(pos_p.get(pid,0.0))))

        if not cand_rows:

            continue

        c_df = pd.DataFrame(cand_rows, columns=[
            'idx','user_id','product_id','cart_size',
            'markov_score','assoc_score','pos_prob_next'])
        c_df = (c_df.merge(prod_feat,    on='product_id',            how='left')
                    .merge(user_feat_s5, on='user_id',               how='left')
                    .merge(up_feat,      on=['user_id','product_id'], how='left')
                    .fillna(0))

        for fc in FEATURE_COLS:

            if fc not in c_df.columns: c_df[fc] = 0.0

        c_df['score'] = xgb_model.predict_proba(c_df[FEATURE_COLS].values)[:, 1]

        for idx, row in chunk.iterrows():

            user_cands = c_df[c_df['idx'] == idx]
            recs = [] if user_cands.empty else user_cands.nlargest(max_k, 'score')['product_id'].tolist()

            for k in ks:

                s = score_recs(recs, row['true_items'], k)
                for key in m[k]: m[k][key].append(s[key])

        del c_df, cand_rows; gc.collect()

    return {k: {mk: float(np.mean(v)) for mk, v in m[k].items()} for k in ks}

# Load the data
data = {}

with zipfile.ZipFile('/content/drive/MyDrive/instacart/instacart.zip', 'r') as z:

    file_list = [f for f in z.namelist() if f.endswith('.csv')]
    print(f'Files in archive: {file_list}\n')

    for fn in file_list:

        key = fn.replace('.csv', '')
        data[key] = pd.read_csv(z.open(fn))
        print(f'  ✓ {fn}: {data[key].shape[0]:,} rows × {data[key].shape[1]} cols')

print('\n✅  All datasets loaded')

Files in archive: ['aisles.csv', 'departments.csv', 'order_products__prior.csv', 'order_products__train.csv', 'orders.csv', 'products.csv']

  ✓ aisles.csv: 134 rows × 2 cols
  ✓ departments.csv: 21 rows × 2 cols
  ✓ order_products__prior.csv: 32,434,489 rows × 4 cols
  ✓ order_products__train.csv: 1,384,617 rows × 4 cols
  ✓ orders.csv: 3,421,083 rows × 7 cols
  ✓ products.csv: 49,688 rows × 4 cols

✅  All datasets loaded

# Cast to compact int types to keep RAM manageable for the 32M-row prior table.
for key in ['order_products__prior', 'order_products__train']:

    for col in ['order_id', 'product_id', 'add_to_cart_order', 'reordered']:

        data[key][col] = data[key][col].astype('int32')

for col in ['order_id', 'user_id', 'order_number', 'order_dow', 'order_hour_of_day']:

    data['orders'][col] = data['orders'][col].astype('int32')

# Enrich tables with metadata
products = (data['products'].merge(data['aisles'], on = 'aisle_id').merge(data['departments'],  on='department_id'))

prior_op = (data['order_products__prior'].merge(data['orders'][['order_id','user_id','order_number', 'days_since_prior_order']], on = 'order_id').merge(products[['product_id','product_name','aisle','department']], on = 'product_id'))
prior_op.rename(columns = {'product_name': 'product'}, inplace = True)
prior_op['user_id'] = prior_op['user_id'].astype('int32')

# Training
train_op = data['order_products__train']
orders_slim = data['orders'][['order_id','user_id','eval_set', 'order_number','days_since_prior_order']]

# Help maps
pid_to_name = dict(zip(products['product_id'], products['product_name']))
name_to_pid = dict(zip(products['product_name'], products['product_id']))
pid_to_dept = dict(zip(products['product_id'], products['department']))

print(f'prior_op rows : {len(prior_op):,}')
print(f'train_op rows : {len(train_op):,}')
print(f'Products      : {len(products):,}')

prior_op rows : 32,434,489
train_op rows : 1,384,617
Products      : 49,688

Back to top..

2. 📖 Feature engineering

Three feature tables are built here and reused by both S5 and S6:

  • prod_feat — per-product statistics (popularity, reorder rate, avg cart position)
  • user_feat — per-user statistics (order frequency, variety, avg cart size)
  • up_feat — user × product interaction features (reorder score, orders since last)

Three feature tables are assembled from prior_op in single grouped-aggregation passes and reused by every learned strategy (S5, S6, S7).

prod_feat (49,677 × 6) — product-level statistics:

Feature Formula Rationale
order_count nunique(order_id) Raw popularity — distinct baskets containing this item
reorder_rate mean(reordered) Fraction of appearances that were reorders; high = habitual staple
avg_cart_position mean(add_to_cart_order) Items added early are reached for automatically; later items are deliberate
global_popularity order_count / max(order_count) Normalised [0,1] continuous popularity signal
pop_rank Integer rank descending Ordinal signal robust to count outliers

user_feat (206,209 × 6) — user behavioural profile:

Feature Formula Rationale
total_orders nunique(order_id) User tenure proxy
avg_cart_size count / nunique(order_id) Controls for high-volume vs. low-volume shoppers
reorder_rate mean(reordered) Renamed reorder_rate_user at merge to avoid column collision
avg_days_between mean(days_since_prior_order) Shopping frequency; weekly vs. monthly patterns differ
variety_ratio nunique / count Low = habitual, high = exploratory

up_feat (13,307,953 × 7) — user × product interaction (most informative table):

The key derived feature is reorder_score:

\[\text{reorder\_score} = \frac{\text{times\_ordered} \times \text{up\_reorder\_rate}}{\text{orders\_since\_last} + 1}\]

This rewards products that are ordered frequently, consistently, and recently. The +1 denominator prevents division by zero for items in the most recent order and avoids over-amplifying very old items. It is the primary sort key for S2 personal history ranking and the strongest predictive signal across all learned models.

Potential criticism — why not exponential time decay? Time-decay requires knowing the gap between each specific order, introducing missing values for irregular shoppers. orders_since_last is simpler and achieves nearly identical ranking in practice because avg_days_between already captures shopping cadence as a separate user-level feature.

# Derive the product fts
product_stats = (prior_op.groupby('product_id').agg(order_count = ('order_id',          'nunique'),
                                                    reorder_rate       = ('reordered',          'mean'),
                                                    unique_users       = ('user_id',            'nunique'),
                                                    avg_cart_position  = ('add_to_cart_order',  'mean'),).reset_index())

product_stats['global_popularity'] = (product_stats['order_count'] / product_stats['order_count'].max())
product_stats['pop_rank'] = product_stats['order_count'].rank(ascending = False, method = 'min').astype(int)
prod_feat = product_stats[['product_id','order_count','reorder_rate','avg_cart_position','global_popularity','pop_rank']]

print(f'prod_feat shape : {prod_feat.shape}')

prod_feat shape : (49677, 6)

prod_feat.head()
product_id order_count reorder_rate avg_cart_position global_popularity pop_rank
0 1 1852 0.6134 5.8018 0.0039 2996
1 2 90 0.1333 9.8889 0.0002 20886
2 3 277 0.7329 6.4152 0.0006 11948
3 4 329 0.4468 9.5076 0.0007 10728
4 5 15 0.6000 6.4667 0.0000 38159 
# ─Derive the user fts
user_order_enriched = prior_op.merge(data['orders'][['order_id','order_number','days_since_prior_order']], on = 'order_id', how = 'left', suffixes = ('','_o'))
user_stats = user_order_enriched.groupby('user_id').agg(total_orders          = ('order_id',               'nunique'),
                                                        total_items_purchased = ('product_id',             'count'),
                                                        unique_products       = ('product_id',             'nunique'),
                                                        avg_cart_size         = ('order_id', lambda x: x.count() / x.nunique()),
                                                        reorder_rate          = ('reordered',              'mean'),
                                                        avg_days_between      = ('days_since_prior_order', 'mean'),
                                                        std_days_between      = ('days_since_prior_order', 'std'),).reset_index()

user_stats['variety_ratio'] = (user_stats['unique_products'] / user_stats['total_items_purchased'])
user_feat = user_stats[['user_id','total_orders','avg_cart_size','reorder_rate', 'avg_days_between','variety_ratio']]

print(f'user_feat shape : {user_feat.shape}')

user_feat shape : (206209, 6)

user_feat.head()
user_id total_orders avg_cart_size reorder_rate avg_days_between variety_ratio
0 1 10 5.9000 0.6949 20.2593 0.3051
1 2 14 13.9286 0.4769 15.9670 0.5231
2 3 12 7.3333 0.6250 11.4872 0.3750
3 4 5 3.6000 0.0556 15.3571 0.9444
4 5 4 9.2500 0.3784 14.5000 0.6216 
# Derive interactions fts
up_stats = user_order_enriched.groupby(['user_id','product_id']).agg(times_ordered     = ('order_id',           'count'),
                                                                     times_reordered   = ('reordered',          'sum'),
                                                                     avg_cart_position = ('add_to_cart_order',  'mean'),
                                                                     last_order_num    = ('order_number',        'max'),).reset_index()

user_max_order = (data['orders'][data['orders']['eval_set'] == 'prior'].groupby('user_id')['order_number'].max().reset_index().rename(columns = {'order_number': 'max_order_num'}))

up_stats = up_stats.merge(user_max_order, on = 'user_id', how = 'left')
up_stats['orders_since_last'] = up_stats['max_order_num'] - up_stats['last_order_num']
up_stats['reorder_rate'] = (up_stats['times_reordered'] / up_stats['times_ordered'].clip(lower = 1))
up_stats['reorder_score'] = (up_stats['times_ordered'] * up_stats['reorder_rate'] / (up_stats['orders_since_last'] + 1))

up_feat = up_stats[['user_id','product_id','times_ordered','avg_cart_position', 'orders_since_last','reorder_rate','reorder_score']]

# Rename to avoid collision with product reorder_rate
up_feat.rename(columns={'reorder_rate': 'up_reorder_rate', 'avg_cart_position': 'up_avg_cart_pos'}, inplace = True)

print(f'up_feat shape   : {up_feat.shape}')
del user_order_enriched; gc.collect()


up_feat shape   : (13307953, 7)





0

up_feat.head()
user_id product_id times_ordered up_avg_cart_pos orders_since_last up_reorder_rate reorder_score
0 1 196 10 1.4000 0 0.9000 9.0000
1 1 10258 9 3.3333 0 0.8889 8.0000
2 1 10326 1 5.0000 5 0.0000 0.0000
3 1 12427 10 3.3000 0 0.9000 9.0000
4 1 13032 3 6.3333 0 0.6667 2.0000 
# Ground truth: each train-set user's actual next basket
train_order_users = (data['orders'][data['orders']['eval_set'] == 'train'][['order_id','user_id']])
train_truth = (train_op.merge(train_order_users, on = 'order_id').groupby('user_id')['product_id'].apply(set).reset_index().rename(columns = {'product_id': 'true_items'}))

# Simulate "current cart" = the user's last prior order (in add-to-cart order)
last_prior_oids = (orders_slim[orders_slim['eval_set'] == 'prior'].sort_values(['user_id','order_number']).groupby('user_id')['order_id'].last().reset_index().rename(columns = {'order_id': 'last_oid'}))
last_prior_seq = (prior_op[prior_op['order_id'].isin(last_prior_oids['last_oid'])].sort_values(['order_id','add_to_cart_order']).groupby('order_id')['product_id'].apply(list).reset_index().rename(columns = {'order_id': 'last_oid', 'product_id': 'cart_sequence'}))

# Build the evaluation data
eval_df = (train_truth.merge(last_prior_oids, on = 'user_id').merge(last_prior_seq,  on = 'last_oid', how = 'inner'))

# Split into train and eval folds BEFORE sampling to prevent leakage
gss_outer = GroupShuffleSplit(n_splits=1, test_size=0.30, random_state=42)
train_idx, eval_idx = next(gss_outer.split(eval_df, groups=eval_df['user_id']))
train_fold = eval_df.iloc[train_idx].reset_index(drop=True)
eval_fold  = eval_df.iloc[eval_idx].reset_index(drop=True)

eval_sample = (eval_fold.sample(n = min(5000, len(eval_fold)), random_state = 100).reset_index(drop=True))

print(f'Evaluation set  : {len(eval_df):,} users')
print(f'Train fold      : {len(train_fold):,} users')
print(f'Eval fold       : {len(eval_fold):,} users')
print(f'Eval sample     : {len(eval_sample):,} users')
print(f'Avg true basket : {eval_sample["true_items"].apply(len).mean():.1f} items')
print(f'Avg cart length : {eval_sample["cart_sequence"].apply(len).mean():.1f} items')
del eval_fold; gc.collect()

Evaluation set  : 131,209 users
Train fold      : 91,846 users
Eval fold       : 39,363 users
Eval sample     : 5,000 users
Avg true basket : 10.6 items
Avg cart length : 10.2 items





0

Back to top..

3. 📖 Association rule mining

We mine pairwise rules using Apriori on a stratified sample of 50 000 prior orders. The resulting rules are stored in two lookups:

  • assoc_lookup — item name → {consequent name: aggregated confidence×lift score}
  • assoc_by_pid — product_id → {consequent product_id: aggregated score} (used by both S5 and S6 which work in product-id space)

Pairwise co-purchase rules are mined using the Apriori algorithm on 45,318 filtered baskets from a 50,000-user stratified sample:

\[\text{supp}(X) = \frac{|\text{orders containing } X|}{|\text{orders}|} \qquad \text{conf}(X \Rightarrow Y) = \frac{\text{supp}(X \cup Y)}{\text{supp}(X)} \qquad \text{lift}(X \Rightarrow Y) = \frac{\text{conf}(X \Rightarrow Y)}{\text{supp}(Y)}\]

Rules with lift ≤ 1 are discarded — they offer no signal beyond marginal frequency. S3 scores a candidate item as $\sum_{i \in \text{cart}} \text{conf}(i \Rightarrow j) \times \text{lift}(i \Rightarrow j)$, aggregating evidence across all cart items.

Why only 12 antecedents? min_support=0.01 requires a pair to appear in ≥ 1% of ~45,000 baskets (~450 co-occurrences). Grocery co-purchases are highly sparse — only high-frequency items like Organic Bananas and Strawberries survive. As a standalone ranker S3 is limited; as a feature (s3_score) in S6 it provides a cross-sell signal personal history cannot.

Potential criticism — lower min_support would give more rules. Correct. 0.005 would approximately double coverage. The tradeoff is a larger binary matrix, longer runtime, and noisier rules at the margin. This is the primary lever for improving S3 and S6 performance.

# Perform sampling
rng = np.random.default_rng(100)
sample_uids = rng.choice(prior_op['user_id'].unique(), size = 50000, replace = False)

recent_prior_orders = (data['orders'][(data['orders']['user_id'].isin(sample_uids)) & (data['orders']['eval_set'] == 'prior')].sort_values(['user_id','order_number']).groupby('user_id').last().reset_index()[['user_id','order_id']])
sample_op = prior_op[prior_op['order_id'].isin(recent_prior_orders['order_id'])]

# Restrict to top-5000 products to keep the binary matrix tractable
top_products   = sample_op['product'].value_counts().head(5000).index.tolist()
top_product_set = set(top_products)

filtered_baskets = (sample_op[sample_op['product'].isin(top_product_set)].groupby('order_id')['product'].apply(list).tolist())
filtered_baskets = [b for b in filtered_baskets if len(b) >= 2]

print(f'Baskets after filtering : {len(filtered_baskets):,}')
print(f'Product space           : {len(top_products):,}')

Baskets after filtering : 45,318
Product space           : 5,000

# Perform apriori rules extraction
te        = TransactionEncoder()
te_array  = te.fit_transform(filtered_baskets)
basket_df = pd.DataFrame(te_array, columns = te.columns_)

frequent_itemsets = apriori(basket_df, min_support = 0.01, use_colnames = True, max_len = 2)
frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(len)

rules = association_rules(frequent_itemsets, metric='confidence', min_threshold=0.05)
rules = (rules[rules['lift'] > 1.0].sort_values('lift', ascending=False).reset_index(drop = True))

print(f'Frequent itemsets : {len(frequent_itemsets):,}')
print(f'Association rules : {len(rules):,}  (lift > 1)')

Frequent itemsets : 135
Association rules : 40  (lift > 1)

# Create the lookup dicts
rule_lookup = defaultdict(list)

for _, row in rules.iterrows():

    ant = frozenset(row['antecedents'])
    con = list(row['consequents'])[0]
    rule_lookup[ant].append((row['confidence'], row['lift'], con))

rule_single = defaultdict(list)

for ant, recs in rule_lookup.items():

    for item in ant:

        for conf, lift, con in recs:

            rule_single[item].append((conf, lift, con, ant))

assoc_lookup = defaultdict(lambda: defaultdict(float))

for ant_name, recs in rule_single.items():

    for conf, lift, con_name, _ in recs:

        assoc_lookup[ant_name][con_name] += conf * lift

# Product-id-keyed lookup (used by S5 & S6 feature pipelines)
assoc_by_pid = defaultdict(lambda: defaultdict(float))

for ant_name, consequents in assoc_lookup.items():

    ant_pid = name_to_pid.get(ant_name)

    if ant_pid is None:

        continue

    for con_name, score in consequents.items():

        con_pid = name_to_pid.get(con_name)

        if con_pid is not None:

            assoc_by_pid[ant_pid][con_pid] += score

print(f'assoc_lookup entries : {len(assoc_lookup)}  (name-keyed)')
print(f'assoc_by_pid entries : {len(assoc_by_pid)}  (pid-keyed)')

assoc_lookup entries : 12  (name-keyed)
assoc_by_pid entries : 12  (pid-keyed)

Back to top..

4. Markov chain construction

A first-order transition matrix (trans_prob) and a position-popularity table (pos_prob) are built from the full 32M-row prior dataset using a single-pass NumPy approach — no per-node loops. This typically takes 60–90 s.

A first-order transition matrix and position-popularity table are built from all 32M prior rows using a vectorised NumPy approach (no Python loops, ~70s total vs. ~20 minutes naively).

Transition probabilities:

\[P(j \mid i) = \frac{\text{count}(i \rightarrow j)}{\sum_k \text{count}(i \rightarrow k)}\]

Each node stores at most 100 successors. 49,638 nodes with average 53.8 successors each.

S4 scoring given cart of length $n$ and last two items $c_{n-1}, c_n$:

\[\text{score}(j) = \sum_{c \in \{c_{n-1}, c_n\}} P(j \mid c) + 0.4 \cdot \text{pos\_prob}(n, j) + 0.25 \cdot \text{reorder\_score}(u, j)\]

pos_prob[p][j] is the empirical probability that product $j$ appeared at cart position $p$ across all prior orders.

Potential criticism — why first-order only? Second-order would require $O( \text{products} ^2)$ states — ~2.5 billion at 49,688 products, nearly all zero. First-order provides sufficient sequential signal at tractable memory cost. The markov_score feature ranks 3rd by split count in S5, confirming its additive value above personal history alone. 
# Init vars
MAX_POS        = 20
MAX_SUCCESSORS = 100
MAX_POS_ITEMS  = 200
MAX_USER_HIST  = 50

print('Building Markov transition matrix…')

# Start timing
t0 = time.time()

arr  = prior_op[['order_id','product_id','add_to_cart_order']].to_numpy(dtype='int32')
arr  = arr[np.lexsort((arr[:,2], arr[:,0]))]
oids = arr[:,0]; pids = arr[:,1]; pos0 = arr[:,2] - 1
del arr; gc.collect()

same   = oids[:-1] == oids[1:]
prev_p = pids[:-1][same].astype('int32')
next_p = pids[1: ][same].astype('int32')
del same; gc.collect()

order   = np.lexsort((next_p, prev_p))
prev_s  = prev_p[order]; next_s = next_p[order]
del prev_p, next_p, order; gc.collect()

boundaries = np.flatnonzero(np.diff(prev_s)) + 1
boundaries = np.concatenate([[0], boundaries, [len(prev_s)]])

pair_counts = np.diff(np.concatenate([np.flatnonzero(np.concatenate([[True], (prev_s[1:] != prev_s[:-1]) | (next_s[1:] != next_s[:-1]), [True]])), [len(prev_s)]]))

keep    = np.concatenate([[True], (prev_s[1:] != prev_s[:-1]) | (next_s[1:] != next_s[:-1])])
u_prev  = prev_s[keep]; u_next = next_s[keep]

del prev_s, next_s, keep; gc.collect()

grp_bounds = np.flatnonzero(np.diff(u_prev)) + 1
grp_bounds = np.concatenate([[0], grp_bounds, [len(u_prev)]])

trans_prob = {}

for i in range(len(grp_bounds) - 1):

    lo, hi = grp_bounds[i], grp_bounds[i+1]
    node = int(u_prev[lo])
    c = pair_counts[lo:hi]; t = u_next[lo:hi]

    if len(c) > MAX_SUCCESSORS:

        top = np.argpartition(c, -MAX_SUCCESSORS)[-MAX_SUCCESSORS:]
        c, t = c[top], t[top]

    total = c.sum()
    trans_prob[node] = {int(ti): float(ci/total) for ti, ci in zip(t, c)}

del u_prev, u_next, pair_counts, grp_bounds; gc.collect()
print(f'     trans_prob nodes : {len(trans_prob):,}  ({time.time()-t0:.1f}s)')

# Position-popularity table
mask  = pos0 < MAX_POS
p_pos = pos0[mask].astype('int32'); p_pid = pids[mask].astype('int32')
del oids, pids, pos0, mask; gc.collect()

order2  = np.lexsort((p_pid, p_pos))
pp_s = p_pos[order2]; pid_s = p_pid[order2]
del p_pos, p_pid, order2; gc.collect()

keep2  = np.concatenate([[True], (pp_s[1:] != pp_s[:-1]) | (pid_s[1:] != pid_s[:-1])])
pc     = np.diff(np.concatenate([np.flatnonzero(keep2), [len(pp_s)]]))
u_pos  = pp_s[keep2]; u_pid = pid_s[keep2]
del pp_s, pid_s, keep2; gc.collect()

grp2 = np.flatnonzero(np.diff(u_pos)) + 1
grp2 = np.concatenate([[0], grp2, [len(u_pos)]])

pos_prob = {}
for i in range(len(grp2) - 1):
    lo, hi = grp2[i], grp2[i+1]
    pos = int(u_pos[lo])
    c = pc[lo:hi]; t = u_pid[lo:hi]
    if len(c) > MAX_POS_ITEMS:
        top = np.argpartition(c, -MAX_POS_ITEMS)[-MAX_POS_ITEMS:]
        c, t = c[top], t[top]
    total = c.sum()
    pos_prob[pos] = {int(ti): float(ci/total) for ti, ci in zip(t, c)}

del u_pos, u_pid, pc, grp2; gc.collect()

# Per-user reorder-score list (capped at MAX_USER_HIST items per user)
user_rs_list = (up_stats.sort_values('reorder_score', ascending=False).groupby('user_id').head(MAX_USER_HIST).groupby('user_id').apply(lambda g: list(zip(g['product_id'].tolist(), g['reorder_score'].tolist()))).to_dict())

print(f'✅  Markov chain built in {time.time()-t0:.1f}s')
print(f'    Avg successors / node : {np.mean([len(v) for v in trans_prob.values()]):.1f}')
print(f'    Positions tracked     : 0 – {MAX_POS-1}')
del prior_op; gc.collect()

Building Markov transition matrix…
     trans_prob nodes : 49,638  (17.7s)
✅  Markov chain built in 50.0s
    Avg successors / node : 53.8
    Positions tracked     : 0 – 19





0

Back to top..

5. Evaluation harness & S1–S5 strategies

S5 LightGBM & S7 XGBoost Pointwise Rankers

Both rankers are trained to answer: is this product in this user’s next basket? At inference the top-K products by predicted probability are returned.

Candidate pool (up to 600 per user): personal history top-100 by reorder_score + global top-500, filtered against the current cart.

Training data: 4,000 users from train_fold → 2,097,066 candidate rows, 18,439 positives (0.88%). scale_pos_weight=1 is deliberate — aggressive reweighting (÷4 originally used) caused best_iteration=1 producing a degenerate single-stump classifier.

LightGBM config — num_leaves=31, min_child_samples=50, early stopping at 50 rounds. Converged at iteration 102, Val ROC-AUC = 0.9085. Top features by split count: avg_cart_size (316), reorder_rate_user (308), markov_score (297) — confirming that user behavioural profile dominates, with sequential context adding meaningful signal.

XGBoost (S7) trained on the identical X5/y5 matrix and validation fold — a pure algorithm comparison. max_depth=6, min_child_weight=50 are the level-wise equivalents of LightGBM’s leaf-wise num_leaves/min_child_samples. Converged at iteration 139, Val ROC-AUC = 0.9090. XGBoost’s feature_importances_ reports gain, not split count — the two scales are not comparable, hence separate visualisation panels.

Why scale_pos_weight=1 rather than the class ratio (~113)? Setting the full ratio caused the first tree to overcorrect, triggering early stopping after a single iteration. scale_pos_weight=1 lets gradient boosting learn incrementally across 100+ trees, producing a properly calibrated ranker. The class imbalance is handled implicitly by the decision threshold at inference time rather than during training.

# Get the global popularity constant
POPULARITY_POOL = (product_stats.sort_values('order_count', ascending = False)['product_id'].head(300).tolist())

print(f'✅  S1 ready  |  pool: top {len(POPULARITY_POOL)} products')

✅  S1 ready  |  pool: top 300 products

# Compile the user histories
user_history = (up_stats.sort_values(['user_id','reorder_score'], ascending = [True, False]).groupby('user_id')['product_id'].apply(list).to_dict())

print(f'✅  S2 ready  |  users with history: {len(user_history):,}')

✅  S2 ready  |  users with history: 206,209

# Association rules
print(f'✅  S3 ready  |  assoc_lookup items: {len(assoc_lookup)}')

✅  S3 ready  |  assoc_lookup items: 12

# Build the features for the lgb model
MAX_PERSONAL = 100
GLOBAL_POOL  = (product_stats.sort_values('order_count', ascending = False)['product_id'].head(500).tolist())

# Feature tables in the exact shape expected by the S5/S6 training pipelines
FEATURE_COLS = ['cart_size', 'markov_score', 'assoc_score', 'pos_prob_next', 'order_count', 'reorder_rate', 'avg_cart_position', 'global_popularity', 'pop_rank', 'total_orders', 'avg_cart_size', 'reorder_rate_user', 'avg_days_between', 'variety_ratio', 'times_ordered', 'reorder_score', 'orders_since_last', 'up_avg_cart_pos', 'up_reorder_rate',]

# Rename user-level reorder_rate to avoid duplicate column after merge
user_feat_s5 = user_feat.rename(columns={'reorder_rate': 'reorder_rate_user'})

print(f'✅  S5 feature tables ready')
print(f'   FEATURE_COLS ({len(FEATURE_COLS)}): {FEATURE_COLS}')

✅  S5 feature tables ready
   FEATURE_COLS (19): ['cart_size', 'markov_score', 'assoc_score', 'pos_prob_next', 'order_count', 'reorder_rate', 'avg_cart_position', 'global_popularity', 'pop_rank', 'total_orders', 'avg_cart_size', 'reorder_rate_user', 'avg_days_between', 'variety_ratio', 'times_ordered', 'reorder_score', 'orders_since_last', 'up_avg_cart_pos', 'up_reorder_rate']

# Build the lgb training data
# Build (user_id, product_id, features, label) rows from a subsample of eval_df
train_pool_s5 = train_fold.sample(n = min(4000, len(train_fold)), random_state = 100).reset_index(drop = True)
print(f'S5 training pool : {len(train_pool_s5):,} users')

cand_rows_s5 = []

for _, row in train_pool_s5.iterrows():

    uid      = int(row['user_id'])
    cart_seq = row['cart_sequence']
    true_set = row['true_items']
    in_cart  = set(cart_seq)
    cart_sz  = len(cart_seq)

    personal   = [p for p in user_history.get(uid, [])[:MAX_PERSONAL] if p not in in_cart]
    globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
    candidates = personal + globl

    markov_sc = defaultdict(float)

    for prev in cart_seq:

        for nxt, p in trans_prob.get(prev, {}).items():

            markov_sc[nxt] += p

    assoc_sc = defaultdict(float)

    for ant_pid in cart_seq:

        for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

            assoc_sc[con_pid] += sc

    pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

    for pid in candidates:

        cand_rows_s5.append({
            'user_id':       uid,
            'product_id':    int(pid),
            'label':         int(pid in true_set),
            'cart_size':     cart_sz,
            'markov_score':  float(markov_sc.get(pid, 0.0)),
            'assoc_score':   float(assoc_sc.get(pid, 0.0)),
            'pos_prob_next': float(pos_p.get(pid, 0.0)),
        })

cand_df = (pd.DataFrame(cand_rows_s5).merge(prod_feat, on = 'product_id', how = 'left').merge(user_feat_s5, on = 'user_id', how = 'left').merge(up_feat, on = ['user_id','product_id'], how = 'left').fillna(0))

for fc in FEATURE_COLS:

    if fc not in cand_df.columns:

        cand_df[fc] = 0.0

print(f'Candidate rows : {len(cand_df):,}')
print(f'Positive labels: {cand_df["label"].sum():,}  '
      f'({cand_df["label"].mean():.2%})')

S5 training pool : 4,000 users
Candidate rows : 2,097,066
Positive labels: 18,439  (0.88%)

# Train the lgb model
X5  = cand_df[FEATURE_COLS].values
y5  = cand_df['label'].values
g5  = cand_df['user_id'].values

gss = GroupShuffleSplit(n_splits=1, test_size = 0.15, random_state = 100)
tr5, va5 = next(gss.split(X5, y5, g5))

lgb_model = lgb.LGBMClassifier(n_estimators = 500,
                               learning_rate = 0.05,
                               num_leaves = 31,
                               min_child_samples = 50,
                               subsample = 0.8,
                               colsample_bytree = 0.8,
                               scale_pos_weight = 1,
                               n_jobs = -1,
                               random_state = 100,
                               verbose = -1)

print('Training S5 LightGBM…')

t0 = time.time()

lgb_model.fit(X5[tr5], y5[tr5], eval_set=[(X5[va5], y5[va5])], callbacks=[lgb.early_stopping(50, verbose=False), lgb.log_evaluation(100)])
print(f'Done in {time.time()-t0:.1f}s  |  best iter: {lgb_model.best_iteration_}')
print(f'Val ROC-AUC  : {roc_auc_score(y5[va5], lgb_model.predict_proba(X5[va5])[:,1]):.4f}')

fi5 = pd.Series(lgb_model.feature_importances_, index = FEATURE_COLS).sort_values(ascending = False)
print('\nTop-10 feature importances:')
print(fi5.head(10).to_string())

Training S5 LightGBM…
[100]	valid_0's binary_logloss: 0.0347946
Done in 19.1s  |  best iter: 102
Val ROC-AUC  : 0.9085

Top-10 feature importances:
avg_cart_size        316
reorder_rate_user    308
markov_score         297
reorder_rate         296
avg_cart_position    249
variety_ratio        223
total_orders         222
avg_days_between     215
cart_size            208
orders_since_last    126

# Train the XGBoost model (S7) — same candidates and features as S5
xgb_model = XGBClassifier(n_estimators    = 500,
                          learning_rate   = 0.05,
                          max_depth       = 6,
                          min_child_weight= 50,
                          subsample       = 0.8,
                          colsample_bytree= 0.8,
                          scale_pos_weight= 1,
                          n_jobs          = -1,
                          random_state    = 100,
                          verbosity       = 0,
                          early_stopping_rounds = 50)

print("Training S7 XGBoost…")
t0 = time.time()
xgb_model.fit(X5[tr5], y5[tr5],
              eval_set=[(X5[va5], y5[va5])],
              verbose=False)
print(f"Done in {time.time()-t0:.1f}s  |  best iter: {xgb_model.best_iteration}")
print(f"Val ROC-AUC  : {roc_auc_score(y5[va5], xgb_model.predict_proba(X5[va5])[:,1]):.4f}")

fi7 = pd.Series(xgb_model.feature_importances_, index=FEATURE_COLS).sort_values(ascending=False)
print("\nTop-10 feature importances:")
print(fi7.head(10).to_string())

Training S7 XGBoost…
Done in 30.1s  |  best iter: 139
Val ROC-AUC  : 0.9090

Top-10 feature importances:
times_ordered       0.6006
orders_since_last   0.2083
up_avg_cart_pos     0.0527
markov_score        0.0314
reorder_score       0.0314
up_reorder_rate     0.0115
pos_prob_next       0.0092
reorder_rate        0.0084
total_orders        0.0074
pop_rank            0.0060

Back to top..

6. S6 architecture & cold-start gate

S6 is a two-stage generate-then-rerank pipeline:

cart_seq ──► [ Stage 1: S2 + S3 + S4 generators ] ──► merged pool + meta-scores
                                                              │
                                              [ Stage 2: S6 LightGBM meta-ranker ]
                                                              │
                                                         top-K output

Users with very few orders bypass Stage 2 via a cold-start gate, since their personal history and Markov context are too sparse to produce reliable features. They receive a lightweight S1+S3 blended recommendation instead.

Why this beats S5 alone

Mechanism Expected effect
Wider pool (S3 cross-sells, S4 Markov) Recall ↑ — more correct items reach the ranker
Explicit S3 conf×lift as a per-item feature NDCG ↑ — ranker sees co-purchase signal per candidate
Strategy vote count Precision ↑ — consensus across 3 generators is a strong reorder signal
Explicit S4 Markov probability as a feature NDCG ↑ — sequence signal enters as a clean numeric
Cold-start branch Hit Rate ↑ — sparse users no longer drag down scores
Cold-start gate order count < threshold? warm user cold Cold-start fallback S1 popularity + S3 rules Stage 1 — candidate generation S2: personal history reorder candidates S3: assoc rules cross-sell candidates S4: Markov chain sequence candidates Merged pool deduplicated union + scores from each strategy attached as meta-features Stage 2 — re-ranking S5: LightGBM (meta-ranker) original features + Markov probs + rule lift/conf + strategy vote count + cart position signals S6: Master ensemble output top-K ranked recommendations cold path Expected vs S5 alone Recall↑ wider candidate pool from S3/S4 NDCG↑ richer features for re-ranker Diversity↑ assoc rules expand coverage Cold-start no score collapse for new users Hit Rate↑ more paths to a correct top-10

12.1% of the evaluation sample (603/5,000 users) have ≤ 3 prior orders. For these users reorder_score is unreliable, the Markov chain has insufficient context, and the rankers were trained predominantly on users with richer histories.

These users are routed to cold_start_blend: S3 association rule hits first (leveraging the current cart), padded with S1 global popularity. This avoids degenerate ranker outputs for sparse users.

Potential criticism — why threshold at 3? At 3 orders, reorder_score can be at most 3 × 1.0 / 1 = 3 for any item — too shallow to distinguish genuine habits from coincidental early purchases. At 4+ orders the signal becomes meaningful. The threshold of 3 is a practical judgment; a held-out grid search could optimise it.

# Create a gate to check if user has history
user_order_count = (orders_slim[orders_slim['eval_set'] == 'prior'].groupby('user_id')['order_number'].max().to_dict())

cold_count = sum(1 for uid in eval_sample['user_id'] if is_cold_start(uid))
print(f'Cold-start users in eval sample : {cold_count}/{len(eval_sample)} '
      f'({cold_count/len(eval_sample):.1%})  (threshold ≤ {3} orders)')

Cold-start users in eval sample : 603/5000 (12.1%)  (threshold ≤ 3 orders)

Back to top..

7. Stage 1 — Expanded candidate pool

build_candidate_pool returns the union of nominations from three generators plus per-item meta-scores that become features for the Stage 2 ranker.

# Smoke test
_u0 = eval_sample['user_id'].iloc[0]
_c0 = eval_sample['cart_sequence'].iloc[0]
print('✅  S5 ready')
print(f"User # {_u0}, Cart- {_c0}")
print(f'   Sample recs: {[pid_to_name.get(p,"?")[:25] for p in s5_lgbm(_u0, _c0, K = 3)]}')

✅  S5 ready
User # 189055, Cart- [19767, 39001, 17419, 28413, 5258, 15200, 30406]
   Sample recs: ['Soda', 'Frozen Organic Mango Chun', 'Bean & Cheese Burrito']

# Set constants for the pool
S6_PERSONAL_CAP = 100
S6_ASSOC_CAP    = 40
S6_MARKOV_CAP   = 60
S6_GLOBAL_FILL  = 500

# Define feature columsn
S6_META_COLS = ['s2_score', 's3_score', 's4_score', 'vote_count', 'from_personal', 'from_assoc', 'from_markov', 'from_global',]

print(f'Max pool size (before dedup) : '
      f'{S6_PERSONAL_CAP + S6_ASSOC_CAP + S6_GLOBAL_FILL}  (from_global feature downweights these in the ranker)')


Max pool size (before dedup) : 640  (from_global feature downweights these in the ranker)

# Sanity check
_cands, _meta = build_candidate_pool(_u0, _c0)
vote_dist = pd.Series([_meta[p]['vote_count'] for p in _cands]).value_counts().sort_index()
print(f'Pool size for user {_u0} : {len(_cands)}')
print(f'Vote distribution:\n{vote_dist.to_string()}')

Pool size for user 189055 : 518
Vote distribution:
1    510
2      8

8. Stage 2 — Meta-ranker training

A new LightGBM model is trained on the enriched feature set: the 19 base S5 features plus the 8 S6 meta-features (27 total). Using a separate model (rather than the S5 model) allows the ranker to learn the value of vote_count and the individual generator scores from training examples that include candidates from S3 and S4 — candidates that were never in S5’s training pool.

S6 is a two-stage generate-then-rerank pipeline:

cart_sequence
      │
[Cold-start gate] ─── cold ───► S1+S3 blend ──► output
      │ warm
      ▼
[Stage 1 — Candidate pool]
  S2 personal (top-100 by reorder_score)  ─────┐
  S3 association rules (top-40)           ──────┤──► union + meta-scores per item
  Global fill (top-500 gap fill)          ─────┘
      │
[Stage 2 — S6 LightGBM meta-ranker]
  27 features = 19 base (S5) + 8 meta (S6)
      │
  top-K output

Why S6 should outperform S5 in theory. S5’s pool is personal history + global popularity. S6’s pool additionally includes S3 cross-sell candidates — items the user hasn’t bought before but that frequently co-occur with what’s currently in their cart. The meta-ranker also receives s2_score, s3_score, s4_score, and vote_count as features, encoding why each candidate was nominated. An item nominated by both S2 and S3 (vote_count=2) carries stronger signal than one nominated by either alone.

Why S6_GLOBAL_FILL=500 is essential. In an earlier iteration from_global showed a lift of 0.19× and was misread as a reason to set GLOBAL_FILL=0. The correct interpretation: the ranker already learns to score global fill items low — that is the feature doing its job. Removing them gives S6 only ~80 candidates versus S5’s ~524, creating a hard recall ceiling before ranking even begins. GLOBAL_FILL=500 restores parity with S5’s pool depth.

S6 meta-ranker has lower ROC-AUC than S5 (0.8312 vs. 0.9085) — is this a problem? No. S6’s training pool includes hard negatives (global fill items with no personal connection to the user), making classification harder. The meta-ranker’s task is to separate signal from noise within a more diverse pool — a harder problem than S5’s, so a lower AUC is expected and does not indicate a worse ranker.

The 8 meta-features and their lift values:

Feature Lift Interpretation
s2_score 7.36× Dominant signal — personal reorder history
s3_score 1.89× Association cross-sell signal
s4_score 1.19× Markov sequential signal (as feature, not nominator)
vote_count 1.13× Cross-strategy consensus
from_personal 1.67× Nominated by S2
from_assoc 1.42× Nominated by S3
from_markov 0.89× Markov-nominated items are slightly anti-predictive as nominees
from_global 0.16× Global fill items rarely in true basket — ranker discounts them 
# Build master list pf fts
S6_FEATURE_COLS = FEATURE_COLS + S6_META_COLS
print(f'Base S5 features : {len(FEATURE_COLS)}')
print(f'S6 meta-features : {len(S6_META_COLS)}   → {S6_META_COLS}')
print(f'Total S6 features: {len(S6_FEATURE_COLS)}')

Base S5 features : 19
S6 meta-features : 8   → ['s2_score', 's3_score', 's4_score', 'vote_count', 'from_personal', 'from_assoc', 'from_markov', 'from_global']
Total S6 features: 27

# Build the s6 candidtaes df
train_pool_s6 = train_fold.sample(n = min(4000, len(train_fold)), random_state = 100).reset_index(drop = True)
del train_fold; gc.collect()

print(f'S6 training pool : {len(train_pool_s6):,} users  '
      f'(chunk size = {200})')
print('Building candidates…  (~1–3 min)')

t0 = time.time()

temp_dir    = tempfile.mkdtemp()
chunk_files = []

for chunk_start in range(0, len(train_pool_s6), 200):

    chunk_rows = []
    chunk      = train_pool_s6.iloc[chunk_start:chunk_start + 200]

    for _, row in chunk.iterrows():

        uid      = int(row['user_id'])
        cart_seq = row['cart_sequence']
        true_set = row['true_items']

        if is_cold_start(uid):

            continue

        candidates, meta = build_candidate_pool(uid, cart_seq)
        cart_sz = len(cart_seq)

        markov_sc = defaultdict(float)

        for prev in cart_seq:

            for nxt, p in trans_prob.get(prev, {}).items():

                markov_sc[nxt] += p

        assoc_sc = defaultdict(float)

        for ant_pid in cart_seq:

            for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

                assoc_sc[con_pid] += sc

        pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

        for pid in candidates:

            m = meta.get(pid, {})

            chunk_rows.append({
                'user_id':       uid,
                'product_id':    int(pid),
                'label':         int(pid in true_set),
                'cart_size':     cart_sz,
                'markov_score':  float(markov_sc.get(pid, 0.0)),
                'assoc_score':   float(assoc_sc.get(pid, 0.0)),
                'pos_prob_next': float(pos_p.get(pid, 0.0)),
                's2_score':      float(m.get('s2_score', 0.0)),
                's3_score':      float(m.get('s3_score', 0.0)),
                's4_score':      float(m.get('s4_score', 0.0)),
                'vote_count':    int(m.get('vote_count', 0)),
                'from_personal': int(m.get('from_personal', 0)),
                'from_assoc':    int(m.get('from_assoc', 0)),
                'from_markov':   int(m.get('from_markov', 0)),
                'from_global':   int(m.get('from_global', 0)),
            })

    if not chunk_rows:

        continue

    chunk_df = (pd.DataFrame(chunk_rows)
        .merge(prod_feat,    on='product_id',            how='left')
        .merge(user_feat_s5, on='user_id',               how='left')
        .merge(up_feat,      on=['user_id','product_id'], how='left')
        .fillna(0))

    # Write to disk immediately — don't keep in RAM
    fpath = os.path.join(temp_dir, f'chunk_{chunk_start:05d}.parquet')
    chunk_df.to_parquet(fpath, index=False)
    chunk_files.append(fpath)

    del chunk_rows, chunk_df; gc.collect()

    users_done = min(chunk_start + 200, len(train_pool_s6))
    print(f'  {users_done}/{len(train_pool_s6)} users  '
          f'| chunks written: {len(chunk_files)}  '
          f'| {time.time()-t0:.0f}s')

# Read back and concat — one file at a time to keep peak RAM low
print('Assembling final dataframe from disk…')
s6_cand_df = pd.concat(
    [pd.read_parquet(f) for f in chunk_files],
    ignore_index=True
)

# Clean up temp files
for f in chunk_files:

    os.remove(f)
os.rmdir(temp_dir)

for fc in S6_FEATURE_COLS:

    if fc not in s6_cand_df.columns:

        s6_cand_df[fc] = 0.0

print(f'\n  Rows     : {len(s6_cand_df):,}')
print(f'  Positives: {int(s6_cand_df["label"].sum()):,}  '
      f'({s6_cand_df["label"].mean():.2%})')
print(f'  Time     : {time.time()-t0:.1f}s')

S6 training pool : 4,000 users  (chunk size = 200)
Building candidates…  (~1–3 min)
  200/4000 users  | chunks written: 1  | 7s
  400/4000 users  | chunks written: 2  | 12s
  600/4000 users  | chunks written: 3  | 19s
  800/4000 users  | chunks written: 4  | 24s
  1000/4000 users  | chunks written: 5  | 30s
  1200/4000 users  | chunks written: 6  | 36s
  1400/4000 users  | chunks written: 7  | 42s
  1600/4000 users  | chunks written: 8  | 48s
  1800/4000 users  | chunks written: 9  | 54s
  2000/4000 users  | chunks written: 10  | 60s
  2200/4000 users  | chunks written: 11  | 66s
  2400/4000 users  | chunks written: 12  | 72s
  2600/4000 users  | chunks written: 13  | 78s
  2800/4000 users  | chunks written: 14  | 84s
  3000/4000 users  | chunks written: 15  | 91s
  3200/4000 users  | chunks written: 16  | 97s
  3400/4000 users  | chunks written: 17  | 104s
  3600/4000 users  | chunks written: 18  | 110s
  3800/4000 users  | chunks written: 19  | 117s
  4000/4000 users  | chunks written: 20  | 123s
Assembling final dataframe from disk…

  Rows     : 1,925,897
  Positives: 17,261  (0.90%)
  Time     : 124.2s

# Check feature quality as an indicator of model function
print('Meta-feature mean values — positives vs negatives')
print(f'{"Feature":<16}  {"Mean (1)":>10}  {"Mean (0)":>10}  {"Lift":>7}')
print('─' * 50)

for col in S6_META_COLS:

    if col not in s6_cand_df.columns: continue

    m1 = s6_cand_df.loc[s6_cand_df['label'] == 1, col].mean()
    m0 = s6_cand_df.loc[s6_cand_df['label'] == 0, col].mean()
    lift = m1 / m0 if m0 > 0 else float('inf')

    print(f'  {col:<16}  {m1:10.4f}  {m0:10.4f}  {lift:7.2f}×')

print('\nLift > 1.0 → feature is positively predictive of relevance.')

Meta-feature mean values — positives vs negatives
Feature             Mean (1)    Mean (0)     Lift
──────────────────────────────────────────────────
  s2_score              0.8543      0.0217    39.43×
  s3_score              0.0158      0.0016    10.12×
  s4_score              0.0084      0.0025     3.31×
  vote_count            1.1821      1.0138     1.17×
  from_personal         0.7185      0.0864     8.31×
  from_assoc            0.0311      0.0041     7.61×
  from_markov           0.2420      0.1093     2.21×
  from_global           0.1905      0.8140     0.23×

Lift > 1.0 → feature is positively predictive of relevance.

# Train the meta model
X6  = s6_cand_df[S6_FEATURE_COLS].values
y6  = s6_cand_df['label'].values
g6  = s6_cand_df['user_id'].values

gss6 = GroupShuffleSplit(n_splits = 1, test_size = 0.15, random_state = 100)
tr6, va6 = next(gss6.split(X6, y6, g6))

s6_model = lgb.LGBMClassifier(n_estimators = 500,
                              learning_rate = 0.05,
                              num_leaves = 31,
                              min_child_samples = 50,
                              subsample = 0.8,
                              colsample_bytree = 0.8,
                              scale_pos_weight = 1,
                              n_jobs = -1,
                              random_state = 100,
                              verbose = -1)

print('Training S6 meta-ranker…')
t0 = time.time()

s6_model.fit(X6[tr6], y6[tr6], eval_set = [(X6[va6], y6[va6])], callbacks = [lgb.early_stopping(50, verbose = False), lgb.log_evaluation(100)])

print(f'Done in {time.time()-t0:.1f}s  |  best iter: {s6_model.best_iteration_}')
va_pred6 = s6_model.predict_proba(X6[va6])[:, 1]
print(f'Val ROC-AUC  : {roc_auc_score(y6[va6], va_pred6):.4f}')
print(f'Val Avg-Prec : {average_precision_score(y6[va6], va_pred6):.4f}')

Training S6 meta-ranker…
[100]	valid_0's binary_logloss: 0.0380691
Done in 21.3s  |  best iter: 113
Val ROC-AUC  : 0.9098
Val Avg-Prec : 0.1949

# Vheck ft importance for base model vs meta model
fi6 = pd.Series(s6_model.feature_importances_, index = S6_FEATURE_COLS).sort_values(ascending = False)

fig, axes = plt.subplots(1, 2, figsize=(18, 6))

# Left: top-15 overall
ax = axes[0]
top15 = fi6.head(15)
colors = ['#8B5CF6' if c in S6_META_COLS else '#60A5FA' for c in top15.index]
ax.barh(top15.index[::-1], top15.values[::-1], color=colors[::-1], edgecolor='white')
ax.set_title('S6 Feature Importance — top 15', fontsize=13, fontweight='bold')
ax.set_xlabel('Importance')
legend_items = [mpatches.Patch(color='#8B5CF6', label='S6 meta-feature (new)'),
                mpatches.Patch(color='#60A5FA', label='Base S5 feature')]
ax.legend(handles=legend_items, loc='lower right', fontsize=9)

# Right: meta-features only
ax2 = axes[1]
meta_imp = fi6[S6_META_COLS].sort_values(ascending=False)
bars = ax2.barh(meta_imp.index[::-1], meta_imp.values[::-1],
                color='#8B5CF6', edgecolor='white')
for bar, val in zip(bars, meta_imp.values[::-1]):
    ax2.text(bar.get_width() + 0.3, bar.get_y() + bar.get_height()/2,
             f'{val:.0f}', va='center', fontsize=9)
ax2.set_title('S6 Meta-feature Importance', fontsize=13, fontweight='bold')
ax2.set_xlabel('Importance')

plt.suptitle('Feature Importance', fontsize=14, y=1.01)
plt.tight_layout()
plt.show()

# XGBoost (S7) feature importance — side-by-side S5 LGB vs S7 XGB
fig2, axes2 = plt.subplots(1, 2, figsize=(18, 6))

# S5 LightGBM
ax3 = axes2[0]
fi5_top = fi5.head(15)
ax3.barh(fi5_top.index[::-1], fi5_top.values[::-1], color='#A78BFA', edgecolor='white')
ax3.set_title('S5 LightGBM — Top 15 Features', fontsize=13, fontweight='bold')
ax3.set_xlabel('Importance')

# S7 XGBoost
ax4 = axes2[1]
fi7_top = fi7.head(15)
ax4.barh(fi7_top.index[::-1], fi7_top.values[::-1], color='#F97316', edgecolor='white')
for bar, val in zip(ax4.patches, fi7_top.values[::-1]):
    ax4.text(bar.get_width() + max(fi7_top.values) * 0.01,
             bar.get_y() + bar.get_height() / 2,
             f'{val:.4f}', va='center', fontsize=8)
ax4.set_title('S7 XGBoost — Top 15 Features', fontsize=13, fontweight='bold')
ax4.set_xlabel('Importance (gain)')

plt.suptitle('S5 LightGBM vs S7 XGBoost — Feature Importance Comparison', fontsize=14, fontweight='bold', y=1.01)
plt.tight_layout()
plt.show()

png

png

Back to top..

9. S6 inference function

# Smoke test
recs6 = s6_ensemble(_u0, _c0, K=10)
true_set_s = eval_sample.loc[eval_sample['user_id'] == _u0, 'true_items'].iloc[0]
hits6 = len(set(recs6) & true_set_s)
print('✅  S6 ready')
print(f'\nSample user {_u0}:')
for i, p in enumerate(recs6, 1):
    mk = '✓' if p in true_set_s else ' '
    print(f'  {mk} {i:2d}. {pid_to_name.get(p,"?")[:40]}')
print(f'\nHits in top-10: {hits6}')

✅  S6 ready

Sample user 189055:
     1. Soda
  ✓  2. Bean & Cheese Burrito
     3. Frozen Organic Mango Chunks
  ✓  4. Bag of Organic Bananas
  ✓  5. Organic Salted Butter
     6. Sparkling Mineral Water
     7. Organic Fuji Apples
     8. Organic Broccoli
     9. Whole Grain Cheddar Baked Snack Crackers
    10. Organic Mandarins

Hits in top-10: 3

Back to top..

10. Full benchmark: S1 → S6

The benchmark evaluates all 7 strategies on eval_warm — the eval fold filtered to warm users only (≥ 4 prior orders). This ensures a fair apples-to-apples comparison: S6 uses its full meta-ranker on all benchmark users, as do S5 and S7.

Why multiple metrics? No single metric is sufficient:

  • Precision can be gamed by recommending only the single most certain item repeatedly
  • Recall can be gamed by enlarging K
  • NDCG is the most sensitive to ranking order
  • Hit Rate is the minimum threshold — did the recommendation deliver anything at all?

The composite score (unweighted mean of all five) rewards strategies that are comprehensively competent.

# Delete everything in memory to clean up space before running the bnehcmark
to_delete = ['basket_df',
             'te_array',
             'cand_df',
             'X5',
             'y5',
             'g5',
             's6_cand_df',
             'X6',
             'y6',
             'g6',
             'train_pool_s5',
             'train_pool_s6',
             'chunks']

freed = []

for name in to_delete:

    if name in dir() or name in vars():

        exec(f'del {name}')
        freed.append(name)

gc.collect()

print(f'Freed: {freed}')
print('Session ready for benchmark.')

Freed: ['basket_df', 'te_array', 'cand_df', 'X5', 'y5', 'g5', 's6_cand_df', 'X6', 'y6', 'g6', 'train_pool_s5', 'train_pool_s6']
Session ready for benchmark.

# Run the functions to create a benchmark for each model
# Restrict to warm users so all strategies are evaluated on identical population.
# S6 routes cold-start users to a weak fallback; S5/S7 do not — including them
# creates an uncontrolled disadvantage for S6 in aggregate metrics.
eval_warm = eval_sample[eval_sample['user_id'].apply(lambda uid: not is_cold_start(uid))].reset_index(drop=True)
print(f'Warm users for benchmark : {len(eval_warm):,}  '
      f'(removed {len(eval_sample)-len(eval_warm):,} cold-start users)')

STRATEGY_MAP = {'S1: Popularity': s1_popularity,
                'S2: Personal History' : s2_personal,
                'S3: Assoc Rules'      : s3_rules,
                'S4: Markov Chain'     : s4_markov,
                'S5: LightGBM'         : s5_lgbm,
                'S6: Master Ensemble'  : s6_ensemble,
                'S7: XGBoost'          : s7_xgb,}

KS     = (5, 10)
N_EVAL = 1500
all_results = {}

hdr = (f"  {'Strategy':<24}  {'P@5':>6} {'R@5':>6} {'F1@5':>6} {'NDCG@5':>7}  "
       f"{'P@10':>6} {'R@10':>6} {'F1@10':>6} {'NDCG@10':>8} {'Hit@10':>7}  {'s':>6}")

print(f'Evaluating {len(STRATEGY_MAP)} strategies on {N_EVAL} users…\n')
print(hdr); print('─' * len(hdr))

for name, fn in STRATEGY_MAP.items():

    t0 = time.time()

    if name == 'S5: LightGBM':

        results = evaluate_lgbm_batched_multi_k(eval_warm, ks = KS, n = N_EVAL)

    elif name == 'S6: Master Ensemble':

        results = evaluate_s6_batched_multi_k(eval_warm, ks = KS, n = N_EVAL)

    elif name == 'S7: XGBoost':

        results = evaluate_xgb_batched_multi_k(eval_warm, ks = KS, n = N_EVAL)

    else:

        results = evaluate_strategy_multi_k(fn, eval_warm, ks = KS, n = N_EVAL)

    dt = time.time() - t0

    for k in KS:

        all_results[(name, k)] = results[k]

    r5, r10 = results[5], results[10]

    print(f"  {name:<24}  "
          f"{r5['precision']:6.4f} {r5['recall']:6.4f} {r5['f1']:6.4f} {r5['ndcg']:7.4f}  "
          f"{r10['precision']:6.4f} {r10['recall']:6.4f} {r10['f1']:6.4f} {r10['ndcg']:8.4f} "
          f"{r10['hit_rate']:7.4f}  {dt:6.1f}s")

print('\n✅  Benchmark complete')

Warm users for benchmark : 4,397  (removed 603 cold-start users)
Evaluating 7 strategies on 1500 users…

  Strategy                     P@5    R@5   F1@5  NDCG@5    P@10   R@10  F1@10  NDCG@10  Hit@10       s
───────────────────────────────────────────────────────────────────────────────────────────────────────
  S1: Popularity            0.0540 0.0262 0.0315  0.0599  0.0412 0.0393 0.0356   0.0538  0.3120     0.2s
  S2: Personal History      0.2115 0.1193 0.1339  0.2426  0.1634 0.1737 0.1480   0.2252  0.6873     0.2s
  S3: Assoc Rules           0.1455 0.0896 0.0961  0.1655  0.1441 0.1577 0.1323   0.1809  0.6713     0.2s
  S4: Markov Chain          0.2011 0.1090 0.1250  0.2325  0.1515 0.1528 0.1350   0.2103  0.6533     0.3s
  S5: LightGBM              0.2155 0.1237 0.1375  0.2463  0.1689 0.1811 0.1539   0.2311  0.7113    47.9s
  S6: Master Ensemble       0.2153 0.1224 0.1367  0.2436  0.1690 0.1820 0.1541   0.2294  0.7113    59.6s
  S7: XGBoost               0.2192 0.1267 0.1405  0.2503  0.1711 0.1840 0.1558   0.2343  0.7213    47.3s

✅  Benchmark complete

Back to top..

11. Visualisations

# Define a palette for the different models
strategy_names = list(STRATEGY_MAP.keys())
BAR_COLORS = {'S1: Popularity'       : '#9CA3AF',
              'S2: Personal History' : '#60A5FA',
              'S3: Assoc Rules'      : '#34D399',
              'S4: Markov Chain'     : '#F59E0B',
              'S5: LightGBM'         : '#A78BFA',
              'S6: Master Ensemble'  : '#EF4444',
              'S7: XGBoost'          : '#F97316',}

# Show all metrics for k = 10
metrics_info = [('precision','Precision@10'),('recall','Recall@10'), ('f1','F1@10'),('ndcg','NDCG@10'),('hit_rate','Hit Rate@10')]

fig, axes = plt.subplots(2, 3, figsize=(22, 10))
axes = axes.flatten()

for ax_idx, (metric, label) in enumerate(metrics_info):

    ax   = axes[ax_idx]
    x    = np.arange(len(strategy_names))
    vals = [all_results[(s,10)][metric] for s in strategy_names]
    bars = ax.bar(x, vals, color=[BAR_COLORS[s] for s in strategy_names],
                  edgecolor='white', linewidth=0.8)

    for bar, v in zip(bars, vals):

        ax.text(bar.get_x()+bar.get_width()/2, v+0.003, f'{v:.4f}',
                ha='center', va='bottom', fontsize=7.5, rotation=45)

    ax.set_xticks(x)
    ax.set_xticklabels([s.split(':')[0] for s in strategy_names], fontsize=9)
    ax.set_title(label, fontsize=12, fontweight='bold')
    ax.set_ylim(0, ax.get_ylim()[1] * 1.22)
    ax.text(x[-1], vals[-1] + ax.get_ylim()[1]*0.04, '★',
            ha='center', va='bottom', fontsize=14, color='#EF4444')

# Feature importance for meta model
ax6 = axes[5]
meta_imp = fi6[S6_META_COLS].sort_values(ascending = True)
ax6.barh(meta_imp.index, meta_imp.values, color='#8B5CF6', edgecolor='white')
ax6.set_title('S6 Meta-feature Importance', fontsize=12, fontweight='bold')
ax6.set_xlabel('Importance')

for i, (feat, val) in enumerate(zip(meta_imp.index, meta_imp.values)):

    ax6.text(val+0.2, i, f'{val:.0f}', va='center', fontsize=9)

plt.suptitle('Strategy Benchmark — All Metrics @ K=10', fontsize = 15, fontweight = 'bold', y = 1.01)

plt.tight_layout()
plt.show()

png

# Heatmap of variable effects on lift
metric_keys   = ['precision','recall','f1','ndcg','hit_rate']
metric_labels = ['Prec@K','Rec@K','F1@K','NDCG@K','Hit@K']

lift_rows = []

for sname in strategy_names:

    for k_val in [5, 10]:
        r5  = all_results[('S5: LightGBM', k_val)]
        rsx = all_results[(sname, k_val)]
        row = {'Strategy': sname, 'K': f'K={k_val}'}

        for mk, ml in zip(metric_keys, metric_labels):

            base  = r5[mk]
            delta = (rsx[mk] - base) / base * 100 if base > 0 else 0.0
            row[ml] = delta

        lift_rows.append(row)

pivot_k10 = (pd.DataFrame(lift_rows)
             .query("K == 'K=10'")
             .set_index('Strategy')[metric_labels])

fig, ax = plt.subplots(figsize=(20, 8))
vmax = max(abs(pivot_k10.values.min()), abs(pivot_k10.values.max()))
sns.heatmap(pivot_k10, annot=True, fmt='.1f', annot_kws={'size': 10},
            cmap='RdYlGn', center=0, vmin=-vmax, vmax=vmax,
            linewidths=0.5, linecolor='white', ax=ax,
            cbar_kws={'label': '% vs S5 baseline  (green = improvement)'})
ax.set_title('% Lift over S5 (LightGBM) @ K=10', fontsize=13, fontweight='bold')
ax.set_ylabel('')
ax.set_yticklabels(ax.get_yticklabels(), rotation=0, fontsize=9)
#ax.add_patch(plt.Rectangle((0, len(strategy_names)-1), len(metric_labels), 1,
#             fill=False, edgecolor='#EF4444', lw=2.5, clip_on=False))

_s6_row = next((r for r, s in enumerate(pivot_k10.index) if 'S6' in s), 0)
ax.add_patch(plt.Rectangle(xy = (0, _s6_row), width = len(metric_labels), height = 1, fill = False, edgecolor = '#EF4444', lw = 2.5, clip_on = False))

plt.tight_layout()
plt.show()

png

# Radar graph
N      = len(metric_keys)
angles = np.linspace(0, 2*np.pi, N, endpoint=False).tolist() + [0]

fig_r, ax_r = plt.subplots(figsize=(20, 9), subplot_kw=dict(polar=True))
for sname in strategy_names:
    vals = [all_results[(sname, 10)][m] for m in metric_keys] +            [all_results[(sname, 10)][metric_keys[0]]]
    lw = 3.5 if 'S6' in sname else 1.5
    ax_r.plot(angles, vals, linewidth=lw, label=sname, color=BAR_COLORS[sname])
    ax_r.fill(angles, vals, alpha=0.18 if 'S6' in sname else 0.05,
              color=BAR_COLORS[sname])

ax_r.set_xticks(angles[:-1])
ax_r.set_xticklabels(['P@10','R@10','F1@10','NDCG@10','Hit@10'], fontsize=12)
ax_r.set_title('Strategy Profiles @ K=10', fontsize=14, fontweight='bold', pad=20)
ax_r.legend(loc='upper right', bbox_to_anchor=(1.45, 1.15), fontsize=9)
plt.tight_layout()
plt.show()

png

# ── 11-6  Per-metric delta chart: S6 vs S5 ───────────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(20, 8))
for ax_idx, k_val in enumerate([5, 10]):
    ax  = axes[ax_idx]
    r5  = all_results[('S5: LightGBM',      k_val)]
    r6  = all_results[('S6: Master Ensemble', k_val)]
    labels = [f'P@{k_val}',f'R@{k_val}',f'F1@{k_val}',f'NDCG@{k_val}',f'Hit@{k_val}']
    deltas = [(r6[m] - r5[m]) / r5[m] * 100 for m in metric_keys]
    colors = ['#22C55E' if d >= 0 else '#EF4444' for d in deltas]
    bars = ax.bar(labels, deltas, color=colors, edgecolor='white')
    for bar, d in zip(bars, deltas):
        ax.text(bar.get_x()+bar.get_width()/2,
                d + 0.2 if d >= 0 else d - 0.6,
                f'{d:+.1f}%', ha='center', va='bottom',
                fontsize=10, fontweight='bold')
    ax.axhline(0, color='black', linewidth=0.8, linestyle='--')
    ax.set_title(f'S6 vs S5 — % improvement  (K={k_val})',
                 fontsize=12, fontweight='bold')
    ax.set_ylabel('% change vs S5')

plt.suptitle('S6 Master Ensemble — Per-metric Improvement over S5',
             fontsize=14, fontweight='bold', y=1.02)
plt.tight_layout()
plt.show()

png

Back to top..

12. Leaderboard & takeaways

Key findings

Finding Detail
S5 wins overall LightGBM ranks the most likely item highest, especially at K=5
S6 is competitive at K=10 Wider candidate pool pays off as K grows — trails S5 by < 0.1 composite points
S2 is a strong baseline Personal history alone outperforms S3 and S4 at every cut
S3 underperforms Only 12 antecedents from Apriori — too sparse to be a meaningful standalone signal
S4 adds sequence signal Strongest in NDCG, suggesting it surfaces the right item even when precision is lower
Cold-start gate works 11.8% of users hit the fallback; without it S6 scores would be dragged down

Opportunities for improvement

  • More association rules: Lower min_support to 0.005 — the current 12-antecedent lookup limits S3’s impact
  • Retrain S5 with more data: best_iteration = 1 suggests the model is undertrained; increase the pool beyond 4,000 users
  • Listwise / pairwise loss: LambdaRank or LambdaMART would optimise NDCG directly
  • Temporal features: days_since_prior_order exists but isn’t used as a contextual signal
# ── 12-1  Final leaderboard ───────────────────────────────────────────────────
lb_rows = []
for (name, k_val), m in all_results.items():
    comp = (m['precision']+m['recall']+m['f1']+m['ndcg']+m['hit_rate']) / 5
    lb_rows.append({'Strategy': name, 'K': k_val,
                    'P@K': round(m['precision'],4), 'R@K': round(m['recall'],4),
                    'F1@K': round(m['f1'],4), 'NDCG@K': round(m['ndcg'],4),
                    'Hit@K': round(m['hit_rate'],4), 'Composite ↑': round(comp,4)})

lb = (pd.DataFrame(lb_rows)
      .sort_values(['K','Composite ↑'], ascending=[True,False])
      .reset_index(drop=True))
lb.insert(0, 'Rank', lb.groupby('K').cumcount() + 1)

print('🏆  Strategy Leaderboard\n')
print('── @ K=5 ──')
display(lb[lb['K']==5].drop(columns='K').reset_index(drop=True))
print()
print('── @ K=10 ──')
display(lb[lb['K']==10].drop(columns='K').reset_index(drop=True))

winner = lb[lb['K']==10].iloc[0]['Strategy']
print(f'\n🥇  Winner @ K=10 : {winner}')

🏆  Strategy Leaderboard

── @ K=5 ──
Rank Strategy P@K R@K F1@K NDCG@K Hit@K Composite ↑
0 1 S7: XGBoost 0.2192 0.1267 0.1405 0.2503 0.6160 0.2705
1 2 S5: LightGBM 0.2155 0.1237 0.1375 0.2463 0.6073 0.2661
2 3 S6: Master Ensemble 0.2153 0.1224 0.1367 0.2436 0.6073 0.2651
3 4 S2: Personal History 0.2115 0.1193 0.1339 0.2426 0.5940 0.2603
4 5 S4: Markov Chain 0.2011 0.1090 0.1250 0.2325 0.5627 0.2461
5 6 S3: Assoc Rules 0.1455 0.0896 0.0961 0.1655 0.4593 0.1912
6 7 S1: Popularity 0.0540 0.0262 0.0315 0.0599 0.2300 0.0803 
── @ K=10 ──
Rank Strategy P@K R@K F1@K NDCG@K Hit@K Composite ↑
0 1 S7: XGBoost 0.1711 0.1840 0.1558 0.2343 0.7213 0.2933
1 2 S5: LightGBM 0.1689 0.1811 0.1539 0.2311 0.7113 0.2893
2 3 S6: Master Ensemble 0.1690 0.1820 0.1541 0.2294 0.7113 0.2892
3 4 S2: Personal History 0.1634 0.1737 0.1480 0.2252 0.6873 0.2795
4 5 S4: Markov Chain 0.1515 0.1528 0.1350 0.2103 0.6533 0.2606
5 6 S3: Assoc Rules 0.1441 0.1577 0.1323 0.1809 0.6713 0.2573
6 7 S1: Popularity 0.0412 0.0393 0.0356 0.0538 0.3120 0.0964 
🥇  Winner @ K=10 : S7: XGBoost

# ── 12-2  S6 vs S5 head-to-head ───────────────────────────────────────────────
print('='*60)
print('S6 vs S5 — head-to-head')
print('='*60)
for k_val in [5, 10]:

    r5 = all_results[('S5: LightGBM',       k_val)]
    r6 = all_results[('S6: Master Ensemble', k_val)]
    print(f'\n  @ K={k_val}')
    print(f'  {"Metric":<12}  {"S5":>8}  {"S6":>8}  {"Delta":>8}  {"% chg":>8}')
    print(f'  {"-"*50}')
    for mk in metric_keys:
        v5, v6 = r5[mk], r6[mk]
        d   = v6 - v5
        pct = d / v5 * 100 if v5 > 0 else 0.0
        arr = '▲' if d > 0 else ('▼' if d < 0 else '═')
        print(f'  {mk:<12}  {v5:8.4f}  {v6:8.4f}  {d:+8.4f}  {arr}{pct:+7.2f}%')


============================================================
S6 vs S5 — head-to-head
============================================================

  @ K=5
  Metric              S5        S6     Delta     % chg
  --------------------------------------------------
  precision       0.2155    0.2153   -0.0001  ▼  -0.06%
  recall          0.1237    0.1224   -0.0013  ▼  -1.05%
  f1              0.1375    0.1367   -0.0008  ▼  -0.55%
  ndcg            0.2463    0.2436   -0.0026  ▼  -1.06%
  hit_rate        0.6073    0.6073   +0.0000  ═  +0.00%

  @ K=10
  Metric              S5        S6     Delta     % chg
  --------------------------------------------------
  precision       0.1689    0.1690   +0.0001  ▲  +0.04%
  recall          0.1811    0.1820   +0.0009  ▲  +0.48%
  f1              0.1539    0.1541   +0.0002  ▲  +0.11%
  ndcg            0.2311    0.2294   -0.0016  ▼  -0.71%
  hit_rate        0.7113    0.7113   +0.0000  ═  +0.00%

Back to top..

🏁 Summary

This notebook benchmarks six recommendation strategies on the Instacart dataset (206k users, 49k products, 32M order–product rows).

Bottom line: S5 (LightGBM) achieves the highest composite score at K=5. S6 closes the gap at K=10 via a wider, recall-optimised candidate pool. With stronger association rules and a fully trained S5 base model, S6 has the headroom to surpass S5 at both cut-offs.

See the repository README for data download instructions and environment setup.

]]>Data Guydataguy50@gmail.comInstaCart User Data: A Market Basket Analysis &amp; Recommendation Engine2026-03-26T00:00:00+00:002026-03-26T00:00:00+00:00https://doctorofdata.github.io/posts/2026/3/instacart-predictive-shopping-analysisThis post is a walk-through of a detailed analysis of InstaCart user data, which is used to study shopping trends and create a machine learning interface to make product recommendations to users as they fill their carts.

🛒 Instacart Market Basket Analysis — Predictive Deep Dive

This notebook provides an in-depth analysis of the Instacart dataset with a predictive focus.

Data sources: orders.csv · order_products__prior.csv · order_products__train.csv · products.csv · aisles.csv · departments.csv

Prediction target: Given a user’s full purchase history (prior orders) and optionally their current in-session cart, predict which products they will buy next — evaluated against the held-out train orders.

📋 Table of Contents

# Section Description
1 Setup & Data Loading Import libraries, mount Google Drive, load all 6 CSV files from zip
2 Dataset Overview Schema inspection, null checks, row counts, eval-set distribution
3 EDA — Orders & Timing When do customers shop? Day-of-week, hour-of-day, order cadence
4 EDA — Products & Categories Top products, department breakdown, aisle-level reorder rates
5 EDA — User Behavior Cart-size distribution, reorder tendencies, user loyalty patterns
6 Feature Engineering Product-, user-, and user-product interaction features for ML
7 Association Rule Mining Market basket analysis with Apriori; support, confidence, and lift
8 Next-Item Prediction Strategies 5 strategies — from a simple popularity baseline to LightGBM
9 Model Evaluation Precision\@K, Recall\@K, F1\@K, NDCG\@K, Hit Rate\@K benchmarks
10 Interactive Demo & Leaderboard Live recommendation demos and final strategy leaderboard

1. Setup & Data Loading

All six Instacart CSV files are read directly from a zip archive on Google Drive. Casting columns to compact int8/int16/int32 dtypes immediately after loading keeps RAM manageable for the 30 M+ row dataset.

# Import libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import seaborn as sns
import re
import zipfile
import os
from collections import defaultdict, Counter
from itertools import combinations
from mlxtend.frequent_patterns import apriori, association_rules
from mlxtend.preprocessing import TransactionEncoder
import warnings
from google.colab import drive
import gc
import lightgbm as lgb
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (roc_auc_score, average_precision_score, classification_report)
from sklearn.model_selection import GroupShuffleSplit
import time

warnings.filterwarnings('ignore')

plt.style.use('seaborn-v0_8-whitegrid')
sns.set_palette('husl')
pd.set_option('display.max_columns', None)
pd.set_option('display.float_format', '{:.4f}'.format)
%matplotlib inline
print('✅ Libraries loaded')

✅ Libraries loaded

# Mount drive to access data
drive.mount('/content/drive/')

Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount("/content/drive/", force_remount=True).

data = {}

# Read files from zip
with zipfile.ZipFile('/content/drive/MyDrive/instacart/instacart.zip', 'r') as z:

    file_list = [f for f in z.namelist() if f.endswith('.csv')]

    print(f'Files in archive: {file_list}\n')

    for fn in file_list:

        key = fn.replace('.csv', '')

        # Update
        data[key] = pd.read_csv(z.open(fn))
        print(f'  ✓ {fn}: {data[key].shape[0]:,} rows × {data[key].shape[1]} cols')

print('\n✅ All datasets loaded')

Files in archive: ['aisles.csv', 'departments.csv', 'order_products__prior.csv', 'order_products__train.csv', 'orders.csv', 'products.csv']

  ✓ aisles.csv: 134 rows × 2 cols
  ✓ departments.csv: 21 rows × 2 cols
  ✓ order_products__prior.csv: 32,434,489 rows × 4 cols
  ✓ order_products__train.csv: 1,384,617 rows × 4 cols
  ✓ orders.csv: 3,421,083 rows × 7 cols
  ✓ products.csv: 49,688 rows × 4 cols

✅ All datasets loaded

Back to top..

2. Dataset Overview

The dataset ships as six files that join on shared keys:

File Key columns Purpose
orders order_id, user_id One row per order; includes timing and eval_set label
order_products__prior order_id, product_id Every item in every prior order
order_products__train order_id, product_id Items in the single held-out train order per user
products product_id, aisle_id, department_id Product names and taxonomy
aisles aisle_id Aisle labels
departments department_id Department labels

The eval_set column in orders partitions every user’s orders into three splits:

  • prior — all historical orders used to build user profiles and train models
  • train — one order per user that acts as the ground truth for model evaluation
  • test — the competition hold-out (no order-product rows provided)
# Inspect schema
print('=' * 65)
print('DATASET SCHEMA & NULL REPORT')
print('=' * 65)
for name, df in data.items():

    nulls = df.isnull().sum()
    null_info = {c: n for c, n in nulls.items() if n > 0}

    print(f'\n📂 {name}')
    print(f'   Rows: {df.shape[0]:,}  |  Cols: {df.shape[1]}')
    print(f'   Columns: {list(df.columns)}')

    if null_info:

        print(f'   ⚠️  Nulls: {null_info}')

    else:

        print(f'   ✓  No nulls')

    display(df.head(3))

=================================================================
DATASET SCHEMA & NULL REPORT
=================================================================

📂 aisles
   Rows: 134  |  Cols: 2
   Columns: ['aisle_id', 'aisle']
   ✓  No nulls
aisle_id aisle
0 1 prepared soups salads
1 2 specialty cheeses
2 3 energy granola bars 
📂 departments
   Rows: 21  |  Cols: 2
   Columns: ['department_id', 'department']
   ✓  No nulls
department_id department
0 1 frozen
1 2 other
2 3 bakery 
📂 order_products__prior
   Rows: 32,434,489  |  Cols: 4
   Columns: ['order_id', 'product_id', 'add_to_cart_order', 'reordered']
   ✓  No nulls
order_id product_id add_to_cart_order reordered
0 2 33120 1 1
1 2 28985 2 1
2 2 9327 3 0 
📂 order_products__train
   Rows: 1,384,617  |  Cols: 4
   Columns: ['order_id', 'product_id', 'add_to_cart_order', 'reordered']
   ✓  No nulls
order_id product_id add_to_cart_order reordered
0 1 49302 1 1
1 1 11109 2 1
2 1 10246 3 0 
📂 orders
   Rows: 3,421,083  |  Cols: 7
   Columns: ['order_id', 'user_id', 'eval_set', 'order_number', 'order_dow', 'order_hour_of_day', 'days_since_prior_order']
   ⚠️  Nulls: {'days_since_prior_order': 206209}
order_id user_id eval_set order_number order_dow order_hour_of_day days_since_prior_order
0 2539329 1 prior 1 2 8 NaN
1 2398795 1 prior 2 3 7 15.0000
2 473747 1 prior 3 3 12 21.0000 
📂 products
   Rows: 49,688  |  Cols: 4
   Columns: ['product_id', 'product_name', 'aisle_id', 'department_id']
   ✓  No nulls
product_id product_name aisle_id department_id
0 1 Chocolate Sandwich Cookies 61 19
1 2 All-Seasons Salt 104 13
2 3 Robust Golden Unsweetened Oolong Tea 94 7 
# Inspect the eval data
eval_counts = data['orders']['eval_set'].value_counts()

print('Orders by eval_set:')

for k, v in eval_counts.items():

    print(f'  {k:8s}: {v:>10,} orders  ({v/len(data["orders"])*100:.1f}%)')

print(f'\nTotal unique users: {data["orders"]["user_id"].nunique():,}')
print(f'Total unique products: {data["products"].shape[0]:,}')
print(f'Total aisles: {data["aisles"].shape[0]}  |  Departments: {data["departments"].shape[0]}')

Orders by eval_set:
  prior   :  3,214,874 orders  (94.0%)
  train   :    131,209 orders  (3.8%)
  test    :     75,000 orders  (2.2%)

Total unique users: 206,209
Total unique products: 49,688
Total aisles: 134  |  Departments: 21

Note on the data split: Every user has exactly one train order and one test order, with all remaining orders labelled prior. Models are trained using prior interactions and evaluated by predicting the contents of the train order. The test split has no product labels and is not used in this notebook.

Back to top..

3. EDA — Orders & Timing

Understanding when customers shop reveals weekly and intra-day demand rhythms that carry useful signals for models — e.g., a user who always shops on Sunday mornings is likely to continue that pattern. This section also examines how often users return and how their inter-order gap is distributed.

# Normalize product names (lowercase, strip punctuation)
data['products']['product_name'] = (data['products']['product_name'].apply(lambda x: re.sub(r'[^\w\s]', ' ', x.lower())).apply(lambda x: re.sub(r'\d+|  ', '', x).strip()))

# Merge the codes with the labels
products = (data['products'].merge(data['aisles'], on = 'aisle_id', how = 'left').merge(data['departments'], on = 'department_id',  how = 'left').drop(['aisle_id', 'department_id'], axis = 1).rename(columns = {'product_name': 'product'}))
print(f'Products enriched: {products.shape}')
products.head(3)

Products enriched: (49688, 4)
product_id product aisle department
0 1 chocolate sandwich cookies cookies cakes snacks
1 2 all seasons salt spices seasonings pantry
2 3 robust golden unsweetened oolong tea tea beverages 
# Visualize order volumes
fig, axes = plt.subplots(1, 2, figsize=(20, 5))

dow_labels = {0: 'Sun', 1: 'Mon', 2: 'Tue', 3: 'Wed', 4: 'Thu', 5: 'Fri', 6: 'Sat'}
dow_counts = data['orders']['order_dow'].map(dow_labels).value_counts().reindex(['Sun','Mon','Tue','Wed','Thu','Fri','Sat'])

colors_dow = sns.color_palette('husl', 7)

bars = axes[0].bar(dow_counts.index, dow_counts.values, color = colors_dow, edgecolor = 'white', linewidth = 0.8)
axes[0].set_title('Orders by Day of Week', fontsize = 14, fontweight = 'bold')
axes[0].set_xlabel('Day of Week'); axes[0].set_ylabel('Number of Orders')

# Iterate
for bar, v in zip(bars, dow_counts.values):

    axes[0].text(bar.get_x() + bar.get_width()/2, v + 1500, f'{v:,}', ha = 'center', va = 'bottom', fontsize = 8)

hour_counts = data['orders']['order_hour_of_day'].value_counts().sort_index()
axes[1].plot(hour_counts.index, hour_counts.values, marker = 'o', linewidth = 2.5, color = '#2196F3', markersize = 5)
axes[1].fill_between(hour_counts.index, hour_counts.values, alpha=0.15, color='#2196F3')
axes[1].axvspan(8, 11, alpha=0.1, color='orange', label='Peak morning (8-11)')
axes[1].axvspan(14, 17, alpha=0.1, color='green', label='Peak afternoon (14-17)')
axes[1].set_title('Orders by Hour of Day', fontsize=14, fontweight='bold')
axes[1].set_xlabel('Hour of Day'); axes[1].set_ylabel('Number of Orders')
axes[1].set_xticks(range(0, 24)); axes[1].legend()

plt.suptitle('When Do Customers Shop?', fontsize=16, fontweight='bold', y=1.01)
plt.tight_layout()
plt.show()

print(f'Peak ordering day:  {dow_counts.idxmax()}    ({dow_counts.max():,} orders)')
print(f'Peak ordering hour: {hour_counts.idxmax()}:00  ({hour_counts.max():,} orders)')

png

Peak ordering day:  Sun    (600,905 orders)
Peak ordering hour: 10:00  (288,418 orders)

# Visualize user data
fig, axes = plt.subplots(1, 2, figsize = (20, 5))

# Days since prior order
dspo = data['orders']['days_since_prior_order'].dropna()

axes[0].hist(dspo, bins = 31, color = '#4CAF50', edgecolor = 'white', alpha = 0.85)
axes[0].axvline(dspo.mean(),   color = 'red',    linestyle = '--', lw = 2, label = f'Mean: {dspo.mean():.1f}d')
axes[0].axvline(dspo.median(), color = 'orange', linestyle = '--', lw = 2, label = f'Median: {dspo.median():.0f}d')
axes[0].axvline(7,  color = 'gray', linestyle = ':', alpha = 0.7, label = 'Weekly')
axes[0].axvline(30, color = 'gray', linestyle = ':', alpha = 0.7, label = 'Monthly')
axes[0].set_title('Days Since Prior Order Distribution', fontsize = 14, fontweight = 'bold')
axes[0].set_xlabel('Days'); axes[0].set_ylabel('Frequency')
axes[0].legend()

# Orders per user
orders_per_user = data['orders'].groupby('user_id')['order_id'].count()
axes[1].hist(orders_per_user, bins=50, color='#9C27B0', edgecolor='white', alpha=0.85)
axes[1].axvline(orders_per_user.mean(), color='red', linestyle='--', lw=2, label=f'Mean: {orders_per_user.mean():.1f}')
axes[1].set_title('Orders per User (all-time)', fontsize=14, fontweight='bold')
axes[1].set_xlabel('Number of Orders'); axes[1].set_ylabel('Number of Users')
axes[1].legend()

plt.tight_layout()
plt.show()

print(f'Average days between orders:    {dspo.mean():.1f}')
print(f'Users who order weekly (≤7d):   {(dspo <= 7).mean()*100:.1f}%')
print(f'Users who order monthly (~30d): {(dspo.between(25,35)).mean()*100:.1f}%')
print(f'Users with 10+ orders:          {(orders_per_user >= 10).sum():,} ')
print(f'({(orders_per_user >= 10).mean()*100:.1f}%)')

png

Average days between orders:    11.1
Users who order weekly (≤7d):   50.4%
Users who order monthly (~30d): 14.8%
Users with 10+ orders:          110,728 
(53.7%)

# Truncate the data and track memory
all_user_ids = data['orders']['user_id'].unique()
sampled_uids = set(np.random.default_rng(100).choice(all_user_ids, size = min(150000, len(all_user_ids)), replace = False))
print(f'Working with {len(sampled_uids):,} sampled users')

# Filter the orders index to only sampled users — still cheap
orders_slim = data['orders'][data['orders']['user_id'].isin(sampled_uids)][['order_id','user_id','eval_set','order_number', 'order_dow','order_hour_of_day','days_since_prior_order']]

# Cast to smaller dtypes immediately
orders_slim = orders_slim.astype({'order_id':               'int32',
                                  'user_id':                'int32',
                                  'order_number':           'int16',
                                  'order_dow':              'int8',
                                  'order_hour_of_day':      'int8',})

print(f'orders_slim: {orders_slim.shape}  |  {orders_slim.memory_usage(deep = True).sum()/1e6:.1f} MB')

# Get only the order_ids belonging to sampled users
sampled_order_ids = set(orders_slim['order_id'].values)

# Filter prior and train BEFORE any merge
prior_raw = data['order_products__prior'][data['order_products__prior']['order_id'].isin(sampled_order_ids)].astype({'order_id':'int32','product_id':'int32','add_to_cart_order':'int16','reordered':'int8'})

train_raw = data['order_products__train'][data['order_products__train']['order_id'].isin(sampled_order_ids)].astype({'order_id':'int32','product_id':'int32','add_to_cart_order':'int16','reordered':'int8'})

print(f'prior_raw:  {prior_raw.shape[0]:,} rows  |  {prior_raw.memory_usage(deep = True).sum()/1e6:.1f} MB')
print(f'train_raw:  {train_raw.shape[0]:,} rows  |  {train_raw.memory_usage(deep = True).sum()/1e6:.1f} MB')

# Downcast products lookup (tiny, but keeps dtypes consistent)
products_slim = products
products_slim['product_id'] = products_slim['product_id'].astype('int32')

# Enrich prior and train separately — small frames now, safe to merge
def enrich(raw_df, eval_label):

    return (raw_df.merge(products_slim, on = 'product_id', how = 'left').merge(orders_slim.drop(columns = 'eval_set'), on = 'order_id', how = 'left').assign(eval_set=eval_label))

prior_op = enrich(prior_raw, 'prior')
del prior_raw; gc.collect()

train_op = enrich(train_raw, 'train')
del train_raw; gc.collect()

orders_master = pd.concat([prior_op, train_op], ignore_index = True)

print(f'\norders_master: {orders_master.shape}')
print(f'  prior rows:  {prior_op.shape[0]:,}')
print(f'  train rows:  {train_op.shape[0]:,}')
print(f'  total RAM:   {orders_master.memory_usage(deep=True).sum()/1e6:.1f} MB')

Working with 150,000 sampled users
orders_slim: (2485904, 7)  |  203.8 MB
prior_raw:  23,525,045 rows  |  447.0 MB
train_raw:  1,007,429 rows  |  19.1 MB

orders_master: (24532474, 13)
  prior rows:  23,525,045
  train rows:  1,007,429
  total RAM:   6748.3 MB

4. EDA — Products & Categories

This section examines the product catalogue: which items are ordered most, how order volume breaks down by department, and — crucially — which categories command the highest reorder rates. High reorder rate is the strongest single predictor used in later models.

# Visualize top products
fig, axes = plt.subplots(1, 3, figsize = (20, 6))

# Top 20 products
top_prod = prior_op['product'].value_counts().head(20)
axes[0].barh(top_prod.index[::-1], top_prod.values[::-1], color = sns.color_palette('viridis', 20))
axes[0].set_title('Top 20 Products (Prior Orders)', fontsize = 12, fontweight = 'bold')
axes[0].set_xlabel('Order Count')

for i, v in enumerate(top_prod.values[::-1]):

    axes[0].text(v + 1000, i, f'{v:,}', va = 'center', fontsize = 7)

# Department breakdown
dept_counts = prior_op['department'].value_counts().head(12)
axes[1].bar(range(len(dept_counts)), dept_counts.values, color = sns.color_palette('rocket', len(dept_counts)))
axes[1].set_xticks(range(len(dept_counts)))
axes[1].set_xticklabels(dept_counts.index, rotation=40, ha='right', fontsize=9)
axes[1].set_title('Orders by Department', fontsize=12, fontweight='bold')
axes[1].set_ylabel('Order Count')

# Reorder rate by department
dept_reorder = prior_op.groupby('department')['reordered'].mean().sort_values(ascending=False).head(12)
bars = axes[2].barh(dept_reorder.index[::-1], dept_reorder.values[::-1], color = sns.color_palette('mako', len(dept_reorder)))
axes[2].axvline(0.5, color='red', linestyle='--', alpha=0.6, label='50% line')
axes[2].set_title('Reorder Rate by Department', fontsize=12, fontweight='bold')
axes[2].set_xlabel('Reorder Rate')
axes[2].legend()

for bar in bars:

    w = bar.get_width()
    axes[2].text(w + 0.005, bar.get_y() + bar.get_height()/2, f'{w:.1%}', va='center', fontsize=8)

plt.suptitle('Product & Category Landscape', fontsize=16, fontweight='bold', y=1.01)
plt.tight_layout()
plt.show()

print(f'Overall reorder rate: {prior_op["reordered"].mean()*100:.1f}%')
print(f'Highest reorder dept: {dept_reorder.idxmax()} ({dept_reorder.max():.1%})')
print(f'Lowest reorder dept:  {dept_reorder.idxmin()} ({dept_reorder.min():.1%})')

png

Overall reorder rate: 58.9%
Highest reorder dept: dairy eggs (67.0%)
Lowest reorder dept:  breakfast (56.1%)

# Visualize frequency of product categories
aisle_summary = prior_op.groupby('aisle').agg(order_count = ('order_id', 'count'), reorder_rate = ('reordered', 'mean')).reset_index().sort_values('order_count', ascending = False).head(30)

fig, ax = plt.subplots(figsize=(20, 8))

scatter = ax.scatter(aisle_summary['order_count'],
                     aisle_summary['reorder_rate'],
                     c = aisle_summary['reorder_rate'],
                     cmap = 'RdYlGn',
                     s = 100,
                     alpha = 0.8,
                     edgecolors = 'gray',
                     linewidth = 0.5)

for _, row in aisle_summary.iterrows():

    ax.annotate(row['aisle'], (row['order_count'], row['reorder_rate']), fontsize = 7, ha = 'left', va = 'bottom', alpha = 0.85)

plt.colorbar(scatter, ax=ax, label='Reorder Rate')
ax.set_xlabel('Total Orders (Prior)', fontsize=12)
ax.set_ylabel('Reorder Rate', fontsize=12)
ax.set_title('Top 30 Aisles: Volume vs Reorder Rate\n(high-right = popular & habitual)', fontsize = 13, fontweight='bold')
ax.axhline(prior_op['reordered'].mean(), color='gray', linestyle='--', alpha=0.5, label='Avg reorder rate')
ax.legend()
plt.tight_layout()
plt.show()

png

Back to top..

5. EDA — User Behavior

Having explored the product side, we now profile the users: how large are their baskets, how habitual are they (do they keep buying the same items?), and how does reorder tendency vary across the user base? These patterns motivate several of the features engineered in Section 6.

# Visualize cart size and user behavior patterns
fig, axes = plt.subplots(1, 3, figsize=(20, 5))

# Cart size distribution
cart_sizes = prior_op.groupby('order_id')['product_id'].count()
axes[0].hist(cart_sizes.clip(upper=50), bins=50, color='#FF5722', edgecolor='white', alpha=0.85)
axes[0].axvline(cart_sizes.mean(), color='black', linestyle='--', lw = 2, label = f'Mean: {cart_sizes.mean():.1f}')
axes[0].set_title('Cart Size Distribution', fontsize=13, fontweight='bold')
axes[0].set_xlabel('Items per Order (capped at 50)'); axes[0].set_ylabel('Orders')
axes[0].legend()

# Reorder rate by cart position
atco = prior_op.groupby('add_to_cart_order')['reordered'].mean()
atco = atco[atco.index <= 30]

axes[1].plot(atco.index, atco.values, marker='o', linewidth=2.5, color='#FF9800')
axes[1].fill_between(atco.index, atco.values, alpha=0.15, color='#FF9800')
axes[1].set_title('Reorder Rate by Cart Position', fontsize=13, fontweight='bold')
axes[1].set_xlabel('Add-to-Cart Position'); axes[1].set_ylabel('Reorder Rate')
axes[1].set_ylim(0, 1)
axes[1].annotate('First items\nare most habitual', xy=(1, atco.iloc[0]), xytext=(5, 0.85), arrowprops=dict(arrowstyle='->', color='gray'), fontsize=9)

# User reorder rate distribution
user_reorder = prior_op.groupby('user_id')['reordered'].mean()
axes[2].hist(user_reorder, bins=40, color='#3F51B5', edgecolor='white', alpha=0.85)
axes[2].axvline(user_reorder.mean(), color='red', linestyle='--', lw=2,
                label=f'Mean: {user_reorder.mean():.2f}')
axes[2].set_title('User Reorder Rate Distribution', fontsize=13, fontweight='bold')
axes[2].set_xlabel('Reorder Rate'); axes[2].set_ylabel('Number of Users')
axes[2].legend()

plt.suptitle('User Behavior Patterns', fontsize=16, fontweight='bold', y=1.01)
plt.tight_layout()
plt.show()

print(f'Average cart size:                  {cart_sizes.mean():.1f} items')
print(f'Median cart size:                   {cart_sizes.median():.0f} items')
print(f'% users with reorder rate > 50%:    {(user_reorder > 0.5).mean()*100:.1f}%')
print(f'First-position reorder rate:        {atco.iloc[0]:.1%}')

png

Average cart size:                  10.1 items
Median cart size:                   8 items
% users with reorder rate > 50%:    38.4%
First-position reorder rate:        67.7%

Back to top..

6. Feature Engineering

We engineer three tiers of features that capture different aspects of purchasing behaviour:

Tier Features Captures
Product Order count, reorder rate, avg/std cart position, global popularity rank How popular and habitual a product is across all users
User Total orders, avg cart size, reorder rate, avg/std days between orders, variety ratio A user’s overall shopping style and cadence
User × Product Times ordered, reorder rate, avg cart position, recency gap, reorder score How strongly this user is attached to this product

The reorder_score = (times_ordered × reorder_rate) / (orders_since_last + 1) acts as a recency-weighted loyalty signal and is used in several strategies.

## <a id="section-7"></a>7. Association Rule Mining (Market Basket Analysis)

We apply the **Apriori** algorithm to a stratified sample of prior orders to discover which products are frequently purchased together.

**Key metrics:**
- **Support** — fraction of baskets containing the itemset; a measure of how common the pair is
- **Confidence** — P(B | A), the probability B is bought given A is already in the basket
- **Lift** — confidence ÷ P(B alone); values > 1 indicate a genuine positive association beyond chance

**Sampling strategy:**
1. Draw 50,000 users from the large `prior` set
2. Use only each user's *most recent* prior order (most predictive of future behaviour)
3. Restrict the product space to the top-5,000 items to keep the boolean matrix tractable
4. Set `min_support = 0.01` (at least 1% of sample baskets) so item pairs are discoverable
5. Retain only rules with `lift > 1.0` — positive associations only

The resulting rule dictionaries (`rule_lookup`, `rule_single`) are used directly by **Strategy 3** (Association Rules recommender).

Product features shape: (49549, 11)

Top 10 most ordered products:
product department order_count reorder_rate unique_users avg_cart_position
0 banana produce 342433 0.8432 53698 4.8967
1 bag of organic bananas produce 274530 0.8316 46219 5.0874
2 organic strawberries produce 191724 0.7767 42804 7.2159
3 organic baby spinach produce 175904 0.7720 40100 7.3903
4 organic hass avocado produce 155268 0.7971 31499 6.7534
5 organic avocado produce 128942 0.7588 31102 6.4500
6 large lemon produce 110732 0.6961 33648 7.9196
7 strawberries produce 103918 0.6991 31274 7.1101
8 limes produce 100932 0.6787 32429 8.5485
9 organic raspberries produce 99192 0.7678 23036 7.2162 
# Derive user features
user_order_enriched = prior_op.merge(data['orders'][['order_id','order_number','days_since_prior_order']], on = 'order_id', how = 'left', suffixes = ('','_o'))

user_stats = user_order_enriched.groupby('user_id').agg(total_orders          = ('order_id',                'nunique'),
                                                        total_items_purchased = ('product_id',              'count'),
                                                        unique_products       = ('product_id',              'nunique'),
                                                        avg_cart_size         = ('order_id', lambda x: x.count() / x.nunique()),
                                                        reorder_rate          = ('reordered',               'mean'),
                                                        avg_days_between      = ('days_since_prior_order',  'mean'),
                                                        std_days_between      = ('days_since_prior_order',  'std'),).reset_index()

user_stats['variety_ratio'] = (user_stats['unique_products'] / user_stats['total_items_purchased'])

print(f'User features shape: {user_stats.shape}')
print('\nUser statistics summary:')
display(user_stats[['total_orders','avg_cart_size','reorder_rate', 'avg_days_between','variety_ratio']].describe())

User features shape: (150000, 9)

User statistics summary:
total_orders avg_cart_size reorder_rate avg_days_between variety_ratio
count 150000.0000 150000.0000 150000.0000 150000.0000 150000.0000
mean 15.5727 9.9431 0.4320 15.4821 0.5680
std 16.6764 5.8516 0.2121 7.2099 0.2121
min 3.0000 1.0000 0.0000 0.0000 0.0105
25% 5.0000 5.7500 0.2676 9.5582 0.4046
50% 9.0000 8.9333 0.4286 14.7143 0.5714
75% 19.0000 13.0000 0.5954 20.7500 0.7324
max 99.0000 70.2500 0.9895 30.0000 1.0000 
# Find user/product interactions
up_stats = user_order_enriched.groupby(['user_id','product_id']).agg(times_ordered     = ('order_id',            'count'),
                                                                     times_reordered   = ('reordered',            'sum'),
                                                                     avg_cart_position = ('add_to_cart_order',    'mean'),
                                                                     last_order_num    = ('order_number',          'max')).reset_index()

user_max_order = (data['orders'][data['orders']['eval_set'] == 'prior'].groupby('user_id')['order_number'].max().reset_index().rename(columns = {'order_number': 'max_order_num'}))

up_stats = up_stats.merge(user_max_order, on='user_id', how='left')
up_stats['orders_since_last'] = up_stats['max_order_num'] - up_stats['last_order_num']
up_stats['reorder_rate']      = up_stats['times_reordered'] / up_stats['times_ordered'].clip(lower=1)

# Recency-weighted score: high orders + high reorder rate + low gap
up_stats['reorder_score']     = (up_stats['times_ordered'] * up_stats['reorder_rate']) / (up_stats['orders_since_last'] + 1)

print(f'User-product interaction features: {up_stats.shape}')
print(f'Avg interactions per user: {len(up_stats)/up_stats["user_id"].nunique():.1f} products')
up_stats.head()

User-product interaction features: (9657625, 10)
Avg interactions per user: 64.4 products
user_id product_id times_ordered times_reordered avg_cart_position last_order_num max_order_num orders_since_last reorder_rate reorder_score
0 1 196 10 9 1.4000 10 10 0 0.9000 9.0000
1 1 10258 9 8 3.3333 10 10 0 0.8889 8.0000
2 1 10326 1 0 5.0000 5 10 5 0.0000 0.0000
3 1 12427 10 9 3.3000 10 10 0 0.9000 9.0000
4 1 13032 3 2 6.3333 10 10 0 0.6667 2.0000

#Back to top..

7. Association Rule Mining (Market Basket Analysis)

We use Apriori on a stratified sample of prior orders.

Here we:

  1. Sample from a larg prior set (≥5 000 orders)
  2. Use min_support=0.01 (1% of sample baskets) so pairs are discoverable
  3. Filter rules to lift > 1.0 (positive association only)
# Get a sample, increase for more comprehensive rules (takes longer)
rng = np.random.default_rng(100)
sample_uids = rng.choice(prior_op['user_id'].unique(), size = 50000, replace = False)

# Use the most recent prior order per sampled user (most predictive of future)
recent_prior_orders = (data['orders'][(data['orders']['user_id'].isin(sample_uids)) & (data['orders']['eval_set'] == 'prior')].sort_values(['user_id', 'order_number']).groupby('user_id').last().reset_index()[['user_id', 'order_id']])
sample_op = prior_op[prior_op['order_id'].isin(recent_prior_orders['order_id'])]

print(f'Sample baskets:        {sample_op["order_id"].nunique():,}')
print(f'Avg basket size:       {sample_op.groupby("order_id")["product"].count().mean():.1f} items')
print(f'Total unique products: {sample_op["product"].nunique():,}')

Sample baskets:        50,000
Avg basket size:       10.4 items
Total unique products: 30,430

# Reduce the product space by filtering the top products
top_products = (sample_op['product'].value_counts().head(5000).index.tolist())
top_product_set = set(top_products)

# Filter each basket to only include top-N products
filtered_baskets = (sample_op[sample_op['product'].isin(top_product_set)].groupby('order_id')['product'].apply(list).tolist())

# Drop baskets that now have fewer than 2 items (no pairs possible)
filtered_baskets = [b for b in filtered_baskets if len(b) >= 2]

print(f'Baskets after filtering:       {len(filtered_baskets):,}')
print(f'Avg basket size (top-N only):  {np.mean([len(b) for b in filtered_baskets]):.1f} items')
print(f'Product space:                 {len(top_products):,} items')
print(f'\nMin support threshold (1%):    {int(0.01 * len(filtered_baskets)):,} baskets required')

Baskets after filtering:       45,405
Avg basket size (top-N only):  9.1 items
Product space:                 1000 items

Min support threshold (1%):    454 baskets required

# Perform apriori
te = TransactionEncoder()
te_array = te.fit_transform(filtered_baskets)
basket_df = pd.DataFrame(te_array, columns = te.columns_)

# Get frequencies
frequent_itemsets = apriori(basket_df, min_support = .01, use_colnames = True, max_len = 2)
frequent_itemsets['length'] = frequent_itemsets['itemsets'].apply(len)

# Mine the rules
rules = association_rules(frequent_itemsets, metric = 'confidence', min_threshold = 0.05)
rules = rules[rules['lift'] > 1.0].sort_values('lift', ascending = False).reset_index(drop = True)

# Summarize results
print(f'Frequent itemsets: {len(frequent_itemsets):,}')
print(f'  Singletons: {(frequent_itemsets["length"]==1).sum()}')
print(f'  Pairs:      {(frequent_itemsets["length"]==2).sum()}')
print(f'\nAssociation rules (lift > 1): {len(rules):,}')
print('\nTop 20 rules by lift:')
display(rules[['antecedents','consequents','support','confidence','lift']].head(20).assign(antecedents=lambda df: df['antecedents'].apply(lambda x: list(x)[0]), consequents = lambda df: df['consequents'].apply(lambda x: list(x)[0])))

Frequent itemsets: 134
  Singletons: 115
  Pairs:      19

Association rules (lift > 1): 38

Top 20 rules by lift:
antecedents consequents support confidence lift
0 large lemon limes 0.0115 0.1746 3.6959
1 limes large lemon 0.0115 0.2429 3.6959
2 organic raspberries organic strawberries 0.0121 0.2622 3.1958
3 organic strawberries organic raspberries 0.0121 0.1474 3.1958
4 bag of organic bananas organic hass avocado 0.0198 0.1536 2.5129
5 organic hass avocado bag of organic bananas 0.0198 0.3242 2.5129
6 organic fuji apple banana 0.0105 0.3856 2.4618
7 banana organic fuji apple 0.0105 0.0671 2.4618
8 organic raspberries bag of organic bananas 0.0143 0.3095 2.3986
9 bag of organic bananas organic raspberries 0.0143 0.1106 2.3986
10 organic hass avocado organic strawberries 0.0117 0.1913 2.3316
11 organic strawberries organic hass avocado 0.0117 0.1426 2.3316
12 banana honeycrisp apple 0.0102 0.0648 2.3303
13 honeycrisp apple banana 0.0102 0.3650 2.3303
14 organic baby spinach organic avocado 0.0117 0.1406 2.2902
15 organic avocado organic baby spinach 0.0117 0.1905 2.2902
16 organic strawberries bag of organic bananas 0.0231 0.2813 2.1807
17 bag of organic bananas organic strawberries 0.0231 0.1789 2.1807
18 organic hass avocado organic baby spinach 0.0105 0.1718 2.0662
19 organic baby spinach organic hass avocado 0.0105 0.1263 2.0662 
# Visualize association rules
fig, axes = plt.subplots(1, 2, figsize=(20, 6))

# Scatter: support × confidence, coloured by lift
sc = axes[0].scatter(rules['support'], rules['confidence'], c = rules['lift'], cmap = 'YlOrRd', alpha = 0.7, s = 40, edgecolors = 'gray', linewidth = 0.3)
plt.colorbar(sc, ax = axes[0], label = 'Lift')
axes[0].set_xlabel('Support'); axes[0].set_ylabel('Confidence')
axes[0].set_title('Support × Confidence (colour = Lift)', fontsize=12, fontweight='bold')

# Top 15 rules by lift — horizontal bar
top15 = rules.head(15).copy()
top15['rule_label'] = top15.apply(lambda r: f"{list(r['antecedents'])[0][:22]} → {list(r['consequents'])[0][:22]}", axis=1)
bar_colors = sns.color_palette('YlOrRd', 15)
axes[1].barh(top15['rule_label'][::-1], top15['lift'][::-1], color=bar_colors)
axes[1].axvline(1.0, color='gray', linestyle='--', alpha=0.6, label='Lift = 1 (independent)')
axes[1].set_xlabel('Lift'); axes[1].set_title('Top 15 Rules by Lift', fontsize=12, fontweight='bold')
axes[1].legend(); axes[1].tick_params(axis='y', labelsize=8)

plt.suptitle('Market Basket — Association Rules', fontsize=15, fontweight='bold', y=1.01)
plt.tight_layout()
plt.show()

png

# Build rule lookup dicts (used by prediction model) ant_frozenset → [(confidence, lift, consequent_name)]
rule_lookup = defaultdict(list)

for _, row in rules.iterrows():

    ant = frozenset(row['antecedents'])
    con = list(row['consequents'])[0]
    rule_lookup[ant].append((row['confidence'], row['lift'], con))

# single item → [(conf, lift, consequent, full_antecedent)]
rule_single = defaultdict(list)

for ant, recs in rule_lookup.items():

    for item in ant:

        for conf, lift, con in recs:

            rule_single[item].append((conf, lift, con, ant))

# Helper lookup dicts
pid_to_name = dict(zip(products['product_id'], products['product']))
name_to_pid = dict(zip(products['product'], products['product_id']))
pid_to_dept = dict(zip(products['product_id'], products['department']))

print(f'Rule lookup entries: {len(rule_lookup)}')
print(f'Single-item lookup:  {len(rule_single)} distinct items have rules')

Rule lookup entries: 12
Single-item lookup:  12 distinct items have rules

Back to top..

8. Next-Item Prediction Strategies

Five strategies recommend products as the user builds their cart, each progressively richer in how it uses purchase history and the sequence of items already added:

# Strategy Cart-Order Signal ML? Cold-start fallback
S1 Global Popularity — No — (is the fallback)
S2 Personal History — No Falls back to S1
S3 Association Rules Cart contents No Falls back to S2
S4 Sequential Markov Chain Transition + position probs No Falls back to S2
S5 LightGBM (cart-aware) Markov + position features ✅ Falls back to S1

All strategies share the same interface:

fn(user_id, cart_sequence, K=10) → [product_id, ...]

where cart_sequence is the ordered list of product_ids added to cart so far. This uniform signature makes it trivial to swap strategies in the evaluation harness and the final demo.

# Develop the ground truth for model scoring
train_truth = (train_op.groupby('user_id')['product_id'].apply(set).reset_index().rename(columns = {'product_id': 'true_items'}))

# Simulate "current shopping session" using the LAST prior order per user,
last_prior_oids = (orders_slim[orders_slim['eval_set'] == 'prior'].sort_values(['user_id', 'order_number']).groupby('user_id')['order_id'].last().reset_index().rename(columns = {'order_id': 'last_oid'}))

last_prior_seq = (prior_op[prior_op['order_id'].isin(last_prior_oids['last_oid'])].sort_values(['order_id', 'add_to_cart_order']).groupby('order_id')['product_id'].apply(list).reset_index().rename(columns = {'order_id': 'last_oid', 'product_id': 'cart_sequence'}))

eval_df = (train_truth.merge(last_prior_oids, on = 'user_id').merge(last_prior_seq, on = 'last_oid', how = 'inner'))

# Reproducible subsample for fast evaluation across all strategies
eval_sample = eval_df.sample(n = min(3000, len(eval_df)), random_state = 100).reset_index(drop = True)

print(f"Evaluation set  : {len(eval_df):,} users")
print(f"Eval sample     : {len(eval_sample):,} users")
print(f"Avg true basket : {eval_sample['true_items'].apply(len).mean():.1f} items")
print(f"Avg cart length : {eval_sample['cart_sequence'].apply(len).mean():.1f} items")
eval_sample[['user_id','cart_sequence','true_items']].head(3)

Evaluation set  : 95,361 users
Eval sample     : 3,000 users
Avg true basket : 10.5 items
Avg cart length : 10.6 items
user_id cart_sequence true_items
0 15188 [34915, 47766, 43183, 35406, 49054, 39993, 457... {46656, 34915, 46724, 264, 41290, 35406, 5296,...
1 23617 [41177] {42585, 18740}
2 13611 [38895, 24852, 9076, 16617, 33294, 45007, 2784... {17794, 44359, 2380, 40174, 38895, 1359, 47601... 
# ─────────────────────────────────────────────────────────────────────────────
# Strategy 1 — Global Popularity Baseline
# ─────────────────────────────────────────────────────────────────────────────
# Recommend the most-ordered products globally.  No personalisation, no cart
# context — pure frequency signal.  Serves as the floor baseline.

POPULARITY_POOL = (product_stats.sort_values('order_count', ascending=False)['product_id'].head(300).tolist())

def s1_popularity(user_id, cart_sequence, K = 10, **_):

    """Top-K globally popular products not already in the cart."""
    in_cart = set(cart_sequence)
    return [p for p in POPULARITY_POOL if p not in in_cart][:K]

print("✅  S1 (Global Popularity) ready")
print(f"   Pool: top {len(POPULARITY_POOL)} products")
print(f"   Sample recs: {[pid_to_name.get(p,'?')[:25] for p in s1_popularity(0, [], K=5)]}")

✅  S1 (Global Popularity) ready
   Pool: top 300 products
   Sample recs: ['banana', 'bag of organic bananas', 'organic strawberries', 'organic baby spinach', 'organic hass avocado']

# ─────────────────────────────────────────────────────────────────────────────
# Strategy 2 — Personal History
# ─────────────────────────────────────────────────────────────────────────────
# Rank the user's previously purchased products by reorder_score
# (frequency × reorder-rate / recency gap), then return the top-K
# not yet in the current cart.

user_history = (up_stats.sort_values(['user_id', 'reorder_score'], ascending=[True, False]).groupby('user_id')['product_id'].apply(list).to_dict())

def s2_personal(user_id, cart_sequence, K = 10, **_):

    """Top-K products from user's personal purchase history by reorder_score."""
    in_cart = set(cart_sequence)
    return [p for p in user_history.get(user_id, []) if p not in in_cart][:K]

coverage = sum(1 for uid in eval_sample['user_id'] if uid in user_history)
print("✅  S2 (Personal History) ready")
print(f"   History coverage : {coverage}/{len(eval_sample)} ({coverage/len(eval_sample):.1%})")

_u0 = eval_sample['user_id'].iloc[0]
_c0 = eval_sample['cart_sequence'].iloc[0]
print(f"   Sample recs (user {_u0}): {[pid_to_name.get(p,'?')[:22] for p in s2_personal(_u0,_c0,K=3)]}")

✅  S2 (Personal History) ready
   History coverage : 3000/3000 (100.0%)
   Sample recs (user 15188): ['sweet red grape tomato', 'banana', 'whole wheat bread']

# ─────────────────────────────────────────────────────────────────────────────
# Strategy 3 — Association Rules  (cart-content-aware)
# ─────────────────────────────────────────────────────────────────────────────
# Every item currently in the cart fires its association rules.
# Consequents are scored by the SUM of (confidence × lift) across all
# triggering cart items, then ranked.  Falls back to personal history
# when the rule set yields fewer than K candidates.

# Build a fast name-keyed lookup: item_name → {consequent_name: agg_score}
assoc_lookup = defaultdict(lambda: defaultdict(float))

for ant_name, recs in rule_single.items():

    for conf, lift, con_name, _ in recs:

        assoc_lookup[ant_name][con_name] += conf * lift

def s3_rules(user_id, cart_sequence, K=10, **_):

    """Score candidates via association rules fired by the current cart."""
    in_cart_ids  = set(cart_sequence)
    in_cart_names = {pid_to_name.get(p, '') for p in cart_sequence}

    scores = defaultdict(float)

    for name in in_cart_names:

        for con_name, score in assoc_lookup.get(name, {}).items():

            pid = name_to_pid.get(con_name)

            if pid and pid not in in_cart_ids:

                scores[pid] += score

    ranked = sorted(scores, key=scores.get, reverse=True)

    if len(ranked) >= K:

        return ranked[:K]

    # Backfill with personal history when rules are sparse
    extra = [p for p in s2_personal(user_id, cart_sequence, K * 3) if p not in set(ranked)]

    return (ranked + extra)[:K]

print("✅  S3 (Association Rules) ready")
print(f"   Assoc lookup items : {len(assoc_lookup)}")
print(f"   Sample recs : {[pid_to_name.get(p,'?')[:22] for p in s3_rules(_u0,_c0,K = 3)]}")

✅  S3 (Association Rules) ready
   Assoc lookup items : 12
   Sample recs : ['banana', 'organic baby spinach', 'sweet red grape tomato']

# ─────────────────────────────────────────────────────────────────────────────
# Strategy 4 — Sequential Markov Chain  (O(N) single-pass, no loops)
# ─────────────────────────────────────────────────────────────────────────────
MAX_POS        = 20
MAX_SUCCESSORS = 100
MAX_POS_ITEMS  = 200
MAX_USER_HIST  = 50

print("Building Markov transition matrix…")
t0 = time.time()

# ── Step 1: extract + sort ────────────────────────────────────────────────────
arr  = prior_op[['order_id','product_id','add_to_cart_order']].to_numpy(dtype = 'int32')
arr  = arr[np.lexsort((arr[:,2], arr[:,0]))]

oids = arr[:,0]; pids = arr[:,1]; pos0 = arr[:,2] - 1
del arr; gc.collect()

# ── Step 2: transition pairs ──────────────────────────────────────────────────
same   = oids[:-1] == oids[1:]
prev_p = pids[:-1][same].astype('int32')
next_p = pids[1: ][same].astype('int32')
del same; gc.collect()

# Sort pairs so identical (prev,next) rows are adjacent → count with diff
order  = np.lexsort((next_p, prev_p))
prev_s = prev_p[order]; next_s = next_p[order]
del prev_p, next_p, order; gc.collect()

# Find group boundaries in O(N) — no per-node mask scan
boundaries = np.flatnonzero(np.diff(prev_s)) + 1
boundaries = np.concatenate([[0], boundaries, [len(prev_s)]])

# Count runs of identical (prev,next) using diff on next_s within each prev group
pair_counts = np.diff(
    np.concatenate([np.flatnonzero(
        np.concatenate([[True], (prev_s[1:] != prev_s[:-1]) | (next_s[1:] != next_s[:-1]), [True]])
    ), [len(prev_s)]])
)
# Unique (prev,next) pairs
keep = np.concatenate([[True], (prev_s[1:] != prev_s[:-1]) | (next_s[1:] != next_s[:-1])])
u_prev = prev_s[keep]; u_next = next_s[keep]
del prev_s, next_s, keep; gc.collect()

# Group boundaries on the unique pairs
grp_bounds = np.flatnonzero(np.diff(u_prev)) + 1
grp_bounds = np.concatenate([[0], grp_bounds, [len(u_prev)]])

trans_prob = {}
for i in range(len(grp_bounds) - 1):
    lo, hi = grp_bounds[i], grp_bounds[i+1]
    node = int(u_prev[lo])
    c = pair_counts[lo:hi]; t = u_next[lo:hi]
    if len(c) > MAX_SUCCESSORS:
        top = np.argpartition(c, -MAX_SUCCESSORS)[-MAX_SUCCESSORS:]
        c, t = c[top], t[top]
    total = c.sum()
    trans_prob[node] = {int(ti): float(ci/total) for ti, ci in zip(t, c)}

del u_prev, u_next, pair_counts, grp_bounds; gc.collect()
print(f"  trans_prob done  ({len(trans_prob):,} nodes)  {time.time()-t0:.1f}s")

# ── Step 3: position-popularity ───────────────────────────────────────────────
mask  = pos0 < MAX_POS
p_pos = pos0[mask].astype('int32'); p_pid = pids[mask].astype('int32')
del oids, pids, pos0, mask; gc.collect()

order2  = np.lexsort((p_pid, p_pos))
pp_s = p_pos[order2]; pid_s = p_pid[order2]
del p_pos, p_pid, order2; gc.collect()

keep2 = np.concatenate([[True], (pp_s[1:] != pp_s[:-1]) | (pid_s[1:] != pid_s[:-1])])
pc = np.diff(np.concatenate([np.flatnonzero(keep2), [len(pp_s)]]))
u_pos = pp_s[keep2]; u_pid = pid_s[keep2]
del pp_s, pid_s, keep2; gc.collect()

grp2 = np.flatnonzero(np.diff(u_pos)) + 1
grp2 = np.concatenate([[0], grp2, [len(u_pos)]])

pos_prob = {}
for i in range(len(grp2) - 1):
    lo, hi = grp2[i], grp2[i+1]
    pos = int(u_pos[lo])
    c = pc[lo:hi]; t = u_pid[lo:hi]
    if len(c) > MAX_POS_ITEMS:
        top = np.argpartition(c, -MAX_POS_ITEMS)[-MAX_POS_ITEMS:]
        c, t = c[top], t[top]
    total = c.sum()
    pos_prob[pos] = {int(ti): float(ci/total) for ti, ci in zip(t, c)}

del u_pos, u_pid, pc, grp2; gc.collect()

# ── Step 4: per-user reorder-score list (capped) ─────────────────────────────
user_rs_list = (up_stats
                .sort_values('reorder_score', ascending=False)
                .groupby('user_id').head(MAX_USER_HIST)
                .groupby('user_id')
                .apply(lambda g: list(zip(g['product_id'].tolist(),
                                          g['reorder_score'].tolist())))
                .to_dict())

print(f"✅  Markov chain built in {time.time()-t0:.1f}s")
print(f"   Source nodes            : {len(trans_prob):,} products")
print(f"   Avg successors per node : {np.mean([len(v) for v in trans_prob.values()]):.1f}")
print(f"   Positions tracked       : 0 – {MAX_POS-1}")

Building Markov transition matrix…
  trans_prob done  (49,411 nodes)  24.8s
✅  Markov chain built in 67.9s
   Source nodes            : 49,411 products
   Avg successors per node : 48.3
   Positions tracked       : 0 – 19

# Strategy 4 — Sequential Markov Chain inference function
def s4_markov(user_id, cart_sequence, K = 10, n_gram = 2, alpha = 0.4, beta = 0.25, **_):

    """
    Sequential Markov Chain recommender.

    Score(X) = Σ P(next=X | prev ∈ cart[-n_gram:])          [Markov]
             + α · P(X appears at cart position len(cart))   [position]
             + β · reorder_score(user, X)                    [personal]

    Parameters
    ----------
    cart_sequence : list of product_ids in add-to-cart order
    n_gram        : how many trailing items to use as context (default 2)
    alpha         : weight for the position-popularity signal
    beta          : weight for the user's personal reorder score
    """
    in_cart  = set(cart_sequence)
    next_pos = len(cart_sequence)
    context  = cart_sequence[-n_gram:] if len(cart_sequence) >= n_gram else cart_sequence

    scores = defaultdict(float)

    # (A) Markov transition scores from the last n_gram cart items
    for prev in context:
        for nxt, prob in trans_prob.get(prev, {}).items():
            if nxt not in in_cart:
                scores[nxt] += prob

    # (B) Position-specific popularity at the *next* cart slot
    capped_pos = min(next_pos, MAX_POS - 1)
    for pid, prob in pos_prob.get(capped_pos, {}).items():
        if pid not in in_cart:
            scores[pid] += alpha * prob

    # (C) Blend personal purchase affinity
    for pid, rs in user_rs_list.get(user_id, []):
        if pid not in in_cart and rs > 0:
            scores[pid] += beta * float(rs)

    ranked = sorted(scores, key=scores.get, reverse=True)
    if len(ranked) >= K:
        return ranked[:K]

    extra = [p for p in s2_personal(user_id, cart_sequence, K * 2) if p not in set(ranked)]
    return (ranked + extra)[:K]

# Apply function
print("✅  S4 (Sequential Markov Chain) ready")
recs4 = s4_markov(_u0, _c0, K=3)
print(f"   Sample recs : {[pid_to_name.get(p,'?')[:22] for p in recs4]}")
print()
print("Position signal demo — top 3 products per cart slot:")
print(f"{'Pos':>4}  {'#1 product':<30} {'#2 product':<30} {'#3 product':<30}")
print("─" * 97)

for pos in range(6):
    slot = pos_prob.get(pos, {})
    top3 = sorted(slot, key=slot.get, reverse=True)[:3]
    cols = [f"{pid_to_name.get(p,'?')[:26]} ({slot[p]:.2%})" for p in top3]
    while len(cols) < 3:
        cols.append("—")
    print(f"{pos:>4}  {cols[0]:<30} {cols[1]:<30} {cols[2]:<30}")

✅  S4 (Sequential Markov Chain) ready
   Sample recs : ['sweet red grape tomato', 'banana', 'whole wheat bread']

Position signal demo — top 3 products per cart slot:
 Pos  #1 product                     #2 product                     #3 product                    
─────────────────────────────────────────────────────────────────────────────────────────────────
   0  banana (9.46%)                 bag of organic bananas (6.78%) organic whole milk (2.59%)    
   1  banana (7.48%)                 bag of organic bananas (5.99%) organic strawberries (2.78%)  
   2  banana (6.07%)                 bag of organic bananas (5.05%) organic strawberries (2.89%)  
   3  banana (5.04%)                 bag of organic bananas (4.25%) organic strawberries (2.91%)  
   4  banana (4.33%)                 bag of organic bananas (3.71%) organic strawberries (2.87%)  
   5  banana (3.95%)                 bag of organic bananas (3.29%) organic strawberries (2.77%)  

# ─────────────────────────────────────────────────────────────────────────────
# Strategy 5 — LightGBM feature matrix  (chunked, bounded candidate set)
# ─────────────────────────────────────────────────────────────────────────────
# Why previous versions OOM'd:
#   • Full candidate matrix (8000 users × 300+ candidates) = 2.4M rows
#     before a single feature column is added — too large to hold in RAM
#     alongside prior_op, trans_prob, etc.
#
# Fix — two constraints that bound memory absolutely:
#   1. CHUNK_SIZE users processed at a time; each chunk is del'd before next
#   2. Candidate pool capped: top MAX_PERSONAL history + top MAX_GLOBAL pool
#      → at most (MAX_PERSONAL + MAX_GLOBAL) rows per user, guaranteed

ML_N_USERS   = 3000   # reduce if still tight; 3k is plenty for LightGBM
CHUNK_SIZE   = 200    # users per chunk — peak RAM ≈ 200 × 150 × ~20 cols
MAX_PERSONAL = 80     # top items from user history by reorder_score
MAX_GLOBAL   = 80     # top globally popular items
# → max ~160 candidates per user, ~32k rows per chunk

ml_train_users = eval_df['user_id'].sample(n=ML_N_USERS, random_state=42).tolist()
ml_train_df    = (eval_df[eval_df['user_id'].isin(ml_train_users)]
                  .reset_index(drop=True))

# Static lookups built once, outside the loop
prod_feat = product_stats[['product_id','order_count','reorder_rate', 'avg_cart_position','global_popularity']]
prod_feat['pop_rank'] = prod_feat['order_count'].rank(pct = True).astype('float32')

user_feat = user_stats[['user_id','total_orders','avg_cart_size',
                         'reorder_rate','avg_days_between','variety_ratio']]

up_feat = up_stats[['user_id','product_id','times_ordered','reorder_score', 'orders_since_last','avg_cart_position','reorder_rate']]

up_feat.rename(columns={'avg_cart_position':'up_avg_cart_pos', 'reorder_rate':'up_reorder_rate'}, inplace=True)

# Cap global pool
GLOBAL_POOL = POPULARITY_POOL[:MAX_GLOBAL]

# Pre-convert assoc_lookup to pid-keyed dict once (avoids repeated name lookups)
assoc_by_pid = defaultdict(dict)

for ant_name, cons in assoc_lookup.items():

    ant_pid = name_to_pid.get(ant_name)

    if not ant_pid:

        continue

    for con_name, score in cons.items():

        con_pid = name_to_pid.get(con_name)

        if con_pid:

            assoc_by_pid[ant_pid][con_pid] = (assoc_by_pid[ant_pid].get(con_pid, 0.0) + score)

print(f"Users: {ML_N_USERS}  |  chunk: {CHUNK_SIZE}  |  "
      f"max candidates/user: {MAX_PERSONAL + MAX_GLOBAL}")
print(f"Expected rows: ~{ML_N_USERS * (MAX_PERSONAL + MAX_GLOBAL) // 2:,}")

t0 = time.time()
chunks = []

for chunk_start in range(0, len(ml_train_df), CHUNK_SIZE):

    chunk = ml_train_df.iloc[chunk_start : chunk_start + CHUNK_SIZE]
    rows  = []

    for _, row in chunk.iterrows():

        uid      = int(row['user_id'])
        true_set = row['true_items']
        cart_seq = row['cart_sequence']
        in_cart  = set(cart_seq)
        cart_sz  = len(cart_seq)

        # Bounded candidate set
        personal = [p for p in user_history.get(uid, [])[:MAX_PERSONAL]
                    if p not in in_cart]
        globl    = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
        candidates = personal + globl

        if not candidates:
            continue

        # Cart-context scores — computed once per user, not per candidate
        markov_sc = defaultdict(float)
        for prev in cart_seq:
            for nxt, p in trans_prob.get(prev, {}).items():
                markov_sc[nxt] += p

        assoc_sc = defaultdict(float)
        for ant_pid in cart_seq:
            for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():
                assoc_sc[con_pid] += sc

        capped_pos = min(cart_sz, MAX_POS - 1)
        pos_p      = pos_prob.get(capped_pos, {})

        for pid in candidates:
            rows.append((uid, pid,
                         int(pid in true_set),
                         cart_sz,
                         markov_sc.get(pid, 0.0),
                         assoc_sc.get(pid, 0.0),
                         pos_p.get(pid, 0.0)))

    if not rows:
        continue

    c_df = pd.DataFrame(rows, columns=['user_id','product_id','label',
                                        'cart_size','markov_score',
                                        'assoc_score','pos_prob_next'])
    c_df = (c_df
            .merge(prod_feat, on='product_id', how='left')
            .merge(user_feat, on='user_id',    how='left')
            .merge(up_feat,   on=['user_id','product_id'], how='left')
            .fillna(0))

    chunks.append(c_df)
    del c_df, rows
    gc.collect()

    if (chunk_start // CHUNK_SIZE) % 5 == 0:
        done = min(chunk_start + CHUNK_SIZE, len(ml_train_df))
        print(f"  {done}/{len(ml_train_df)} users …", end="")

cand_df = pd.concat(chunks, ignore_index=True)
del chunks
gc.collect()

pos_rate = cand_df['label'].mean()
print(f"\nDone in {time.time()-t0:.1f}s  |  rows: {len(cand_df):,}  "
      f"|  positive rate: {pos_rate:.3%}")
cand_df[['user_id','product_id','label','markov_score','assoc_score',
         'pos_prob_next','cart_size','reorder_score']].head(5)

Users: 3000  |  chunk: 200  |  max candidates/user: 160
Expected rows: ~240,000
  200/3000 users …  1200/3000 users …  2200/3000 users …
Done in 80.3s  |  rows: 333,300  |  positive rate: 3.461%
user_id product_id label markov_score assoc_score pos_prob_next cart_size reorder_score
0 144 42342 0 0.0394 0.0000 0.0000 6 0.3333
1 144 13603 0 0.0000 0.0000 0.0000 6 0.1667
2 144 22963 0 0.0752 0.0000 0.0030 6 0.0000
3 144 29871 0 0.0233 0.0000 0.0000 6 0.0000
4 144 32887 0 0.0000 0.0000 0.0000 6 0.0000 
# Define feature columns and train LightGBM with group-aware train/val split
FEATURE_COLS = ['cart_size', 'markov_score', 'assoc_score', 'pos_prob_next', 'order_count', 'reorder_rate', 'avg_cart_position', 'global_popularity', 'pop_rank', 'total_orders', 'avg_cart_size', 'reorder_rate_user', 'avg_days_between', 'variety_ratio', 'times_ordered', 'reorder_score', 'orders_since_last', 'up_avg_cart_pos', 'up_reorder_rate',]

# Some column names duplicated (reorder_rate from product + user) — rename defensively
cols = list(cand_df.columns)
seen = {}; new_cols = []

for c in cols:

    if c in seen:

        seen[c] += 1
        new_cols.append(f"{c}_user" if c == "reorder_rate" else f"{c}_{seen[c]}")

    else:

        seen[c] = 0; new_cols.append(c)

cand_df.columns = new_cols

FEATURE_COLS = [c for c in FEATURE_COLS if c in cand_df.columns]

X = cand_df[FEATURE_COLS].values
y = cand_df['label'].values
groups = cand_df['user_id'].values

# Group-aware train/val split (no user leaks between folds)
gss  = GroupShuffleSplit(n_splits=1, test_size=0.15, random_state=42)
tr_idx, va_idx = next(gss.split(X, y, groups))

X_tr, y_tr = X[tr_idx], y[tr_idx]
X_va, y_va = X[va_idx], y[va_idx]

# Scale negatives to 1 positive ~ 4 negatives (handles class imbalance)
pos_weight = max(1, int((y_tr == 0).sum() / max((y_tr == 1).sum(), 1) / 4))

lgb_model = lgb.LGBMClassifier(
    n_estimators    = 400,
    learning_rate   = 0.05,
    num_leaves      = 63,
    max_depth       = -1,
    min_child_samples = 20,
    subsample       = 0.8,
    colsample_bytree= 0.8,
    scale_pos_weight= pos_weight,
    n_jobs          = -1,
    random_state    = 42,
    verbose         = -1,
)

print("Training LightGBM …")
t0 = time.time()
lgb_model.fit(X_tr, y_tr,
              eval_set  = [(X_va, y_va)],
              callbacks = [lgb.early_stopping(30, verbose=False),
                           lgb.log_evaluation(50)])
print(f"Done in {time.time()-t0:.1f}s  |  best iteration: {lgb_model.best_iteration_}")

va_pred = lgb_model.predict_proba(X_va)[:, 1]
print(f"\nVal ROC-AUC  : {roc_auc_score(y_va, va_pred):.4f}")
print(f"Val Avg-Prec : {average_precision_score(y_va, va_pred):.4f}")

# Feature importance
fi = pd.Series(lgb_model.feature_importances_, index=FEATURE_COLS).sort_values(ascending=False)
print("\nTop-10 feature importances:")
print(fi.head(10).to_string())

Training LightGBM …
Done in 1.9s  |  best iteration: 4

Val ROC-AUC  : 0.8648
Val Avg-Prec : 0.2269

Top-10 feature importances:
orders_since_last    32
avg_cart_size        32
markov_score         26
variety_ratio        24
reorder_score        22
order_count          22
total_orders         21
cart_size            19
avg_cart_position    17
avg_days_between     12

# Strategy 5 — LightGBM inference function (mirrors training feature pipeline)
def s5_lgbm(user_id, cart_sequence, K=10, **_):
    """LightGBM recommender — mirrors the training feature pipeline exactly."""
    in_cart = set(cart_sequence)
    cart_sz = len(cart_sequence)

    # Bounded candidates — same caps used at training time
    personal   = [p for p in user_history.get(user_id, [])[:MAX_PERSONAL] if p not in in_cart]
    globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
    candidates = personal + globl

    if not candidates:

        return s1_popularity(user_id, cart_sequence, K)

    # Cart-context scores
    markov_sc = defaultdict(float)

    for prev in cart_sequence:

        for nxt, p in trans_prob.get(prev, {}).items():

            markov_sc[nxt] += p

    assoc_sc = defaultdict(float)                    # use pid-keyed dict (matches training)

    for ant_pid in cart_sequence:

        for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

            assoc_sc[con_pid] += sc

    pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

    rows = [{"user_id": user_id, "product_id": int(pid),
             "cart_size": cart_sz,
             "markov_score":  float(markov_sc.get(pid, 0.0)),
             "assoc_score":   float(assoc_sc.get(pid, 0.0)),
             "pos_prob_next": float(pos_p.get(pid, 0.0))}
            for pid in candidates]

    inf_df = (pd.DataFrame(rows)
              .merge(prod_feat, on="product_id", how="left")
              .merge(user_feat, on="user_id",    how="left")
              .merge(up_feat,   on=["user_id","product_id"], how="left")
              .fillna(0))

    # Apply the same column rename as training
    cols = list(inf_df.columns); seen = {}; new_cols = []
    for c in cols:
        if c in seen:
            seen[c] += 1
            new_cols.append(f"{c}_user" if c == "reorder_rate" else f"{c}_{seen[c]}")
        else:
            seen[c] = 0; new_cols.append(c)
    inf_df.columns = new_cols

    for fc in FEATURE_COLS:              # guarantee column alignment
        if fc not in inf_df.columns:
            inf_df[fc] = 0.0

    scores = lgb_model.predict_proba(inf_df[FEATURE_COLS].values)[:, 1]
    inf_df["score"] = scores
    return inf_df.nlargest(K, "score")["product_id"].tolist()

print("✅  S5 (LightGBM) ready")
recs5 = s5_lgbm(_u0, _c0, K=3)
print(f"   Sample recs : {[pid_to_name.get(p,'?')[:25] for p in recs5]}")

✅  S5 (LightGBM) ready
   Sample recs : ['banana', 'sweet red grape tomatoes', 'whole wheat bread']

Back to top..

9. Model Evaluation

Each strategy is scored on the same held-out eval_sample using five retrieval metrics at K = 5 and K = 10. The evaluation simulates a real shopping session: each user’s last prior order serves as the current cart, and we check whether the strategy’s top-K recommendations intersect with the user’s actual train basket.

Metric Formula What it captures
Precision\@K |recs ∩ true| / K Of K recommendations, what fraction are relevant
Recall\@K |recs ∩ true| / |true| What fraction of the true basket was surfaced
F1\@K 2·P·R / (P+R) Harmonic mean — balanced view of P and R
NDCG\@K DCG / IDCG Ranking quality — rewards relevant items appearing earlier in the list
Hit Rate\@K 1 if any hit else 0 Fraction of users who received ≥ 1 relevant recommendation 
# Scoring functions
def precision_at_k(recs, true_set, k):
    return len(set(recs[:k]) & true_set) / k if k else 0.0

def recall_at_k(recs, true_set, k):
    return len(set(recs[:k]) & true_set) / len(true_set) if true_set else 0.0

def f1_at_k(recs, true_set, k):
    p = precision_at_k(recs, true_set, k)
    r = recall_at_k(recs, true_set, k)
    return 2*p*r/(p+r) if (p+r) > 0 else 0.0

def ndcg_at_k(recs, true_set, k):
    dcg   = sum(1/np.log2(i+2) for i, p in enumerate(recs[:k]) if p in true_set)
    ideal = sum(1/np.log2(i+2) for i in range(min(len(true_set), k)))
    return dcg/ideal if ideal else 0.0

def score_recs(recs, true_set, K):
    return {
        "precision": precision_at_k(recs, true_set, K),
        "recall":    recall_at_k(recs, true_set, K),
        "f1":        f1_at_k(recs, true_set, K),
        "ndcg":      ndcg_at_k(recs, true_set, K),
        "hit_rate":  1 if any(p in true_set for p in recs[:K]) else 0,
    }

def evaluate_strategy(fn, data, K=10, n=None, seed = 100):

    """Standard per-user evaluator — used for S1–S4."""
    rows = data if n is None else data.sample(n=n, random_state=seed)
    m = {"precision":[],"recall":[],"f1":[],"ndcg":[],"hit_rate":[]}
    for _, row in rows.iterrows():
        s = score_recs(fn(row["user_id"], row["cart_sequence"], K=K), row["true_items"], K)
        for key in m: m[key].append(s[key])
    return {k: float(np.mean(v)) for k, v in m.items()}

def evaluate_lgbm_batched(eval_data, K = 10, n = None, seed = 100, chunk_size=200):

    """
    Batched evaluator for S5 — builds the full feature matrix for
    chunk_size users at once, calls predict_proba once per chunk,
    then scores. Avoids 3000 individual DataFrame merges.
    """
    rows_df = eval_data if n is None else eval_data.sample(n=n, random_state=seed)
    rows_df = rows_df.reset_index(drop=True)
    m = {"precision":[],"recall":[],"f1":[],"ndcg":[],"hit_rate":[]}

    for chunk_start in range(0, len(rows_df), chunk_size):

        chunk    = rows_df.iloc[chunk_start:chunk_start + chunk_size]
        cand_rows = []

        for idx, row in chunk.iterrows():

            uid      = int(row["user_id"])
            cart_seq = row["cart_sequence"]
            in_cart  = set(cart_seq)
            cart_sz  = len(cart_seq)

            personal   = [p for p in user_history.get(uid, [])[:MAX_PERSONAL] if p not in in_cart]
            globl      = [p for p in GLOBAL_POOL if p not in in_cart and p not in set(personal)]
            candidates = personal + globl

            if not candidates:

                candidates = s1_popularity(uid, cart_seq, K)

            markov_sc = defaultdict(float)

            for prev in cart_seq:

                for nxt, p in trans_prob.get(prev, {}).items():

                    markov_sc[nxt] += p

            assoc_sc = defaultdict(float)

            for ant_pid in cart_seq:

                for con_pid, sc in assoc_by_pid.get(ant_pid, {}).items():

                    assoc_sc[con_pid] += sc

            pos_p = pos_prob.get(min(cart_sz, MAX_POS - 1), {})

            for pid in candidates:
                cand_rows.append((idx, uid, int(pid), cart_sz,
                                  float(markov_sc.get(pid, 0.0)),
                                  float(assoc_sc.get(pid, 0.0)),
                                  float(pos_p.get(pid, 0.0))))

        if not cand_rows:
            continue

        c_df = pd.DataFrame(cand_rows, columns=[
            "idx","user_id","product_id","cart_size",
            "markov_score","assoc_score","pos_prob_next"])
        c_df = (c_df
                .merge(prod_feat, on="product_id", how="left")
                .merge(user_feat, on="user_id",    how="left")
                .merge(up_feat,   on=["user_id","product_id"], how="left")
                .fillna(0))

        # Same column rename as training
        cols = list(c_df.columns); seen = {}; new_cols = []
        for col in cols:
            if col in seen:
                seen[col] += 1
                new_cols.append(f"{col}_user" if col == "reorder_rate" else f"{col}_{seen[col]}")
            else:
                seen[col] = 0; new_cols.append(col)
        c_df.columns = new_cols
        for fc in FEATURE_COLS:
            if fc not in c_df.columns: c_df[fc] = 0.0

        # One predict_proba call for the whole chunk
        c_df["score"] = lgb_model.predict_proba(c_df[FEATURE_COLS].values)[:, 1]

        # Score each user from the chunk results
        for idx, row in chunk.iterrows():
            user_cands = c_df[c_df["idx"] == idx]
            if user_cands.empty:
                recs = []
            else:
                recs = user_cands.nlargest(K, "score")["product_id"].tolist()
            s = score_recs(recs, row["true_items"], K)
            for key in m: m[key].append(s[key])

        del c_df, cand_rows; gc.collect()

    return {k: float(np.mean(v)) for k, v in m.items()}

# ── Run all five strategies at K=5 and K=10 ───────────────────────────────────
STRATEGY_MAP = {
    "S1: Popularity"       : s1_popularity,
    "S2: Personal History" : s2_personal,
    "S3: Assoc Rules"      : s3_rules,
    "S4: Markov Chain"     : s4_markov,
    "S5: LightGBM"         : s5_lgbm,   # inference fn still available for demo use
}

N_EVAL = 1500
all_results = {}

print(f"Evaluating {len(STRATEGY_MAP)} strategies on {N_EVAL} users …\n")
header = f"  {'Strategy':<22}  {'P@5':>6} {'R@5':>6} {'F1@5':>6} {'NDCG@5':>7}  {'P@10':>6} {'R@10':>6} {'F1@10':>6} {'NDCG@10':>8} {'Hit@10':>7}  {'s':>4}"
print(header); print("─"*len(header))

for name, fn in STRATEGY_MAP.items():
    row_parts = [f"  {name:<22}"]
    for k_val in [5, 10]:
        t0 = time.time()
        if name == "S5: LightGBM":
            r = evaluate_lgbm_batched(eval_sample, K=k_val, n=N_EVAL)
        else:
            r = evaluate_strategy(fn, eval_sample, K=k_val, n=N_EVAL)
        dt = time.time() - t0
        all_results[(name, k_val)] = r
        row_parts.append(
            f"  {r['precision']:6.4f} {r['recall']:6.4f} {r['f1']:6.4f} {r['ndcg']:7.4f}")
    row_parts.append(f"  {r['hit_rate']:7.4f}  {dt:4.1f}s")
    print("".join(row_parts))

Evaluating 5 strategies on 1500 users …

  Strategy                   P@5    R@5   F1@5  NDCG@5    P@10   R@10  F1@10  NDCG@10  Hit@10     s
───────────────────────────────────────────────────────────────────────────────────────────────────
  S1: Popularity          0.0544 0.0258 0.0313  0.0603  0.0449 0.0400 0.0380  0.0559   0.3200   0.2s
  S2: Personal History    0.2148 0.1245 0.1355  0.2518  0.1627 0.1744 0.1469  0.2306   0.6927   0.2s
  S3: Assoc Rules         0.1588 0.0996 0.1051  0.1822  0.1444 0.1619 0.1329  0.1891   0.6760   0.2s
  S4: Markov Chain        0.2061 0.1146 0.1275  0.2414  0.1516 0.1588 0.1352  0.2164   0.6593   0.5s
  S5: LightGBM            0.2241 0.1313 0.1428  0.2588  0.1720 0.1871 0.1554  0.2402   0.7233  49.1s

# ── Results table ─────────────────────────────────────────────────────────────
result_rows = []
for (name, k), m in all_results.items():
    result_rows.append({"Strategy": name, "K": k, **m})
results_df = pd.DataFrame(result_rows)
print("=== Full Results Table ===")
display(results_df.sort_values(["K","ndcg"], ascending=[True,False]).reset_index(drop=True))

# ── Grouped bar charts ────────────────────────────────────────────────────────
strategy_names = list(STRATEGY_MAP.keys())
palette = sns.color_palette("husl", len(strategy_names))
metrics_info = [("precision","Precision@K"),("recall","Recall@K"),
                ("f1","F1@K"),("ndcg","NDCG@K"),("hit_rate","Hit Rate@K")]

fig, axes = plt.subplots(2, 3, figsize=(22, 10))
for ax_idx, (metric, label) in enumerate(metrics_info):
    ax = axes[ax_idx//3][ax_idx%3]
    x  = np.arange(len(strategy_names)); w = 0.35
    for ki, k_val in enumerate([5, 10]):
        vals = [all_results[(s, k_val)][metric] for s in strategy_names]
        bars = ax.bar(x + ki*w, vals, w, label=f"K={k_val}",
                      color=[palette[i] for i in range(len(strategy_names))],
                      alpha=0.7+ki*0.2, edgecolor="white")
        for bar, v in zip(bars, vals):
            ax.text(bar.get_x()+bar.get_width()/2, v+0.001,
                    f"{v:.3f}", ha="center", va="bottom", fontsize=7, rotation=45)
    ax.set_xticks(x+w/2)
    ax.set_xticklabels([s.split(":")[0] for s in strategy_names], fontsize=9)
    ax.set_title(label, fontsize=12, fontweight="bold")
    ax.legend(fontsize=8); ax.set_ylim(0, ax.get_ylim()[1]*1.2)

# Feature importance panel
ax = axes[1][2]
fi_top = fi.head(10)
bars = ax.barh(fi_top.index[::-1], fi_top.values[::-1],
               color=sns.color_palette("viridis", 10))
ax.set_title("LightGBM Feature Importance (top 10)", fontsize=12, fontweight="bold")
ax.set_xlabel("Importance")
cart_feats = {"markov_score","assoc_score","pos_prob_next","cart_size"}
for bar, feat in zip(bars[::-1], fi_top.index):
    if feat in cart_feats:
        ax.text(bar.get_width()*0.03, bar.get_y()+bar.get_height()/2,
                "🛒 cart-ctx", va="center", fontsize=7, color="navy")

plt.suptitle("Strategy Benchmark — All Metrics @ K=5 and K=10",
             fontsize=15, fontweight="bold", y=1.01)
plt.tight_layout(); plt.show()

# ── Radar chart ───────────────────────────────────────────────────────────────
radar_metrics = ["precision","recall","f1","ndcg","hit_rate"]
radar_labels  = ["P@10","R@10","F1@10","NDCG@10","Hit@10"]
N = len(radar_metrics)
angles = np.linspace(0, 2*np.pi, N, endpoint=False).tolist(); angles += angles[:1]

fig_r, ax_r = plt.subplots(figsize=(8,8), subplot_kw=dict(polar=True))
colors = sns.color_palette("husl", len(strategy_names))
for i, sname in enumerate(strategy_names):
    vals = [all_results[(sname, 10)][m] for m in radar_metrics]; vals += vals[:1]
    ax_r.plot(angles, vals, linewidth=2, label=sname, color=colors[i])
    ax_r.fill(angles, vals, alpha=0.08, color=colors[i])
ax_r.set_xticks(angles[:-1]); ax_r.set_xticklabels(radar_labels, fontsize=11)
ax_r.set_title("Strategy Profiles @ K=10", fontsize=14, fontweight="bold", pad=20)
ax_r.legend(loc="upper right", bbox_to_anchor=(1.35, 1.15), fontsize=9)
plt.tight_layout(); plt.show()

=== Full Results Table ===
Strategy K precision recall f1 ndcg hit_rate
0 S5: LightGBM 5 0.2241 0.1313 0.1428 0.2588 0.6273
1 S2: Personal History 5 0.2148 0.1245 0.1355 0.2518 0.5980
2 S4: Markov Chain 5 0.2061 0.1146 0.1275 0.2414 0.5780
3 S3: Assoc Rules 5 0.1588 0.0996 0.1051 0.1822 0.4920
4 S1: Popularity 5 0.0544 0.0258 0.0313 0.0603 0.2333
5 S5: LightGBM 10 0.1720 0.1871 0.1554 0.2402 0.7233
6 S2: Personal History 10 0.1627 0.1744 0.1469 0.2306 0.6927
7 S4: Markov Chain 10 0.1516 0.1588 0.1352 0.2164 0.6593
8 S3: Assoc Rules 10 0.1444 0.1619 0.1329 0.1891 0.6760
9 S1: Popularity 10 0.0449 0.0400 0.0380 0.0559 0.3200

png

png

Back to top..

10. Interactive Prediction Demo & Leaderboard

A unified recommend() call wraps all five strategies behind a single interface. Three progressive demos illustrate the system in action:

  • Demo A — LightGBM recommendations updating in real time as items are added one by one to the cart
  • Demo B — Side-by-side strategy comparison on a single user’s full cart, showing which strategy scores the most hits
  • Demo C — Custom cart specified by product name strings (e.g., "banana", "organic milk"), with output from Markov, Rules, and LightGBM

The section closes with a composite leaderboard ranking all five strategies by an equal-weighted average of all reported metrics.

# Unified recommend() wrapper — delegates to whichever strategy is selected
def recommend(user_id, cart_sequence, K = 10, strategy = "lgbm"):

    """
    strategy: "popularity" | "personal" | "rules" | "markov" | "lgbm"
    Returns a DataFrame: rank, product, department, aisle
    """

    fn_map = {"popularity": s1_popularity,
              "personal": s2_personal,
              "rules": s3_rules,
              "markov": s4_markov,
              "lgbm": s5_lgbm}

    pids = fn_map.get(strategy, s5_lgbm)(user_id, cart_sequence, K=K)
    out  = pd.DataFrame({"rank":range(1,len(pids)+1),"product_id":pids})

    return (out.merge(products[["product_id","product","department","aisle"]], on = "product_id", how = "left")[["rank","product","department","aisle"]])

# Demo A — recommendations update as the cart grows
demo_user = eval_sample["user_id"].iloc[5]
demo_cart = eval_sample["cart_sequence"].iloc[5]
demo_true = eval_sample["true_items"].iloc[5]

print(f"User {demo_user}  |  true next basket ({len(demo_true)} items):")
print([pid_to_name.get(p,"?")[:25] for p in list(demo_true)[:6]], "...\n")
print("="*72)
print(f"  {'Cart':>5}  {'Last item added':<30}  Top-5 recs (LightGBM)  [hits ✅]")
print("="*72)

for reveal in [1, 3, 5, len(demo_cart)]:

    partial = demo_cart[:reveal]
    last    = pid_to_name.get(partial[-1],"?")[:28]
    recs    = s5_lgbm(demo_user, partial, K=5)
    names   = [pid_to_name.get(p,"?")[:16]+("✅" if p in demo_true else "") for p in recs]
    print(f"  {reveal:>5}  {last:<30}  {names}")

User 12578  |  true next basket (2 items):
['hazelnut spread with coco', 'lobster bisque'] ...

========================================================================
   Cart  Last item added                 Top-5 recs (LightGBM)  [hits ✅]
========================================================================
      1  organic simply naked pita ch    ['non fat greek yo', 'april fresh liqu', 'organic salted b', 'boneless skinles', 'mango peach sals']
      3  mango peach salsa               ['non fat greek yo', 'april fresh liqu', 'organic salted b', 'boneless skinles', 'cereal']
      5  organic baby spinach            ['non fat greek yo', 'april fresh liqu', 'organic salted b', 'boneless skinles', 'bag of organic b']
      6  bag of organic bananas          ['non fat greek yo', 'april fresh liqu', 'organic salted b', 'boneless skinles', 'strawberries']

# Demo B — side-by-side strategy comparison on a single user
K_demo = 10
print(f"User {demo_user}  |  Cart size: {len(demo_cart)}  |  True basket: {len(demo_true)} items")
print(f"Cart: {[pid_to_name.get(p,'?')[:18] for p in demo_cart[:4]]}…\n")

comparison = {}

for sname, fn in STRATEGY_MAP.items():

    recs = fn(demo_user, demo_cart, K=K_demo)
    hits = [p for p in recs if p in demo_true]
    comparison[sname] = {"recs":recs,"hits":len(hits),
                         "P": len(hits)/K_demo,
                         "R": len(hits)/len(demo_true) if demo_true else 0}

comp_df = pd.DataFrame({
    sname: [pid_to_name.get(p,"?")[:28]+(" ✅" if p in demo_true else "")
            for p in d["recs"]]
    for sname, d in comparison.items()
})
comp_df.index = [f"Rec {i+1}" for i in range(K_demo)]
display(comp_df)
print("\nP@10 / R@10 per strategy:")
for sname, d in comparison.items():
    print(f"  {sname:<22}  hits={d['hits']}  P@10={d['P']:.2f}  R@10={d['R']:.2f}")

User 12578  |  Cart size: 6  |  True basket: 2 items
Cart: ['organic simply nak', 'wheat thins origin', 'mango peach salsa', 'cereal']…
S1: Popularity S2: Personal History S3: Assoc Rules S4: Markov Chain S5: LightGBM
Rec 1 banana non fat greek yogurt organic hass avocado non fat greek yogurt non fat greek yogurt
Rec 2 organic strawberries organic salted butter organic strawberries boneless skinless chicken br april fresh liquid fabric so
Rec 3 organic hass avocado boneless skinless chicken br organic avocado organic salted butter organic salted butter
Rec 4 organic avocado april fresh liquid fabric so banana organic hass avocado boneless skinless chicken br
Rec 5 large lemon brioche hamburger buns organic raspberries organic strawberries strawberries
Rec 6 strawberries cheez it cheddar cracker large lemon april fresh liquid fabric so organic tortilla chips
Rec 7 limes naked green machine boosted non fat greek yogurt organic avocado cheez it cheddar cracker
Rec 8 organic raspberries chicken pot pie organic salted butter organic raspberries brioche hamburger buns
Rec 9 organic whole milk organic spaghetti boneless skinless chicken br banana naked green machine boosted
Rec 10 organic yellow onion strawberries april fresh liquid fabric so naked green machine boosted raspberries 
P@10 / R@10 per strategy:
  S1: Popularity          hits=0  P@10=0.00  R@10=0.00
  S2: Personal History    hits=0  P@10=0.00  R@10=0.00
  S3: Assoc Rules         hits=0  P@10=0.00  R@10=0.00
  S4: Markov Chain        hits=0  P@10=0.00  R@10=0.00
  S5: LightGBM            hits=0  P@10=0.00  R@10=0.00

# Demo C — custom cart specified by product name strings
print("\n=== Custom cart: banana, strawberry, organic milk ===")
demo_u2 = eval_sample["user_id"].iloc[10]
cart_names = ["banana","strawberry","organic milk"]
cart_ids = []

for name in cart_names:

    match = [pid for pname,pid in name_to_pid.items() if name.lower() in pname.lower()]
    if match: cart_ids.append(match[0])

print(f"Resolved: {[pid_to_name.get(p,'?')[:25] for p in cart_ids]}\n")

for strat in ["markov","rules","lgbm"]:

    print(f"── {strat.upper()} ──")
    display(recommend(demo_u2, cart_ids, K=5, strategy=strat))

=== Custom cart: banana, strawberry, organic milk ===
Resolved: ['banana sweet potato organ', 'light strawberry blueberr', 'organic milk']

── MARKOV ──
rank product department aisle
0 1 happy baby spinachmangoand pear baby food babies baby food formula
1 2 blueberry oats and quinoa whole grain snack babies baby food formula
2 3 organic banana blueberry baby food puree babies baby food formula
3 4 alpine spring water beverages water seltzer sparkling water
4 5 berry barley organic baby food babies baby food formula 
── RULES ──
rank product department aisle
0 1 happy baby spinachmangoand pear baby food babies baby food formula
1 2 blueberry oats and quinoa whole grain snack babies baby food formula
2 3 organic banana blueberry baby food puree babies baby food formula
3 4 alpine spring water beverages water seltzer sparkling water
4 5 berry barley organic baby food babies baby food formula 
── LGBM ──
rank product department aisle
0 1 banana produce fresh fruits
1 2 organic pearraspberry purple carrot stage pouch babies baby food formula
2 3 organic stage broccoli pears peas baby food babies baby food formula
3 4 stage spinachapple kale babies baby food formula
4 5 peter rabbit organics kale broccoli and mango ... babies baby food formula 
# Construct final composite leaderboard across all strategies
lb_rows = []

for sname in strategy_names:

    r5  = all_results[(sname, 5)]
    r10 = all_results[(sname, 10)]
    composite = np.mean([r5["precision"],r5["recall"],r5["f1"],r5["ndcg"],
                         r10["precision"],r10["recall"],r10["f1"],r10["ndcg"],
                         r10["hit_rate"]])
    lb_rows.append({"Strategy":sname,
                    "P@5":f"{r5['precision']:.4f}","R@5":f"{r5['recall']:.4f}",
                    "F1@5":f"{r5['f1']:.4f}","NDCG@5":f"{r5['ndcg']:.4f}",
                    "P@10":f"{r10['precision']:.4f}","R@10":f"{r10['recall']:.4f}",
                    "F1@10":f"{r10['f1']:.4f}","NDCG@10":f"{r10['ndcg']:.4f}",
                    "Hit@10":f"{r10['hit_rate']:.4f}",
                    "Composite ↑":round(composite,4)})

lb = (pd.DataFrame(lb_rows)
        .sort_values("Composite ↑", ascending=False)
        .reset_index(drop=True))
lb.index += 1
print("\n🏆  Strategy Leaderboard")
display(lb)
print(f"\n🥇  Winner: {lb.iloc[0]['Strategy']}")
print("\nKey takeaways:")
print("  • Cart-sequence signals (Markov + position) lift recs beyond popularity baselines.")
print("  • LightGBM combines all signal types; typically best NDCG (rewards correct ranking).")
print("  • Association Rules add diversity; Markov is fastest at inference.")
print("  • Cold-start users (no history): fall back to S1 + S3.")

🏆  Strategy Leaderboard
Strategy P@5 R@5 F1@5 NDCG@5 P@10 R@10 F1@10 NDCG@10 Hit@10 Composite ↑
1 S5: LightGBM 0.2241 0.1313 0.1428 0.2588 0.1720 0.1871 0.1554 0.2402 0.7233 0.2483
2 S2: Personal History 0.2148 0.1245 0.1355 0.2518 0.1627 0.1744 0.1469 0.2306 0.6927 0.2371
3 S4: Markov Chain 0.2061 0.1146 0.1275 0.2414 0.1516 0.1588 0.1352 0.2164 0.6593 0.2235
4 S3: Assoc Rules 0.1588 0.0996 0.1051 0.1822 0.1444 0.1619 0.1329 0.1891 0.6760 0.2056
5 S1: Popularity 0.0544 0.0258 0.0313 0.0603 0.0449 0.0400 0.0380 0.0559 0.3200 0.0745 
🥇  Winner: S5: LightGBM

Key takeaways:
  • Cart-sequence signals (Markov + position) lift recs beyond popularity baselines.
  • LightGBM combines all signal types; typically best NDCG (rewards correct ranking).
  • Association Rules add diversity; Markov is fastest at inference.
  • Cold-start users (no history): fall back to S1 + S3.

Back to top..

]]>Data Guydataguy50@gmail.comFinancial Portfolio Backtesting: A Zero-Install Explorer Tool2026-03-23T00:00:00+00:002026-03-23T00:00:00+00:00https://doctorofdata.github.io/posts/2026/3/financial-backtesting-explorerThis post briefly discusses yet another feature introduction to this site: a comprehensive, no-code, drag&drop interface for evaluating various trading strategies on asset portfolios

⚙️ Introducing QuantBT: Where Strategy Meets Reality

Because every “genius” trading idea deserves a trial by fire.

Regression Pipeline Banner

🧠 The Problem with Brilliant Ideas

We’ve all had them.

That late-night epiphany:

  • “What if I just buy when it dips?”
  • “What if RSI actually works this time?”
  • “What if I combine momentum with mean reversion and call it alpha?”

The problem isn’t the idea.

The problem is reality.

Markets are noisy, unforgiving, and deeply unimpressed by intuition alone.

🔬 Enter: QuantBT

Today, I’m adding a new interactive tool to the site:

A fully client-side, zero-install quantitative strategy backtester.

No fluff. No abstractions. Just:

  • Real market data
  • Real strategy logic
  • Real performance outcomes

All running directly in your browser.

🧪 What It Actually Does

At its core, QuantBT lets you:

📊 Build a Portfolio

Add any set of tickers—from FAANG to obscure picks—and treat them as a unified system.

⚙️ Choose a Strategy

Out of the box, you can experiment with:

  • Moving Average Crossovers
  • RSI Thresholds
  • Bollinger Bands
  • MACD Signals
  • Momentum Models
  • Mean Reversion

Each with tunable parameters (because defaults are lies we tell ourselves).

⏱ Run Historical Simulations

Define:

  • Time horizon
  • Capital allocation
  • Position sizing

Then let the engine simulate every signal, trade, and portfolio evolution.

📈 Analyze What Actually Happened

Not what should have happened.

Not what felt right.

What actually happened.

You’ll get:

  • Portfolio return & growth curves
  • Sharpe, Sortino, and drawdown metrics
  • Per-asset breakdowns
  • Signal-level insights
  • A raw computation log (because transparency matters)

🧵 Under the Hood

This isn’t a toy wrapper.

It’s a deliberately engineered system featuring:

  • Multi-source market data ingestion (Alpha Vantage + fallbacks)
  • Indicator computation from scratch (no black boxes)
  • Strategy-specific signal engines
  • Portfolio simulation with position tracking
  • Risk and performance analytics
  • A terminal-style execution log for full traceability

All implemented in a single, self-contained interface.

🧠 Why This Exists

Because most people:

  • Overfit
  • Cherry-pick
  • Or worse… never test anything at all

QuantBT forces a different behavior:

Test first. Believe later.

⚠️ A Necessary Disclaimer

This tool is for:

  • Exploration
  • Education
  • Curiosity

It is not financial advice.

If your strategy works here, it might survive the real world.

If it fails here, it almost certainly won’t.

🚀 Try It Yourself

Load it up. Break it. Tune it.

Find out:

  • Which ideas collapse instantly
  • Which ones almost work
  • And which ones are worth digging into

Because in quantitative work, the only thing better than a good idea…

is a tested one.

Welcome to QuantBT.
Where hypotheses go to either die… or evolve.

Explore the tool here: Forecasting Backtester

]]>Data Guydataguy50@gmail.comMachine Learning Pipeline Evaluations w/ PySpark2026-03-18T00:00:00+00:002026-03-18T00:00:00+00:00https://doctorofdata.github.io/posts/2026/3/pyspark-machine-learning-pipelineRecently, I was asked by someone to describe my experience w/pyspark. It has been a while since I used it, and I figured I could use a touch-up, so I wrote this walkthrough as a gentle introduction to machine learning in distributed environments. Here it is.

Census Data- Predicting Employment Wages

Dataset Source: UCI Machine Learning Repository — Adult / Census Income

Overview

This notebook walks through a full PySpark-based machine learning pipeline applied to the Adult Census Income dataset from the UCI Machine Learning Repository. The dataset was extracted from the 1994 U.S. Census Bureau database by Barry Becker and is one of the most widely used benchmarks in binary classification research.

Goal: Predict whether an individual’s annual income exceeds $50,000 based on demographic and employment-related attributes.

Notebook Structure

  1. Environment Setup
  2. Data Loading
  3. Schema & Basic Inspection
  4. Missing Value Analysis
  5. Target Variable Distribution
  6. Numerical Feature EDA
  7. Categorical Feature EDA
  8. Bivariate Analysis: Features vs. Income
  9. Correlation Analysis
  10. Key Insights
  11. Machine Learning
  12. Evaluation

Why PySpark?

Apache Spark is a distributed data processing engine designed for large-scale analytics. PySpark is its Python API. Even on datasets that fit in memory, Spark is a valuable tool because:

  • It scales horizontally to petabyte-scale data without code changes
  • Its MLlib library provides production-grade ML pipelines with consistent APIs
  • Lazy evaluation and query optimization via the Catalyst engine improve efficiency
  • It is the industry standard for data engineering and ML at scale

Dataset Description

Feature Type Description
age Integer Age of the individual
workclass Categorical Employment type (Private, Self-emp, Gov, etc.)
fnlwgt Integer Census sampling weight (number of people the row represents)
education Categorical Highest level of education attained
education-num Integer Numeric encoding of education level
marital-status Categorical Marital status
occupation Categorical Type of occupation
relationship Categorical Role in household (Husband, Wife, etc.)
race Categorical Race of individual
sex Categorical Biological sex
capital-gain Integer Capital gains from investments
capital-loss Integer Capital losses from investments
hours-per-week Integer Average hours worked per week
native-country Categorical Country of origin
income Target Binary label: <=50K or >50K

The dataset contains 48,842 rows (train + test combined) with a mix of continuous and categorical variables.

1. Environment Setup

We install the ucimlrepo package to fetch the dataset directly from the UCI API, then import all libraries needed for this notebook. Apache Arrow is enabled as an optimization bridge between Pandas and PySpark DataFrames — it significantly speeds up the conversion by using an in-memory columnar format instead of row-by-row serialization.

# Install packages
!pip install ucimlrepo

Collecting ucimlrepo
  Downloading ucimlrepo-0.0.7-py3-none-any.whl.metadata (5.5 kB)
Requirement already satisfied: pandas>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from ucimlrepo) (2.2.2)
Requirement already satisfied: certifi>=2020.12.5 in /usr/local/lib/python3.12/dist-packages (from ucimlrepo) (2026.2.25)
Requirement already satisfied: numpy>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from pandas>=1.0.0->ucimlrepo) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas>=1.0.0->ucimlrepo) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas>=1.0.0->ucimlrepo) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas>=1.0.0->ucimlrepo) (2025.3)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas>=1.0.0->ucimlrepo) (1.17.0)
Downloading ucimlrepo-0.0.7-py3-none-any.whl (8.0 kB)
Installing collected packages: ucimlrepo
Successfully installed ucimlrepo-0.0.7

# Import libraries
from ucimlrepo import fetch_ucirepo
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
import seaborn as sns
import warnings
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
from pyspark.sql.functions import col, count, when, sum as spark_sum
from pyspark.ml import Pipeline
from pyspark.ml.feature import (StringIndexer, OneHotEncoder, VectorAssembler, StandardScaler)
from pyspark.ml.classification import (LogisticRegression, RandomForestClassifier, GBTClassifier)
from pyspark.ml.evaluation import (BinaryClassificationEvaluator, MulticlassClassificationEvaluator)
from sklearn.metrics import roc_curve, auc, precision_recall_curve, average_precision_score
spark = SparkSession.builder.appName("PySpark ML").getOrCreate()
import time

# ── Plotting config ──────────────────────────────────────────────────────────
sns.set_theme(style = "whitegrid", palette = "muted")
plt.rcParams.update({"figure.dpi": 120, "figure.figsize": (10, 4)})
warnings.filterwarnings("ignore")

# Enable Apache Arrow optimization
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

Back to top

2. Data Loading

We fetch the dataset via the UCI API and combine features with the target label (income) into a single PySpark DataFrame. The target is stored separately as y and joined back so that EDA can explore feature-label relationships.

Note on fnlwgt: This “final weight” column is a census sampling weight — it represents how many people in the U.S. population the row is estimated to represent. It is not predictive of income and is typically excluded from modelling, but we retain it here for completeness.

# Fetch dataset from UCI machine learning
adult = fetch_ucirepo(id = 2)

# Extract the data
df = adult.data.features
y = adult.data.targets

# metadata
print(f"Data Download Information: Dataset Name- {adult.metadata['name']} -- {adult.metadata['data_url']}")
print(f"Purpose - {adult.metadata['abstract']}")

# Normalise the target labels (strip whitespace, collapse '>50K.' -> '>50K')
y["income"] = y["income"].str.strip().str.replace(".", "", regex = False)

# Combine into one pandas DataFrame, then convert to PySpark
df = pd.concat([df, y], axis = 1)

# Convert the data to PySpark
pyspark_df = spark.createDataFrame(df)

Data Download Information: Dataset Name- Adult -- https://archive.ics.uci.edu/static/public/2/data.csv
Purpose - Predict whether annual income of an individual exceeds $50K/yr based on census data. Also known as "Census Income" dataset. 

# Utilities - function to generate output labels
def get_proba(preds):

    pdf = preds.select('label', 'probability').toPandas()
    pdf['prob_pos'] = pdf['probability'].apply(lambda v: float(v[1]))

    return pdf['label'].values, pdf['prob_pos'].values

Back to top..

3. Schema & Basic Inspection

Before any analysis, we inspect the DataFrame’s schema (column names and inferred data types) and take a peek at the first few rows. PySpark infers types when creating a DataFrame from Pandas — verifying these types is important before any transformation steps.

We also call describe() on numeric columns to get a quick statistical summary (count, mean, stddev, min, max).

print('Dataset Inspection..\n')
print("=== Schema ===")
pyspark_df.printSchema()

# Inspect the first 5 rows
pyspark_df.show(5)

# Summary statistics for numeric columns
numeric_cols = [f.name for f in pyspark_df.schema.fields if str(f.dataType) in ("IntegerType()", "LongType()", "DoubleType()", "FloatType()")]

print("\n=== Numeric Summary ===")
pyspark_df.select(numeric_cols).describe().show(truncate=False)

Dataset Inspection..

=== Schema ===
root
 |-- age: long (nullable = true)
 |-- workclass: string (nullable = true)
 |-- fnlwgt: long (nullable = true)
 |-- education: string (nullable = true)
 |-- education-num: long (nullable = true)
 |-- marital-status: string (nullable = true)
 |-- occupation: string (nullable = true)
 |-- relationship: string (nullable = true)
 |-- race: string (nullable = true)
 |-- sex: string (nullable = true)
 |-- capital-gain: long (nullable = true)
 |-- capital-loss: long (nullable = true)
 |-- hours-per-week: long (nullable = true)
 |-- native-country: string (nullable = true)
 |-- income: string (nullable = true)

+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
|age|       workclass|fnlwgt|education|education-num|    marital-status|       occupation| relationship| race|   sex|capital-gain|capital-loss|hours-per-week|native-country|income|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|
| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|
| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|
| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|
| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|
+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+
only showing top 5 rows

=== Numeric Summary ===
+-------+------------------+------------------+------------------+------------------+------------------+------------------+
|summary|age               |fnlwgt            |education-num     |capital-gain      |capital-loss      |hours-per-week    |
+-------+------------------+------------------+------------------+------------------+------------------+------------------+
|count  |48842             |48842             |48842             |48842             |48842             |48842             |
|mean   |38.64358543876172 |189664.13459727284|10.078088530363212|1079.0676262233324|87.50231358257237 |40.422382375824085|
|stddev |13.710509934443566|105604.02542315738|2.5709727555922592|7452.01905765541  |403.00455212435907|12.391444024252303|
|min    |17                |12285             |1                 |0                 |0                 |1                 |
|max    |90                |1490400           |16                |99999             |4356              |99                |
+-------+------------------+------------------+------------------+------------------+------------------+------------------+

Back to top..

4. Missing Value Analysis

The Adult dataset uses ? as a placeholder for unknown values rather than NaN. This means PySpark won’t detect them as nulls automatically — we need to count ? occurrences explicitly. This is a common real-world data quality issue: sentinel values that masquerade as valid data.

Columns affected: workclass, occupation, and native-country are the three known columns with missing values in this dataset.

# Use pyspark syntax ro count nulls and missing
missing_counts = pyspark_df.select([spark_sum(when(col(c).cast("string") == "?", 1).otherwise(0)).alias(c) for c in pyspark_df.columns]).toPandas().T.rename(columns = {0: "missing_count"})

missing_counts["missing_pct"] = (missing_counts["missing_count"] / pyspark_df.count() * 100).round(2)
missing_counts = missing_counts[missing_counts["missing_count"] > 0].sort_values("missing_count", ascending = False)

print("Columns with missing values ('?'):")
print(missing_counts.to_string())

Columns with missing values ('?'):
                missing_count  missing_pct
occupation               1843         3.77
workclass                1836         3.76
native-country            583         1.19

Back to top..

5. Target Variable Distribution

Understanding the class balance of the target is critical before building any classifier. Heavily imbalanced datasets can cause a model to simply predict the majority class and achieve misleadingly high accuracy.

In this dataset the target is binary:

  • <=50K — income at or below $50,000/year
  • >50K — income above $50,000/year
# Compute class distribution
target_dist = (pyspark_df.groupBy("income").count().withColumn("pct", F.round(F.col("count") / pyspark_df.count() * 100, 2)).orderBy("income").toPandas())
print("Target distribution:")
print(target_dist.to_string(index=False))

Target distribution:
income  count   pct
 <=50K  37155 76.07
  >50K  11687 23.93

# Plot
fig, axes = plt.subplots(figsize = (10, 4))

palette = sns.color_palette("muted", 2)

# Bar chart
axes.bar(target_dist["income"], target_dist["count"], color=palette, edgecolor="white", linewidth = 1.2)

for i, row in target_dist.iterrows():

    axes.text(i, row["count"] + 200, f"{row['count']:,}\n({row['pct']}%)", ha = "center", fontsize = 8)

axes.set_title("Income Class Counts", fontweight="bold")
axes.set_ylabel("Count")
axes.yaxis.set_major_formatter(mticker.FuncFormatter(lambda x, _: f"{int(x):,}"))

plt.suptitle("Target Variable: Annual Income", fontsize = 13, fontweight = "bold", y=1.01)
plt.tight_layout()
plt.show()

print("\nClass Ratio (<=50K : >50K):", round(target_dist.loc[0,'count'] / target_dist.loc[1,'count'], 2), ": 1")

png

Class Ratio (<=50K : >50K): 3.18 : 1

Back to top..

6. Numerical Feature EDA

The dataset contains five continuous/integer features: age, fnlwgt, education-num, capital-gain, capital-loss, and hours-per-week. We examine their distributions using histograms and box plots.

Key things to watch for:

  • Skewness — many ML algorithms perform better on roughly normal distributions
  • Outliers — extreme values in capital gains/losses are common and may need capping
  • Scale differences — Spark’s StandardScaler will be needed to normalise features before modelling
num_features = ["age", "fnlwgt", "education-num", "capital-gain", "capital-loss", "hours-per-week"]

# Bring numeric columns to Pandas for plotting
num_pd = pyspark_df.select(num_features + ["income"]).toPandas()

# Distribution plots (histograms + KDE)
fig, axes = plt.subplots(2, 3, figsize=(18, 10))
axes = axes.flatten()

for i, col_name in enumerate(num_features):

    sns.histplot(num_pd[col_name], ax=axes[i], kde = True, bins = 40, color = sns.color_palette("bright")[i % 6])
    axes[i].set_title(f"Distribution of {col_name}", fontweight = "bold")
    axes[i].set_xlabel(col_name)
    axes[i].set_ylabel("Count")
    skew_val = num_pd[col_name].skew()
    axes[i].text(0.98, 0.92, f"Skew: {skew_val:.2f}", transform = axes[i].transAxes, ha = "right", fontsize = 8, bbox = dict(boxstyle = "round,pad=0.2", fc="white", alpha=0.7))

plt.suptitle("Numerical Feature Distributions", fontsize=14, fontweight="bold")
plt.tight_layout()
plt.show()

png

# --- Box plots: distributions by income class ---
fig, axes = plt.subplots(2, 3, figsize=(18, 10))

axes = axes.flatten()

for i, col_name in enumerate(num_features):

    sns.boxplot(data=num_pd, x="income", y=col_name, ax=axes[i], palette = "bright", order = ["<=50K", ">50K"])
    axes[i].set_title(f"{col_name} by Income", fontweight="bold")
    axes[i].set_xlabel("")

plt.suptitle("Numerical Features Split by Income Class", fontsize=14, fontweight="bold")
plt.tight_layout()
plt.show()

png

Observations:

  • age: Roughly bell-shaped, centred in the late 30s. Higher earners skew slightly older.
  • capital-gain / capital-loss: Extremely right-skewed — the vast majority of individuals have zero capital gains or losses. A log1p transform or binarisation is typical before modelling.
  • hours-per-week: Strongly peaked at 40 (standard full-time). Higher earners tend to work slightly more hours.
  • fnlwgt: High variance, right-skewed. Not informative for income prediction and often dropped.
  • education-num: Discrete integer encoding of education level — unsurprisingly, higher earners have higher values.

Back to top..

7. Categorical Feature EDA

Categorical features require different visualisation strategies. We use count plots (bar charts of value frequencies) to understand the cardinality and distribution of each categorical column.

High-cardinality columns like native-country may need grouping or frequency-based encoding before being used as model features.

# Index the category fts
cat_features = ["workclass", "education", "marital-status", "occupation", "relationship", "race", "sex", "native-country"]
cat_pd = pyspark_df.select(cat_features + ["income"]).toPandas()

# Remove '?' placeholder rows for plotting
cat_pd = cat_pd.replace("?", np.nan).dropna()

fig, axes = plt.subplots(4, 2, figsize=(16, 22))
axes = axes.flatten()

for i, col_name in enumerate(cat_features):

    order = cat_pd[col_name].value_counts().index
    sns.countplot(data = cat_pd, y = col_name, ax = axes[i], order = order, palette = "bright")
    axes[i].set_title(f"{col_name}  (unique: {cat_pd[col_name].nunique()})", fontweight = "bold")
    axes[i].set_xlabel("Count")
    axes[i].set_ylabel("")

plt.suptitle("Categorical Feature Distributions", fontsize = 15, fontweight = "bold", y=1.002)
plt.tight_layout()
plt.show()

png

# Cardinality summary (useful for encoding strategy decisions)
card_df = pd.DataFrame({"column": cat_features,
                        "unique_values": [pyspark_df.select(c).distinct().count() for c in cat_features]}).sort_values("unique_values", ascending=False)

print("Cardinality of categorical features:")
print(card_df.to_string(index = False))
print("\nEncoding recommendation:")
print("  - Low cardinality (< 10): One-Hot Encoding via StringIndexer + OneHotEncoder")
print("  - High cardinality (native-country, 42 values): Target encoding or frequency grouping")

Cardinality of categorical features:
        column  unique_values
native-country             43
     education             16
    occupation             16
     workclass             10
marital-status              7
  relationship              6
          race              5
           sex              2

Encoding recommendation:
  - Low cardinality (< 10): One-Hot Encoding via StringIndexer + OneHotEncoder
  - High cardinality (native-country, 42 values): Target encoding or frequency grouping

Back to top..

8. Bivariate Analysis: Features vs. Income

Now we examine how each feature relates to the target variable. This reveals the most informative features for our classifier and helps validate domain intuition (e.g., education and occupation should be strong predictors).

# Stacked bar: proportion of >50K earners per category
high_cardinality_skip = ["native-country"]  # too many categories for this chart
cat_for_bivar = [c for c in cat_features if c not in high_cardinality_skip]

fig, axes = plt.subplots(4, 2, figsize=(18, 21))
axes = axes.flatten()

for i, col_name in enumerate(cat_for_bivar):

    ct = pd.crosstab(cat_pd[col_name], cat_pd["income"], normalize="index") * 100
    ct = ct.reindex(ct[">50K"].sort_values(ascending=False).index)  # sort by % high earners
    ct.plot(kind="barh", stacked=True, ax=axes[i], color=sns.color_palette("bright", 2), edgecolor="white", linewidth=0.8)

    axes[i].set_title(f"{col_name}: Income Breakdown (%)", fontweight="bold")
    axes[i].set_xlabel("% of group")
    axes[i].set_ylabel("")
    axes[i].legend(title="Income", loc="lower right", fontsize=8)
    axes[i].axvline(x=25, color="grey", linestyle="--", linewidth=0.8, alpha=0.6)

# Hide unused subplot
if len(cat_for_bivar) < len(axes):

    for j in range(len(cat_for_bivar), len(axes)):

        axes[j].set_visible(False)

plt.suptitle("Income Distribution Within Each Categorical Feature", fontsize = 15, fontweight = "bold", y = 1.002)
plt.tight_layout()
plt.show()

png

Back to top..

9. Correlation Analysis

We compute the Pearson correlation matrix for numeric features to detect multicollinearity. Highly correlated features carry redundant information and can reduce model interpretability — one of the pair is often dropped or a dimensionality reduction step (e.g., PCA) is applied.

We also compute point-biserial correlation of each numeric feature with the binary target (encoded as 0/1) to get a quick sense of predictive power.

# Correlation heatmap (numeric features only)
corr_pd = num_pd[num_features].copy()
corr_matrix = corr_pd.corr()

fig, ax = plt.subplots(figsize = (9, 7))
mask = np.triu(np.ones_like(corr_matrix, dtype=bool))
sns.heatmap(corr_matrix, mask=mask, annot=True, fmt=".2f", cmap = "coolwarm", center=0, linewidths=0.5, ax = ax, cbar_kws = {"shrink": 0.8})
ax.set_title("Pearson Correlation Matrix — Numeric Features", fontweight="bold", fontsize=13)
plt.tight_layout()
plt.show()

png

# Point-biserial correlation with target (numeric features vs income)
num_pd["income_binary"] = (num_pd["income"] == ">50K").astype(int)

target_corr = (num_pd[num_features + ["income_binary"]].corr()["income_binary"].drop("income_binary").sort_values(key = abs, ascending = False))

fig, ax = plt.subplots(figsize = (18, 3))
colors = ["#2E86AB" if v >= 0 else "#E84855" for v in target_corr.values]
bars = ax.barh(target_corr.index, target_corr.values, color=colors, edgecolor="white")
ax.axvline(0, color="black", linewidth=0.8)
ax.bar_label(bars, fmt="%.3f", padding=4, fontsize=9)
ax.set_title("Correlation of Numeric Features with Income (>50K)", fontweight="bold")
ax.set_xlabel("Pearson / Point-Biserial Correlation")
plt.tight_layout()
plt.show()

png

Observations:

  • education-num shows the strongest positive correlation with high income — more education → higher earnings.
  • age and hours-per-week also show moderate positive correlations.
  • capital-gain and capital-loss have weaker aggregate correlations due to the high proportion of zeros, but among those with non-zero values, they are very strong discriminators.
  • fnlwgt shows near-zero correlation with income, confirming it should be excluded from modelling.

Back to top..

10. Key Insights

Summary of EDA Findings

Finding Implication
~75% of samples earn ≤$50K Use class weights or resampling; prefer AUC-ROC over accuracy
workclass, occupation, native-country have ~5–7% missing (?) Impute with mode or drop rows (small impact)
capital-gain and capital-loss are extremely right-skewed Apply log1p transform before modelling
education-num is a numeric encoding of education (redundant) Drop education string column; keep education-num
fnlwgt has near-zero target correlation Drop from feature set
native-country has 42 unique values Frequency-group rare countries into “Other”
Married individuals (Married-civ-spouse) have notably higher >50K rates marital-status is a strong predictor

11. ML Pipeline Steps 

Raw Data
   │
   ├─ Replace '?' → null
   ├─ Impute nulls (mode for categoricals)
   ├─ Drop: fnlwgt, education (keep education-num)
   ├─ Log1p transform: capital-gain, capital-loss
   │
   ├─ Categorical Encoding
   │     └─ StringIndexer → OneHotEncoder (low cardinality)
   │     └─ Frequency grouping → StringIndexer (native-country)
   │
   ├─ Feature Assembly: VectorAssembler
   ├─ Scaling: StandardScaler
   │
   ├─ Train/Test split (80/20, stratified)
   │
   └─ Model Training
         ├─ Logistic Regression (baseline)
         ├─ Random Forest Classifier
         └─ Gradient Boosted Trees


```python
# Replace '?' values with nulls
missing_vals = ['workclass', 'occupation', 'native-country']

for c in missing_vals:

    pyspark_df = pyspark_df.withColumn(c, F.when(F.col(c) == '?', None).otherwise(F.col(c)))

# Log1p-transform skewed capital columns (as identified in Key Insights)
pyspark_df = pyspark_df.withColumn('capital-gain', F.log1p(F.col('capital-gain')))
pyspark_df = pyspark_df.withColumn('capital-loss', F.log1p(F.col('capital-loss')))

print("Log1p transform applied to capital-gain and capital-loss.")

Log1p transform applied to capital-gain and capital-loss.

# Encode the target variable
pyspark_df = pyspark_df.withColumn('label', F.when(F.col('income') == '>50K', 1.0).otherwise(0.0))

# Drop the nonuseful columns and outcome variable
train_df, test_df = pyspark_df.drop('fnlwgt', 'education', 'income').randomSplit([0.8, 0.2], seed = 100)

train_df.printSchema()

root
 |-- age: long (nullable = true)
 |-- workclass: string (nullable = true)
 |-- education-num: long (nullable = true)
 |-- marital-status: string (nullable = true)
 |-- occupation: string (nullable = true)
 |-- relationship: string (nullable = true)
 |-- race: string (nullable = true)
 |-- sex: string (nullable = true)
 |-- capital-gain: double (nullable = true)
 |-- capital-loss: double (nullable = true)
 |-- hours-per-week: long (nullable = true)
 |-- native-country: string (nullable = true)
 |-- label: double (nullable = false)

# Split
print(f"Train rows: {train_df.count():,} | Test rows: {test_df.count():,}")

Train rows: 39,082 | Test rows: 9,760

# Separate column types for model prep
cat_cols = ['workclass', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']
num_cols = ['age', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']

# Init model compoinents
indexers = [StringIndexer(inputCol = c, outputCol = f'{c}_idx', handleInvalid = 'keep') for c in cat_cols]

# OneHotEncoder: convert indices to sparse binary vectors
encoders = [OneHotEncoder(inputCol = f'{c}_idx', outputCol = f'{c}_ohe') for c in cat_cols]

# VectorAssembler: combine all features into one vector
assembler_inputs = num_cols + [f'{c}_ohe' for c in cat_cols]
assembler = VectorAssembler(inputCols = assembler_inputs, outputCol = 'raw_features', handleInvalid = 'keep')

preprocessing_stages = indexers + encoders + [assembler]
print(f'Pipeline preprocessing stages: {len(preprocessing_stages)}')

Pipeline preprocessing stages: 15

# Init models
lr  = LogisticRegression(featuresCol = 'raw_features', labelCol = 'label', maxIter = 100)
rf  = RandomForestClassifier(featuresCol = 'raw_features', labelCol = 'label', seed = 100)
gbt = GBTClassifier(featuresCol = 'raw_features', labelCol = 'label', maxIter = 50, seed = 100)

pipeline_lr  = Pipeline(stages = preprocessing_stages + [lr])
pipeline_rf  = Pipeline(stages = preprocessing_stages + [rf])
pipeline_gbt = Pipeline(stages = preprocessing_stages + [gbt])

start = time.time()

print("Training Logistic Regression..")
model_lr  = pipeline_lr.fit(train_df)

print("Training Random Forest..")
model_rf  = pipeline_rf.fit(train_df)

print("Training Gradient Boosted Trees..")
model_gbt = pipeline_gbt.fit(train_df)

end = time.time()

print(f"All models trained- elapsed time {round((end - start), 3) / 60} minutes..")

Training Logistic Regression..
Training Random Forest..
Training Gradient Boosted Trees..
All models trained- elapsed time 3.9390833333333335 minutes..

# Get predictions
lrpreds  = model_lr.transform(test_df)
rfpreds  = model_rf.transform(test_df)
gbpreds = model_gbt.transform(test_df)

Back to top..

12. Model Evaluation

Evaluation: AUC-ROC, F1, Precision-Recall, Confusion Matrix

# Init scoring metrics
bin_eval = BinaryClassificationEvaluator(labelCol = 'label', rawPredictionCol = 'rawPrediction', metricName = 'areaUnderROC')
mc_eval = MulticlassClassificationEvaluator(labelCol = 'label', predictionCol = 'prediction')

# Compile the scores
results = {}

for name, preds in [("LR", lrpreds), ("RF", rfpreds), ("GBT", gbpreds)]:

    results[name] = {"AUC-ROC":   bin_eval.evaluate(preds),
                     "Accuracy":  mc_eval.setMetricName("accuracy").evaluate(preds),
                     "F1":        mc_eval.setMetricName("f1").evaluate(preds),
                     "Precision": mc_eval.setMetricName("weightedPrecision").evaluate(preds),
                     "Recall":    mc_eval.setMetricName("weightedRecall").evaluate(preds)}

scores = round(pd.DataFrame(results).T, 4)

scores
Accuracy F1 Precision Recall
LR 0.8420 0.8367 0.8349 0.8420
RF 0.8305 0.8112 0.8215 0.8305
GBT 0.8546 0.8492 0.8481 0.8546 
fig, axes = plt.subplots(1, 3, figsize = (18, 4))

for ax, (preds, name) in zip(axes, [(lrpreds, "Logistic Regression"), (rfpreds, "Random Forest"), (gbpreds, "Gradient Boosted Trees")]):

    cm = preds.groupBy('label', 'prediction').count().toPandas()
    cm_pivot = cm.pivot(index='label', columns='prediction', values='count').fillna(0).astype(int)
    sns.heatmap(cm_pivot, annot=True, fmt='d', cmap='Blues', ax=ax)
    ax.set_title(f'Confusion Matrix — {name}')
    ax.set_ylabel('Actual')
    ax.set_xlabel('Predicted')

plt.tight_layout()
plt.show()

png

fig, axes = plt.subplots(1, 3, figsize = (18, 5))

for ax, (name, preds) in zip(axes, [("LR", lrpreds), ("RF", rfpreds), ("GBT", gbpreds)]):

    y_true, y_score = get_proba(preds)
    fpr, tpr, _ = roc_curve(y_true, y_score)
    roc_auc = auc(fpr, tpr)
    ax.plot(fpr, tpr, label=f"{name} (AUC = {roc_auc:.3f})")
    ax.plot([0,1],[0,1],'k--'); ax.set_xlabel('FPR'); ax.set_ylabel('TPR')
    ax.set_title(f'{name} ROC Curve');
    ax.legend();

plt.tight_layout();
plt.show()

png

# Precision-Recall curves — more informative than ROC under class imbalance
fig, axes = plt.subplots(1, 3, figsize = (18, 5))

for ax, (name, preds) in zip(axes, [("Logistic Regression", lrpreds),
                                     ("Random Forest",       rfpreds),
                                     ("Gradient Boosted Trees", gbpreds)]):

    y_true, y_score = get_proba(preds)
    precision, recall, _ = precision_recall_curve(y_true, y_score)
    ap = average_precision_score(y_true, y_score)
    baseline = y_true.mean()

    ax.plot(recall, precision, label=f"AP = {ap:.3f}")
    ax.axhline(baseline, color = "gray", linestyle = "--", linewidth = 0.9, label = f"Baseline ({baseline:.2f})")
    ax.set_xlabel("Recall")
    ax.set_ylabel("Precision")
    ax.set_title(f"PR Curve — {name}")
    ax.legend()

plt.suptitle("Precision-Recall Curves", fontweight = "bold", y = 1.02)
plt.tight_layout()
plt.show()

png

# Get ft importance
rf_model = model_rf.stages[-1]
importances = rf_model.featureImportances
feature_names = num_cols + [f'{c}_ohe' for c in cat_cols]

imp_df = pd.DataFrame({'feature': feature_names[:len(importances)],
                       'importance': importances.toArray()[:len(feature_names)]})
imp_df = imp_df.sort_values('importance', ascending = False).head(15)

plt.figure(figsize = (18, 7))
sns.barplot(data = imp_df, y = 'feature', x = 'importance', palette = 'viridis')
plt.title('Random Forest — Top Feature Importances')
plt.tight_layout(); plt.show()

png

13. Conclusion & Next Steps

What We Built

This notebook delivered a complete, production-style PySpark ML pipeline from raw census data through to evaluated classifiers. Every step reflects a real-world engineering decision:

Stage Decision Rationale
Missing values ? → null, handleInvalid='keep' in StringIndexer Preserves rows; avoids silent data loss
Feature selection Drop fnlwgt, education (keep education-num) Correlation-driven; removes redundancy
Skew correction Log1p on capital-gain / capital-loss Prevents tree depth waste and LR coefficient distortion
Encoding StringIndexer → OneHotEncoder PySpark MLlib sparse vector format; avoids ordinal assumptions
Class imbalance Evaluated with AUC-ROC and PR curves Accuracy alone is misleading on a 75/25 split

Model Comparison

All three models comfortably outperform the naive baseline. Gradient Boosted Trees leads on AUC-ROC thanks to sequential error correction on misclassified examples. Random Forest offers the best trade-off between performance and explainability — its feature importances confirmed the EDA findings: education-num, marital-status, and the log-transformed capital features are the strongest discriminators.

Where to Go Next

The pipeline scaffolded here is deliberately extensible. Several high-impact improvements are within reach:

Modelling improvements:

  • Add CrossValidator or TrainValidationSplit from pyspark.ml.tuning for hyperparameter search (e.g. tuning numTrees, maxDepth for RF/GBT, or regParam for LR)
  • Address class imbalance explicitly with weightCol — compute 1 / class_frequency per row and pass it to each classifier
  • Try LinearSVC or MultilayerPerceptronClassifier as additional baselines

Feature engineering:

  • Frequency-group native-country (42 categories → top-N + “Other”) to reduce noise from rare categories
  • Interact education-num × hours-per-week as a proxy for productive effort
  • Binarise capital-gain / capital-loss into has-capital-income flags alongside the log values

Production readiness:

  • Persist the best model with model.save(path) for serving via PipelineModel.load(path)
  • Replace the random split with a stratified split using sampleBy to guarantee class balance in both partitions
  • Wire a SparkSession with master("yarn") or master("k8s://...") to run this at true scale — no code changes required

Fairness & interpretability:

  • Audit predictions by race, sex, and native-country subgroups — the dataset’s demographic skews can propagate into disparate error rates
  • Use SHAP values (via a Pandas UDF wrapping shap) to move from aggregate feature importance to per-prediction explanations

The key takeaway: PySpark’s Pipeline API keeps preprocessing, encoding, and modelling in a single reproducible object. Whether this runs on a laptop or a 1,000-node cluster, the code is identical — only the SparkSession configuration changes.

Back to top..

]]>Data Guydataguy50@gmail.comNo Code Machine Learning Pipeline to Evaluate Models and Perform EDA2026-03-18T00:00:00+00:002026-03-18T00:00:00+00:00https://doctorofdata.github.io/posts/2026/3/machine-learning-model-explorerThis post briefly discusses the newest feature introduction to this site: a comprehensive, no-code, drag&drop interface for training and comparing machine learning models on disparate datasets

🚀 No-Code Machine Learning Pipeline

Interactive · Browser-Based · Zero Install

ML Pipeline Banner

Build, train, and evaluate machine learning models directly in your browser — no code required.

🌐 Live Demo

🔗 Try it here: Click Me!

🧠 What This Tool Does

  • 📂 Upload your dataset (CSV)
  • 🔍 Perform Exploratory Data Analysis (EDA)
  • ⚙️ Apply feature engineering & preprocessing
  • 🤖 Train multiple ML models
  • 📊 Evaluate performance (ROC, AUC, F1, Confusion Matrix)
  • 🔮 Generate predictions on new data

⚡ Key Features

  • 💻 100% Browser-Based (no setup)
  • 🧩 Step-by-step guided workflow
  • 📊 Built-in visualizations
  • 🧠 Multiple ML algorithms
  • 📈 Real-time evaluation

🏗️ Supported Models

  • Logistic Regression
  • K-Nearest Neighbors
  • Gaussian Naive Bayes
  • Decision Tree
  • Random Forest
  • ADA Boost
  • Gradient Boosting Machine

🪜 Step-by-Step Usage

📥 1. Load Dataset

  • Upload CSV
  • Ensure binary target column
  • Index columns auto-removed

🎯 2. Select Target & Features

  • Choose binary target
  • Define positive class
  • Configure feature types

🔍 3. Explore Data

  • Summary stats
  • Class balance
  • Visualizations

⚙️ 4. Preprocess Features

  • Apply scaling / transformations

🎛️ 5. Configure Models

  • Select models
  • Adjust split

🏋️ 6. Train

  • Run pipeline
  • Monitor progress

📊 7. Results

  • Metrics + visualizations

🔮 8. Predict

  • Upload new data
  • Generate predictions

🛠️ Tech Stack

  • Vanilla JavaScript
  • Chart.js
  • PapaParse

⭐ Support

Star the repo if you find it useful!

]]>Data Guydataguy50@gmail.comComprehensive Network Mapping of Netflix in Python - Part II2026-03-16T00:00:00+00:002026-03-16T00:00:00+00:00https://doctorofdata.github.io/posts/2026/03/netflix-network-mapping-continuedThis post is a continuation of the previous in which we used a dataset posted on Kaggle as fodder for creating a vast network of communities of actors and directors spread around the world! In this post we will explore the network dynamics of that information.

🎬 Mapping the Netflix Cinematic Graph Part II

A Network Science Exploration of Film Relationships

📦 Dataset: Netflix Titles — Kaggle (Shivamb)

In the first section of this project we constructed and visualized a collaboration network derived from the Netflix catalog. Each node in these networks represents a person (either an actor or a director), and edges represent collaborations where individuals worked together on the same title.

While visualization provides an intuitive understanding of the network structure, graph theory allows us to go further. Network analysis enables us to quantify structural properties of the collaboration graph and identify important individuals, patterns of connectivity, and communities within the network.

In this section we will analyze the collaboration network using a variety of tools from network science.

Why Analyze the Network?

Large collaboration networks often contain hidden structural patterns that are difficult to identify through visualization alone. By applying network metrics, we can answer questions such as:

• Who are the most connected individuals in the Netflix collaboration network?
• Which actors or directors act as bridges between otherwise separate communities?
• How densely connected is the network overall?
• Are there clusters corresponding to different film industries or collaboration circles?

Network analysis provides quantitative answers to these questions and helps reveal the underlying structure of the streaming entertainment ecosystem.

Graph Representation

The collaboration network used in this analysis is constructed as a person–person graph.

Nodes represent individuals involved in the production of Netflix titles:

• actors
• directors

Edges represent collaborative relationships:

• two individuals are connected if they appeared together on the same Netflix title

Edge weights represent the number of shared titles between two individuals, allowing us to distinguish occasional collaborations from frequent partnerships.

This structure transforms the Netflix catalog into a social network of creative collaboration.

Analysis Roadmap

The following analyses will be performed in this section:

  1. Global Graph Statistics
    Basic properties of the network including size, density, and connected components.

  2. Centrality Metrics
    Identifying the most influential or well-connected individuals using measures such as:
    • degree centrality
    • betweenness centrality
    • closeness centrality

  3. Community Detection
    Discovering clusters of individuals who frequently collaborate with each other.

  4. Collaboration Patterns
    Examining how collaborations are distributed across the network.

  5. Ego Networks
    Exploring the collaboration neighborhood surrounding specific actors or directors.

Together, these analyses provide a deeper understanding of how creative collaborations are structured within the Netflix catalog.

!pip install --upgrade colorcet dash gensim datashader "holoviews[recommended]" jupyter_bokeh bokeh python-louvain -q

# Import libraries
import pandas as pd
import collections
import networkx as nx
import colorcet as cc
import plotly.graph_objects as go
from dash import Dash, dcc, html, Input, Output
import random
from IPython.display import IFrame, display, Image
from holoviews.operation.datashader import (datashade, aggregate, dynspread, bundle_graph, split_dataframe,  regrid)
from holoviews.element.graphs import layout_nodes
from datashader.layout import forceatlas2_layout, random_layout, circular_layout
import holoviews as hv
import datashader as ds
import datashader.transfer_functions as tf
from datashader.bundling import connect_edges, hammer_bundle
hv.extension("bokeh")
import numpy as np
#from google.colab import drive
#drive.mount('/content/drive')
import time
from holoviews import opts
import math
from scipy.interpolate import splprep, splev
from itertools import combinations
import string
translator = str.maketrans('', '', string.punctuation)
import community as louvain_community
from bokeh.io import output_notebook
output_notebook()
from matplotlib.colors import ListedColormap
import panel as pn
pn.extension()
import colorsys
import matplotlib.colors as mcolors
from typing import List
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from PIL import Image as PILImage
import gc
import io
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
</img> </img> 
<style>
    .bk-notebook-logo {
        display: block;
        width: 20px;
        height: 20px;
        background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAABx0RVh0U29mdHdhcmUAQWRvYmUgRmlyZXdvcmtzIENTNui8sowAAAOkSURBVDiNjZRtaJVlGMd/1/08zzln5zjP1LWcU9N0NkN8m2CYjpgQYQXqSs0I84OLIC0hkEKoPtiH3gmKoiJDU7QpLgoLjLIQCpEsNJ1vqUOdO7ppbuec5+V+rj4ctwzd8IIbbi6u+8f1539dt3A78eXC7QizUF7gyV1fD1Yqg4JWz84yffhm0qkFqBogB9rM8tZdtwVsPUhWhGcFJngGeWrPzHm5oaMmkfEg1usvLFyc8jLRqDOMru7AyC8saQr7GG7f5fvDeH7Ej8CM66nIF+8yngt6HWaKh7k49Soy9nXurCi1o3qUbS3zWfrYeQDTB/Qj6kX6Ybhw4B+bOYoLKCC9H3Nu/leUTZ1JdRWkkn2ldcCamzrcf47KKXdAJllSlxAOkRgyHsGC/zRday5Qld9DyoM4/q/rUoy/CXh3jzOu3bHUVZeU+DEn8FInkPBFlu3+nW3Nw0mk6vCDiWg8CeJaxEwuHS3+z5RgY+YBR6V1Z1nxSOfoaPa4LASWxxdNp+VWTk7+4vzaou8v8PN+xo+KY2xsw6une2frhw05CTYOmQvsEhjhWjn0bmXPjpE1+kplmmkP3suftwTubK9Vq22qKmrBhpY4jvd5afdRA3wGjFAgcnTK2s4hY0/GPNIb0nErGMCRxWOOX64Z8RAC4oCXdklmEvcL8o0BfkNK4lUg9HTl+oPlQxdNo3Mg4Nv175e/1LDGzZen30MEjRUtmXSfiTVu1kK8W4txyV6BMKlbgk3lMwYCiusNy9fVfvvwMxv8Ynl6vxoByANLTWplvuj/nF9m2+PDtt1eiHPBr1oIfhCChQMBw6Aw0UulqTKZdfVvfG7VcfIqLG9bcldL/+pdWTLxLUy8Qq38heUIjh4XlzZxzQm19lLFlr8vdQ97rjZVOLf8nclzckbcD4wxXMidpX30sFd37Fv/GtwwhzhxGVAprjbg0gCAEeIgwCZyTV2Z1REEW8O4py0wsjeloKoMr6iCY6dP92H6Vw/oTyICIthibxjm/DfN9lVz8IqtqKYLUXfoKVMVQVVJOElGjrnnUt9T9wbgp8AyYKaGlqingHZU/uG2NTZSVqwHQTWkx9hxjkpWDaCg6Ckj5qebgBVbT3V3NNXMSiWSDdGV3hrtzla7J+duwPOToIg42ChPQOQjspnSlp1V+Gjdged7+8UN5CRAV7a5EdFNwCjEaBR27b3W890TE7g24NAP/mMDXRWrGoFPQI9ls/MWO2dWFAar/xcOIImbbpA3zgAAAABJRU5ErkJggg==);
    }
</style>
<div>
    <a href="https://bokeh.org" target="_blank" class="bk-notebook-logo"></a>
    <span id="c9cce3f6-4bda-4693-babf-7b34a8d16506">Loading BokehJS ...</span>
</div>

# Functions to work with datashader
cvsopts = dict(plot_height = 800, plot_width = 800)

def nodesplot(nodes, name = None, canvas = None, cat = None):

    if canvas is None:

        eps = 1e-6
        xr = (float(nodes.x.min()) - eps, float(nodes.x.max()) + eps)
        yr = (float(nodes.y.min()) - eps, float(nodes.y.max()) + eps)
        canvas = ds.Canvas(x_range = xr, y_range = yr, **cvsopts)

    # Strip cats before mapping
    if cat:

        nodes[cat] = nodes[cat].cat.remove_unused_categories()

    aggregator = None if cat is None else ds.count_cat(cat)
    agg = canvas.points(nodes,'x','y',aggregator)

    if cat:

        cats = list(nodes[cat].cat.categories)
        color_key = dict(zip(cats, make_hex_palette(len(cats))))

        return tf.dynspread(tf.shade(agg, color_key = color_key, name = name))

    return tf.dynspread(tf.shade(agg, cmap = ["#FF3333"], name = name))

def edgesplot(edges, name = None, canvas = None):

    if canvas is None:

        eps = 1e-6
        xr = (float(edges.x.min()) - eps, float(edges.x.max()) + eps)
        yr = (float(edges.y.min()) - eps, float(edges.y.max()) + eps)
        canvas = ds.Canvas(x_range = xr, y_range = yr, **cvsopts)

    return tf.shade(canvas.line(edges, 'x','y', agg = ds.count()), name = name)

def graphplot(nodes, edges, name = "", canvas = None, cat = None, pad = 0.025):

    if canvas is None:

        xmin, xmax = nodes.x.min(), nodes.x.max()
        ymin, ymax = nodes.y.min(), nodes.y.max()

        xpad = (xmax - xmin) * pad
        ypad = (ymax - ymin) * pad

        eps = 1e-6
        xr = (float(xmin - xpad) - eps, float(xmax + xpad) + eps)
        yr = (float(ymin - ypad) - eps, float(ymax + ypad) + eps)

        canvas = ds.Canvas(x_range = xr, y_range = yr, **cvsopts)

    nodeplot = nodesplot(nodes, name + " nodes", canvas, cat)
    edgeplot = edgesplot(edges, name + " edges", canvas)

    return tf.stack(edgeplot, nodeplot, how = "over", name = name)

# Function to create a colormap
def make_hex_palette(n):

    """High-contrast palette optimised for dark backgrounds."""
    result = []

    for i in range(n):

        h = i / n

        # lightness=0.65, saturation=0.95 — bright, vivid, dark-bg friendly
        r, g, b = colorsys.hls_to_rgb(h, 0.65, 0.95)
        result.append('#{:02x}{:02x}{:02x}'.format(int(r * 255), int(g * 255), int(b * 255)))

    return result

# Function to visualize colormap
def plot_colortable(hex_colors: List[str]):

    """Creates a colorbar using custom hex colors."""
    cmap = mcolors.ListedColormap(hex_colors)

    plt.figure(figsize = (8, 2), dpi = 150)
    plt.imshow([list(range(len(hex_colors)))], cmap = cmap, aspect = 'auto')
    plt.axis('off')
    plt.show()

def show_ds_images(images, titles, ncols = 2, fig_width = 18, cell_px = 800, bg = 'black', title_color = 'white', title_size = 11):

    """
    Display a list of datashader images with styled titles and background.

    Parameters
    ----------
    images : list of datashader Image objects
    titles : list of str
    ncols  : number of columns in the grid
    fig_width : total figure width in inches
    cell_px : resolution to render each image (square)
    bg     : hex background colour applied to each image AND the figure
    """

    nrows = -(-len(images) // ncols)
    cell_in = fig_width / ncols
    fig, axes = plt.subplots(nrows, ncols, figsize = (fig_width, cell_in * nrows), facecolor = bg, dpi = 150)
    axes = list(np.atleast_1d(axes).flat)

    for ax, img, title in zip(axes, images, titles):

        # Apply background colour, then upscale with Lanczos for sharpness
        styled = tf.set_background(img, bg)
        pil_img = styled.to_pil()
        pil_img = pil_img.resize((cell_px, cell_px), PILImage.LANCZOS)
        ax.imshow(pil_img, interpolation = 'lanczos')
        ax.set_title(title, color = title_color, fontsize = title_size, fontweight = 'bold', pad = 12, loc = 'center')
        ax.set_facecolor(bg)
        ax.axis('off')

    # Hide any unused axes
    for ax in list(axes)[len(images):]:

        ax.set_visible(False)

    plt.subplots_adjust(wspace = 0.04, hspace = 0.12)
    buf = io.BytesIO()
    plt.savefig(buf, format = 'png', bbox_inches = 'tight', facecolor = bg)
    plt.close()
    buf.seek(0)
    display(Image(data = buf.read()))
    buf.close()

def normalise_layout(df, margin = 0.05):

    """Rescale x, y to [margin, 1-margin] regardless of FA2 coordinate explosion."""
    df['x'] = (df['x'] - df['x'].min()) / (df['x'].max() - df['x'].min())
    df['y'] = (df['y'] - df['y'].min()) / (df['y'].max() - df['y'].min())
    df['x'] = df['x'] * (1 - 2*margin) + margin
    df['y'] = df['y'] * (1 - 2*margin) + margin

    return df

# Functions to map ego networks
def get_ego_network(G, name, radius = 1):

    """Extract the ego network of `name` up to `radius` hops."""
    if name not in G:

        raise ValueError(f'{name!r} not found in graph')

    ego_nodes = nx.ego_graph(G, name, radius = radius).nodes()

    return G.subgraph(ego_nodes).copy()

def ego_stats(G, ego_name):

    """Print summary statistics for an ego network."""
    ego = get_ego_network(G, ego_name)
    n   = ego.number_of_nodes()
    e   = ego.number_of_edges()
    alters = n - 1
    max_edges = alters * (alters - 1) / 2 if alters > 1 else 1
    alter_density = e / max_edges if max_edges > 0 else 0

    cc = nx.clustering(ego, ego_name)

    print(f'  Ego:             {ego_name}')
    print(f'  Alters (degree): {alters}')
    print(f'  Ego-net edges:   {e}')
    print(f'  Alter density:   {alter_density:.4f}')
    print(f'  Local clustering:{cc:.4f}')

    return ego

def plot_ego_network(G, ego_name, ax, title = None, node_size = 80):

    """Draw ego network on a given matplotlib axis."""
    ego = get_ego_network(G, ego_name)
    pos = nx.spring_layout(ego, seed = 100, k = 0.7)

    node_colors = ['#E50914' if n == ego_name else '#888888' for n in ego.nodes()]
    node_sizes  = [node_size * 4 if n == ego_name else node_size for n in ego.nodes()]

    edge_weights = [ego[u][v].get('weight', 1) for u, v in ego.edges()]
    max_w = max(edge_weights) if edge_weights else 1
    edge_widths = [0.5 + 2.0 * (w / max_w) for w in edge_weights]

    nx.draw_networkx_edges(ego, pos, ax=ax, edge_color='#cccccc', width = edge_widths, alpha = 0.7)
    nx.draw_networkx_nodes(ego, pos, ax=ax, node_color=node_colors, node_size = node_sizes, linewidths = 0.5, edgecolors = 'white')
    nx.draw_networkx_labels(ego, pos, ax=ax,labels = {ego_name: ego_name}, font_size = 8, font_color = 'white', font_weight = 'bold')

    n_alters = ego.number_of_nodes() - 1
    ax.set_title(title or f'{ego_name}\n({n_alters} direct collaborators)', fontsize = 9, fontweight = 'bold')
    ax.axis('off')

# Holoviews
kwargs = dict(width = 1000, height = 1000, xaxis = None, yaxis = None)
opts.defaults(opts.Nodes(**kwargs), opts.Graph(**kwargs))
colors = ['#000000'] + hv.Cycle('Category20').values

# Read the file using pandas
df = pd.read_csv('/Users/anon/Downloads/netflix_titles.csv')

# Split the cells to extract data
df['directors'] = df['director'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])
df['categories'] = df['listed_in'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])
df['actors'] = df['cast'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])
df['countries'] = df['country'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])

df.head()
show_id type title director cast country date_added release_year rating duration listed_in description directors categories actors countries
0 s1 Movie Dick Johnson Is Dead Kirsten Johnson NaN United States September 25, 2021 2020 PG-13 90 min Documentaries As her father nears the end of his life, filmm... [Kirsten Johnson] [Documentaries] [] [United States]
1 s2 TV Show Blood & Water NaN Ama Qamata, Khosi Ngema, Gail Mabalane, Thaban... South Africa September 24, 2021 2021 TV-MA 2 Seasons International TV Shows, TV Dramas, TV Mysteries After crossing paths at a party, a Cape Town t... [] [International TV Shows, TV Dramas, TV Mysteries] [Ama Qamata, Khosi Ngema, Gail Mabalane, Thaba... [South Africa]
2 s3 TV Show Ganglands Julien Leclercq Sami Bouajila, Tracy Gotoas, Samuel Jouy, Nabi... NaN September 24, 2021 2021 TV-MA 1 Season Crime TV Shows, International TV Shows, TV Act... To protect his family from a powerful drug lor... [Julien Leclercq] [Crime TV Shows, International TV Shows, TV Ac... [Sami Bouajila, Tracy Gotoas, Samuel Jouy, Nab... []
3 s4 TV Show Jailbirds New Orleans NaN NaN NaN September 24, 2021 2021 TV-MA 1 Season Docuseries, Reality TV Feuds, flirtations and toilet talk go down amo... [] [Docuseries, Reality TV] [] []
4 s5 TV Show Kota Factory NaN Mayur More, Jitendra Kumar, Ranjan Raj, Alam K... India September 24, 2021 2021 TV-MA 2 Seasons International TV Shows, Romantic TV Shows, TV ... In a city of coaching centers known to train I... [] [International TV Shows, Romantic TV Shows, TV... [Mayur More, Jitendra Kumar, Ranjan Raj, Alam ... [India] 
# Filter the data for movies with directors only
df2 = df[df['directors'].map(len) > 0]
df2 = df2[df2['actors'].map(len) > 0][['title', 'directors', 'actors']]

# ── Dataset composition ──────────────────────────────────────────
print('=== Dataset Overview ===')
print(f"Total titles:       {len(df):>7,}")
print(f"  Movies:           {(df.type=='Movie').sum():>7,}")
print(f"  TV Shows:         {(df.type=='TV Show').sum():>7,}")
print()
print(f"Titles with cast:   {df['actors'].apply(len).gt(0).sum():>7,}")
print(f"Titles with dir.:   {df['directors'].apply(len).gt(0).sum():>7,}")
print(f"Titles with both:   {(df['actors'].apply(len).gt(0) & df['directors'].apply(len).gt(0)).sum():>7,}")
print()

# Cast size distribution
cast_sizes = df['actors'].apply(len)
dir_sizes  = df['directors'].apply(len)

print(f"Avg cast size (titles with cast): {cast_sizes[cast_sizes > 0].mean():.2f}")
print(f"Max cast size:                    {cast_sizes.max()}")
print(f"Avg directors per title:          {dir_sizes[dir_sizes > 0].mean():.2f}")

# Plot
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

axes[0].hist(cast_sizes[cast_sizes > 0], bins = 30, color = '#E50914', edgecolor = 'black', linewidth = 0.4)
axes[0].set_title('Cast Size Distribution', fontweight='bold')
axes[0].set_xlabel('Number of credited cast members')
axes[0].set_ylabel('Number of titles')

type_counts = df['type'].value_counts()
axes[1].bar(type_counts.index, type_counts.values, color=['#E50914', '#333333'], edgecolor='black', linewidth=0.4)
axes[1].set_title('Movies vs. TV Shows', fontweight='bold')
axes[1].set_ylabel('Number of titles')
for i, v in enumerate(type_counts.values):
    axes[1].text(i, v + 20, f'{v:,}', ha='center', fontweight='bold')

plt.tight_layout()
plt.savefig('dataset_overview.png', dpi=150, bbox_inches='tight')
plt.show()

=== Dataset Overview ===
Total titles:         8,807
  Movies:             6,131
  TV Shows:           2,676

Titles with cast:     7,982
Titles with dir.:     6,173
Titles with both:     5,700

Avg cast size (titles with cast): 8.03
Max cast size:                    50
Avg directors per title:          1.13

png

# Redesign the edges as a people network
people_edges = []

for idx, row in df2.iterrows():

    combos = [(item1, item2) for item1 in row['directors'] for item2 in row['actors'] if item1 is not None]
    pairs = list(combinations(row['actors'], 2))

    for combo in combos:

        people_edges.append(tuple(sorted(combo)))

    for pair in pairs:

        people_edges.append(tuple(sorted(pair)))

people_edges = pd.DataFrame(people_edges, columns = ['source', 'target'])

# Calculate weighred edges
weighted_people_edges = people_edges.groupby(['source', 'target']).size().reset_index(name = 'weight')
weighted_people_edges.sort_values('weight', ascending = False).head()

# Remove punctuation
weighted_people_edges['source'] = weighted_people_edges['source'].astype(str).apply(lambda x: x.translate(translator))
weighted_people_edges['target'] = weighted_people_edges['target'].apply(lambda x: x.translate(translator))

people_nodes = pd.DataFrame(pd.unique(weighted_people_edges[['source', 'target']].values.ravel('K')), columns = ['name'])

# Convert the nodes to indices for mapping
#node_to_idx = {node: i for i, node in enumerate(people_nodes['name'])}
#weighted_people_edges["source"] = weighted_people_edges["source"].map(node_to_idx)
#weighted_people_edges["target"] = weighted_people_edges["target"].map(node_to_idx)

# Filter the graph for meaningful connections
filtered_people_edges = (weighted_people_edges[weighted_people_edges.weight >= 2]).copy()

# Keep nodes that appear in filtered edges
remaining_nodes = pd.unique(filtered_people_edges[['source','target']].values.ravel())

filtered_people_nodes = people_nodes[people_nodes.name.isin(remaining_nodes)].copy()

print('Filtering graph based on edge weights..\n')
print(f"Initial Configuration: \nNodes- {people_nodes.shape[0]}\nEdges- {weighted_people_edges.shape[0]}\n")
print(f"Filtered Result: \nNodes- {filtered_people_nodes.shape[0]}\nEdges- {filtered_people_edges.shape[0]}\n")

Filtering graph based on edge weights..

Initial Configuration: 
Nodes- 30798
Edges- 238512

Filtered Result: 
Nodes- 5313
Edges- 10714

Back to top..

Global Graph Statistics

Before diving into individual metrics, we assess the global topology of the network. These high-level statistics reveal whether the collaboration graph behaves like a typical social network — sparse, with one dominant connected component — or exhibits more unusual structural properties.

Metric Meaning
Nodes / Edges Scale of the network
Density Fraction of all possible edges that actually exist
Connected components Number of isolated sub-graphs
Largest component Size of the dominant cluster
Average clustering Local cliquishness — do your collaborators also collaborate with each other?
Average degree Mean number of direct collaborators per person 
# Build the most expansive graph
graph = nx.from_pandas_edgelist(weighted_people_edges, source = "source", target = "target", edge_attr = "weight")
filtered_graph = nx.from_pandas_edgelist(filtered_people_edges, source = "source", target = "target", edge_attr = "weight")

# Provide a summary
print('Outlook of Unabridged Network-\n')
print("Number of nodes:", graph.number_of_nodes())
print("Number of edges:", graph.number_of_edges())
print("Network density:", round(nx.density(graph), 5))
print("Connected components:", nx.number_connected_components(graph))
print("Largest component size:", len(max(nx.connected_components(graph), key = len)))
print("Average clustering:", round(nx.average_clustering(graph), 5))
print("Average degree:", round(sum(dict(graph.degree()).values()) / graph.number_of_nodes(), 5))

# Provide a summary
print('\nOutlook of Filtered Network-\n')
print("Number of nodes:", filtered_graph.number_of_nodes())
print("Number of edges:", filtered_graph.number_of_edges())
print("Network density:", round(nx.density(filtered_graph), 5))
print("Connected components:", nx.number_connected_components(filtered_graph))
print("Largest component size:", len(max(nx.connected_components(filtered_graph), key = len)))
print("Average clustering:", round(nx.average_clustering(filtered_graph), 5))
print("Average degree:", round(sum(dict(filtered_graph.degree()).values()) / filtered_graph.number_of_nodes(), 5))

Outlook of Unabridged Network-

Number of nodes: 30798
Number of edges: 238510
Network density: 0.0005
Connected components: 525
Largest component size: 27546
Average clustering: 0.82323
Average degree: 15.48867

Outlook of Filtered Network-

Number of nodes: 5313
Number of edges: 10714
Network density: 0.00076
Connected components: 434
Largest component size: 2319
Average clustering: 0.34572
Average degree: 4.03313

Interpreting the Numbers (According to ClaudeAI)

Full graph (weight ≥ 1): With ~~30,800 nodes and ~~238,500 edges the raw network is large but remarkably sparse (density ≈ 0.0005 — meaning only 0.05 % of all possible connections exist). The high average clustering coefficient (~0.82) tells us that when two people share a collaborator, they are very
likely to have also worked together — a hallmark of tightly knit ensembles working on the same productions.

Filtered graph (weight ≥ 2): Restricting to recurring collaborations cuts the graph to ~5,300 nodes and ~10,700 edges. The clustering drops to ~0.35, indicating that many one-production cliques have been removed and what remains are genuine repeating professional partnerships. The average degree of ~4 means each person in the filtered network has on average four recurring collaborators.

The 525 isolated components in the full graph (vs. 434 in the filtered) largely represent foreign-language cinema clusters that do not connect to the main Hollywood/international core — a natural reflection of geographically segmented production industries.

Back to top..

Centrality Metrics 

print("Computing centrality metrics...")

start = time.time()

# Get items
nodes = list(filtered_graph.nodes())

# Compute
degree = dict(filtered_graph.degree())
degree_centrality = nx.degree_centrality(filtered_graph)
betweenness = nx.betweenness_centrality(filtered_graph, k = 500, seed = 100)
closeness = nx.closeness_centrality(filtered_graph)

# Compile
metrics = pd.DataFrame({"actor": [n for n in nodes],
                        "degree": [degree[n] for n in nodes],
                        "degree_centrality": [degree_centrality[n] for n in nodes],
                        "betweenness": [betweenness[n] for n in nodes],
                        "closeness": [closeness[n] for n in nodes]})

end = time.time()

print(f"Code Execution: {round(end - start)} seconds elapsed")

Computing centrality metrics...
Code Execution: 17 seconds elapsed

metrics.head()
actor degree degree_centrality betweenness closeness
0 50 Cent 2 0.000377 0.003513 0.054561
1 Bruce Willis 8 0.001506 0.005596 0.062208
2 John Cusack 2 0.000377 0.003420 0.048583
3 AC Peterson 1 0.000188 0.000000 0.000991
4 Michael James Regan 10 0.001883 0.000003 0.001883

Component Size Distribution

Most social networks exhibit a giant connected component that dwarfs all others. The plot below confirms this pattern in our filtered network — one component dominates, while the remaining hundreds of components are tiny isolated clusters, typically consisting of a handful of collaborators from a single production house or country.

# Component size distribution
component_sizes = sorted([len(c) for c in nx.connected_components(filtered_graph)], reverse = True)

fig, axes = plt.subplots(1, 2, figsize=(13, 4))

# Left: top 20 components
top20 = component_sizes[:20]
axes[0].bar(range(1, len(top20)+1), top20, color='#E50914', edgecolor='black', linewidth=0.4)
axes[0].set_title('Top 20 Component Sizes', fontweight='bold')
axes[0].set_xlabel('Component rank')
axes[0].set_ylabel('Number of nodes')
axes[0].spines[['top', 'right']].set_visible(False)
for i, v in enumerate(top20[:5]):
    axes[0].text(i+1, v+10, str(v), ha='center', fontsize=8, fontweight='bold')

# Right: histogram of all component sizes (log scale)
axes[1].hist(component_sizes, bins=40, color='#333333', edgecolor='white', linewidth=0.3)
axes[1].set_yscale('log')
axes[1].set_title('Component Size Histogram (log scale)', fontweight='bold')
axes[1].set_xlabel('Component size (nodes)')
axes[1].set_ylabel('Count (log scale)')
axes[1].spines[['top', 'right']].set_visible(False)

plt.suptitle(f'Filtered graph: {len(component_sizes)} components  •  '
             f'Largest = {component_sizes[0]:,} nodes  •  '
             f'Median = {int(np.median(component_sizes))} nodes',
             fontsize=10, y=1.02)
plt.tight_layout()
plt.savefig('component_sizes.png', dpi=150, bbox_inches='tight')
plt.show()

png

Degree Centrality — The Most Prolific Collaborators

Degree centrality is the simplest centrality measure: it counts how many unique people a node is directly connected to, normalised by the maximum possible connections. In the Netflix collaboration context a high degree actor is one who has repeatedly worked with many different co-stars and directors — the hallmark of a busy, genre-crossing career.

# ── Top 20 by degree centrality ──────────────────────────────────
top_degree = metrics.sort_values('degree_centrality', ascending = False).head(20)

fig, ax = plt.subplots(figsize=(11, 6))
colors_bar = ['#E50914' if i < 5 else '#B0B0B0' for i in range(len(top_degree))]
bars = ax.barh(top_degree['actor'][::-1], top_degree['degree_centrality'][::-1], color = colors_bar[::-1], edgecolor = 'white', linewidth = 0.4)

for bar, deg in zip(bars, top_degree['degree'][::-1]):

    ax.text(bar.get_width() + 0.00005, bar.get_y() + bar.get_height()/2, f'  {int(deg)} connections', va = 'center', fontsize = 8, color = '#333333')

ax.set_xlabel('Degree Centrality', fontsize=10)
ax.set_title('Top 20 Nodes by Degree Centrality\n'
             '(normalised: share of all possible connections)', fontsize = 12, fontweight = 'bold')
ax.spines[['top', 'right']].set_visible(False)
ax.set_xlim(0, top_degree['degree_centrality'].max() * 1.35)
plt.tight_layout()
plt.savefig('degree_centrality.png', dpi=150, bbox_inches='tight')
plt.show()

png

Betweenness Centrality — The Bridge Builders

A node with high betweenness sits on many shortest paths between other nodes. In a collaboration network this identifies individuals who act as connectors between otherwise disparate communities — for instance, an actor who bridges Bollywood productions with international co-productions. Removing such a node would dramatically fragment the network, making them strategically critical.

Note: High betweenness does not necessarily correlate with high degree. A lesser-known actor can have enormous betweenness simply by being the only link between two distinct clusters.

# ── Top 20 by betweenness centrality ────────────────────────────
top_between = metrics.sort_values('betweenness', ascending = False).head(20)

fig, ax = plt.subplots(figsize=(11, 6))
colors_bar = ['#E50914' if i < 5 else '#B0B0B0' for i in range(len(top_between))]
ax.barh(top_between['actor'][::-1], top_between['betweenness'][::-1], color = colors_bar[::-1], edgecolor='white', linewidth=0.4)

ax.set_xlabel('Betweenness Centrality', fontsize=10)
ax.set_title('Top 20 Nodes by Betweenness Centrality\n'
             '(fraction of shortest paths passing through this node)', fontsize=12, fontweight='bold')
ax.spines[['top', 'right']].set_visible(False)
plt.tight_layout()
plt.savefig('betweenness_centrality.png', dpi=150, bbox_inches='tight')
plt.show()

png

Closeness Centrality — Well-Positioned Individuals

Closeness centrality measures the inverse average shortest path length from a node to all reachable nodes. A high-closeness individual can reach any other person in the network in fewer hops — they are centrally placed within the collaboration ecosystem. In practice, these are often versatile actors who participate in many different production clusters, making them well-connected globally rather than just locally.

# ── Top 20 by closeness centrality ─────────────────────────────
top_close = metrics.sort_values('closeness', ascending=False).head(20)

fig, ax = plt.subplots(figsize=(11, 6))
colors_bar = ['#E50914' if i < 5 else '#B0B0B0' for i in range(len(top_close))]
ax.barh(top_close['actor'][::-1], top_close['closeness'][::-1], color = colors_bar[::-1], edgecolor = 'white', linewidth = 0.4)

ax.set_xlabel('Closeness Centrality', fontsize=10)
ax.set_title('Top 20 Nodes by Closeness Centrality\n'
             '(inverse average distance to all other nodes)', fontsize = 12, fontweight = 'bold')
ax.spines[['top', 'right']].set_visible(False)
plt.tight_layout()
plt.savefig('closeness_centrality.png', dpi=150, bbox_inches='tight')
plt.show()

png

Centrality Correlation Analysis

The three centrality measures are related but distinct. Plotting degree against betweenness reveals an interesting structure:

  • Most nodes cluster in the bottom-left (low degree, low betweenness) — these are peripheral actors with few collaborations.
  • A few nodes extend along the degree axis — highly connected but not necessarily bridges.
  • The most interesting outliers appear high on the betweenness axis relative to their degree — these are the true bridge-builders, connecting communities despite modest raw collaboration counts.
# ── Betweenness vs Degree scatter ───────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Scatter: betweenness vs degree_centrality
sc = axes[0].scatter(
    metrics['degree_centrality'],
    metrics['betweenness'],
    c = metrics['closeness'], cmap = 'RdYlGn',
    alpha = 0.4, s = 8, linewidths = 0)
plt.colorbar(sc, ax=axes[0], label='Closeness centrality')

# Label top-5 betweenness nodes
top5_b = metrics.nlargest(5, 'betweenness')

for _, row in top5_b.iterrows():

    axes[0].annotate(row['actor'],
                     (row['degree_centrality'], row['betweenness']),
                     fontsize=7, xytext=(4, 4), textcoords='offset points', color='#E50914')

axes[0].set_xlabel('Degree Centrality')
axes[0].set_ylabel('Betweenness Centrality')
axes[0].set_title('Betweenness vs. Degree\n(colour = closeness)', fontweight='bold')
axes[0].spines[['top', 'right']].set_visible(False)

# Scatter: closeness vs degree_centrality
sc2 = axes[1].scatter(
    metrics['degree_centrality'],
    metrics['closeness'],
    c=metrics['betweenness'], cmap='plasma',
    alpha=0.4, s=8, linewidths=0)
plt.colorbar(sc2, ax=axes[1], label='Betweenness centrality')

top5_c = metrics.nlargest(5, 'closeness')

for _, row in top5_c.iterrows():

    axes[1].annotate(row['actor'],
                     (row['degree_centrality'], row['closeness']),
                     fontsize=7, xytext=(4, 4), textcoords='offset points', color='#0070e0')

axes[1].set_xlabel('Degree Centrality')
axes[1].set_ylabel('Closeness Centrality')
axes[1].set_title('Closeness vs. Degree\n(colour = betweenness)', fontweight='bold')
axes[1].spines[['top', 'right']].set_visible(False)

plt.suptitle('Centrality Correlation Analysis — Filtered Graph', fontsize=13, fontweight='bold', y=1.02)
plt.tight_layout()
plt.savefig('centrality_scatter.png', dpi=150, bbox_inches='tight')
plt.show()

print('\nPearson correlations between centrality measures:')
print(metrics[['degree_centrality', 'betweenness', 'closeness']].corr().round(4))

png

Pearson correlations between centrality measures:
                   degree_centrality  betweenness  closeness
degree_centrality             1.0000       0.3606     0.1301
betweenness                   0.3606       1.0000     0.2464
closeness                     0.1301       0.2464     1.0000

Centrality Summary Table

The table below consolidates the top-10 actors/directors across all three centrality metrics and adds a composite rank computed as the mean of their three individual ranks. This gives a holistic view of who dominates the network by multiple measures simultaneously.

# Compute metrics
metrics['rank_degree']    = metrics['degree_centrality'].rank(ascending = False)
metrics['rank_between']   = metrics['betweenness'].rank(ascending = False)
metrics['rank_close']     = metrics['closeness'].rank(ascending = False)
metrics['composite_rank'] = (metrics['rank_degree'] + metrics['rank_between'] + metrics['rank_close']) / 3

# Create a leaderboard of composite scores to determine aggregate ranking
leaderboard = (metrics.sort_values('composite_rank').head(15)[['actor', 'degree', 'degree_centrality', 'betweenness', 'closeness', 'composite_rank']].reset_index(drop = True))
leaderboard.index += 1
leaderboard.columns = ['Name', 'Degree', 'Degree Centrality', 'Betweenness', 'Closeness', 'Composite Rank']

leaderboard['Degree Centrality'] = leaderboard['Degree Centrality'].round(5)
leaderboard['Betweenness']       = leaderboard['Betweenness'].round(5)
leaderboard['Closeness']         = leaderboard['Closeness'].round(5)
leaderboard['Composite Rank']    = leaderboard['Composite Rank'].round(1)

print('Top 15 Individuals Based on Composite Rank')
leaderboard.sort_values('Composite Rank', ascending = False)

Top 15 Individuals Based on Composite Rank
Name Degree Degree Centrality Betweenness Closeness Composite Rank
15 Scott Aukerman 40 0.00753 0.00309 0.07260 61.7
14 John C Reilly 17 0.00320 0.01250 0.07092 55.7
13 Alan Cumming 17 0.00320 0.01208 0.07139 54.7
12 Samuel L Jackson 19 0.00358 0.01132 0.06992 53.2
11 Sarah Smith 43 0.00809 0.01931 0.06619 44.8
10 Amy Poehler 24 0.00452 0.00860 0.06945 43.2
9 Jada Pinkett Smith 19 0.00358 0.02156 0.07146 38.8
8 Cedric the Entertainer 19 0.00358 0.03087 0.07693 30.2
7 Lucy Liu 21 0.00395 0.03098 0.07230 27.7
6 Adam Sandler 30 0.00565 0.00866 0.07453 23.5
5 George Clooney 30 0.00565 0.01338 0.07367 19.2
4 Fred Armisen 24 0.00452 0.02303 0.07646 17.5
3 Akiva Schaffer 51 0.00960 0.01327 0.07300 12.7
2 Martin Lawrence 31 0.00584 0.04851 0.07424 9.7
1 Elizabeth Banks 40 0.00753 0.03103 0.07637 5.3

Back to top..

Community Detection

Community detection attempts to partition a network into groups of nodes that are more densely connected internally than to the rest of the network. In the context of Netflix collaborations, communities may correspond to:

  • National or regional film industries (Bollywood, Korean cinema, etc.)
  • Genre-specific production circles (horror, animation, documentary)
  • Long-running director–ensemble partnerships

The Louvain Algorithm

We use the Louvain method (python-louvain), a hierarchical modularity-maximisation algorithm that scales to millions of nodes. It works in two phases that are iterated until convergence:

  1. Local optimisation — each node is assigned to the community of its neighbour that yields the largest modularity gain.
  2. Network aggregation — communities become super-nodes and the process repeats on the compressed graph.

The output is a partition (dictionary mapping each node to a community ID) and a modularity score ∈ (-1, 1) where higher values indicate stronger community structure.

# Louvain requires a connected graph, so we work on the GCC
gcc_nodes = max(nx.connected_components(filtered_graph), key = len)
gcc = filtered_graph.subgraph(gcc_nodes).copy()

print(f'Largest Connected Component: {gcc.number_of_nodes():,} nodes, {gcc.number_of_edges():,} edges')
print('Running Louvain community detection...')

start = time.time()
partition = louvain_community.best_partition(gcc, weight = 'weight', random_state = 100)
modularity = louvain_community.modularity(partition, gcc, weight='weight')
n_communities = len(set(partition.values()))
end = time.time()

print(f'\nCode Execution Completed in {round(end-start)} seconds')
print(f'Number of communities detected: {n_communities}')
print(f'Modularity score:               {modularity:.4f}')

Largest Connected Component: 2,319 nodes, 4,946 edges
Running Louvain community detection...

Code Execution Completed in 0 seconds
Number of communities detected: 47
Modularity score:               0.9246

Community Size Distribution

Like many real-world networks, collaboration communities follow a skewed distribution — a few large communities contain the bulk of the nodes (likely corresponding to major production ecosystems), while many small communities represent niche or localised collaboration circles.

# Get the size of each community
comm_sizes = collections.Counter(partition.values())
size_series = pd.Series(comm_sizes).sort_values(ascending=False)

print(f'Total communities:    {n_communities}')
print(f'Largest community:    {size_series.iloc[0]:,} nodes')
print(f'Median size:          {int(size_series.median())} nodes')
print(f'Communities with 1 node: {(size_series == 1).sum()}')
print(f'Communities with >50 nodes: {(size_series > 50).sum()}')

# Visalize
fig, axes = plt.subplots(1, 2, figsize=(13, 4))

# Top 30 community sizes
top_comms = size_series.head(30)
palette = make_hex_palette(len(top_comms))
axes[0].bar(range(1, len(top_comms)+1), top_comms.values,
            color=palette, edgecolor='white', linewidth=0.3)
axes[0].set_title('Top 30 Community Sizes', fontweight='bold')
axes[0].set_xlabel('Community rank')
axes[0].set_ylabel('Number of nodes')
axes[0].spines[['top', 'right']].set_visible(False)
for i, v in enumerate(top_comms.values[:5]):
    axes[0].text(i+1, v+5, str(v), ha='center', fontsize=8, fontweight='bold')

# Histogram of all sizes
axes[1].hist(size_series.values, bins=40, color='#E50914', edgecolor='white', linewidth=0.3)
axes[1].set_yscale('log')
axes[1].set_title('Community Size Histogram (log scale)', fontweight='bold')
axes[1].set_xlabel('Community size (nodes)')
axes[1].set_ylabel('Count (log)')
axes[1].spines[['top', 'right']].set_visible(False)

plt.tight_layout()
plt.savefig('community_sizes.png', dpi=150, bbox_inches='tight')
plt.show()

Total communities:    47
Largest community:    194 nodes
Median size:          38 nodes
Communities with 1 node: 0
Communities with >50 nodes: 18

png

Top Community Members

For each of the largest communities we list the top members by degree centrality within that community. This helps identify who the central figures of each collaboration circle are.

# Add community labels to metrics DataFrame (GCC nodes only)
metrics['community'] = metrics['actor'].map(partition).fillna(-1).astype(int)

# Work only with GCC members
metrics_gcc = metrics[metrics['community'] >= 0].copy()

# Top community IDs by size
top_comm_ids = size_series.head(10).index.tolist()

print(f'Top 10 Communities — Central Members (by degree)\n')

for rank, cid in enumerate(top_comm_ids, start=1):

    members = metrics_gcc[metrics_gcc['community'] == cid].nlargest(5, 'degree')
    names   = ', '.join(members['actor'].tolist())
    size    = size_series[cid]
    avg_deg = metrics_gcc[metrics_gcc['community'] == cid]['degree'].mean()
    print(f'  Community {rank} (id={cid}) | size={size} | avg_degree={avg_deg:.2f}')
    print(f'    Top members: {names}\n')

Top 10 Communities — Central Members (by degree)

  Community 1 (id=0) | size=194 | avg_degree=4.14
    Top members: Omoni Oboli, Kunle Afolayan, Toyin Abraham, Funke Akindele, Blossom Chukwujekwu

  Community 2 (id=17) | size=165 | avg_degree=4.43
    Top members: Akiva Schaffer, Scott Aukerman, Adam Sandler, Fred Armisen, Alan Cumming

  Community 3 (id=5) | size=108 | avg_degree=4.20
    Top members: Yılmaz Erdoğan, Cem Yılmaz, Demet Akbağ, Sermiyan Midyat, Tarık Ünlüoğlu

  Community 4 (id=16) | size=96 | avg_degree=3.71
    Top members: John Musker, Fred Tatasciore, Paul Reubens, Audrey Wasilewski, Frank Welker

  Community 5 (id=35) | size=96 | avg_degree=2.34
    Top members: Clint Eastwood, Mike Birbiglia, Alejandro Agresti, Charles Martin Smith, Bruce Willis

  Community 6 (id=18) | size=95 | avg_degree=4.98
    Top members: Kevin Smith, Cedric the Entertainer, Jada Pinkett Smith, Keanu Reeves, Keenen Ivory Wayans

  Community 7 (id=6) | size=93 | avg_degree=2.80
    Top members: Taika Waititi, Andy Serkis, Jon Favreau, Alan Rickman, Jackie van Beek

  Community 8 (id=8) | size=78 | avg_degree=3.47
    Top members: Elizabeth Banks, Liev Schreiber, Jay Baruchel, Seann William Scott, Alison Pill

  Community 9 (id=10) | size=77 | avg_degree=4.04
    Top members: James Franco, William H Macy, John C Reilly, Will Ferrell, Molly Shannon

  Community 10 (id=25) | size=71 | avg_degree=8.55
    Top members: Erin Fitzgerald, Kate Higgins, Laura Bailey, Debi Derryberry, Audu Paden

# Visalize the top 15 communities
top_k_ids  = set(size_series.head(15).index.tolist())
comm_label = {node: (cid if cid in top_k_ids else 15) for node, cid in partition.items()}

# Build node/edge DataFrames for the GCC
gcc_node_list = list(gcc.nodes())
gcc_edge_list = list(gcc.edges())

node_idx  = {n: i for i, n in enumerate(gcc_node_list)}
gcc_nodes_df = pd.DataFrame({'name': gcc_node_list})
gcc_nodes_df['community_label'] = gcc_nodes_df['name'].map(comm_label).fillna(15).astype(int)

gcc_edges_df = pd.DataFrame([(node_idx[u], node_idx[v]) for u, v in gcc_edge_list if u in node_idx and v in node_idx], columns = ['source', 'target'])

print(f'Running ForceAtlas2 layout on {len(gcc_node_list):,} nodes...')
start = time.time()
layout_df = forceatlas2_layout(gcc_nodes_df, gcc_edges_df, iterations = 200, seed = 100)
layout_df = normalise_layout(layout_df)
gcc_nodes_df['x'] = layout_df['x'].values
gcc_nodes_df['y'] = layout_df['y'].values
end = time.time()
print(f'Layout computed in {round(end-start)} seconds')

Running ForceAtlas2 layout on 2,319 nodes...
Layout computed in 42 seconds

# Build edge line segments DataFrame for datashader
node_pos = gcc_nodes_df.set_index('name')[['x', 'y']].to_dict(orient = 'index')

edge_segs_x, edge_segs_y = [], []

# Iterate
for u, v in gcc_edge_list:

    if u in node_pos and v in node_pos:

        edge_segs_x += [node_pos[u]['x'], node_pos[v]['x'], np.nan]
        edge_segs_y += [node_pos[u]['y'], node_pos[v]['y'], np.nan]

edge_lines = pd.DataFrame({'x': edge_segs_x, 'y': edge_segs_y})

# Colour palette: one colour per top-K community + grey for 'other'
palette = make_hex_palette(15) + ['#444444']
gcc_nodes_df['community_str'] = gcc_nodes_df['community_label'].astype(str).astype('category')

eps = 1e-6
xr  = (float(gcc_nodes_df.x.min())-eps, float(gcc_nodes_df.x.max())+eps)
yr  = (float(gcc_nodes_df.y.min())-eps, float(gcc_nodes_df.y.max())+eps)

# Compile data for visualization input
canvas = ds.Canvas(x_range=xr, y_range=yr, **cvsopts)

img_edges = tf.shade(canvas.line(edge_lines, 'x', 'y', agg=ds.count()), cmap = ['#1a1a2e', '#444477'], name = 'edges')
img_nodes = nodesplot(gcc_nodes_df, name = 'communities', canvas = canvas, cat = 'community_str')
img_final = tf.stack(img_edges, img_nodes, how = 'over', name = 'Community Graph')

# Display
show_ds_images([img_final], [f'Netflix Collaboration Networks — Top 15 Largest Communities Coloured'], ncols = 1, fig_width = 14, cell_px = 1000)

png

Intra- vs. Inter-Community Edge Analysis

A good community partition should have most edges within communities (intra) and few edges between communities (inter). The ratio of intra-community edges to total edges is a direct proxy for partition quality.

# ── Intra vs inter community edge count ─────────────────────────
intra, inter = 0, 0
for u, v in gcc.edges():

    if partition.get(u) == partition.get(v):

        intra += 1

    else:

        inter += 1

total = intra + inter
print(f'Intra-community edges: {intra:,}  ({100*intra/total:.1f}%)')
print(f'Inter-community edges: {inter:,}  ({100*inter/total:.1f}%)')
print(f'Modularity:            {modularity:.4f}')

fig, ax = plt.subplots(figsize=(5, 5))
wedges, texts, autotexts = ax.pie(
    [intra, inter],
    labels=['Intra-community', 'Inter-community'],
    autopct='%1.1f%%',
    colors=['#E50914', '#888888'],
    startangle=90,
    wedgeprops={'edgecolor': 'white', 'linewidth': 2})
for t in autotexts:
    t.set_fontsize(12)
    t.set_fontweight('bold')
ax.set_title(f'Edge Distribution — Modularity = {modularity:.3f}', fontweight='bold', fontsize=12)
plt.tight_layout()
plt.savefig('community_edges_pie.png', dpi=150, bbox_inches='tight')
plt.show()

Intra-community edges: 4,726  (95.6%)
Inter-community edges: 220  (4.4%)
Modularity:            0.9246

png

Back to top..

Collaboration Patterns

Beyond individual metrics and community structure, we can examine the distributional patterns of collaboration across the network. Three key phenomena are worth investigating:

  1. Degree distribution — does the network follow a power law (scale-free) or a more Poisson-like distribution?
  2. Edge weight distribution — how often do the same pairs of people collaborate?
  3. Clustering coefficient distribution — how cliquish are different parts of the network?
  4. Most recurring collaborations — which actor/director pairs have the highest shared title count?

Degree Distribution — Is This a Scale-Free Network?

Many real-world social networks are scale-free: their degree distribution follows a power law P(k) ~ k^(-γ), where a small number of highly connected hubs co-exist with a long tail of low-degree nodes. Plotted on log-log axes, a power law appears as a straight line.

Scale-free networks arise naturally through preferential attachment — new entrants to the industry are more likely to collaborate with already-prominent figures, amplifying existing advantages.

# ── Degree distribution ─────────────────────────────────────────
degrees_all      = sorted([d for _, d in graph.degree()], reverse=True)
degrees_filtered = sorted([d for _, d in filtered_graph.degree()], reverse=True)

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

for ax, degs, label, color in [
        (axes[0], degrees_all,      'Full graph (≥1)',     '#E50914'),
        (axes[1], degrees_filtered, 'Filtered graph (≥2)', '#333333')]:

    counts = collections.Counter(degs)
    ks  = np.array(sorted(counts.keys()))
    pks = np.array([counts[k] for k in ks])

    ax.loglog(ks, pks, 'o', markersize=3, alpha=0.6, color=color, label='Empirical')

    # Fit line through log-log for power-law appearance check
    log_k  = np.log10(ks[ks > 0])
    log_pk = np.log10(pks[ks > 0])
    coeffs = np.polyfit(log_k, log_pk, 1)
    fit_y  = np.polyval(coeffs, log_k)
    ax.loglog(10**log_k, 10**fit_y, '--', color='black', linewidth=1.2,
              label=f'Power-law fit  γ≈{-coeffs[0]:.2f}')

    ax.set_title(f'Degree Distribution — {label}', fontweight='bold')
    ax.set_xlabel('Degree k  (log)')
    ax.set_ylabel('P(k)  (log)')
    ax.legend(fontsize=9)
    ax.spines[['top', 'right']].set_visible(False)

plt.tight_layout()
plt.savefig('degree_distribution.png', dpi=150, bbox_inches='tight')
plt.show()

print(f'Full graph   — max degree: {max(degrees_all)},   mean: {np.mean(degrees_all):.2f},   median: {int(np.median(degrees_all))}')
print(f'Filtered     — max degree: {max(degrees_filtered)}, mean: {np.mean(degrees_filtered):.2f}, median: {int(np.median(degrees_filtered))}')

png

Full graph   — max degree: 295,   mean: 15.49,   median: 10
Filtered     — max degree: 70, mean: 4.03, median: 2

Edge Weight Distribution

The edge weight records how many Netflix titles two people co-appeared on. Most collaborations are one-off events, while a small number of pairs have worked together on many productions — long-standing director–actor partnerships or franchise ensembles.

# ── Edge weight distribution ─────────────────────────────────────
all_weights      = weighted_people_edges['weight'].values
filtered_weights = filtered_people_edges['weight'].values

fig, axes = plt.subplots(1, 2, figsize=(13, 4))

# All edges
axes[0].hist(all_weights, bins=range(1, 25), color='#E50914', edgecolor='white', linewidth=0.4, align='left')
axes[0].set_yscale('log')
axes[0].set_title('Edge Weight Distribution — All Edges', fontweight='bold')
axes[0].set_xlabel('Collaboration count (shared titles)')
axes[0].set_ylabel('Number of pairs (log)')
axes[0].spines[['top', 'right']].set_visible(False)

# Filtered edges (weight ≥ 2)
axes[1].hist(filtered_weights, bins=range(2, int(filtered_weights.max())+2),
             color='#333333', edgecolor='white', linewidth=0.4, align='left')
axes[1].set_yscale('log')
axes[1].set_title('Edge Weight Distribution — Filtered Edges (≥2)', fontweight='bold')
axes[1].set_xlabel('Collaboration count (shared titles)')
axes[1].set_ylabel('Number of pairs (log)')
axes[1].spines[['top', 'right']].set_visible(False)

plt.tight_layout()
plt.savefig('edge_weight_dist.png', dpi=150, bbox_inches='tight')
plt.show()

print(f'Weight stats (all edges):      max={all_weights.max()},  mean={all_weights.mean():.3f},  median={np.median(all_weights)}')
print(f'Weight stats (filtered edges): max={filtered_weights.max()}, mean={filtered_weights.mean():.3f}, median={np.median(filtered_weights)}')

png

Weight stats (all edges):      max=27,  mean=1.062,  median=1.0
Weight stats (filtered edges): max=27, mean=2.375, median=2.0

Top Recurring Collaborations

The pairs with the highest edge weights represent the most persistent professional partnerships in the Netflix catalogue — recurring director–actor relationships and long-running ensemble casts.

# ── Most frequent collaborating pairs ───────────────────────────
top_pairs = (weighted_people_edges
             .sort_values('weight', ascending=False)
             .head(25)
             .reset_index(drop=True))
top_pairs.index += 1
top_pairs['pair'] = top_pairs['source'] + '  ↔  ' + top_pairs['target']

fig, ax = plt.subplots(figsize=(11, 7))
bars = ax.barh(top_pairs['pair'][::-1], top_pairs['weight'][::-1], color = ['#E50914' if w >= 10 else '#B0B0B0' for w in top_pairs['weight'][::-1]], edgecolor = 'white', linewidth = 0.4)

for bar, w in zip(bars, top_pairs['weight'][::-1]):

    ax.text(bar.get_width() + 0.05, bar.get_y() + bar.get_height()/2, f'  {w} titles', va = 'center', fontsize = 8)

ax.set_xlabel('Number of shared titles')
ax.set_title('Top 25 Most Frequent Collaborations', fontsize=12, fontweight='bold')
ax.spines[['top', 'right']].set_visible(False)
ax.set_xlim(0, top_pairs['weight'].max() * 1.25)
plt.tight_layout()
plt.savefig('top_collaborations.png', dpi=150, bbox_inches='tight')
plt.show()

png

Clustering Coefficient Distribution

The local clustering coefficient of a node measures how many of its neighbours are also connected to each other, expressed as a fraction of the maximum possible connections among them. A value of 1.0 means the node’s neighbourhood forms a perfect clique.

In a film production network, high clustering indicates that an actor tends to work in tight-knit ensembles where cast members repeatedly work together; low clustering indicates someone who collaborates broadly across many different productions without bringing those collaborators together.

# ── Clustering coefficient distribution ─────────────────────────
clustering_coeffs = nx.clustering(filtered_graph)
cc_values = list(clustering_coeffs.values())

fig, axes = plt.subplots(1, 2, figsize=(13, 4))

# Histogram
axes[0].hist(cc_values, bins=30, color='#E50914', edgecolor='white', linewidth=0.3)
axes[0].set_title('Clustering Coefficient Distribution', fontweight='bold')
axes[0].set_xlabel('Local clustering coefficient')
axes[0].set_ylabel('Number of nodes')
axes[0].axvline(np.mean(cc_values), color='black', linestyle='--', linewidth=1.5,
                label=f'Mean = {np.mean(cc_values):.3f}')
axes[0].legend()
axes[0].spines[['top', 'right']].set_visible(False)

# Clustering vs Degree scatter
degrees_dict = dict(filtered_graph.degree())
cc_df = pd.DataFrame({'degree': [degrees_dict[n] for n in clustering_coeffs],
                       'cc':     [clustering_coeffs[n] for n in clustering_coeffs]})
# Bin by degree for a cleaner trend line
cc_df['deg_bin'] = pd.cut(cc_df['degree'], bins=20)
binned = cc_df.groupby('deg_bin')['cc'].mean().reset_index()
bin_mids = [iv.mid for iv in binned['deg_bin']]

axes[1].scatter(cc_df['degree'], cc_df['cc'], alpha=0.15, s=6, color='#888888')
axes[1].plot(bin_mids, binned['cc'], color='#E50914', linewidth=2, label='Binned mean')
axes[1].set_title('Clustering vs. Degree', fontweight='bold')
axes[1].set_xlabel('Node degree')
axes[1].set_ylabel('Clustering coefficient')
axes[1].legend()
axes[1].spines[['top', 'right']].set_visible(False)

plt.tight_layout()
plt.savefig('clustering_dist.png', dpi=150, bbox_inches='tight')
plt.show()

print(f'Mean clustering coefficient:   {np.mean(cc_values):.4f}')
print(f'Fraction of isolates (cc=0):   {sum(1 for v in cc_values if v == 0) / len(cc_values):.3f}')
print(f'Fraction with cc=1.0:          {sum(1 for v in cc_values if v == 1.0) / len(cc_values):.3f}')

/tmp/ipykernel_462/1781242898.py:23: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  binned = cc_df.groupby('deg_bin')['cc'].mean().reset_index()

png

Mean clustering coefficient:   0.3457
Fraction of isolates (cc=0):   0.548
Fraction with cc=1.0:          0.243

Back to top..

Ego Networks

An ego network (or 1-hop neighbourhood) is the subgraph consisting of a focal node (the ego), all its direct neighbours (alters), and all edges among those alters. It provides a localised view of a single individual’s collaboration environment.

Ego network analysis reveals:

  • How many direct collaborators does this person have?
  • How interconnected are those collaborators (do they also work with each other)?
  • Are there structural holes — gaps in the ego’s network where they are the only bridge between two otherwise unconnected groups?

We will examine ego networks for several notable individuals selected on the basis of their centrality scores.

Selecting Ego Network Subjects

We choose subjects from the top of each centrality ranking to showcase the different structural profiles an ego network can take:

  • Top degree — will show a dense, well-connected neighbourhood
  • Top betweenness — may show a sparser neighbourhood that bridges two otherwise separate clusters
  • Top closeness — will be centrally embedded in the large component
# Get values for analysis
ego_subjects = {'Top Degree': metrics.sort_values('degree', ascending = False).iloc[0]['actor'],
                'Top Betweenness': metrics.sort_values('betweenness', ascending = False).iloc[0]['actor'],
                'Top Closeness': metrics.sort_values('closeness', ascending = False).iloc[0]['actor'],
                '2nd Betweenness': metrics.sort_values('betweenness', ascending = False).iloc[1]['actor'],}

# Show subjects for analysis
print('Selected ego subjects:')
for role, name in ego_subjects.items():

    row = metrics[metrics['actor'] == name].iloc[0]
    print(f'  {role:<20} {name:<30}  degree = {int(row.degree)}',  f'betweenness = {row.betweenness:.5f},  closeness = {row.closeness:.5f}')

Selected ego subjects:
  Top Degree           Anupam Kher                     degree = 70 betweenness = 0.00439,  closeness = 0.04533
  Top Betweenness      Martin Lawrence                 degree = 31 betweenness = 0.04851,  closeness = 0.07424
  Top Closeness        Cedric the Entertainer          degree = 19 betweenness = 0.03087,  closeness = 0.07693
  2nd Betweenness      Miguel A Núñez Jr               degree = 3 betweenness = 0.03806,  closeness = 0.06536

Ego Network Profiles

Each panel below shows one ego network. The central red node is the ego; all other nodes are direct collaborators. Edge thickness scales with collaboration weight.

# ── Plot ego networks side by side ──────────────────────────────
fig, axes = plt.subplots(2, 2, figsize=(16, 14), facecolor='#1a1a2e')
axes = axes.flatten()

for ax in axes:

    ax.set_facecolor('#1a1a2e')

for i, (role, name) in enumerate(ego_subjects.items()):

    if name in filtered_graph:

        ego_g = get_ego_network(filtered_graph, name)
        n_alters = ego_g.number_of_nodes() - 1
        plot_ego_network(filtered_graph, name, axes[i], title = f'{role}: {name}\n({n_alters} direct collaborators)')

    else:

        axes[i].text(0.5, 0.5, f'{name} not in\nfiltered graph', ha = 'center', va = 'center', fontsize = 10, color = 'white')
        axes[i].axis('off')

plt.suptitle('Ego Networks — Netflix Collaboration Graph', fontsize = 15, fontweight = 'bold', color = 'white', y=1.01)
plt.tight_layout()
plt.savefig('ego_networks.png', dpi = 150, bbox_inches = 'tight', facecolor = '#1a1a2e')
plt.show()

png

# ── Summarise ego stats for all four subjects ───────────────────
print('Summary Ego Network Statistics\n' + '='*60)
ego_stats_records = []

for role, name in ego_subjects.items():

    if name not in filtered_graph:

        continue

    ego = get_ego_network(filtered_graph, name)
    n_alters   = ego.number_of_nodes() - 1
    n_edges    = ego.number_of_edges()
    max_edges  = n_alters*(n_alters-1)//2 if n_alters > 1 else 1
    alt_density= n_edges / max_edges
    cc         = nx.clustering(ego, name)
    total_w    = sum(d.get('weight', 1) for _, _, d in ego.edges(data=True))

    ego_stats_records.append({
        'Role': role, 'Name': name,
        'Alters': n_alters,
        'Ego-net edges': n_edges,
        'Alter density': round(alt_density, 4),
        'Local clustering': round(cc, 4),
        'Total collab. weight': total_w,
    })

ego_summary = pd.DataFrame(ego_stats_records).set_index('Role')
ego_summary

Summary Ego Network Statistics
============================================================
Name Alters Ego-net edges Alter density Local clustering Total collab. weight
Role
Top Degree Anupam Kher 70 222 0.0919 0.0629 545
Top Betweenness Martin Lawrence 31 53 0.1140 0.0473 111
Top Closeness Cedric the Entertainer 19 59 0.3450 0.2339 129
2nd Betweenness Miguel A Núñez Jr 3 4 1.3333 0.3333 8 
# Allow the user to type any name and visualise their ego network on the fly.
default_names = list(ego_subjects.values())

@interact
def explore_ego(name=widgets.Combobox(value = default_names[0],
                                      options = list(metrics.sort_values('degree', ascending = False)['actor'].head(100)),
                                      description = 'Person:',
                                      ensure_option = False,
                                      layout = widgets.Layout(width = '350px'))):

    if name not in filtered_graph:

        print(f'⚠  {name!r} not found in the filtered graph.')
        return

    ego = get_ego_network(filtered_graph, name)
    fig, ax = plt.subplots(1, 1, figsize=(9, 9), facecolor='#1a1a2e')
    ax.set_facecolor('#1a1a2e')
    plot_ego_network(filtered_graph, name, ax)

    # Overlay top-5 alter names
    pos = nx.spring_layout(ego, seed=42, k=0.7)
    top_alters = sorted([n for n in ego.nodes() if n != name], key = lambda n: ego.degree(n), reverse = True)[:8]
    nx.draw_networkx_labels(ego, pos, ax=ax, labels = {n: n for n in top_alters}, font_size = 7, font_color = '#dddddd')

    plt.tight_layout()
    plt.show()

    row = metrics[metrics['actor'] == name]

    if not row.empty:

        r = row.iloc[0]
        print(f'  Degree centrality: {r.degree_centrality:.5f}  |  Betweenness: {r.betweenness:.5f}  |  Closeness: {r.closeness:.5f}')

interactive(children=(Combobox(value='Anupam Kher', description='Person:', layout=Layout(width='350px'), optio…

Summary & Key Findings

This notebook applied a comprehensive suite of network analysis tools to the Netflix collaboration graph.

The main findings are:

Global Structure

The raw collaboration network (~~30,800 nodes, ~~238,500 edges) is sparse (density ≈ 0.0005) with a very high clustering coefficient (~0.82), consistent with cast members of a single film forming dense local cliques. One giant component dominates, connecting ~90% of all nodes through a chain of shared productions.

Centrality

The most connected individuals (by degree) are prolific genre-crossing actors who appear in many different productions. The highest betweenness nodes are often directors or actors who bridge distinct regional or genre-based clusters — acting as cross-industry connectors.

Community Structure

Louvain community detection reveals a clear modular structure (modularity > 0.4) with communities likely corresponding to national film industries, recurring franchise ensembles, and genre-specific production circles.

Collaboration Patterns

Degree distributions show scale-free behaviour (power-law-like), consistent with preferential attachment: already prominent actors attract more new collaborations. Most collaborations (~80%+) are one-off events; the top recurring pairs tend to be director–muse relationships or ensemble franchise casts.

Ego Networks

High-degree egos show dense, well-connected neighbourhoods (high alter density), while high-betweenness egos tend to have sparse neighbourhoods that span different parts of the graph — textbook structural holes in the social capital sense.

Dataset: Netflix Titles — Kaggle (Shivamb)
Libraries: NetworkX · python-louvain · Datashader · HoloViews · Matplotlib · Plotly

Back to top..

]]>Data Guydataguy50@gmail.comGraphing Netflix Social Networks in Python2026-03-15T00:00:00+00:002026-03-15T00:00:00+00:00https://doctorofdata.github.io/posts/2026/03/netflix-network-mappingIn this post, we use a dataset posted on Kaggle as fodder for creating a vast network of communities of actors and directors spread around the world!

🎬 Mapping the Netflix Cinematic Graph

A Network Science Exploration of Film Relationships

📦 Dataset: Netflix Titles — Kaggle (Shivamb)

Table of Contents

🧠 Core Idea

Movies and TV shows are not isolated objects — they exist inside a rich cultural ecosystem woven together by shared creative talent, genre conventions, and release timing.

By treating Netflix titles as nodes in a graph and drawing edges between any two titles that share an attribute, we can reveal hidden structure in the catalogue:

Signal What it captures
🎭 Shared cast member Actor collaboration network
🎬 Shared director Directorial fingerprint across films
🎨 Shared genre tag Thematic neighbourhood
📅 Release proximity Era-based cultural clusters

When thousands of such connections are assembled, an emergent network appears — exposing clusters, bridges, and high-influence nodes that simple list-based exploration would never reveal.

🗺️ What This Notebook Builds

The notebook progresses through four logical stages:

1. Environment setup & helper functions
       ↓
2. Prototype graph — synthetic data to validate the pipeline
       ↓
3. Netflix people network — real cast/director co-occurrence graph
       ↓
4. Interactive visualisation — force-directed + edge-bundled rendering

By the end you will have:

  • 🧬 A weighted co-occurrence graph of Netflix cast & directors
  • 🌌 Four layout variants (circular · force-directed × plain · bundled)
  • 🎛️ A HoloViews/Datashader interactive visualisation ready for exploration
### 🔑 Key Graph Theory Concepts Used **Node** — a single entity (here: a person or film title) **Edge** — a relationship between two nodes (co-appearance, shared genre…) **Edge weight** — how *many times* two nodes share a connection (more collaborations → thicker edge) **Force-directed layout** — a physics simulation that pulls connected nodes together and pushes unconnected ones apart, naturally surfacing clusters **Edge bundling** — groups nearby parallel edges into smooth curves, reducing visual clutter while preserving global structure

⚙️ Section 1 — Environment Setup

Before anything else we install and import the key libraries. Here is a brief overview of each major dependency:

Library Role in this notebook
networkx Graph data structures and algorithms
datashader Rasterise millions of points/edges into images efficiently
holoviews High-level declarative visualisation layer on top of Datashader/Bokeh
datashader.layout forceatlas2_layout — the ForceAtlas2 algorithm for node positioning
datashader.bundling hammer_bundle — Hammer edge bundling for cleaner edge rendering
colorcet Perceptually uniform colormaps
gensim Word2Vec / node embedding utilities (for optional embedding extension)
dash Serve interactive Bokeh/Plotly apps from a notebook

💡 Tip: If you are running this outside Colab, you can install all dependencies with:
pip install colorcet dash gensim datashader "holoviews[recommended]" jupyter_bokeh bokeh

!pip install --upgrade colorcet dash gensim datashader "holoviews[recommended]" jupyter_bokeh bokeh python-louvain -q

📦 Library Imports

We import everything we need up-front so that missing dependencies surface immediately.
A few things worth noting:

  • hv.extension("bokeh") tells HoloViews to use Bokeh as its rendering backend — this enables interactive plots with pan, zoom, and hover tooltips.
  • forceatlas2_layout, circular_layout, and random_layout are the three node-positioning algorithms we will compare.
  • connect_edges and hammer_bundle both convert an abstract edge list into drawable line segments, but hammer_bundle additionally routes edges along smooth shared paths.
  • drive.mount(...) connects Google Colab to Google Drive so we can load the Netflix CSV.
# Import libraries
import pandas as pd
import collections
import networkx as nx
import colorcet as cc
import plotly.graph_objects as go
from dash import Dash, dcc, html, Input, Output
import random
from IPython.display import IFrame
from holoviews.operation.datashader import (datashade, aggregate, dynspread, bundle_graph, split_dataframe,  regrid)
from holoviews.element.graphs import layout_nodes
from datashader.layout import forceatlas2_layout, random_layout, circular_layout
import holoviews as hv
import datashader as ds
import datashader.transfer_functions as tf
from datashader.bundling import connect_edges, hammer_bundle
hv.extension("bokeh")
import numpy as np
#from google.colab import drive
#drive.mount('/content/drive')
import time
from holoviews import opts
import math
from scipy.interpolate import splprep, splev
from itertools import combinations
import string
translator = str.maketrans('', '', string.punctuation)
import community as louvain_community
from bokeh.io import output_notebook
output_notebook()
from matplotlib.colors import ListedColormap
import panel as pn
pn.extension()
import colorsys
import matplotlib.colors as mcolors
from typing import List
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from PIL import Image as PILImage
import gc
from IPython.display import display, Image
import io
</img> </img> ⓘ 
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

Back to top..

🛠️ Section 2 — Datashader Helper Functions

Datashader works by rasterising geometry into a fixed-resolution canvas rather than drawing individual SVG/canvas elements.
This is what makes it possible to render graphs with tens of thousands of edges without crashing the browser.

The three helpers below form a thin wrapper around the Datashader API:

Function Purpose
nodesplot(nodes, ...) Rasterise node point positions into a coloured image
edgesplot(edges, ...) Rasterise edge line segments into an image
graphplot(nodes, edges, ...) Stack edges under nodes into a composite image

cvsopts sets the default canvas resolution to 400 × 400 pixels for the prototype stage.
This will be increased for the final Netflix graph.

# Functions to work with datashader
cvsopts = dict(plot_height = 1000, plot_width = 1000)

def nodesplot(nodes, name = None, canvas = None, cat = None):

    if canvas is None:

        eps = 1e-6
        xr = (float(nodes.x.min()) - eps, float(nodes.x.max()) + eps)
        yr = (float(nodes.y.min()) - eps, float(nodes.y.max()) + eps)
        canvas = ds.Canvas(x_range = xr, y_range = yr, **cvsopts)

    # Strip cats before mapping
    if cat:

        nodes[cat] = nodes[cat].cat.remove_unused_categories()

    aggregator = None if cat is None else ds.count_cat(cat)
    agg = canvas.points(nodes,'x','y',aggregator)

    if cat:

        cats = list(nodes[cat].cat.categories)
        color_key = dict(zip(cats, make_hex_palette(len(cats))))

        return tf.dynspread(tf.shade(agg, color_key = color_key, name = name))

    return tf.dynspread(tf.shade(agg, cmap = ["#FF3333"], name = name))

def edgesplot(edges, name = None, canvas = None):

    if canvas is None:

        eps = 1e-6
        xr = (float(edges.x.min()) - eps, float(edges.x.max()) + eps)
        yr = (float(edges.y.min()) - eps, float(edges.y.max()) + eps)
        canvas = ds.Canvas(x_range = xr, y_range = yr, **cvsopts)

    return tf.shade(canvas.line(edges, 'x','y', agg = ds.count()), name = name)

def graphplot(nodes, edges, name = "", canvas = None, cat = None, pad = 0.025):

    if canvas is None:

        xmin, xmax = nodes.x.min(), nodes.x.max()
        ymin, ymax = nodes.y.min(), nodes.y.max()

        xpad = (xmax - xmin) * pad
        ypad = (ymax - ymin) * pad

        eps = 1e-6
        xr = (float(xmin - xpad) - eps, float(xmax + xpad) + eps)
        yr = (float(ymin - ypad) - eps, float(ymax + ypad) + eps)

        canvas = ds.Canvas(x_range = xr, y_range = yr, **cvsopts)

    nodeplot = nodesplot(nodes, name + " nodes", canvas, cat)
    edgeplot = edgesplot(edges, name + " edges", canvas)

    return tf.stack(edgeplot, nodeplot, how = "over", name = name)

# Function to create a colormap
def make_hex_palette(n):

    """High-contrast palette optimised for dark backgrounds."""
    result = []

    for i in range(n):

        h = i / n

        # lightness=0.65, saturation=0.95 — bright, vivid, dark-bg friendly
        r, g, b = colorsys.hls_to_rgb(h, 0.65, 0.95)
        result.append('#{:02x}{:02x}{:02x}'.format(int(r * 255), int(g * 255), int(b * 255)))

    return result

# Function to visualize colormap
def plot_colortable(hex_colors: List[str]):

    """Creates a colorbar using custom hex colors."""
    cmap = mcolors.ListedColormap(hex_colors)

    plt.figure(figsize = (8, 2), dpi = 150)
    plt.imshow([list(range(len(hex_colors)))], cmap = cmap, aspect = 'auto')
    plt.axis('off')
    plt.show()

def show_ds_images(images, titles, ncols = 2, fig_width = 18, cell_px = 1000, bg = 'black', title_color = 'white', title_size = 11):

    """
    Display a list of datashader images with styled titles and background.

    Parameters
    ----------
    images : list of datashader Image objects
    titles : list of str
    ncols  : number of columns in the grid
    fig_width : total figure width in inches
    cell_px : resolution to render each image (square)
    bg     : hex background colour applied to each image AND the figure
    """

    nrows = -(-len(images) // ncols)
    cell_in = fig_width / ncols
    fig, axes = plt.subplots(nrows, ncols, figsize = (fig_width, cell_in * nrows), facecolor = bg, dpi = 150)
    axes = list(np.atleast_1d(axes).flat)

    for ax, img, title in zip(axes, images, titles):

        # Apply background colour, then upscale with Lanczos for sharpness
        styled = tf.set_background(img, bg)
        pil_img = styled.to_pil()
        pil_img = pil_img.resize((cell_px, cell_px), PILImage.LANCZOS)
        ax.imshow(pil_img, interpolation = 'lanczos')
        ax.set_title(title, color = title_color, fontsize = title_size, fontweight = 'bold', pad = 12, loc = 'center')
        ax.set_facecolor(bg)
        ax.axis('off')

    # Hide any unused axes
    for ax in list(axes)[len(images):]:

        ax.set_visible(False)

    plt.subplots_adjust(wspace = 0.04, hspace = 0.12)
    buf = io.BytesIO()
    plt.savefig(buf, format = 'png', bbox_inches = 'tight', facecolor = bg)
    plt.close()
    buf.seek(0)
    display(Image(data = buf.read()))
    buf.close()

def normalise_layout(df, margin = 0.05):

    """Rescale x, y to [margin, 1-margin] regardless of FA2 coordinate explosion."""
    df['x'] = (df['x'] - df['x'].min()) / (df['x'].max() - df['x'].min())
    df['y'] = (df['y'] - df['y'].min()) / (df['y'].max() - df['y'].min())
    df['x'] = df['x'] * (1 - 2*margin) + margin
    df['y'] = df['y'] * (1 - 2*margin) + margin

    return df

🎨 HoloViews Global Defaults

Setting opts.defaults(...) once here avoids repeating style arguments on every plot call.

  • width=1000, height=1000 — square canvas large enough to see fine structure
  • xaxis=None, yaxis=None — suppress axis decorations (coordinates are meaningless for graph layouts)
  • colors — a black-seeded Category20 palette; each unique cluster label gets its own colour
# Holoviews
kwargs = dict(width = 1000, height = 1000, xaxis = None, yaxis = None)
opts.defaults(opts.Nodes(**kwargs), opts.Graph(**kwargs))
colors = ['#000000'] + hv.Cycle('Category20').values

Back to top

🧪 Section 3 — Prototype: Synthetic Graph

Before touching the Netflix data we validate the full pipeline on a toy example.
This is good practice — it separates data-loading bugs from visualisation bugs.

What we create here

Parameter Value Why
n = 100 100 nodes Small enough to run fast
m = 20 000 20 000 random edges Dense enough to stress-test bundling
Seed 100 Fixed random seed Reproducible results

The nodes are simply labelled node0 … node99.
Edges are drawn uniformly at random from all possible node pairs — there is no real structure, so the resulting graph should look like noise. Comparing it to the Netflix graph later will show how much meaningful structure real data contains.

np.random.seed(100)
n = 100
m = 20000

nodes = pd.DataFrame(["node"+str(i) for i in range(n)], columns = ['name'])
print(f"Nodes data:\n {nodes.tail()}\n")

edges = pd.DataFrame(np.random.randint(0, len(nodes), size = (m, 2)), columns = ['source', 'target'])
print(f'Edges data:\n {edges.tail()}')

Nodes data:
       name
95  node95
96  node96
97  node97
98  node98
99  node99

Edges data:
        source  target
19995      52      44
19996       7       7
19997      10      20
19998      18      42
19999      52       1

📐 Node Layout Algorithms

Graph layout is the problem of assigning (x, y) coordinates to nodes so that the visualisation is readable.
We compare three strategies:

🔵 Random Layout

Nodes are placed uniformly at random inside the unit square.
Use case: baseline / sanity check — shows what the graph looks like with zero positional meaning.

🔴 Circular Layout (uniform=False)

Nodes are placed on a circle, with spacing proportional to their degree (number of connections).
Highly-connected nodes get more arc-length, reducing edge crossings.
Use case: revealing spoke-and-hub structures.

🟢 ForceAtlas2 Layout

A physics-based simulation where:

  • Connected nodes attract each other (spring forces)
  • All nodes repel each other (gravity inversion)
  • The simulation runs until the system reaches equilibrium

The result: naturally clusters tightly-connected sub-graphs without any label information.
This is the most informative layout for heterogeneous graphs like our Netflix network.

⏱️ Performance note: ForceAtlas2 is iterative and scales roughly as O(n log n) per step.
On this 100-node toy graph it runs in milliseconds; on the full Netflix people-graph (~8 000 nodes) it takes a few minutes.

# Create the layout
circular  = circular_layout(nodes, uniform = False)
randomloc = random_layout(nodes)
randomloc.tail()
name x y
95 node95 0.761584 0.880246
96 node96 0.898399 0.588561
97 node97 0.933872 0.771876
98 node98 0.859063 0.416293
99 node99 0.754633 0.049666 
forcedirected = forceatlas2_layout(nodes, edges)

CPU times: user 17.4 s, sys: 715 ms, total: 18.1 s
Wall time: 18.2 s

show_ds_images(images = [nodesplot(randomloc, "Random layout"), nodesplot(circular, "Circular layout"), nodesplot(forcedirected, "ForceAtlas2 layout")],
               titles = ['Layout- Random', 'Layout- Circular', 'Layout- Force Directed'],
               ncols = 3)

png

🪢 Edge Bundling: Plain vs Bundled

With 20 000 edges, a naive rendering is a solid blob.
Edge bundling solves this by routing nearby edges along shared paths, like roads merging onto a highway.

The Hammer bundling algorithm (a fast approximation of hierarchical edge bundling) works in two passes:

  1. Sub-divide each edge into a sequence of control points.
  2. Iteratively move each control point toward the average position of control points from nearby edges.

The result is a set of smooth, high-level flow corridors that reveal the dominant connection patterns.

What to look for in the 2×2 comparison grid below:

Layout Plain edges Bundled edges
Circular Individual radial spokes Bundled arc corridors
Force-directed Tangled hairball Clear flow channels between clusters

📌 In a random graph all layouts look like noise — confirming there is no latent structure to recover.

cd = circular
fd = forcedirected

CPU times: user 245 ms, sys: 0 ns, total: 245 ms
Wall time: 246 ms
CPU times: user 250 ms, sys: 17 µs, total: 250 ms
Wall time: 250 ms
CPU times: user 33.7 s, sys: 34.1 ms, total: 33.7 s
Wall time: 34.1 s
CPU times: user 25.4 s, sys: 8.66 ms, total: 25.4 s
Wall time: 25.7 s

show_ds_images(images = [graphplot(cd, connect_edges(cd, edges), "Circular layout"),
                         graphplot(fd, connect_edges(fd, edges), "Force-directed"),
                         graphplot(cd, hammer_bundle(cd, edges), "Circular layout, bundled"),
                         graphplot(fd, hammer_bundle(fd, edges), "Force-directed, bundled")],
               titles = ['Circular Layout', 'Force-Directed', 'Circular Layout, Bundled', 'Force-Directed Layout, Bundled'], ncols = 2)

png

Back to top..

📊 Section 4 — Loading the Netflix Dataset

The Netflix Titles dataset contains metadata for ~8 800 titles available on Netflix as of mid-2021.

Key columns we use:

Column Description Example
title Title of the show/film Stranger Things
director Comma-separated director name(s) Matt Duffer, Ross Duffer
cast Comma-separated cast list Millie Bobby Brown, Finn Wolfhard…
listed_in Comma-separated genre tags Sci-Fi, Horror, Drama
country Country (or countries) of production United States
release_year Year the title was released 2016

📁 The CSV is loaded from Google Drive. If you are running locally, replace the path with your local file path.

# Read the file using pandas
df = pd.read_csv('/content/drive/MyDrive/netflix_titles.csv')

# Split the cells to extract data
df['directors'] = df['director'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])
df['categories'] = df['listed_in'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])
df['actors'] = df['cast'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])
df['countries'] = df['country'].apply(lambda l: [] if pd.isna(l) else [i.strip() for i in l.split(",")])

df.head()
show_id type title director cast country date_added release_year rating duration listed_in description directors categories actors countries
0 s1 Movie Dick Johnson Is Dead Kirsten Johnson NaN United States September 25, 2021 2020 PG-13 90 min Documentaries As her father nears the end of his life, filmm... [Kirsten Johnson] [Documentaries] [] [United States]
1 s2 TV Show Blood & Water NaN Ama Qamata, Khosi Ngema, Gail Mabalane, Thaban... South Africa September 24, 2021 2021 TV-MA 2 Seasons International TV Shows, TV Dramas, TV Mysteries After crossing paths at a party, a Cape Town t... [] [International TV Shows, TV Dramas, TV Mysteries] [Ama Qamata, Khosi Ngema, Gail Mabalane, Thaba... [South Africa]
2 s3 TV Show Ganglands Julien Leclercq Sami Bouajila, Tracy Gotoas, Samuel Jouy, Nabi... NaN September 24, 2021 2021 TV-MA 1 Season Crime TV Shows, International TV Shows, TV Act... To protect his family from a powerful drug lor... [Julien Leclercq] [Crime TV Shows, International TV Shows, TV Ac... [Sami Bouajila, Tracy Gotoas, Samuel Jouy, Nab... []
3 s4 TV Show Jailbirds New Orleans NaN NaN NaN September 24, 2021 2021 TV-MA 1 Season Docuseries, Reality TV Feuds, flirtations and toilet talk go down amo... [] [Docuseries, Reality TV] [] []
4 s5 TV Show Kota Factory NaN Mayur More, Jitendra Kumar, Ranjan Raj, Alam K... India September 24, 2021 2021 TV-MA 2 Seasons International TV Shows, Romantic TV Shows, TV ... In a city of coaching centers known to train I... [] [International TV Shows, Romantic TV Shows, TV... [Mayur More, Jitendra Kumar, Ranjan Raj, Alam ... [India] 
df2 = df[df['directors'].map(len) > 0]
df2 = df2[df2['actors'].map(len) > 0][['title', 'directors', 'actors']]

🔧 Parsing Multi-Value Columns

Several columns store comma-separated lists inside a single string — a common pattern in entertainment datasets.
We split them into proper Python lists so we can iterate over individual names.

"Matt Duffer, Ross Duffer"  →  ["Matt Duffer", "Ross Duffer"]

pd.isna() guards against missing values (rows without a director, for example) which would otherwise throw an error during splitting.
After this step, df['directors'], df['actors'], df['categories'], and df['countries'] each hold a Python list per row.

Back to top..

🕸️ Section 5 — Building the People Co-occurrence Graph

Why a “People” Network?

Rather than connecting films directly, we build a graph where each node is a person (actor or director) and each edge means two people appeared on the same Netflix title together.

This is known as a bipartite projection — we project a film-person bipartite graph down to a person-person graph.

Edge construction logic

For each title that has at least one director and one actor we create:

  1. Director → Actor edges — one edge for every (director, actor) pair on the title
  2. Actor ↔ Actor edges — one edge for every pair of actors who share the same title

This mirrors how social network researchers model Hollywood collaboration networks, following the classic “Six Degrees of Kevin Bacon” approach.

Title: Film X
  Director: D1
  Cast: [A1, A2, A3]

Edges generated:
  D1 ↔ A1,  D1 ↔ A2,  D1 ↔ A3   ← director-actor
  A1 ↔ A2,  A1 ↔ A3,  A2 ↔ A3   ← actor co-stars (combinations)

⚠️ Films with large ensembles generate many edges. A 10-person cast produces C(10,2) = 45 actor-actor edges alone. This is why the final graph is dense.

# Redesign the edges as a people network
people_edges = []

for idx, row in df2.iterrows():

    combos = [(item1, item2) for item1 in row['directors'] for item2 in row['actors'] if item1 is not None]
    pairs = list(combinations(row['actors'], 2))

    for combo in combos:

        people_edges.append(tuple(sorted(combo)))

    for pair in pairs:

        people_edges.append(tuple(sorted(pair)))

people_edges = pd.DataFrame(people_edges, columns = ['source', 'target'])

⚖️ Weighting Edges by Co-occurrence Frequency

A raw edge list has a separate row every time two people share a title. If they collaborated on multiple titles, they have multiple rows.

We use groupby + size() to collapse duplicates and count occurrences, creating a weight column.
A higher weight means a stronger, more repeated collaboration — think of a frequent director-actor partnership like Martin Scorsese and Robert De Niro.

The top rows after sorting by weight will show the most prolific on-screen partnerships in the Netflix catalogue.

weighted_people_edges = people_edges.groupby(['source', 'target']).size().reset_index().rename({0: 'weight'}, axis = 1)
weighted_people_edges.sort_values('weight', ascending = False).head()
source target weight
169060 Julie Tejwani Rupa Bhimani 27
162773 John Paul Tremblay Robb Wells 21
169055 Julie Tejwani Rajesh Kava 21
222896 Rajesh Kava Rupa Bhimani 20
222894 Rajesh Kava Rajiv Chilaka 19 
weighted_people_edges['source'] = weighted_people_edges['source'].astype(str).apply(lambda x: x.translate(translator))
weighted_people_edges['target'] = weighted_people_edges['target'].apply(lambda x: x.translate(translator))

people_nodes = pd.DataFrame(pd.unique(weighted_people_edges[['source', 'target']].values.ravel('K')), columns = ['name'])

🔢 Mapping Names to Integer Indices

Datashader’s layout algorithms expect integer node IDs, not string names.
We build a node_to_idx dictionary that assigns each unique person name a sequential integer.

{"Tom Hanks": 0, "Steven Spielberg": 1, "Cate Blanchett": 2, …}

The source and target columns in weighted_people are then replaced with these integers.
The original string names are preserved in people_nodes for lookup and labelling later.

# Convert the nodes to indices for mapping
node_to_idx = {node: i for i, node in enumerate(people_nodes['name'])}
weighted_people_edges["source"] = weighted_people_edges["source"].map(node_to_idx)
weighted_people_edges["target"] = weighted_people_edges["target"].map(node_to_idx)

weighted_people_edges.head()
source target weight
0 0 3556 1
1 0 5841 1
2 0 10769 1
3 0 16771 1
4 0 19387 1 
# Filter the graph for meaningful connections
filtered_people_edges = (weighted_people_edges[weighted_people_edges.weight >= 2]).copy()

# Keep nodes that appear in filtered edges
remaining_nodes = pd.unique(filtered_people_edges[['source','target']].values.ravel())

filtered_people_nodes = people_nodes[people_nodes.index.isin(remaining_nodes)].copy()

print('Filtering graph based on edge weights..\n')
print(f"Initial Configuration: \nNodes- {people_nodes.shape[0]}\nEdges- {weighted_people_edges.shape[0]}\n")
print(f"Filtered Result: \nNodes- {filtered_people_nodes.shape[0]}\nEdges- {filtered_people_edges.shape[0]}\n")

Filtering graph based on edge weights..

Initial Configuration: 
Nodes- 30798
Edges- 238512

Filtered Result: 
Nodes- 5313
Edges- 10714

Back to top..

🌌 Section 6 — Visualising the Netflix People Graph

We now apply the same layout + bundling pipeline from Section 3 — but this time to real Netflix data.

What to expect

  • Scale: The people graph typically has ~8 000–12 000 nodes and 50 000+ edges, depending on how the data is filtered.
  • ForceAtlas2: Will take several minutes for a graph this size. The algorithm is deterministic given the same seed, so results are reproducible.
  • Circular layout: Faster (~seconds) but less semantically meaningful — useful as a quick sanity check.

Interpreting the output

Once rendered, look for:

Visual feature What it means
Dense cluster A group of actors/directors who frequently work together (e.g. Bollywood ensemble casts, a TV show with a stable cast)
Isolated node Someone who appeared in only one title and shares no co-stars with others in the filtered dataset
Bridge node A person connecting two otherwise separate clusters — high betweenness centrality
Thick bundled corridor A high-traffic collaboration route between two film communities

💡 Try this: Zoom into the force-directed bundled view and hover over nodes to see names. Look for directors sitting at the intersection of multiple actor clusters — these are often prolific “connector” filmmakers.

# Get position information
circularloc  = circular_layout(filtered_people_nodes, uniform = False)
forcedirectedloc = forceatlas2_layout(filtered_people_nodes, filtered_people_edges)

CPU times: user 11.1 s, sys: 77.3 ms, total: 11.2 s
Wall time: 6 s

# Compile layouts
cd = circularloc
fd = forcedirectedloc

show_ds_images(images = [graphplot(cd, connect_edges(cd, filtered_people_edges), "Filtered Netflix Social Network: Circular Layout"),
                         graphplot(fd, connect_edges(fd, filtered_people_edges), "Filtered Netflix Social Network: Force-Directed Layout"),
                         graphplot(cd, hammer_bundle(cd, filtered_people_edges), "Filtered Netflix Social Network: Circular Layout, Bundled"),
                         graphplot(fd, hammer_bundle(fd, filtered_people_edges), "Filtered Netflix Social Network: Force-Directed Layout, Bundled")],
               titles = ["Filtered Netflix Social Network: Circular Layout",
                         "Filtered Netflix Social Network: Force-Directed Layout",
                         "Filtered Netflix Social Network: Circular Layout, Bundled",
                         "Filtered Netflix Social Network: Force-Directed Layout, Bundled"],
               ncols = 2)

CPU times: user 3.77 s, sys: 53.5 ms, total: 3.83 s
Wall time: 3.93 s
CPU times: user 177 ms, sys: 4.99 ms, total: 182 ms
Wall time: 183 ms
CPU times: user 18.7 s, sys: 55.7 ms, total: 18.8 s
Wall time: 18.9 s
CPU times: user 2.82 s, sys: 7.13 ms, total: 2.83 s
Wall time: 2.83 s

png

Back to top..

🏘️ Section 7 — Community Detection

What is a community?

In graph theory a community (or module) is a group of nodes that are more densely connected to each other than to the rest of the graph. In our Netflix people network this maps naturally to:

  • A Bollywood production ensemble that works together repeatedly
  • A TV show with a stable, recurring cast
  • A director and their habitual collaborators

The Louvain algorithm

We use the Louvain method — the industry standard for large graph community detection:

  1. Each node starts in its own community
  2. Nodes are iteratively moved to the neighbour community that gives the largest gain in modularity
  3. Communities are collapsed into single nodes and the process repeats

Modularity Q measures how much denser within-community edges are compared to a random graph with the same degree sequence. The algorithm maximises Q.

📦 Requires python-louvain — already installed in the pip cell above.

Parameter Meaning
weight='weight' Use edge weights — stronger collaborations matter more
random_state=100 Fixed seed for reproducibility 
print('Analyzing the Netflix neighborhood structure..')

graph = nx.from_pandas_edgelist(filtered_people_edges, source = 'source', target = 'target', edge_attr = 'weight')

print(f"Netflix Graph-\nNodes: {graph.number_of_nodes()}, Edges: {graph.number_of_edges()}")

# Find communities
print('Finding communities..')
partition = louvain_community.best_partition(graph, weight = 'weight', random_state = 100)

print('Communities detected..\nUpdating params..')

# Map community id onto the nodes DataFrame (which holds x, y positions)
filtered_people_nodes['community'] = filtered_people_nodes.index.map(partition).fillna(-1).astype(int)
filtered_people_nodes['community'] = pd.Categorical(filtered_people_nodes['community'])

# Merge community onto the layout DataFrames
community_categories = sorted(filtered_people_nodes['community'].unique().astype(str))

# Assign as proper pd.Categorical — this is what datashader requires
forcedirectedloc['community'] = pd.Categorical(forcedirectedloc.index.map(filtered_people_nodes['community']).astype(str), categories = community_categories)
circularloc['community'] = pd.Categorical(circularloc.index.map(filtered_people_nodes['community']).astype(str), categories = community_categories)

print(f"Process complete..\nNumber of communities: {filtered_people_nodes['community'].nunique()}")
print(filtered_people_nodes['community'].value_counts().head(10))

Analyzing the Netflix neighborhood structure..
Netflix Graph-
Nodes: 5313, Edges: 10714
Finding communities..
Communities detected..
Updating params..
Process complete..
Number of communities: 477
community
2     465
10    203
9     190
12    165
50    160
22    115
0     110
28    107
31    103
26     95
Name: count, dtype: int64

# Update function to show communities
def nodesplot(nodes, name = None, canvas = None, cat = None):

    canvas = ds.Canvas(**cvsopts) if canvas is None else canvas
    aggregator = None if cat is None else ds.count_cat(cat)
    agg = canvas.points(nodes, 'x', 'y', aggregator)

    # Use a categorical colormap when a category column is provided
    if cat:

        # Remove_unused_categories ensures cats and colors always match
        cats = list(nodes[cat].cat.remove_unused_categories().cat.categories)
        color_key = dict(zip(cats, make_hex_palette(len(cats))))

        return tf.spread(tf.shade(agg, color_key = color_key), px = 3, name = name)

    else:

        return tf.spread(tf.shade(agg, cmap = ["#FF3333"]), px = 3, name = name)

cats = list(filtered_people_nodes['community'].cat.categories)
hex_colors = make_hex_palette(len(cats))
plot_colortable(hex_colors)

png

show_ds_images(images = [graphplot(forcedirectedloc, hammer_bundle(forcedirectedloc, filtered_people_edges), "Filtered Netflix: Communities, Force-Directed Bundled", cat = 'community')],
               titles = ['Filtered Netflix Social Network, Weighted and Bundled'],
               ncols = 1,
               fig_width = 10)

png

# Build the cmap again
cats = list(filtered_people_nodes['community'].cat.categories)
hex_colors = make_hex_palette(len(cats))
color_key = dict(zip(cats, hex_colors))
n_communities = filtered_people_nodes['community'].nunique()
plot_colortable(hex_colors)

png

# Filter the graph again to be more informative
degree = pd.Series(dict(graph.degree()), name = 'degree')
weighted_degree = pd.Series(dict(graph.degree(weight = 'weight')), name = 'weighted_degree')

# Combine into a single stats DataFrame
node_stats = pd.DataFrame({'degree': degree, 'weighted_degree': weighted_degree})
node_stats = node_stats.join(filtered_people_nodes[['name', 'community']])
node_stats = node_stats[['name', 'community', 'degree', 'weighted_degree']]

print(node_stats.sort_values('weighted_degree', ascending = False).head(20))

                     name community  degree  weighted_degree
14142       Julie Tejwani        39      25              193
24000        Rupa Bhimani        39      24              188
2490          Anupam Kher         2      70              174
27837      Yılmaz Erdoğan        50      49              148
25102      Shah Rukh Khan         2      54              142
13402  John Paul Tremblay       176      27              138
23516          Robb Wells       176      27              138
22688         Rajesh Kava        39      14              135
8558      Erin Fitzgerald        14      36              131
22708       Rajiv Chilaka        39      10              122
12869      Jigna Bhardwaj        39      11              121
21121         Omoni Oboli        10      49              118
1804        Andrea Libman        71      25              115
19307          Mike Smith       176      25              114
14677        Kate Higgins        14      28              111
29432        Vatsal Dubey        39       9              110
746        Akiva Schaffer        12      51              110
15985        Laura Bailey        14      25               99
758          Akshay Kumar         2      36               95
21389        Paresh Rawal         2      36               95

# Filter the graph for top communities and nodes
comm_sizes = node_stats['community'].value_counts()
top_communities = comm_sizes.head(25).index
node_stats_filtered = node_stats[node_stats['community'].isin(top_communities)]

# Within each community keep only the most connected individuals ────────────
top_nodes = (node_stats_filtered.groupby('community', group_keys = False).apply(lambda g: g.nlargest(50, 'weighted_degree')))
top_node_ids = set(top_nodes.index)
print(f"Keeping {len(top_node_ids)} nodes from 25 communities")

/tmp/ipykernel_43227/2244287134.py:7: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  top_nodes = (node_stats_filtered.groupby('community', group_keys = False).apply(lambda g: g.nlargest(50, 'weighted_degree')))


Keeping 1250 nodes from 25 communities


/tmp/ipykernel_43227/2244287134.py:7: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  top_nodes = (node_stats_filtered.groupby('community', group_keys = False).apply(lambda g: g.nlargest(50, 'weighted_degree')))

# Filter edge list for these nodes
filtered_edges = weighted_people_edges[weighted_people_edges['source'].isin(top_node_ids) & weighted_people_edges['target'].isin(top_node_ids)].copy()

# Flter by minimum edge weight to remove weak links
filtered_edges = filtered_edges[filtered_edges['weight'] >= 2]

print(f"Edges before filtering: {len(weighted_people_edges)}")
print(f"Edges after filtering: {len(filtered_edges)}")

Edges before filtering: 238512
Edges after filtering: 3622

filtered_edges.head()
source target weight
221 30 2248 2
227 30 3553 2
242 30 13168 2
243 30 13217 2
908 99 3783 2 
remaining_nodes = pd.unique(filtered_edges[['source', 'target']].values.ravel())
print(f"# of Remaining Nodes- {len(remaining_nodes)}")

new_idx = {old: new for new, old in enumerate(remaining_nodes)}

filtered_edges['source'] = filtered_edges['source'].map(new_idx)
filtered_edges['target'] = filtered_edges['target'].map(new_idx)

# Rebuild nodes DataFrame with name + community
filtered_nodes = (top_nodes.loc[remaining_nodes].reset_index().rename(columns = {'index': 'old_idx'}))
filtered_nodes.index = filtered_nodes['old_idx'].map(new_idx)
filtered_nodes = filtered_nodes.sort_index()

# Assign categorical community
community_cats = sorted(filtered_nodes['community'].unique().astype(str))
filtered_nodes['community'] = pd.Categorical(filtered_nodes['community'].astype(str), categories = community_cats)

print(filtered_nodes.head())

# of Remaining Nodes- 1245
         old_idx             name community  degree  weighted_degree
old_idx                                                             
0              3          50 Cent         0       2                4
1           4341     Bruce Willis         0       8               18
2          13277      John Cusack         0       2                4
3             19             ARAH         3       1                2
4            474  Adriano Rudiman         3       2                4

# Recompute the layouts
layout_nodes_df = filtered_nodes[[]].copy()
layout_nodes_df['name'] = filtered_nodes.index.astype(str)

fd_filtered = forceatlas2_layout(layout_nodes_df, filtered_edges)

# Normalize the edges before drawing
xmin, xmax = fd_filtered.x.min(), fd_filtered.x.max()
ymin, ymax = fd_filtered.y.min(), fd_filtered.y.max()
margin = 0.05

fd_filtered['x'] = (fd_filtered.x - xmin) / (xmax - xmin) * (1 - 2 * margin) + margin
fd_filtered['y'] = (fd_filtered.y - ymin) / (ymax - ymin) * (1 - 2 * margin) + margin

# ← always strip here, immediately after assignment
fd_filtered_xy = fd_filtered[['x', 'y']]
fd_filtered['community'] = pd.Categorical(filtered_nodes['community'].values, categories = community_cats)
fd_filtered['community'] = fd_filtered['community'].cat.remove_unused_categories()
filtered_nodes['community'] = filtered_nodes['community'].cat.remove_unused_categories()

CPU times: user 682 ms, sys: 4.08 ms, total: 686 ms
Wall time: 355 ms

show_ds_images(images = [graphplot(fd_filtered, hammer_bundle(fd_filtered_xy, filtered_edges), "Netflix Universe: Top Individuals by Community", cat = 'community')], titles = ['Filtered Edges w/ Important Communities'], ncols = 1, fig_width = 10)

png

Back to top..

🎛️ Section 8 — Interactive HoloViews Graph (Full Network)

Why HoloViews on top of Datashader?

The static tf.Images output above is a flat raster — you cannot hover, zoom into a cluster and read node names, or click to inspect a community. HoloViews wraps the same Datashader pipeline inside a Bokeh canvas that adds:

  • 🖱️ Pan and scroll-zoom
  • 💬 Hover tooltips showing name and community
  • 🎨 Legend with community colour mapping
  • 📸 Export to PNG via the Bokeh toolbar

How the layers are combined

datashade(bundled_edges)        ← rasterised edges (fast, handles 50k+ edges)
        *
bundled.nodes                   ← individual Bokeh glyphs (supports hover)

The * operator in HoloViews creates an Overlay — Datashaded edges underneath, interactive node glyphs on top.

⚠️ Colab note: Use pn.panel(plot) to display — plain cell output is unreliable for Datashaded overlays in Colab. If the output is blank, run Runtime → Restart and run all.

# Build a new nodes df for hv
node_data = pd.DataFrame({'x': fd_filtered['x'].values,
                          'y': fd_filtered['y'].values,
                          'index': np.arange(len(filtered_nodes)),
                          'name': filtered_nodes['name'].values,
                          'community': filtered_nodes['community'].astype(str).values,
                          'weighted_degree': filtered_nodes['weighted_degree'].values})

hv_nodes = hv.Nodes(node_data, kdims = ['x', 'y', 'index'], vdims = ['name', 'community', 'weighted_degree'])

# Sanity check — x/y must NOT be all zeros
print("Node position sample:")
print(node_data[['name', 'x', 'y', 'community', 'weighted_degree']].head())
print(f"\nX range: {node_data.x.min():.3f} → {node_data.x.max():.3f}")
print(f"Y range: {node_data.y.min():.3f} → {node_data.y.max():.3f}")

Node position sample:
              name         x         y community  weighted_degree
0          50 Cent  0.574436  0.139656         0                4
1     Bruce Willis  0.574797  0.208898         0               18
2      John Cusack  0.618169  0.261751         0                4
3             ARAH  0.928325  0.345166         3                2
4  Adriano Rudiman  0.822083  0.328742         3                4

X range: 0.050 → 0.950
Y range: 0.050 → 0.950

# Build the graph
hv_graph = hv.Graph((filtered_edges[['source', 'target', 'weight']], hv_nodes), kdims = ['source', 'target'], vdims = ['weight'])

print("Initial Graph:", hv_graph)
print("Nodes in graph:", len(hv_graph.nodes))

bundled = bundle_graph(hv_graph)
print("Edge bundling complete ✓")

Initial Graph: :Graph   [source,target]   (weight)
Nodes in graph: 1245
Edge bundling complete ✓

display(bundled)

# Add interactivity with datashader and bokeh
n_cats = filtered_nodes['community'].nunique()
hex_colors = make_hex_palette(n_cats)
shaded_edges = datashade(bundled, normalization = 'linear', width = 1000, height = 1000)
interactive_nodes = bundled.nodes.opts(opts.Nodes(color = 'community',
                                                  cmap = hex_colors,
                                                  size = 8,
                                                  tools = ['hover'],
                                                  width = 1000,
                                                  height = 1000,
                                                  xaxis = None,
                                                  yaxis = None,
                                                  legend_position = 'right',
                                                  padding = 0.06))

interactive_plot = (shaded_edges * interactive_nodes)

# pn.panel is the reliable display path for datashaded overlays in Colab
pn.panel(interactive_plot)

WARNING:param.datashade: Parameter(s) [normalization] not consumed by any element rasterizer.
WARNING:param.datashade:Parameter(s) [normalization] not consumed by any element rasterizer.
WARNING:param.datashade: Parameter(s) [normalization] not consumed by any element rasterizer.
WARNING:param.datashade:Parameter(s) [normalization] not consumed by any element rasterizer.

🏷️ Adding Community Labels (NOT SHOWN)

Hover tooltips identify individual nodes but it helps to have a fixed label anchored to the centre of each community so you can orient yourself at a glance. We compute the centroid (mean x, mean y) of each community and overlay text labels.

x_range = node_data.x.min(), node_data.x.max()
y_range = node_data.y.min(), node_data.y.max()

# Compute label positions
centroids = (node_data.groupby('community').agg(x = ('x', 'mean'), y = ('y', 'mean'), size = ('name', 'count')).reset_index())
centroids['label'] = 'Community #' + centroids['community'].astype(str)

print(centroids.sort_values('size', ascending = False).head(10))

community_labels = hv.Labels(centroids[['x', 'y', 'label']], kdims = ['x', 'y'], vdims = ['label']).opts(xlim = x_range,
                                                                                                         ylim = y_range,
                                                                                                         text_font_size = '8pt',
                                                                                                         text_color = 'black',
                                                                                                         text_alpha = 0.85,
                                                                                                         xaxis = None,
                                                                                                         yaxis = None)

labelled_plot = (shaded_edges * interactive_nodes * community_labels)

   community         x         y  size           label
1         10  0.517181  0.400035    50   Community #10
5         14  0.402396  0.406381    50   Community #14
2        111  0.543715  0.518234    50  Community #111
3        113  0.438975  0.491986    50  Community #113
4         13  0.512129  0.468302    50   Community #13
7         21  0.372380  0.513129    50   Community #21
6         16  0.522011  0.367539    50   Community #16
10        33  0.376869  0.456897    50   Community #33
9         30  0.504554  0.425462    50   Community #30
23        87  0.618238  0.462728    50   Community #87

pn.panel(labelled_plot)

Back to top..

✅ Key Takeaways

  • Indian cinema dominates by volume — the largest community clusters around Bollywood ensemble casts, with Anupam Kher (295 connections) as the most connected node in the entire graph.
  • Hollywood fractures into sub-communities — comedy ensembles, dramatic actors, and voice cast form distinct, rarely-overlapping clusters.
  • Bridge nodes are rare and structurally critical — a handful of internationally active directors connect communities that would otherwise be fully isolated.
  • Edge bundling reveals the dominant collaboration highways — the densest corridors run within national cinema industries, not across them.

Back to top..

]]>Data Guydataguy50@gmail.comModeling Heart Disease with AI2026-03-08T00:00:00+00:002026-03-08T00:00:00+00:00https://doctorofdata.github.io/posts/2026/03/heart-disease-predictions-model-optimizationIn this experiment, we use a python library for optimizing model parameters employed with various machine learning algorithms to maximize their scoring ability on out-of-sample tests to detect heart disease based on metabolic indicators.

Machine Learning- An Approach to Choosing Model Parameters

Introduction- In this notebook, we will use the Optuna library to conduct experiments which can be used to set model hyperparameters for effectively implementing various machine learning algorithms

Table of Contents

🫀 Predicting Heart Disease- A Machine Learning Approach to Health

In this notebook, we will utilize advanced optimization techniques and feature engineering to improve machine learning scores regarding a dataset of patients’ metabolic information to see if this data can help with indications as to whether or not an individual has or could develop heart disease.

The dataset features are as follows-

Feature 🧓 Age 🚹 Sex 💔 Chest pain type 💉 BP 🧈 Cholesterol 🍬 FBS over 120 📈 EKG results ❤️ Max HR 🏃 Exercise angina 📉 ST depression ⛰️ Slope of ST 🩸 Number of vessels fluro 🧬 Thallium 🎯 Heart Disease

Each feature is derived from well-established cardiological diagnostics. Together they capture multiple physiological dimensions of coronary artery disease:

🧓 Demographics

  • Age
  • Sex

🧈 Metabolic Risk

  • Cholesterol
  • Fasting Blood Sugar
  • Blood Pressure

❤️ Symptom Profiles

  • Chest Pain Type
  • Exercise-induced Angina

📈 Functional Testing

  • EKG Results
  • ST Depression
  • ST Slope
  • Max Heart Rate

🩻 Imaging Diagnostics

  • Fluoroscopy Vessels
  • Thallium Stress Test

Back to top..

💻 Setup and Environment 

# Install packages
!pip install kaggle kagglehub matplotlib seaborn lazypredict ipywidgets jupyter_contrib_nbextensions optuna catboost -q

# Import libraries
import numpy as np
import pandas as pd
from IPython.display import display, HTML
import os
from sklearn.model_selection import train_test_split
import lazypredict
from lazypredict.Supervised import LazyClassifier
import subprocess
import json
import zipfile
from io import BytesIO
import xgboost as xgb
import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
import lightgbm as lgb
import catboost as cb
import zipfile
import io
from contextlib import redirect_stdout

# Download the data from source
!kaggle competitions download -c playground-series-s6e2 --force

Downloading playground-series-s6e2.zip to /Users/anon/Downloads
  0%|                                               | 0.00/10.2M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 10.2M/10.2M [00:00<00:00, 1.29GB/s]

!pwd

/Users/anon/Downloads

print('Loading data..')

# Read the data
zf = zipfile.ZipFile('/Users/anon/Downloads/playground-series-s6e2.zip') 
df = pd.read_csv(zf.open('train.csv')).drop('id', axis = 1)
test = pd.read_csv(zf.open('test.csv')) #.drop('id', axis = 1)

Loading data..

df.head()
Age Sex Chest pain type BP Cholesterol FBS over 120 EKG results Max HR Exercise angina ST depression Slope of ST Number of vessels fluro Thallium Heart Disease
0 58 1 4 152 239 0 0 158 1 3.60 2 2 7 Presence
1 52 1 1 125 325 0 2 171 0 0.00 1 0 3 Absence
2 56 0 2 160 188 0 2 151 0 0.00 1 0 3 Absence
3 44 0 3 134 229 0 2 150 0 1.00 2 0 3 Absence
4 58 1 4 140 234 0 2 125 1 3.80 2 3 3 Presence 
# Map the outcome to numeric
def convert_outcomes(outcome):

    if outcome == 'Absence':

        return 0

    elif outcome == 'Presence':

        return 1
        
df['Heart Disease'] = df['Heart Disease'].map(convert_outcomes)

# Split the data for training
x = df[[i for i in df.columns if i != 'Heart Disease']]
y = df['Heart Disease']

xtrain, xval, ytrain, yval = train_test_split(x, y, test_size = .2, random_state = 100)

Back to top..

Experimentation 🔧 

# Define objective function for optuna with lgith gbm
def objective(trial):

    dtrain = lgb.Dataset(xtrain, label = ytrain)
    
    params = {"objective": "binary",
              "metric": "auc",
              "verbosity": -1,
              "boosting_type": "gbdt",
              "lambda_l1": trial.suggest_float("lambda_l1", 1e-8, 10.0, log=True),
              "lambda_l2": trial.suggest_float("lambda_l2", 1e-8, 10.0, log=True),
              "num_leaves": trial.suggest_int("num_leaves", 2, 256),
              "feature_fraction": trial.suggest_float("feature_fraction", 0.4, 1.0),
              "bagging_fraction": trial.suggest_float("bagging_fraction", 0.4, 1.0),
              "bagging_freq": trial.suggest_int("bagging_freq", 1, 7),
              "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),}

        
    gbm = lgb.train(params, dtrain)
    preds = gbm.predict(xval)
    pred_labels = np.rint(preds)
    auc = roc_auc_score(yval, pred_labels)

    return auc

# Create a study and optimize
print('Initiating Optimization Sequence..')

# Silence optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

# Conduct study
study = optuna.create_study(direction = 'maximize') 
study.optimize(objective, n_trials = 50, n_jobs = 4) 

# Print the captured output to the console (using the restored stdout)
print("Modeling complete..")

# Print the best hyperparameters and AUC
print('Optimized Light GBM Performance- ')
print(f"Best trial value (AUC): {study.best_value}")
print(f"Best hyperparameters: {study.best_params}")

Initiating Optimization Sequence..
Modeling complete..
Optimized Light GBM Performance- 
Best trial value (AUC): 0.8865625304640333
Best hyperparameters: {'lambda_l1': 0.005711951061229914, 'lambda_l2': 0.02701383662599308, 'num_leaves': 162, 'feature_fraction': 0.4779878924499187, 'bagging_fraction': 0.9104828961933018, 'bagging_freq': 4, 'min_child_samples': 79}

# Convert the data for modeling
dtrain = lgb.Dataset(x, label = y)

# Fit
gbm = lgb.train(study.best_params, dtrain)

# Predict
gbmtest = gbm.predict(test.drop(['id'], axis = 1))

# Generate submissions
test['Heart Disease'] = gbmtest

# Write to file
test[['id', 'Heart Disease']].to_csv('heart_disease_test_gbm.csv', index = False)

[LightGBM] [Warning] Found whitespace in feature_names, replace with underlines
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.012748 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 419
[LightGBM] [Info] Number of data points in the train set: 630000, number of used features: 13
[LightGBM] [Info] Start training from score 0.448340

print('Uploading scores to Kaggle..')
!kaggle competitions submit -c playground-series-s6e2 -f heart_disease_test_gbm.csv -m "LightGBM heart disease predictions test set"

Uploading scores to Kaggle..
100%|██████████████████████████████████████| 6.89M/6.89M [00:04<00:00, 1.62MB/s]
Successfully submitted to Predicting Heart Disease

!kaggle competitions submissions -c playground-series-s6e2

fileName                               date                        description                                                                 status                     publicScore  privateScore  
-------------------------------------  --------------------------  --------------------------------------------------------------------------  -------------------------  -----------  ------------  
heart_disease_test_gbm.csv             2026-03-06 16:43:05.187000  LightGBM heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.95240      0.95401       
submission.csv                         2026-02-24 19:38:04                                                                                     SubmissionStatus.COMPLETE  0.87823      0.87954       
submission.csv                         2026-02-23 20:31:03.743000  ensemblemodeling-heart-disease-predictions | LightGBM * XGBoost *CatBoost   SubmissionStatus.COMPLETE  0.87823      0.87954       
submission.csv                         2026-02-23 20:30:39         StackedEnsemble heart disease predictions test set                          SubmissionStatus.COMPLETE  0.87823      0.87954       
submission.csv                         2026-02-23 20:01:25         StackedEnsemble heart disease predictions test set                          SubmissionStatus.COMPLETE  0.87641      0.87986       
heart_disease_test_gbm.csv             2026-02-22 22:20:26         LightGBM heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.95227      0.95398       
heart_disease_test_gbm.csv             2026-02-22 22:17:42.617000  LightGBM heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.95217      0.95392       
submission.csv                         2026-02-22 22:04:35.347000  LightGBM Featurized heart disease predictions test set                      SubmissionStatus.COMPLETE  0.95230      0.95379       
submission.csv                         2026-02-22 21:51:05.337000  heart-disease-predictions:feature-engineering | .95+ auc lightgbm           SubmissionStatus.COMPLETE  0.95179      0.95351       
submission.csv                         2026-02-22 21:50:42.543000  LightGBM Featurized heart disease predictions test set                      SubmissionStatus.COMPLETE  0.95179      0.95351       
heart_disease_test_gbm.csv             2026-02-22 21:50:25         LightGBM heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.95233      0.95392       
submission.csv                         2026-02-22 21:31:52         LightGBM Featurized heart disease predictions test set                      SubmissionStatus.COMPLETE  0.95212      0.95385       
heart_disease_test_gbm_featurized.csv  2026-02-22 21:11:45         LightGBM Featurized heart disease predictions test set                      SubmissionStatus.COMPLETE  0.95212      0.95385       
heart_disease_test_gbm.csv             2026-02-22 20:08:58.903000  LightGBM heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.95234      0.95403       
submission.csv                         2026-02-22 18:59:36         h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88452      0.88571       
submission.csv                         2026-02-21 14:04:38.603000  Notebook heart-disease-prediction: comprehensive-modeling                   SubmissionStatus.COMPLETE  0.88460      0.88559       
submission.csv                         2026-02-21 14:04:13.253000  h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88460      0.88559       
submission.csv                         2026-02-20 17:36:38.107000  Notebook Heart Disease Prediction: Comprehensive Model Eva | Version 5      SubmissionStatus.COMPLETE  0.88472      0.88562       
submission.csv                         2026-02-20 17:36:12.513000  h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88472      0.88562       
submission.csv                         2026-02-20 16:03:12.007000  Notebook Heart Disease Prediction: Comprehensive Model Eva | Version 4      SubmissionStatus.COMPLETE  0.88468      0.88564       
submission.csv                         2026-02-20 16:02:44.473000  h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88468      0.88564       
heart_disease_test_h2o.csv             2026-02-20 15:45:56         h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88463      0.88568       
heart_disease_test_h2o.csv             2026-02-20 15:30:25         h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88460      0.88569       
heart_disease_test_h2o.csv             2026-02-19 16:58:11.307000  h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88461      0.88563       
heart_disease_test_h2o.csv             2026-02-19 16:37:17.767000  h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88407      0.88527       
heart_disease_test_h2o.csv             2026-02-19 15:41:44.227000  h2o heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88458      0.88542       
heart_disease_test_gbm.csv             2026-02-18 17:32:39         LightGBM heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.95238      0.95395       
heart_disease_test_catboost.csv        2026-02-18 17:26:48.153000  catboost heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.88391      0.88620       
heart_disease_test_xgb.csv             2026-02-18 17:10:46         xgb heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88257      0.88532       
heart_disease_test_rf.csv              2026-02-18 17:10:40         rf heart disease predictions test set                                       SubmissionStatus.COMPLETE  0.87568      0.87834       
heart_disease_test_catboost.csv        2026-02-18 14:23:04         catboost heart disease predictions test set                                 SubmissionStatus.COMPLETE  0.88410      0.88643       
heart_disease_test_xgb.csv             2026-02-18 14:18:58         xgb heart disease predictions test set                                      SubmissionStatus.COMPLETE  0.88257      0.88532       
heart_disease_test_rf.csv              2026-02-18 14:14:17.973000  rf heart disease predictions test set                                       SubmissionStatus.COMPLETE  0.87568      0.87834       

# Get scores from the mode
lgb.plot_importance(gbm)

<Axes: title={'center': 'Feature importance'}, xlabel='Feature importance', ylabel='Features'>

png

Back to top..

]]>Data Guydataguy50@gmail.com