Designing a Production Recommendation System That Actually Moves Revenue

Most recommendation tutorials stop at training a model and reporting an offline AUC. That model never ships, and if it does, nobody can tell you whether it made the business any money. I've built and run recommenders in production for years, and the hard part was never the model — it was the candidate generation, the serving pipeline, and agreeing on a metric that maps to revenue instead of a leaderboard score.

This is how I think about designing one that actually moves the needle.

Start From the Decision, Not the Model

A recommender exists to change a decision: show this user something they're more likely to buy, or keep them from churning. So before any modeling, I write down the decision and the metric. For a marketplace I worked on, the metric was transaction rate uplift (we saw roughly 5-15% lifts depending on surface) and churn reduction (around 4-6%). Those numbers are the contract. AUC is a diagnostic, not a goal.

Once the metric is fixed, the architecture falls out of two questions: how do I narrow millions of items down to a few hundred fast, and how do I order those few hundred well? That's candidate generation versus ranking, and conflating them is the most common design mistake I see.

Candidate Generation vs Ranking

Candidate generation is a recall problem. You have a huge catalog and a tight latency budget, so you use cheap, approximate methods to pull a few hundred plausible items: content-based similarity (embeddings over item attributes), co-visitation counts, recently-popular-in-category, and a few business rules. None of these need to be precise — they just need to not miss the good stuff.

Ranking is a precision problem. Given a small candidate set, you score each item with an expensive model that uses rich user and context features. This is where a propensity model lives: a gradient-boosted model (I usually reach for LightGBM) predicting probability of purchase, click, or another conversion event.

Splitting these lets each stage use the right tool. Content-based generation handles cold-start items that have no interaction history. The propensity ranker handles personalization. A final re-rank applies dynamic pricing and diversity so you don't show ten near-identical items.

The Feature and Serving Pipeline

The model is maybe 20% of the work. The pipeline that computes features consistently between training and serving is the other 80%, and it's where most recommenders quietly rot.

I split features by how fresh they need to be. Slow-moving features (item category embeddings, 30-day purchase aggregates, user lifetime stats) are batch-computed in Airflow, written to BigQuery for training, and pushed to a Redis feature store for serving. Fast-moving features (events in the current session, items viewed in the last few minutes) flow through Kafka and are aggregated in near-real-time.

# Airflow DAG: nightly feature materialization to BigQuery + Redis
from airflow.decorators import dag, task
import pendulum
 
@dag(schedule="0 2 * * *", start_date=pendulum.datetime(2026, 1, 1), catchup=False)
def user_features():
    @task
    def compute_aggregates() -> str:
        # BigQuery does the heavy lifting; we don't pull rows into Python
        sql = """
        CREATE OR REPLACE TABLE feature_store.user_features AS
        SELECT
          user_id,
          COUNTIF(event = 'purchase') AS purchases_30d,
          APPROX_QUANTILES(order_value, 100)[OFFSET(50)] AS median_order_value,
          SAFE_DIVIDE(COUNTIF(event='purchase'), COUNTIF(event='view')) AS cvr_30d
        FROM events.user_events
        WHERE event_ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
        GROUP BY user_id
        """
        return sql
 
    @task
    def push_to_redis(_sql: str) -> None:
        # Stream the materialized table into Redis hashes keyed by user_id
        ...
 
    push_to_redis(compute_aggregates())
 
user_features()

The non-negotiable rule here is that the same feature definition produces training and serving values. If cvr_30d is computed one way in your BigQuery training job and another way in your serving code, you get training-serving skew, and your offline metrics become fiction. I keep the transformation logic in one shared library and call it from both paths.

A Ranking Snippet

Here's the shape of the ranking step. The candidate set arrives from generation, features are joined from the store, and a LightGBM propensity model scores them. The output is a probability, which I can then combine with margin or price elasticity in re-ranking.

import lightgbm as lgb
import numpy as np
 
class PropensityRanker:
    def __init__(self, model_path: str, feature_order: list[str]):
        self.model = lgb.Booster(model_file=model_path)
        self.feature_order = feature_order  # frozen at train time
 
    def rank(self, candidates: list[dict], user_features: dict) -> list[dict]:
        # Build the feature matrix in the exact order the model expects
        rows = []
        for c in candidates:
            feats = {**user_features, **c["item_features"]}
            rows.append([feats.get(name, np.nan) for name in self.feature_order])
 
        scores = self.model.predict(np.asarray(rows, dtype=np.float32))
 
        for c, p in zip(candidates, scores):
            # Blend conversion probability with expected margin for ranking
            c["p_convert"] = float(p)
            c["expected_value"] = float(p) * c["margin"]
 
        return sorted(candidates, key=lambda c: c["expected_value"], reverse=True)

Two details matter more than the model choice. First, feature_order is frozen at training time and reused at serving — reordering columns silently corrupts predictions. Second, ranking by p_convert * margin rather than raw probability is what ties the recommender to revenue. A high-probability, low-margin item shouldn't always win.

Serving and Latency

The recommender sits in the request path of a product surface, so latency is a feature. My target is single-digit to low-tens of milliseconds at the ranking stage. Practically that means: candidates capped at a few hundred, features fetched from Redis in one pipelined batch (never N round-trips), and the model kept small enough to score the whole batch in a few milliseconds.

# p99 is what users feel, not the mean
recommend_latency_ms{stage="generation",quantile="0.99"}  6.2
recommend_latency_ms{stage="ranking",quantile="0.99"}     11.8
recommend_latency_ms{stage="rerank",quantile="0.99"}       1.4
feature_store_hit_ratio                                    0.97

On a 3rd-party data platform I built, sustaining ~300 req/min at under 25% CPU and memory came down to exactly this discipline: batch the I/O, cap the work per request, and put rate limiting and circuit breakers in front of every external dependency so a slow upstream degrades gracefully instead of taking the whole surface down.

Offline vs Online Metrics

Offline, I look at AUC, PR-AUC, and ranking metrics like NDCG and recall-at-K. These are guardrails — a model that regresses badly offline rarely wins online, so they're a cheap filter. But they systematically lie about real impact because they're computed on logged data your old policy generated, with all its biases baked in.

The truth comes from online experiments. I run A/B tests, and where traffic allows, I automate allocation with a multi-armed bandit so winning variants get more traffic without a manual decision. The online metrics that decide ship-or-kill are the business ones: transaction rate, revenue per session, and downstream churn. A variant can lift click-through while flatlining transactions; I've killed models that looked great on every offline number because they nudged users toward cheap, easy clicks instead of purchases.

The loop that makes this work: log every recommendation with the features used and the model version, so you can reconstruct exactly what was served and attribute outcomes back to it. Without that logging you can compute online metrics, but you can't debug them, and a recommender you can't debug is one you'll eventually be afraid to change.

Takeaways

• Design around the decision and a revenue metric first; AUC is a guardrail, not the goal.
• Separate candidate generation (recall, cheap, content-based for cold start) from ranking (precision, propensity model on rich features).
• Invest in the feature pipeline: Airflow plus BigQuery for batch, Kafka for real-time, one shared transform library to kill training-serving skew.
• Rank by expected value (probability times margin), not raw probability, to connect the model to money.
• Treat latency as a product feature: cap candidates, batch your feature reads, and watch p99.
• Decide with online A/B tests and bandits on transaction uplift and churn; log every recommendation so you can attribute and debug.