Multi-Armed Bandits in Production: When A/B Testing Wastes Your Traffic

Classic A/B testing is the right default until it isn't. The moment you have several variants, expensive traffic, and a strong opinion that one arm is winning, a fixed-horizon test starts to feel wasteful — you keep shipping the loser to half your users just to satisfy a sample-size calculator. I've replaced fixed A/B splits with bandits in recommendation and marketing flows more than once, and the win is real, but so are the footguns. Here's how I think about it.

Where fixed-horizon A/B testing wastes traffic

A standard A/B test commits to a sample size up front and splits traffic evenly until you hit it. That's a feature: it protects your statistical guarantees. But it's also the cost. If variant B is clearly better by day three of a fourteen-day test, you still serve A to half your users for eleven more days. With four or five variants, the waste compounds — every user routed to an obviously bad arm is a missed conversion you could have captured.

Bandits trade some statistical rigor for adaptive allocation. They shift traffic toward arms that look good as evidence accumulates, while still exploring enough to avoid locking in on a fluke. The formal objective is minimizing regret — the cumulative gap between the reward you got and the reward you'd have gotten by always playing the best arm.

A few situations where I reach for a bandit instead of A/B:

• Many variants, limited traffic. Banner creatives, pricing nudges, ranking strategies. With six arms, an even split is brutal.
• Continuous optimization, no fixed end date. You're not asking "did B beat A this quarter," you're asking "keep serving the best thing, forever."
• The cost of serving a bad arm is high. Churned users, lost transactions, ad spend.

Epsilon-greedy vs Thompson Sampling

The simplest bandit is epsilon-greedy: with probability 1 - epsilon serve the arm with the best observed mean, and with probability epsilon pick a random arm to keep exploring. It's trivial to implement and reason about. The downside is that exploration is dumb and uniform — once you've collected plenty of data, you're still wasting that epsilon fraction equally across obviously-bad arms, and a fixed epsilon never adapts to your certainty.

Thompson Sampling fixes that. Instead of point estimates, you keep a posterior distribution over each arm's reward rate and, on every request, draw one sample per arm and play the arm with the highest sample. When you're uncertain about an arm, its posterior is wide, so it occasionally samples high and gets explored. As evidence accumulates the posterior tightens, and bad arms naturally stop winning the draw. Exploration is proportional to uncertainty, which is exactly what you want — and there's nothing to tune like epsilon.

For binary rewards (click / no-click, convert / no-convert), the Beta-Bernoulli conjugate pair makes this almost embarrassingly cheap.

A Thompson Sampling snippet (Beta-Bernoulli)

import numpy as np
 
class BetaBernoulliBandit:
    """Thompson Sampling for binary rewards (click, convert, etc.)."""
 
    def __init__(self, arms, prior_alpha=1.0, prior_beta=1.0, rng=None):
        self.arms = list(arms)
        # Beta(1, 1) is a uniform prior: no opinion before we see data.
        self.alpha = {a: prior_alpha for a in self.arms}
        self.beta = {a: prior_beta for a in self.arms}
        self.rng = rng or np.random.default_rng()
 
    def select(self):
        # Draw one sample from each arm's posterior; play the best draw.
        samples = {a: self.rng.beta(self.alpha[a], self.beta[a]) for a in self.arms}
        return max(samples, key=samples.get)
 
    def update(self, arm, reward):
        # reward is 0 or 1. Conjugacy makes the update a single increment.
        self.alpha[arm] += reward
        self.beta[arm] += 1 - reward
 
    def estimates(self):
        # Posterior mean per arm, handy for dashboards.
        return {a: self.alpha[a] / (self.alpha[a] + self.beta[a]) for a in self.arms}
 
 
bandit = BetaBernoulliBandit(["control", "variant_a", "variant_b"])
arm = bandit.select()         # called per request
# ... serve `arm`, observe outcome later ...
bandit.update(arm, reward=1)  # 1 if the user converted, else 0

In production I don't keep this in memory. The alpha and beta counters live in Redis (one hash per experiment), the select call is a few microseconds, and update is an atomic increment. That's the whole point: the decision is cheap enough to run on every request without touching your latency budget.

Delayed rewards

The snippet above pretends the reward shows up immediately. It rarely does. A user clicks now and converts forty minutes later — or never. If you update only on conversion, your counters silently undercount the denominator and the math drifts.

The fix is to separate the decision log from the reward join. Log every select with a request id and the chosen arm. When a reward event arrives, join it back by request id and apply the update. For rewards that may never come, you need an attribution window — say, 24 hours — after which an unconverted impression counts as a reward=0. I run this as a scheduled job: Airflow sweeps the decision log, joins against the conversion table in BigQuery, and emits (arm, reward) pairs that get folded into the posteriors. The bandit serving online stays fast; the bookkeeping happens out of band.

Non-stationarity

Beta-Bernoulli assumes each arm has a fixed conversion rate. Real traffic doesn't. Weekday versus weekend, a new campaign, seasonal demand — the best arm last month may be mediocre today. A vanilla bandit that accumulated millions of observations becomes overconfident and effectively stops exploring, so it can't notice the world changed.

The standard remedy is to decay old evidence. Before each update, pull the counters slightly back toward the prior:

def decayed_update(self, arm, reward, gamma=0.999):
    # gamma < 1 forgets old data so the bandit can react to drift.
    a, b = self.alpha[arm], self.beta[arm]
    self.alpha[arm] = gamma * (a - 1) + 1 + reward
    self.beta[arm]  = gamma * (b - 1) + 1 + (1 - reward)

Pick gamma based on how fast your environment moves. Closer to 1 means longer memory and more stability; lower means faster adaptation and more churn. This is the bandit analogue of the windowing I used on dynamic-pricing models, where last quarter's elasticity genuinely stopped applying.

Guardrails

Adaptive allocation cuts both ways: a bad arm with an early lucky streak can grab traffic fast, and a broken arm can do real damage before the posterior recovers. The guardrails I always wire in:

• Floor allocation. Force a minimum share (e.g. 2-5%) to every live arm, including control, so you keep a clean reference and never fully starve an arm.
• Kill switch on hard metrics. Watch latency, error rate, and a business floor (revenue per session, refund rate). If an arm breaches a threshold, evict it from the candidate set automatically. This is the same circuit-breaker thinking I leaned on for our 3rd-party data platform.
• Cap the change rate. Clamp how fast allocation can swing between snapshots so a noisy hour can't reroute everything.
• Hold out a control. A small, fixed control slice gives you an honest read on lift after the fact — bandit estimates are biased by the adaptive sampling itself.

Wiring it into a recommendation / marketing flow

A bandit slots in as a thin policy layer between candidate generation and serving. It doesn't pick items; it picks strategies — a ranking model, a pricing rule, a creative.

# Pseudocode for the request path.
def serve_recommendations(user, context):
    strategy = bandit.select()                 # e.g. "propensity_v3" vs "content_v2"
    items = STRATEGIES[strategy].rank(user, context)
    log_decision(request_id, user.id, strategy)  # for the delayed reward join
    return items
 
# Asynchronously, when a conversion lands:
def on_conversion(request_id):
    decision = lookup_decision(request_id)
    if within_attribution_window(decision):
        bandit.update(decision.strategy, reward=1)

In a marketing flow the "arm" is a message variant or send-time policy and the reward is open or click. Same skeleton. The hard parts are never the bandit math — they're the decision log, the reward join, and the guardrails.

A short note on evaluation

You can't A/B-test the bandit the way you'd test a single variant, because the policy changes the data distribution as it runs. Two things I rely on:

• Cumulative regret / reward over time. Plot reward-per-session for the bandit against a hypothetical even-split baseline. The gap is your win.
• Offline replay with off-policy estimators. Before going live, replay logged data through estimators like inverse propensity scoring to sanity-check that the policy would have beaten the incumbent. It's noisy, but it catches obvious regressions cheaply.

And keep that small fixed control. When someone asks "how much did this actually lift conversions," the holdout — not the bandit's own posteriors — is the number you can defend.

Takeaways

• Use bandits when you have many variants, expensive traffic, and continuous optimization; use fixed-horizon A/B when you need a defensible causal claim.
• Thompson Sampling beats epsilon-greedy in practice: exploration scales with uncertainty and there's no epsilon to tune.
• Beta-Bernoulli Thompson Sampling is a few lines of code; the real work is the decision log and reward join, not the math.
• Handle delayed rewards with an attribution window and an out-of-band join; handle non-stationarity by decaying old evidence.
• Wire in floor allocation, a kill switch on hard metrics, and a small fixed control before you let allocation move on its own.
• Evaluate with cumulative reward over time and a held-out control — not the bandit's own (biased) posteriors.