top of page
Search

🔍 Step-by-Step Guide to Designing an A/B Test

  • Dr Dilek Celik
  • Jun 13
  • 2 min read

Updated: Jul 9

AB Testing Procedure flowchart with 7 steps: Problem Statement, Hypothesis Testing, Design, Run, Validity, Interpret Results, Launch Decision.
Procedure for A/B Testing

Step 1: Understand the Problem Statement in A/B Test

  • Business context: Online clothing store (Fashion Web Store) wants to test a new ranking algorithm to improve product relevance and sales.

  • User funnel:

    • User visits the website

    • User searches for an item

    • User browses results

    • User clicks on a product

    • User completes a purchase

Funnel diagram with stages: Visit, Search, Browse, View, and Purchase. Colors: blue, green, yellow, orange, red. Arrow shows rank change. A/B Testing
Defining user journey of the product demonstrates your product knowledge and create success metrics

Step 2: Define the Success Metric

To choose an appropriate success metric, it must be:

  • Measurable: Can you reliably collect data on it?

  • Attributable: Is the change in the metric caused by the treatment?

  • Sensitive: Can it detect small but important changes (low variability)?

  • Timely: Can you observe changes quickly (avoid long delays)?

✅ Chosen Metric: Revenue per user per day — a good proxy for long-term sales impact that’s also timely and attributable.

When developing success metrics, four principles are outlined with icons: 1. Measurable, 2. Attributable, 3. Sensitive, 4. Timely. Text poses questions about metrics.
When developing Success Metrics, consdier these princeiples

🧪 Step 3: Hypothesis Testing Framework in A/B Test

  • Null Hypothesis (H₀): There is no difference in average revenue per user per day between the new and old ranking algorithm.

  • Alternative Hypothesis (H₁): There is a difference between the two.

Set statistical parameters:

  • Significance Level (α): 0.05

  • Statistical Power (1 - β): 0.80

  • Minimum Detectable Effect (MDE): ~1% (common for large-scale tests)

Steps for hypothesis testing are outlined: stating hypotheses, setting alpha at 0.05, statistical power at 0.80, and minimum detectable effect. A/B Test.
Hypothesis Testing Steps

🛠️ Step 4: Design the Experiment in A/B Testing

  1. Randomization Unit: Individual users (not sessions or page views).

  2. Target Population: Users who start searching (as this is where the ranking algorithm applies).

  3. Sample Size:

    • Formula: n ≈ 16 × (σ² / Δ²)

    • Where:

      • σ² = variance of the metric (e.g., revenue per user)

      • Δ = minimum detectable difference

  4. Experiment Duration: Typically 1–2 weeks (avoid running for <1 week to capture weekday/weekend patterns).

    Steps for an experiment with a funnel diagram. Includes setting randomization units, population, sample size, and duration details.
    Design the Experiment

✅ Step 5: Run the Experiment in A/B Testing

  • Use appropriate instrumentation and tracking to collect metrics like search behavior, clicks, revenue, etc.

  • Ensure correct assignment logic and data logging.

    Line graph shows "Average Revenue per Day per User" for Treatment (orange) and Control (blue). Includes tips: set up data pipelines, avoid peeking p-values. A/B testing.
    Run the Experiment

⚠️ Step 6: Sanity Checks Before Analysis

  • Validate:

    • Proper randomization

    • Balance of user characteristics across groups

    • No major data quality or logging issues

This helps ensure any observed difference is due to the treatment — not confounding or bias.

Table listing four biases and checks: Instrumentation Effect, External Factors, Selection Bias, Sample Ratio Mismatch. Green headers. AB Testing.
Validity Checks

📊 Step 7: Analyze and Interpret Results

  • Check:

    • Lift (relative increase)

    • P-value

    • Confidence intervals

Also consider:

  • Practical vs. statistical significance

  • Impact on secondary metrics (e.g., bounce rate, conversion rate)

Table comparing control and treatment metrics with revenue differences and a significant p-value. Interpretation text below. AB Testing.
Interpret Results

🚀 Step 8: Launch Decision

  • Combine statistical results with business goals to decide:

    • Launch the new algorithm?

    • Iterate?

    • Rollback?

      Table on business decisions. Two rows: "Metric Trade-Offs" and "Cost of Launching." Key considerations and explanations provided.
      Launch Decisions

🔁 Pro Tip for Interviews:

Always start with business context and user journey before jumping into stats/design.

For company-specific roles (e.g., Meta, Google), analyze their core product’s user funnel in advance so you can adapt this framework confidently during interview A/B test questions.

Comments

Rated 0 out of 5 stars.
No ratings yet
Add a rating

machine learning shap
Data Scientist jobs

business analytics

bottom of page