Creating Experiments

Creating an A/B test in CheckoutOS is straightforward. This guide walks you through setting up experiments from hypothesis to launch.

Before You Start

Pre-Test Checklist

Define your hypothesis — What do you expect to happen?
Choose one variable — Test a single change at a time
Set success metrics — What will you measure?
Estimate sample size — How long will the test need to run?
Get stakeholder buy-in — Ensure everyone agrees on the test

Creating an Experiment

Navigate to A/B Testing

Go to CheckoutOS → A/B Testing → Create Experiment.

Name Your Experiment

Use a descriptive name like "Upsell Headline Test - Jan 2025".

Select What to Test

Choose the element: Upsell offer, checkout extension, bundle, etc.

Create Variants

Set up your control (current) and variant (new) experiences.

Configure Traffic Split

Decide how to divide visitors between variants.

Set Success Metric

Choose your primary metric (conversion, AOV, etc.).

Launch

Review settings and start the experiment.

Experiment Types

Upsell Offer Test

Test different upsell products, discounts, or messaging:

Upsell A/B Test

{
  "experiment": {
    "name": "Premium Product Upsell Test",
    "type": "upsell_offer",
    "variants": [
      {
        "name": "Control",
        "weight": 50,
        "offer": {
          "product_id": "standard-upsell",
          "discount": "15%",
          "headline": "Add this to your order"
        }
      },
      {
        "name": "Premium Offer",
        "weight": 50,
        "offer": {
          "product_id": "premium-upsell",
          "discount": "20%",
          "headline": "Exclusive: Save 20% on this upgrade"
        }
      }
    ],
    "success_metric": "upsell_acceptance_rate"
  }
}

Pricing Test

Test different discount amounts or pricing strategies:

Pricing A/B Test

{
  "experiment": {
    "name": "Discount Format Test",
    "type": "pricing",
    "variants": [
      {
        "name": "Percentage",
        "weight": 50,
        "discount": {
          "type": "percentage",
          "value": 20,
          "display": "20% OFF"
        }
      },
      {
        "name": "Fixed Amount",
        "weight": 50,
        "discount": {
          "type": "fixed",
          "value": 10,
          "display": "$10 OFF"
        }
      }
    ],
    "success_metric": "revenue_per_visitor"
  }
}

Placement Test

Test where offers appear in the checkout flow:

Placement A/B Test

{
  "experiment": {
    "name": "Upsell Placement Test",
    "type": "placement",
    "variants": [
      {
        "name": "Post-Purchase",
        "weight": 50,
        "placement": "post_purchase_page"
      },
      {
        "name": "Thank You Page",
        "weight": 50,
        "placement": "thank_you_page"
      }
    ],
    "success_metric": "conversion_rate"
  }
}

Configuring Variants

Each variant can customize:

Variant Options

Product or offer shown
Discount amount and type
Headline and description text
Button text and color
Image or media
Placement location
Timing (when offer appears)

Keep Variants Distinct

Make sure your variants are meaningfully different. Testing a 14% discount vs 15% discount won't give you actionable insights.

Multivariate Testing

For advanced users, CheckoutOS supports multivariate testing where you test multiple variables simultaneously:

Multivariate Test

{
  "experiment": {
    "name": "Headline + Discount MVT",
    "type": "multivariate",
    "factors": [
      {
        "name": "headline",
        "levels": ["Save Big!", "Exclusive Offer", "Don't Miss Out"]
      },
      {
        "name": "discount",
        "levels": ["10%", "15%", "20%"]
      }
    ],
    "success_metric": "upsell_acceptance_rate"
  }
}

Sample Size Warning

Multivariate tests require much larger sample sizes. The example above has 9 combinations (3 x 3), requiring 9x the traffic for statistical significance.

Scheduling Experiments

Control when experiments run:

Scheduled Experiment

{
  "schedule": {
    "start_date": "2025-02-01T00:00:00Z",
    "end_date": "2025-02-28T23:59:59Z",
    "auto_conclude": true,
    "minimum_runtime_days": 14,
    "minimum_conversions": 500
  }
}

Best Practices

Experiment Setup Tips

Write a hypothesis first — "We believe X will improve Y by Z%"
Use descriptive names — Future you will thank you
Start with 50/50 splits — Equal traffic for fastest results
Set minimum runtime — Avoid peeking and stopping early
Document everything — Record your reasoning and results