Canonical URL: https://chargezen.com/docs/checkoutos/ab-testing/results

# Analyzing Results

How to interpret A/B test data and make decisions

Last updated: 2025-01-16

**Results analysis** is where you determine if your experiment produced meaningful insights. CheckoutOS provides comprehensive analytics to help you understand what happened and make confident decisions.

## Results Dashboard

The experiment results dashboard shows:

*   **Performance summary** — Key metrics for each variant
*   **Statistical significance** — Confidence in the results
*   **Lift calculation** — How much better/worse the variant performed
*   **Sample sizes** — Traffic received by each variant
*   **Trend graphs** — Performance over time

## Understanding Key Metrics

Results Summary

```json
{
  "experiment_results": {
    "status": "significant",
    "winner": "Variant A",
    "runtime_days": 14,
    "total_visitors": 12500,
    "variants": [
      {
        "name": "Control",
        "visitors": 6250,
        "conversions": 312,
        "conversion_rate": "4.99%",
        "revenue": 28750,
        "aov": 92.15
      },
      {
        "name": "Variant A",
        "visitors": 6250,
        "conversions": 387,
        "conversion_rate": "6.19%",
        "revenue": 35418,
        "aov": 91.52,
        "lift": "+24.0%",
        "p_value": 0.003,
        "confidence": "99.7%"
      }
    ]
  }
}
```

## Statistical Significance

Statistical significance tells you whether the observed difference is likely real or could be due to random chance:

#### Significance Indicators

*   95% confidence — Standard threshold for declaring a winner
*   99% confidence — High confidence, very unlikely to be random
*   P-value < 0.05 — Statistically significant at 95% level
*   Confidence interval — Range where true value likely falls

##### Avoid Peeking

Checking results repeatedly and stopping when you see significance inflates your false positive rate. Set a sample size target and stick to it.

## Interpreting Results

### Clear Winner

When results show statistical significance and meaningful lift:

1.  Document the winning variant and key learnings
2.  Roll out the winner to 100% of traffic
3.  Monitor post-rollout metrics to confirm
4.  Plan your next experiment

### No Significant Difference

When variants perform similarly:

*   This is still valuable — you learned the change doesn't matter
*   Consider if the test ran long enough
*   The simpler/cheaper option may be preferable
*   Document and move on to testing bigger changes

### Unexpected Results

When the variant performs worse than expected:

*   Don't dismiss negative results — they're valuable
*   Investigate why it didn't work
*   Check for implementation errors
*   Consider if you tested the right audience

## Segment Analysis

Break down results by customer segments to find hidden insights:

Segment Breakdown

```json
{
  "segment_analysis": {
    "by_device": {
      "mobile": { "control": "4.2%", "variant": "6.8%", "lift": "+62%" },
      "desktop": { "control": "5.5%", "variant": "5.4%", "lift": "-2%" }
    },
    "by_customer_type": {
      "new": { "control": "3.1%", "variant": "5.2%", "lift": "+68%" },
      "returning": { "control": "7.8%", "variant": "7.4%", "lift": "-5%" }
    },
    "by_cart_value": {
      "under_50": { "control": "5.1%", "variant": "6.2%", "lift": "+22%" },
      "50_to_100": { "control": "4.9%", "variant": "6.1%", "lift": "+24%" },
      "over_100": { "control": "4.8%", "variant": "6.3%", "lift": "+31%" }
    }
  }
}
```

##### Segment Insights

Sometimes an overall "no winner" hides a big win in a specific segment. Mobile users might love a change that desktop users hate. Segment analysis reveals these opportunities.

## Exporting Results

Export experiment data for further analysis:

#### Export Options

*   CSV export — Raw data for spreadsheet analysis
*   PDF report — Shareable summary document
*   API access — Programmatic data retrieval
*   Analytics integration — Send to Google Analytics, Mixpanel, etc.

## Concluding Experiments

1

#### Verify Significance

Ensure results have reached statistical significance.

2

#### Document Findings

Record hypothesis, results, and learnings.

3

#### Declare Winner

Choose winner or declare inconclusive.

4

#### Roll Out

Apply winning variant to 100% of traffic.

5

#### Monitor

Watch metrics post-rollout to confirm improvement holds.

## Best Practices

##### Analysis Tips

*   **Wait for significance** — Don't call winners too early
*   **Look at multiple metrics** — Conversion isn't everything
*   **Check segments** — Overall results may hide segment differences
*   **Consider business impact** — Small lift on high traffic = big value
*   **Document everything** — Build an experiment knowledge base

## Related Topics

[

### Analytics Dashboard

Track overall CheckoutOS performance

Learn more



](https://chargezen.com/docs/checkoutos/analytics/dashboard)[

### Creating Experiments

Set up your next A/B test

Learn more



](https://chargezen.com/docs/checkoutos/ab-testing/creating)

Was this page helpful?

Need more help? [Contact support](https://chargezen.com/docs/checkoutos/troubleshooting/support)