Analyzing Results

Results analysis is where you determine if your experiment produced meaningful insights. CheckoutOS provides comprehensive analytics to help you understand what happened and make confident decisions.

Results Dashboard

The experiment results dashboard shows:

Performance summary — Key metrics for each variant
Statistical significance — Confidence in the results
Lift calculation — How much better/worse the variant performed
Sample sizes — Traffic received by each variant
Trend graphs — Performance over time

Understanding Key Metrics

Results Summary

{
  "experiment_results": {
    "status": "significant",
    "winner": "Variant A",
    "runtime_days": 14,
    "total_visitors": 12500,
    "variants": [
      {
        "name": "Control",
        "visitors": 6250,
        "conversions": 312,
        "conversion_rate": "4.99%",
        "revenue": 28750,
        "aov": 92.15
      },
      {
        "name": "Variant A",
        "visitors": 6250,
        "conversions": 387,
        "conversion_rate": "6.19%",
        "revenue": 35418,
        "aov": 91.52,
        "lift": "+24.0%",
        "p_value": 0.003,
        "confidence": "99.7%"
      }
    ]
  }
}

Statistical Significance

Statistical significance tells you whether the observed difference is likely real or could be due to random chance:

Significance Indicators

95% confidence — Standard threshold for declaring a winner
99% confidence — High confidence, very unlikely to be random
P-value < 0.05 — Statistically significant at 95% level
Confidence interval — Range where true value likely falls

Avoid Peeking

Checking results repeatedly and stopping when you see significance inflates your false positive rate. Set a sample size target and stick to it.

Interpreting Results

Clear Winner

When results show statistical significance and meaningful lift:

Document the winning variant and key learnings
Roll out the winner to 100% of traffic
Monitor post-rollout metrics to confirm
Plan your next experiment

No Significant Difference

When variants perform similarly:

This is still valuable — you learned the change doesn't matter
Consider if the test ran long enough
The simpler/cheaper option may be preferable
Document and move on to testing bigger changes

Unexpected Results

When the variant performs worse than expected:

Don't dismiss negative results — they're valuable
Investigate why it didn't work
Check for implementation errors
Consider if you tested the right audience

Segment Analysis

Break down results by customer segments to find hidden insights:

Segment Breakdown

{
  "segment_analysis": {
    "by_device": {
      "mobile": { "control": "4.2%", "variant": "6.8%", "lift": "+62%" },
      "desktop": { "control": "5.5%", "variant": "5.4%", "lift": "-2%" }
    },
    "by_customer_type": {
      "new": { "control": "3.1%", "variant": "5.2%", "lift": "+68%" },
      "returning": { "control": "7.8%", "variant": "7.4%", "lift": "-5%" }
    },
    "by_cart_value": {
      "under_50": { "control": "5.1%", "variant": "6.2%", "lift": "+22%" },
      "50_to_100": { "control": "4.9%", "variant": "6.1%", "lift": "+24%" },
      "over_100": { "control": "4.8%", "variant": "6.3%", "lift": "+31%" }
    }
  }
}

Segment Insights

Sometimes an overall "no winner" hides a big win in a specific segment. Mobile users might love a change that desktop users hate. Segment analysis reveals these opportunities.

Exporting Results

Export experiment data for further analysis:

Export Options

CSV export — Raw data for spreadsheet analysis
PDF report — Shareable summary document
API access — Programmatic data retrieval
Analytics integration — Send to Google Analytics, Mixpanel, etc.

Concluding Experiments

Verify Significance

Ensure results have reached statistical significance.

Document Findings

Record hypothesis, results, and learnings.

Declare Winner

Choose winner or declare inconclusive.

Roll Out

Apply winning variant to 100% of traffic.

Monitor

Watch metrics post-rollout to confirm improvement holds.

Best Practices

Analysis Tips

Wait for significance — Don't call winners too early
Look at multiple metrics — Conversion isn't everything
Check segments — Overall results may hide segment differences
Consider business impact — Small lift on high traffic = big value
Document everything — Build an experiment knowledge base