Analysis Frequentist Approach — abexp 0.0.1 documentation

# Analysis Frequentist Approach

This tutorial shows how to perform post-test analysis of an A/B test experiment with two variants, so called control and treatment groups, using frequentist statistics. It handles both the case of means comparison and conversions comparison with closed-form-solutions. It assumes that sample data are normally distributed.

Let’s import first the tools needed.

    [1]:

    import numpy as np
    from abexp.core.analysis_frequentist import FrequentistAnalyzer
    from abexp.visualization.analysis_plots import AnalysisPlot

## Compare means

Here we want to compare the mean of the control group versus the mean of the treatment group given the sample observations.

    [2]:

    # Define the analyzer
    analyzer = FrequentistAnalyzer()

We will compare the average revenue per user of the control group versus the treatment group, making separate analysis for standard and premium users.

    [3]:

    # Revenue for standard users
    np.random.seed(42)
    revenueS_contr = np.random.normal(270, 200, 1000)
    revenueS_treat = np.random.normal(300, 200, 1000)

    # Revenue for premium users
    revenueP_contr = np.random.normal(300, 200, 1000)
    revenueP_treat = np.random.normal(310, 200, 1000)

    [4]:

    pval_S, ciS_contr, ciS_treat =  analyzer.compare_mean_obs(obs_contr=revenueS_contr,
                                                              obs_treat=revenueS_treat,
                                                              alpha=0.05)

    pval_P, ciP_contr, ciP_treat =  analyzer.compare_mean_obs(obs_contr=revenueP_contr,
                                                              obs_treat=revenueP_treat,
                                                              alpha=0.05)

    [5]:

    print('Standard users: p-value = {:.6f}'.format(pval_S))
    print('Premium  users: p-value = {:.6f}'.format(pval_P))

    Standard users: p-value = 0.000005
    Premium  users: p-value = 0.571544

If `p-value` \(\leq\)`0.05` the test result is statistically significant. There is a significative difference between control and treatment groups.

Otherwise if `p-value` \(>\) `0.05` the test result is not statistically significant. There is not a statistical significant difference between control and treatment groups.

    [6]:

    # Computer groups mean
    meanS_contr = np.mean(revenueS_contr)
    meanS_treat = np.mean(revenueS_treat)
    meanP_contr = np.mean(revenueP_contr)
    meanP_treat = np.mean(revenueP_treat)

Display test results in barplots.

    [42]:

    # Define height of the control group bars
    bars_contr = [meanS_contr, meanP_contr]

    # Define height of the treatment group bars
    bars_treat = [meanS_treat, meanP_treat]

    # Define upper and lower limit of the error bars for the control group
    ci_contr = [[ciS_contr[0], ciP_contr[0]],  #  2.5 percetiles
                [ciS_contr[1], ciP_contr[1]]]  # 97.5 percentiles

    # Define upper and lower limit of the error bars for the treatment group
    ci_treat = [[ciS_treat[0], ciP_treat[0]],  #  2.5 percetiles
                [ciS_treat[1], ciP_treat[1]]]  # 97.5 percentiles

    bars = [bars_contr, bars_treat]
    ci = [ci_contr, ci_treat]

    fig = AnalysisPlot.barplot(bars, ci, title='Barplot',
                               ylabel='average revenue per user',
                               xlabel=['standard', 'premium'],
                               groupslabel=['control', 'treatment'])

## Compare conversions

Here we want to compare the number of user that made a purchase in the control group versus the treatment group.

    [8]:

    # Number of users that made a purchase
    purchase_contr = 400
    purchase_treat = 470

    # Total number of users
    total_usr_treat = 5000
    total_usr_contr = 5000

    [9]:

    p_val, ci_contr, ci_treat = analyzer.compare_conv_stats(conv_contr=purchase_contr,
                                                            conv_treat=purchase_treat,
                                                            nobs_contr=total_usr_treat,
                                                            nobs_treat=total_usr_contr)

    [10]:

    print('p-value = {:.6f}'.format(p_val))

    p-value = 0.013002

In this case `p-value` \(\leq\)`0.05`, the test result is statistically significant. There is a significative difference between control and treatment groups. The treatment applied on the test group was successful.