{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis Frequentist Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial shows how to perform post-test analysis of an A/B test experiment with two variants, so called control and\n",
    "treatment groups, using frequentist statistics. It handles both the case of means comparison and conversions comparison \n",
    "with closed-form-solutions. It assumes that sample data are normally distributed.\n",
    "\n",
    "Let's import first the tools needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from abexp.core.analysis_frequentist import FrequentistAnalyzer\n",
    "from abexp.visualization.analysis_plots import AnalysisPlot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we want to compare the mean of the control group versus the mean of the treatment group given the sample \n",
    "observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the analyzer\n",
    "analyzer = FrequentistAnalyzer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will compare the *average revenue per user*  of the control group versus the treatment group, making separate \n",
    "analysis for standard and premium users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Revenue for standard users\n",
    "np.random.seed(42)\n",
    "revenueS_contr = np.random.normal(270, 200, 1000)\n",
    "revenueS_treat = np.random.normal(300, 200, 1000)\n",
    "\n",
    "# Revenue for premium users\n",
    "revenueP_contr = np.random.normal(300, 200, 1000)\n",
    "revenueP_treat = np.random.normal(310, 200, 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pval_S, ciS_contr, ciS_treat =  analyzer.compare_mean_obs(obs_contr=revenueS_contr, \n",
    "                                                          obs_treat=revenueS_treat, \n",
    "                                                          alpha=0.05)\n",
    "\n",
    "pval_P, ciP_contr, ciP_treat =  analyzer.compare_mean_obs(obs_contr=revenueP_contr, \n",
    "                                                          obs_treat=revenueP_treat, \n",
    "                                                          alpha=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Standard users: p-value = 0.000005\n",
      "Premium  users: p-value = 0.571544\n"
     ]
    }
   ],
   "source": [
    "print('Standard users: p-value = {:.6f}'.format(pval_S))\n",
    "print('Premium  users: p-value = {:.6f}'.format(pval_P))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If ``p-value`` $\\leq$``0.05`` the test result is statistically significant. There is a significative difference between \n",
    "control and treatment groups. \n",
    "\n",
    "Otherwise if ``p-value`` $>$ ``0.05`` the test result is not statistically significant. There is not a statistical \n",
    "significant difference between control and treatment groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Computer groups mean\n",
    "meanS_contr = np.mean(revenueS_contr)\n",
    "meanS_treat = np.mean(revenueS_treat)\n",
    "meanP_contr = np.mean(revenueP_contr)\n",
    "meanP_treat = np.mean(revenueP_treat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display test results in barplots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18060 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Define height of the control group bars\n",
    "bars_contr = [meanS_contr, meanP_contr]\n",
    " \n",
    "# Define height of the treatment group bars\n",
    "bars_treat = [meanS_treat, meanP_treat]\n",
    " \n",
    "# Define upper and lower limit of the error bars for the control group\n",
    "ci_contr = [[ciS_contr[0], ciP_contr[0]],  #  2.5 percetiles\n",
    "            [ciS_contr[1], ciP_contr[1]]]  # 97.5 percentiles\n",
    "\n",
    "# Define upper and lower limit of the error bars for the treatment group\n",
    "ci_treat = [[ciS_treat[0], ciP_treat[0]],  #  2.5 percetiles\n",
    "            [ciS_treat[1], ciP_treat[1]]]  # 97.5 percentiles\n",
    "\n",
    "bars = [bars_contr, bars_treat]\n",
    "ci = [ci_contr, ci_treat]\n",
    "\n",
    "fig = AnalysisPlot.barplot(bars, ci, title='Barplot',\n",
    "                           ylabel='average revenue per user',\n",
    "                           xlabel=['standard', 'premium'],\n",
    "                           groupslabel=['control', 'treatment'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare conversions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we want to compare the number of user that made a purchase in the control group versus the treatment group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of users that made a purchase\n",
    "purchase_contr = 400\n",
    "purchase_treat = 470\n",
    "\n",
    "# Total number of users\n",
    "total_usr_treat = 5000\n",
    "total_usr_contr = 5000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_val, ci_contr, ci_treat = analyzer.compare_conv_stats(conv_contr=purchase_contr,\n",
    "                                                        conv_treat=purchase_treat,\n",
    "                                                        nobs_contr=total_usr_treat,\n",
    "                                                        nobs_treat=total_usr_contr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p-value = 0.013002\n"
     ]
    }
   ],
   "source": [
    "print('p-value = {:.6f}'.format(p_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case ``p-value`` $\\leq$``0.05``, the test result is statistically significant. There is a significative \n",
    "difference between control and treatment groups. The treatment applied on the test group was successful."
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
