{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis Bootstrap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial shows how to perform analysis after A/B test experiments using bootstrapping. This technique makes \n",
    "inference about a certain estimate (e.g. sample mean) for a certain population parameter (e.g. population mean) by \n",
    "resampling with replacement from the observed dataset. It does not make any assumption on the samples distribution.\n",
    "\n",
    "Let's import first the tools needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from abexp.core.analysis_frequentist import FrequentistAnalyzer\n",
    "from abexp.visualization.analysis_plots import AnalysisPlot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple bootstrap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we want to compare a specific metrics of the control group versus the treatment group (e.g. average revenue per \n",
    "user). We will perform bootstrapping on the kpi metrics (revenue) of each group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate random data for revenue control group\n",
    "revenue_contr = np.random.randint(low=50, high=500, size=100)\n",
    "\n",
    "# Generate random data for revenue treatment group\n",
    "revenue_treat = np.random.randint(low=50, high=600, size=100)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the analyzer\n",
    "analyzer = FrequentistAnalyzer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the aggregation function that will be applied on the sample\n",
    "aggregation_func = np.mean\n",
    "\n",
    "# other possibles aggregation functions might be:\n",
    "#  - standard deviation = np.std, \n",
    "#  - sum                = np.sum\n",
    "#  - median             = lambda x: np.median(x, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bootstrapping will generate a sequence of ``N`` values (where ``N`` is the number of repetitions). The bootstrap \n",
    "function returns a table with the median, 2.5 percentile and 97.5 percentile of this sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>median</th>\n",
       "      <th>2.5 percentile</th>\n",
       "      <th>97.5 percentile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>282.475</td>\n",
       "      <td>255.092</td>\n",
       "      <td>307.1285</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   median  2.5 percentile  97.5 percentile\n",
       "  282.475         255.092         307.1285"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform bootstrapping on the control group\n",
    "stats_contr = analyzer.bootstrap(revenue_contr, func=aggregation_func, rep=500)\n",
    "stats_contr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>median</th>\n",
       "      <th>2.5 percentile</th>\n",
       "      <th>97.5 percentile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <td>305.5</td>\n",
       "      <td>276.192</td>\n",
       "      <td>339.1205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  median  2.5 percentile  97.5 percentile\n",
       "   305.5         276.192         339.1205"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform bootstrapping on the treatment group\n",
    "stats_treat = analyzer.bootstrap(revenue_treat, func=aggregation_func, rep=500)\n",
    "stats_treat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define heights of the bars\n",
    "bars = [stats_contr['median'], stats_treat['median']]\n",
    "\n",
    "# Compute the error between median and percentiles\n",
    "ci_contr = [stats_contr['2.5 percentile'], \n",
    "            stats_contr['97.5 percentile']]\n",
    "\n",
    "ci_treat = [stats_treat['2.5 percentile'], \n",
    "            stats_treat['97.5 percentile']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 17900 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot results with confidence interval\n",
    "fig = AnalysisPlot.barplot(bars, [ci_contr, ci_treat], \n",
    "                           groupslabel=['control group', 'treatment group'], \n",
    "                           ylabel='average revenue per user', xlabel='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the barplot above we see that there is no difference between empirical means because the confidence intervals \n",
    "overlap. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time series bootstrap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we want to compare a specific metrics of the control group versus the treatment group (e.g. average revenue per \n",
    "user) across time. We will perform bootstrapping on the kpi metrics (revenue) of each group per each day. Note that the \n",
    "bootstrap function maintains the correlation across days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate random data for revenue control group\n",
    "revenue_contr_ts = pd.DataFrame({'day1': np.random.randint(low=1, high=500, size=1000),\n",
    "                                 'day2': np.random.randint(low=1, high=500, size=1000),\n",
    "                                 'day3': np.random.randint(low=1, high=500, size=1000),\n",
    "                                 'day4': np.random.randint(low=1, high=500, size=1000),\n",
    "                                 'day5': np.random.randint(low=1, high=500, size=1000),\n",
    "                                 'day6': np.random.randint(low=1, high=500, size=1000),\n",
    "                                 'day7': np.random.randint(low=1, high=500, size=1000)})\n",
    "\n",
    "# Generate random data for revenue treatment group\n",
    "revenue_treat_ts = pd.DataFrame({'day1': np.random.randint(low=1, high=600, size=1000),\n",
    "                                 'day2': np.random.randint(low=1, high=600, size=1000),\n",
    "                                 'day3': np.random.randint(low=1, high=600, size=1000),\n",
    "                                 'day4': np.random.randint(low=1, high=600, size=1000),\n",
    "                                 'day5': np.random.randint(low=1, high=600, size=1000),\n",
    "                                 'day6': np.random.randint(low=1, high=600, size=1000),\n",
    "                                 'day7': np.random.randint(low=1, high=600, size=1000)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>median</th>\n",
       "      <th>2.5 percentile</th>\n",
       "      <th>97.5 percentile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>day1</th>\n",
       "      <td>246.6630</td>\n",
       "      <td>237.597475</td>\n",
       "      <td>255.912150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day2</th>\n",
       "      <td>248.0410</td>\n",
       "      <td>239.164575</td>\n",
       "      <td>256.451925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day3</th>\n",
       "      <td>250.4535</td>\n",
       "      <td>241.068275</td>\n",
       "      <td>259.431525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day4</th>\n",
       "      <td>252.0625</td>\n",
       "      <td>244.145850</td>\n",
       "      <td>261.326350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day5</th>\n",
       "      <td>246.4465</td>\n",
       "      <td>237.647800</td>\n",
       "      <td>255.376150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day6</th>\n",
       "      <td>252.0445</td>\n",
       "      <td>243.933075</td>\n",
       "      <td>261.501475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day7</th>\n",
       "      <td>249.0605</td>\n",
       "      <td>240.451950</td>\n",
       "      <td>257.918600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        median  2.5 percentile  97.5 percentile\n",
       "day1  246.6630      237.597475       255.912150\n",
       "day2  248.0410      239.164575       256.451925\n",
       "day3  250.4535      241.068275       259.431525\n",
       "day4  252.0625      244.145850       261.326350\n",
       "day5  246.4465      237.647800       255.376150\n",
       "day6  252.0445      243.933075       261.501475\n",
       "day7  249.0605      240.451950       257.918600"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform bootstrapping on the control group\n",
    "stats_contr_ts = analyzer.bootstrap(revenue_contr_ts, func=aggregation_func, rep=500)\n",
    "stats_contr_ts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>median</th>\n",
       "      <th>2.5 percentile</th>\n",
       "      <th>97.5 percentile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>day1</th>\n",
       "      <td>305.8540</td>\n",
       "      <td>295.303525</td>\n",
       "      <td>315.689025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day2</th>\n",
       "      <td>297.1785</td>\n",
       "      <td>287.122225</td>\n",
       "      <td>308.437950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day3</th>\n",
       "      <td>311.1690</td>\n",
       "      <td>300.329400</td>\n",
       "      <td>322.258075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day4</th>\n",
       "      <td>297.0245</td>\n",
       "      <td>286.180500</td>\n",
       "      <td>307.432525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day5</th>\n",
       "      <td>302.7850</td>\n",
       "      <td>292.530300</td>\n",
       "      <td>313.642875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day6</th>\n",
       "      <td>300.4425</td>\n",
       "      <td>289.563675</td>\n",
       "      <td>311.556775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day7</th>\n",
       "      <td>299.9155</td>\n",
       "      <td>288.822375</td>\n",
       "      <td>310.642600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        median  2.5 percentile  97.5 percentile\n",
       "day1  305.8540      295.303525       315.689025\n",
       "day2  297.1785      287.122225       308.437950\n",
       "day3  311.1690      300.329400       322.258075\n",
       "day4  297.0245      286.180500       307.432525\n",
       "day5  302.7850      292.530300       313.642875\n",
       "day6  300.4425      289.563675       311.556775\n",
       "day7  299.9155      288.822375       310.642600"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform bootstrapping on the treatment group\n",
    "stats_treat_ts = analyzer.bootstrap(revenue_treat_ts, func=aggregation_func, rep=500)\n",
    "stats_treat_ts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Plot results with confidence intervals__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define heights of the bars\n",
    "y = [stats_contr_ts['median'], stats_treat_ts['median']]\n",
    "\n",
    "# Compute the error between median and percentiles\n",
    "ci_treat_ts = [stats_treat_ts['median'] - stats_treat_ts['2.5 percentile'],\n",
    "               stats_treat_ts['97.5 percentile'] - stats_treat_ts['median']]\n",
    "ci_contr_ts = [stats_contr_ts['median'] - stats_contr_ts['2.5 percentile'],\n",
    "               stats_contr_ts['97.5 percentile'] - stats_contr_ts['median']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32900 characters]\n",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = AnalysisPlot.timeseries_plot(y, [ci_contr_ts, ci_treat_ts])"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
