Lecture 1 exercises and typo fixes in Lectures 1-3

Lecture1.ipynb

@@ -526,6 +526,18 @@
     "<small>For a normal distribution $y^2\\sim\\chi^2_{n-1}$ with $y=\\sqrt{n-1}s/\\sigma$ by [Cochran's theorem](https://en.wikipedia.org/wiki/Cochran%27s_theorem), so in principle it is possible to have an analytical expression for an unbiased estimator</small>"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Exercise: bias of maximum likelihood estimators\n",
+    "You measure $n$ decay times $t_i\\sim\\text{Exp}(\\lambda)$.\n",
+    "1. Show that the estimated mean lifetime $\\hat\\tau=\\bar t$ is unbiased.\n",
+    "2. Show that the estimated rate $\\hat\\lambda=1/\\bar t$ is biased: $E[\\hat\\lambda]=\\lambda\\,n/(n-1)$ (for $n>1$). Hint: $\\sum_i t_i\\sim\\text{Gamma}(n, 1/\\lambda)$.\n",
+    "3. Check the result with toys, for $n=5$ and $\\lambda=2$.\n",
+    "4. The maximum likelihood estimator is invariant under reparametrization ($\\hat\\lambda=1/\\hat\\tau$), but its bias is not."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -648,7 +660,7 @@
     "\n",
     "$$ H_0: M\\sim B(m| N, 1/2)$$\n",
     "\n",
-    "Compute the p-value of the data, assuming $H_0$ to be true. What \"more extreme\" means in this case? Since usually there are more male students, \"more extreme\" means \"more male students than observed\".\n",
+    "Compute the p-value of the data, assuming $H_0$ to be true. What does \"more extreme\" mean in this case? Since usually there are more male students, \"more extreme\" means \"more male students than observed\".\n",
     "\n",
     "$$\\text{p-value} = P(m\\geq m_{obs}|H_0) = \\sum_{m\\geq m_{obs}} B(m|N, 1/2)$$"
    ]
@@ -746,7 +758,7 @@
     }
    },
    "source": [
-    "What if all the students are female? Using the previous test we would find a very high p-value, and so we would not be able to reject the null hypothesis. The problem is that we are looking to the right tail (more males). In that case we should change our definition of \"more extreme\".\n",
+    "What if all the students are female? Using the previous test we would find a very high p-value, and so we would not be able to reject the null hypothesis. The problem is that we are looking at the right tail (more males). In that case we should change our definition of \"more extreme\".\n",
     "\n",
     "Another solution is to perform a two-tailed test. In general the definition is not unique. One common approach is to define \"more extreme\" as \"less probable\" under the null hypothesis. What is the probability of observing a number of males less probable than the one actually observed? Note that in this case we are counting as \"more extreme\" cases where there are too many males (as before) or too few males.\n",
     "\n",
@@ -840,6 +852,16 @@
     "What is the distribution of the p-value if the null hypothesis is true?"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Exercise: the distribution of the p-value\n",
+    "1. Generate $10^5$ gaussian toys under $H_0$ and histogram the one-tail p-value. What distribution do you expect?\n",
+    "2. Repeat for a Poisson counting experiment with $b=3$, using the p-value $P(n\\geq n_\\text{obs})$. Is the histogram uniform? Compute $P(\\text{p-value}\\leq 0.05)$ and $P(\\text{p-value}\\leq 0.01)$ and compare them with 5% and 1%.\n",
+    "3. Why is a test based on discrete counts conservative? What happens for $b=100$?"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -897,7 +919,7 @@
     "$$ z = \\text{SF}^{-1}(\\text{p-value})$$\n",
     "\n",
     "### Question\n",
-    "What is the significance if p-value $\\geq 0.5$? Note that in this case it means you are looking to the wrong tail."
+    "What is the significance if p-value $\\geq 0.5$? Note that in this case it means you are looking at the wrong tail."
    ]
   },
   {
@@ -1000,7 +1022,7 @@
     "$$p_0 = \\text{p-value for no signal} = \\sum_{n\\geq N_{obs}} P[n|H_0] =\\sum_{n\\geq N_{obs}} \\text{Poisson}[n|N_{bkg}^{exp}]$$\n",
     "\n",
     "#### Question\n",
-    "What \"more extreme\" means in this case? Why?\n",
+    "What does \"more extreme\" mean in this case? Why?\n",
     "If we observe $n\\ll N_{bkg}^{exp}$ will we observe a small p-value?"
    ]
   },
@@ -1332,6 +1354,18 @@
     "Derive the expression for the median expected significance and its approximation when $s\\ll b$. Hint: use likelihood ratio and then use gaussian approximation of the Poisson distribution."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Exercise: categorization pays off\n",
+    "Consider two categories: A with $s=10$, $b=100$ and B with $s=5$, $b=5$.\n",
+    "1. Compute the median expected significance $Z_A=\\sqrt{2((s+b)\\ln(1+s/b)-s)}$ for each category.\n",
+    "2. Compute $Z_A$ merging the two categories into a single counting experiment ($s=15$, $b=105$).\n",
+    "3. Compute the significance of the combination using a joint likelihood of the two categories, $\\sqrt{Z_A^2+Z_B^2}$, and compare it with the previous results.\n",
+    "4. Show that for the Asimov dataset $-2\\ln\\lambda$ is additive over the categories. This is the principle behind shape analyses and the BDT categories of analyses such as $H\\to\\gamma\\gamma$."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -2043,7 +2077,7 @@
    "source": [
     "# the sb model is not really needed in this case\n",
     "hypoCalc = RooStats.FrequentistCalculator(data, sbModel, bModel)\n",
-    "hypoCalc.SetToys(20000, 1000)  # generate much more toys for bkg-only\n",
+    "hypoCalc.SetToys(20000, 1000)  # generate many more toys for bkg-only\n",
     "\n",
     "toy_sampler = hypoCalc.GetTestStatSampler()\n",
     "toy_sampler.SetTestStatistic(test)  # our test statistic\n",
@@ -2234,7 +2268,7 @@
     }
    },
    "source": [
-    "Define the test statistic: $\\chi^2 = \\sum_{i,j}\\frac{(O_{i,j}-E_{i,j})^2}{E_{i,j}}$ where $O$ and $E$ are the observed and expected values, while the index $i,j$ identify a cell of the table. The test statistic $\\chi^2$ is approximately distributed as a $\\chi^2$ distribution with $(\\text{nrows} - 1)(\\text{ncolumns} - 1)=1$ degree of freedom."
+    "Define the test statistic: $\\chi^2 = \\sum_{i,j}\\frac{(O_{i,j}-E_{i,j})^2}{E_{i,j}}$ where $O$ and $E$ are the observed and expected values, while the indices $i,j$ identify a cell of the table. The test statistic $\\chi^2$ is approximately distributed as a $\\chi^2$ distribution with $(\\text{nrows} - 1)(\\text{ncolumns} - 1)=1$ degree of freedom."
    ]
   },
   {