diff --git a/Lecture1.ipynb b/Lecture1.ipynb
index 387818e..1a3ffb2 100755
--- a/Lecture1.ipynb
+++ b/Lecture1.ipynb
@@ -526,6 +526,18 @@
"For a normal distribution $y^2\\sim\\chi^2_{n-1}$ with $y=\\sqrt{n-1}s/\\sigma$ by [Cochran's theorem](https://en.wikipedia.org/wiki/Cochran%27s_theorem), so in principle it is possible to have an analytical expression for an unbiased estimator"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Exercise: bias of maximum likelihood estimators\n",
+ "You measure $n$ decay times $t_i\\sim\\text{Exp}(\\lambda)$.\n",
+ "1. Show that the estimated mean lifetime $\\hat\\tau=\\bar t$ is unbiased.\n",
+ "2. Show that the estimated rate $\\hat\\lambda=1/\\bar t$ is biased: $E[\\hat\\lambda]=\\lambda\\,n/(n-1)$ (for $n>1$). Hint: $\\sum_i t_i\\sim\\text{Gamma}(n, 1/\\lambda)$.\n",
+ "3. Check the result with toys, for $n=5$ and $\\lambda=2$.\n",
+ "4. The maximum likelihood estimator is invariant under reparametrization ($\\hat\\lambda=1/\\hat\\tau$), but its bias is not."
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {
@@ -648,7 +660,7 @@
"\n",
"$$ H_0: M\\sim B(m| N, 1/2)$$\n",
"\n",
- "Compute the p-value of the data, assuming $H_0$ to be true. What \"more extreme\" means in this case? Since usually there are more male students, \"more extreme\" means \"more male students than observed\".\n",
+ "Compute the p-value of the data, assuming $H_0$ to be true. What does \"more extreme\" mean in this case? Since usually there are more male students, \"more extreme\" means \"more male students than observed\".\n",
"\n",
"$$\\text{p-value} = P(m\\geq m_{obs}|H_0) = \\sum_{m\\geq m_{obs}} B(m|N, 1/2)$$"
]
@@ -746,7 +758,7 @@
}
},
"source": [
- "What if all the students are female? Using the previous test we would find a very high p-value, and so we would not be able to reject the null hypothesis. The problem is that we are looking to the right tail (more males). In that case we should change our definition of \"more extreme\".\n",
+ "What if all the students are female? Using the previous test we would find a very high p-value, and so we would not be able to reject the null hypothesis. The problem is that we are looking at the right tail (more males). In that case we should change our definition of \"more extreme\".\n",
"\n",
"Another solution is to perform a two-tailed test. In general the definition is not unique. One common approach is to define \"more extreme\" as \"less probable\" under the null hypothesis. What is the probability of observing a number of males less probable than the one actually observed? Note that in this case we are counting as \"more extreme\" cases where there are too many males (as before) or too few males.\n",
"\n",
@@ -840,6 +852,16 @@
"What is the distribution of the p-value if the null hypothesis is true?"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Exercise: the distribution of the p-value\n",
+ "1. Generate $10^5$ gaussian toys under $H_0$ and histogram the one-tail p-value. What distribution do you expect?\n",
+ "2. Repeat for a Poisson counting experiment with $b=3$, using the p-value $P(n\\geq n_\\text{obs})$. Is the histogram uniform? Compute $P(\\text{p-value}\\leq 0.05)$ and $P(\\text{p-value}\\leq 0.01)$ and compare them with 5% and 1%.\n",
+ "3. Why is a test based on discrete counts conservative? What happens for $b=100$?"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {
@@ -897,7 +919,7 @@
"$$ z = \\text{SF}^{-1}(\\text{p-value})$$\n",
"\n",
"### Question\n",
- "What is the significance if p-value $\\geq 0.5$? Note that in this case it means you are looking to the wrong tail."
+ "What is the significance if p-value $\\geq 0.5$? Note that in this case it means you are looking at the wrong tail."
]
},
{
@@ -1000,7 +1022,7 @@
"$$p_0 = \\text{p-value for no signal} = \\sum_{n\\geq N_{obs}} P[n|H_0] =\\sum_{n\\geq N_{obs}} \\text{Poisson}[n|N_{bkg}^{exp}]$$\n",
"\n",
"#### Question\n",
- "What \"more extreme\" means in this case? Why?\n",
+ "What does \"more extreme\" mean in this case? Why?\n",
"If we observe $n\\ll N_{bkg}^{exp}$ will we observe a small p-value?"
]
},
@@ -1332,6 +1354,18 @@
"Derive the expression for the median expected significance and its approximation when $s\\ll b$. Hint: use likelihood ratio and then use gaussian approximation of the Poisson distribution."
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Exercise: categorization pays off\n",
+ "Consider two categories: A with $s=10$, $b=100$ and B with $s=5$, $b=5$.\n",
+ "1. Compute the median expected significance $Z_A=\\sqrt{2((s+b)\\ln(1+s/b)-s)}$ for each category.\n",
+ "2. Compute $Z_A$ merging the two categories into a single counting experiment ($s=15$, $b=105$).\n",
+ "3. Compute the significance of the combination using a joint likelihood of the two categories, $\\sqrt{Z_A^2+Z_B^2}$, and compare it with the previous results.\n",
+ "4. Show that for the Asimov dataset $-2\\ln\\lambda$ is additive over the categories. This is the principle behind shape analyses and the BDT categories of analyses such as $H\\to\\gamma\\gamma$."
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {
@@ -2043,7 +2077,7 @@
"source": [
"# the sb model is not really needed in this case\n",
"hypoCalc = RooStats.FrequentistCalculator(data, sbModel, bModel)\n",
- "hypoCalc.SetToys(20000, 1000) # generate much more toys for bkg-only\n",
+ "hypoCalc.SetToys(20000, 1000) # generate many more toys for bkg-only\n",
"\n",
"toy_sampler = hypoCalc.GetTestStatSampler()\n",
"toy_sampler.SetTestStatistic(test) # our test statistic\n",
@@ -2234,7 +2268,7 @@
}
},
"source": [
- "Define the test statistic: $\\chi^2 = \\sum_{i,j}\\frac{(O_{i,j}-E_{i,j})^2}{E_{i,j}}$ where $O$ and $E$ are the observed and expected values, while the index $i,j$ identify a cell of the table. The test statistic $\\chi^2$ is approximately distributed as a $\\chi^2$ distribution with $(\\text{nrows} - 1)(\\text{ncolumns} - 1)=1$ degree of freedom."
+ "Define the test statistic: $\\chi^2 = \\sum_{i,j}\\frac{(O_{i,j}-E_{i,j})^2}{E_{i,j}}$ where $O$ and $E$ are the observed and expected values, while the indices $i,j$ identify a cell of the table. The test statistic $\\chi^2$ is approximately distributed as a $\\chi^2$ distribution with $(\\text{nrows} - 1)(\\text{ncolumns} - 1)=1$ degree of freedom."
]
},
{
diff --git a/Lecture2.ipynb b/Lecture2.ipynb
index bf9016d..e37e131 100755
--- a/Lecture2.ipynb
+++ b/Lecture2.ipynb
@@ -148,7 +148,7 @@
"## Content of the lecture\n",
"\n",
" * Hypothesis testing\n",
- " * Type or errors\n",
+ " * Types of errors\n",
" * Likelihood ratio\n",
" * Neyman–Pearson lemma\n",
" * Exclusions\n",
@@ -163,7 +163,7 @@
}
},
"source": [
- "## Simple and composite hypothesis\n",
+ "## Simple and composite hypotheses\n",
"\n",
"* A simple hypothesis is one in which all parameters of the distribution are specified"
]
@@ -314,7 +314,7 @@
" * Do the experiment\n",
" * If the outcome is outside the acceptance region reject the null-hypothesis\n",
" \n",
- "In complex example one don't compute the acceptance region, but just compute the observed p-value"
+ "In complex examples one doesn't compute the acceptance region, but just computes the observed p-value"
]
},
{
@@ -325,11 +325,11 @@
}
},
"source": [
- "### Exercize: Lady tasting tea (Fisher \"The Design of Experiments\" - 1935)\n",
+ "### Exercise: Lady tasting tea (Fisher \"The Design of Experiments\" - 1935)\n",
"\n",
"A lady claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups, four of each variety, in random order. What is the critical region at 5%? If we ask for a $5\\sigma$ evidence, what's the minimum amount of cups we have to prepare? (Hint: look at the [Hypergeometric distribution](https://en.wikipedia.org/wiki/Hypergeometric_distribution), answer should be 26)\n",
"\n",
- "Are we interested in the rare case when the lady never guess? What is \"more extreme\" in this case?"
+ "Are we interested in the rare case when the lady never guesses? What is \"more extreme\" in this case?"
]
},
{
@@ -482,7 +482,7 @@
},
"source": [
"### Example\n",
- "Suppose you have a rock that seems to be opal (density 2.2 g/cm3), but may be also quartz (2.6 g/cm3). You have two instruments to measure the density, the first with resolution 0.2 g/cm3, the second with resolution 0.5 g/cm3. Opal is much expensive, we want to check if it is opal. $H_0:$quartz. What is the rejection region at 5%? Assume gaussian errors."
+ "Suppose you have a rock that seems to be opal (density 2.2 g/cm3), but may be also quartz (2.6 g/cm3). You have two instruments to measure the density, the first with resolution 0.2 g/cm3, the second with resolution 0.5 g/cm3. Opal is much more expensive, we want to check if it is opal. $H_0:$quartz. What is the rejection region at 5%? Assume gaussian errors."
]
},
{
@@ -560,7 +560,7 @@
"test.EnableDetailedOutput(True)\n",
"\n",
"hypoCalc = RooStats.FrequentistCalculator(data, opal_model, quartz_model) # alt, null\n",
- "hypoCalc.SetToys(20000, 1000) # generate much more toys for bkg-only\n",
+ "hypoCalc.SetToys(20000, 1000) # generate many more toys for bkg-only\n",
"\n",
"toy_sampler = hypoCalc.GetTestStatSampler()\n",
"toy_sampler.SetTestStatistic(test) # our test statistics\n",
@@ -636,8 +636,8 @@
}
},
"source": [
- "### Exercize\n",
- "Show that the in this case the likelihood ratio is equivalent to the weighted mean of the two measurements"
+ "### Exercise\n",
+ "Show that in this case the likelihood ratio is equivalent to the weighted mean of the two measurements"
]
},
{
@@ -727,7 +727,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Exercize\n",
+ "### Exercise\n",
"Compute the acceptance region using the simple mean as test-statistics: show that it is less powerful. What kind of shape is the acceptance region in this case?"
]
},
@@ -785,10 +785,10 @@
}
},
"source": [
- "### Exercize\n",
+ "### Exercise\n",
"Compute $\\text{power}(\\alpha)$ for the simple mean and compare with the weighted mean.\n",
- "### Exercize\n",
- "Explain why the previous plot remembers the ROC curve (bkg-rejection vs signal-efficiency)."
+ "### Exercise\n",
+ "Explain why the previous plot resembles the ROC curve (bkg-rejection vs signal-efficiency)."
]
},
{
@@ -801,15 +801,15 @@
"source": [
"## Exclusion and frequentist intervals\n",
"\n",
- "By now we have done statistical test to discover something (a signal), defining the null hypothesis to be the background only case. Suppose we have counted $n_{obs}$ events in a Poissonian process with mean $\\lambda$. What are the values $[\\lambda_{up}, \\infty]$ that we can exclude? This is something called hypothesis inversion. As usual we define the null hypothesis: $H_0: \\lambda = N$ with $N > n_{obs}$. The likelihood:\n",
+ "By now we have done statistical tests to discover something (a signal), defining the null hypothesis to be the background only case. Suppose we have counted $n_{obs}$ events in a Poissonian process with mean $\\lambda$. What are the values $[\\lambda_{up}, \\infty]$ that we can exclude? This is something called hypothesis inversion. As usual we define the null hypothesis: $H_0: \\lambda = N$ with $N > n_{obs}$. The likelihood:\n",
"\n",
"$$L = \\text{Pois}(n|N)$$\n",
"\n",
- "than the p-value of our observation $n_{obs}$:\n",
+ "Then the p-value of our observation $n_{obs}$ is:\n",
"\n",
"$$\\text{p-value} = \\sum_{n\\geq n_{obs}} \\text{Pois}(n|N)$$\n",
"\n",
- "if p-value $<\\alpha$ we consider $N$ to be inside the exluded region $[\\lambda_{up}, \\infty]$. The general approach is to scan for many value of $N$ until we found the one with p-value $=\\alpha$, which is $n_{up}$."
+ "if p-value $<\\alpha$ we consider $N$ to be inside the excluded region $[\\lambda_{up}, \\infty]$. The general approach is to scan for many values of $N$ until we find the one with p-value $=\\alpha$, which is $\\lambda_{up}$."
]
},
{
@@ -822,7 +822,7 @@
"source": [
"### Example: RooFit asymmetric error bars\n",
"\n",
- "Let's do a two-side test and compute the excluded intervals $[-\\infty, \\lambda_{down}]$ and $[\\lambda_{up}, \\infty]$ with $\\alpha=1 - 0.6827$. This means that every single interval should be tested with $\\alpha=(1-0.6827)/2.=0.15865$. We can solve it analytically or with a scan. Suppose $n_{obs} = 4$."
+ "Let's do a two-sided test and compute the excluded intervals $[-\\infty, \\lambda_{down}]$ and $[\\lambda_{up}, \\infty]$ with $\\alpha=1 - 0.6827$. This means that every single interval should be tested with $\\alpha=(1-0.6827)/2.=0.15865$. We can solve it analytically or with a scan. Suppose $n_{obs} = 4$."
]
},
{
@@ -943,7 +943,7 @@
"metadata": {},
"source": [
"### Question:\n",
- "why we assumed that the central value for $\\lambda$ is $n_{obs}=4$?"
+ "Why did we assume that the central value for $\\lambda$ is $n_{obs}=4$?"
]
},
{
@@ -967,13 +967,13 @@
}
],
"source": [
- "def get_confiderence_poisson(n):\n",
+ "def get_confidence_poisson(n):\n",
" ym = np.array([0], dtype=np.float64)\n",
" yp = np.array([0], dtype=np.float64)\n",
" ROOT.RooHistError.instance().getPoissonInterval(n, ym, yp, 1)\n",
" return ym[0], yp[0]\n",
"\n",
- "get_confiderence_poisson(4)"
+ "get_confidence_poisson(4)"
]
},
{
@@ -1010,7 +1010,7 @@
"v = np.zeros((len(trues), len(obss)))\n",
"limits = []\n",
"for ix, obs in enumerate(obss):\n",
- " limits.append(get_confiderence_poisson(int(obs)))\n",
+ " limits.append(get_confidence_poisson(int(obs)))\n",
" for iy, true in enumerate(trues):\n",
" v[iy, ix] = stats.poisson(true).pmf(obs)\n",
"ax.pcolormesh(obss, trues, v, vmax=np.percentile(v, 99))\n",
@@ -1036,11 +1036,11 @@
"\n",
"$$P(\\lambda|n_{obs}) = P(n_{obs}|\\lambda)P(\\lambda)$$\n",
"\n",
- "We assume the prior $P(\\lambda)$ to be flat (and zero for negative values), equivalent to a constant we can ignore. Everybody agree?\n",
+ "We assume the prior $P(\\lambda)$ to be flat (and zero for negative values), equivalent to a constant we can ignore. Does everybody agree?\n",
"\n",
"$$P(\\lambda|n_{obs}) = P(n_{obs}|\\lambda)$$\n",
"\n",
- "note that $n_{obs}$ is fixed, so this is a real function $f(\\lambda)$. Let find a region [low, high] such that\n",
+ "note that $n_{obs}$ is fixed, so this is a real function $f(\\lambda)$. Let's find a region [low, high] such that\n",
"\n",
"$$\\int_{low}^{high} P(\\lambda|n_{obs}) = 68\\%$$"
]
@@ -1224,7 +1224,7 @@
"source": [
"## Profile likelihood ratio\n",
"\n",
- "Suppose you have composite hypothesis as \n",
+ "Suppose you have composite hypotheses such as \n",
"\n",
"$$H_0: \\theta\\in\\Theta_0 \\quad H_1: \\theta\\in\\Theta_0^C \\quad \\Theta_0\\subset\\Theta$$\n",
"\n",
@@ -1237,7 +1237,7 @@
"$$\\lambda(x) = \\frac{\\sup_{\\theta}L(s=0, \\theta|x)}{\\sup_{\\theta,s} L(s, \\theta|x)}=\\frac{L(s=0, \\hat{\\hat\\theta}(0)|x)}{L(\\hat{s}, \\hat\\theta|x)}$$\n",
"\n",
"\n",
- "where $\\hat{\\hat\\theta}(0)$ is the value of $\\theta$ which optimize the likelihood for $s=0$ (conditioned likelihood), while $\\hat{s}$ and $\\hat{\\theta}$ are the values that optimize the likelihood without any constrains (unconditioned likelihood).\n",
+ "where $\\hat{\\hat\\theta}(0)$ is the value of $\\theta$ which optimizes the likelihood for $s=0$ (conditioned likelihood), while $\\hat{s}$ and $\\hat{\\theta}$ are the values that optimize the likelihood without any constraints (unconditioned likelihood).\n",
"\n",
"\n",
"It varies between 0 and 1, low values mean that the observed result is less likely to occur under the null hypothesis as compared to the alternative. (numerator $\\leq$ denominator)\n",
@@ -1253,11 +1253,11 @@
}
},
"source": [
- "As shown it is important to have an analytically expression $f_q$ of the distribution of the test-statistics $q$ to compute the p-value: $\\text{p-value} = \\int_{q^\\text{obs}}^{\\infty} f_q(q) dq$. Otherwise toys must be run.\n",
+ "As shown it is important to have an analytical expression $f_q$ of the distribution of the test-statistics $q$ to compute the p-value: $\\text{p-value} = \\int_{q^\\text{obs}}^{\\infty} f_q(q) dq$. Otherwise toys must be run.\n",
"\n",
"## Wilks's theorem\n",
"\n",
- "The quantity $t=-2\\log(\\lambda)$ is aymptotically (large sample) distributed as a $\\chi^2$ distribution with $n=\\text{dim}(\\Theta)-\\text{dim}(\\Theta_0)$ degrees of freedom when $H_0$ is true."
+ "The quantity $t=-2\\log(\\lambda)$ is asymptotically (large sample) distributed as a $\\chi^2$ distribution with $n=\\text{dim}(\\Theta)-\\text{dim}(\\Theta_0)$ degrees of freedom when $H_0$ is true."
]
},
{
@@ -1268,7 +1268,7 @@
}
},
"source": [
- "Let resurrect the simple Poisson model $\\text{Poisson}(n|s + b)$. This time, instead of using the number of observed events as test-statistics, let use the profile likelihood ratio."
+ "Let's resurrect the simple Poisson model $\\text{Poisson}(n|s + b)$. This time, instead of using the number of observed events as test-statistics, let's use the profile likelihood ratio."
]
},
{
@@ -1332,11 +1332,11 @@
"outputs": [],
"source": [
"profll = RooStats.ProfileLikelihoodTestStat(bModel.GetPdf())\n",
- "# this modify a bit our test statistics\n",
+ "# this modifies a bit our test statistics\n",
"profll.SetOneSidedDiscovery(1)\n",
"\n",
"hypoCalc = RooStats.FrequentistCalculator(data, sbModel, bModel)\n",
- "hypoCalc.SetToys(100000, 5000) # generate much more toys for bkg-only\n",
+ "hypoCalc.SetToys(100000, 5000) # generate many more toys for bkg-only\n",
"\n",
"toy_sampler = hypoCalc.GetTestStatSampler()\n",
"toy_sampler.SetTestStatistic(profll)\n",
@@ -1390,7 +1390,7 @@
}
},
"source": [
- "The distribution under the $H_0$ has a peak at 0, which does not follow the $\\chi^2$ distribution. Explain why (see test statistics for one-side-discovery and SetOneSideDiscovery(True) option)."
+ "The distribution under the $H_0$ has a peak at 0, which does not follow the $\\chi^2$ distribution. Explain why (see test statistics for one-sided discovery and SetOneSidedDiscovery(True) option)."
]
},
{
@@ -1401,7 +1401,7 @@
}
},
"source": [
- "We can also use the AsymtoticCalculator, which instead of using the empirical distribution of the likelihood ratio from toys, it uses the asymptotic distribution ($\\chi^2_1$ distribution for the simple likelihood ratio)"
+ "We can also use the AsymptoticCalculator, which instead of using the empirical distribution of the likelihood ratio from toys, uses the asymptotic distribution ($\\chi^2_1$ distribution for the simple likelihood ratio)"
]
},
{
@@ -1450,9 +1450,9 @@
"source": [
"## On/off problem\n",
"\n",
- "Let's introduce a classical problem in high-energy / astro / ... physics. Suppose you have a source that can contain signal and background events. You have an other source that contains only background events and the background in the two source is correlated.\n",
+ "Let's introduce a classical problem in high-energy / astro / ... physics. Suppose you have a source that can contain signal and background events. You have another source that contains only background events and the background in the two sources is correlated.\n",
"\n",
- "This is the case when you have a control region, with only background, and a signal region with signal plus background. Usually the correlation of the background between the two region comes from a simulation. In astrophysics this can be the case when with telescope one wants to see if there is source in a particular region of the sky, using as control region the average of regions where there are no known source.\n",
+ "This is the case when you have a control region, with only background, and a signal region with signal plus background. Usually the correlation of the background between the two regions comes from a simulation. In astrophysics this can be the case when with a telescope one wants to see if there is a source in a particular region of the sky, using as control region the average of regions where there are no known sources.\n",
"\n"
]
},
@@ -1464,7 +1464,7 @@
}
},
"source": [
- "In term of random variables:\n",
+ "In terms of random variables:\n",
"\n",
"$$ N_{SR} = S + B \\qquad N_{CR} = \\alpha B$$\n",
"\n",
@@ -1775,7 +1775,7 @@
"outputs": [],
"source": [
"profll = RooStats.ProfileLikelihoodTestStat(bModel.GetPdf())\n",
- "# this modify a bit our test statistics\n",
+ "# this modifies a bit our test statistics\n",
"profll.SetOneSidedDiscovery(True)\n",
"\n",
"hypoCalc = RooStats.FrequentistCalculator(data, sbModel, bModel)\n",
diff --git a/Lecture3.ipynb b/Lecture3.ipynb
index d1fab77..425d185 100755
--- a/Lecture3.ipynb
+++ b/Lecture3.ipynb
@@ -161,7 +161,7 @@
"source": [
"## Systematics\n",
"\n",
- "How we can incorporate systematics inside our model? Suppose we know the parameter $\\alpha$ with a relative uncertainty $\\sigma_\\alpha$. We can imagine that there is another measurement (auxiliary measurement), that measured $\\alpha$ (the parameter of the model of the auxiliary measurement) and observed $a$. We can write the likelihood of the auxiliary measurement as $L(a|\\alpha) = N(a|\\alpha, \\delta_\\alpha)$, if assume that $a$ is normally distributed aroud the true value $\\alpha$ and $\\delta\\alpha$ is the absolute error on $\\alpha$ ($\\delta\\alpha=\\sigma_\\alpha a$)\n",
+ "How can we incorporate systematics inside our model? Suppose we know the parameter $\\alpha$ with a relative uncertainty $\\sigma_\\alpha$. We can imagine that there is another measurement (auxiliary measurement), that measured $\\alpha$ (the parameter of the model of the auxiliary measurement) and observed $a$. We can write the likelihood of the auxiliary measurement as $L(a|\\alpha) = N(a|\\alpha, \\delta_\\alpha)$, if we assume that $a$ is normally distributed around the true value $\\alpha$ and $\\delta\\alpha$ is the absolute error on $\\alpha$ ($\\delta\\alpha=\\sigma_\\alpha a$)\n",
"\n",
"\n",
"\n"
@@ -179,7 +179,7 @@
"\n",
"$$L(s, b, a|N_{SR}, N_{CR}, \\alpha) = L(s, b|N_{SR}, N_{CR}) L(a|\\alpha)$$\n",
"\n",
- "or more explicitely:\n",
+ "or more explicitly:\n",
"\n",
"$$\\text{Pois}(N_{SR}|s + b) \\text{Pois}(N_{CR}|\\alpha b) \\times N(a|\\alpha, \\delta_\\alpha)$$\n",
"\n",
@@ -698,7 +698,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Excercize\n",
+ "## Exercise\n",
"Explain why the profile likelihood curve is larger when adding systematics"
]
},
@@ -921,9 +921,9 @@
}
},
"source": [
- "### (Big) exercize\n",
+ "### (Big) exercise\n",
"\n",
- "Suppose you are looking for a signal in a SR. There are two main background on the signal region, due to two different physical processes, B1 and B2. You also have two CRs. The simulation predicts the following countings for an equivalent luminosity of 1/fb.\n",
+ "Suppose you are looking for a signal in a SR. There are two main backgrounds in the signal region, due to two different physical processes, B1 and B2. You also have two CRs. The simulation predicts the following event counts for an equivalent luminosity of 1/fb.\n",
"\n",
"\n",
"| `` | SR | CR1 | CR2 |\n",
@@ -938,7 +938,7 @@
"\n",
"where $L$ is the observed luminosity. Note that we are using the correlation between SR/CR1/CR2 of the simulation. Assume L=10/fb +/- 5%. \n",
"\n",
- "Suppose you analysis is blinded (you haven't look to the signal region) and you observe: CR1=1509, CR2=5017 events.\n",
+ "Suppose your analysis is blinded (you haven't looked at the signal region) and you observe: CR1=1509, CR2=5017 events.\n",
"\n",
"How many events do you have to observe in the signal region to claim a discovery at 3 sigma (what is the acceptance region)? What is the impact of the systematics? Bonus: take into account the MC statistical uncertainty of 3% for every prediction.\n",
"\n",
@@ -953,11 +953,11 @@
}
},
"source": [
- "Costraints can be gaussian:\n",
+ "Constraints can be gaussian:\n",
" \n",
" $$m\\to m(1 + \\sigma\\theta) \\qquad L \\to L \\times G(0, \\theta, 1)$$\n",
" \n",
- "or log-normal, for example for quantity that cannot be negative (e.g. efficiencies):\n",
+ "or log-normal, for example for quantities that cannot be negative (e.g. efficiencies):\n",
"\n",
" $$\\varepsilon\\to \\varepsilon\\exp(\\sigma\\theta)\\qquad L \\to L \\times G(0, \\theta, 1)$$\n",
" \n"
@@ -971,9 +971,9 @@
}
},
"source": [
- "### Exercize (big)\n",
+ "### Exercise (big)\n",
"\n",
- "Find a better constrain for log-normal variable. Suppose you have a random variable $Y$ that is log-normal distributed. Do a change of variable and write it as $Y = a\\exp(b \\Theta)$. How is distributed $\\Theta$? How can I constrain $\\Theta$? Suppose I know the median of $Y$ ($=Y_0$) and the standard deviation $\\sigma\\times Y_0$. What are the values of $a$ and $b$ if I want to conserve the median and the standard deviation? What is the value of $y$ if $\\theta=1$? Is it $Y_0 + \\sigma Y_0$? If not, give an approximation in terms of $O(\\sigma)$ for small $\\sigma$."
+ "Find a better constraint for a log-normal variable. Suppose you have a random variable $Y$ that is log-normally distributed. Do a change of variable and write it as $Y = a\\exp(b \\Theta)$. How is $\\Theta$ distributed? How can I constrain $\\Theta$? Suppose I know the median of $Y$ ($=Y_0$) and the standard deviation $\\sigma\\times Y_0$. What are the values of $a$ and $b$ if I want to conserve the median and the standard deviation? What is the value of $y$ if $\\theta=1$? Is it $Y_0 + \\sigma Y_0$? If not, give an approximation in terms of $O(\\sigma)$ for small $\\sigma$."
]
},
{
@@ -986,11 +986,11 @@
"source": [
"## Shape analysis\n",
"\n",
- "In more complicate analysis you don't look just to the number of events in specific region but you look to the distribution of the events as a function of a continuos variable (e.g. invariant mass).\n",
+ "In more complicated analyses you don't look just at the number of events in a specific region but you look at the distribution of the events as a function of a continuous variable (e.g. invariant mass).\n",
"\n",
"$$L(\\alpha|\\{m_i\\}_{i=1}^n) = \\text{Pois}[n|N_s(\\alpha) + N_b(\\alpha)] \\prod_{i=1}^n\\left(\\frac{N_s(\\alpha) f_s[m_i|\\alpha] + N_b(\\alpha) f_b[m_i|\\alpha]}{N_s(\\alpha)+N_b(\\alpha)}\\right)\\times\\prod_j L(a_j|\\alpha_j)$$\n",
"\n",
- "Where $\\alpha$ is a set of nuisance parameters, $\\{m_i\\}_{i=1}^n$ are the data, $N_s$ and $N_b$ the predicted number of signal and background events, $f_s$ and $f_b$ the pdf describing the continuos observable.\n",
+ "Where $\\alpha$ is a set of nuisance parameters, $\\{m_i\\}_{i=1}^n$ are the data, $N_s$ and $N_b$ the predicted numbers of signal and background events, $f_s$ and $f_b$ the pdfs describing the continuous observable.\n",
"\n",
"The last term is the product of all the constraints of the nuisance parameters (not all the nuisance parameters need to be constrained by auxiliary measurements). The other term is the extended likelihood of the s+b model.\n",
"\n",
@@ -1007,7 +1007,7 @@
"source": [
"### Example model\n",
"\n",
- "Suppose we observe the invariant mass distribution and we are searching for a narrow resonance in the spectrum. Our signal + background model is (after many careful studies) an exponential plus a gaussian. We also know the expected number of events for the signal under a particular theory. We want to be able to parametrize also similar model, where the number of signal events is multiplied by the \"signal strength\", $\\mu$. So we will write the number of signal events as $\\mu \\times n_{exp}$ where $n_{exp}$ are the one from the nominal theory. In this way we can also write the background-only model as the special case for $\\mu=0$. The number of background events are not well know by the theory, so we can estimate them from data, which means that $n_b$ is a nuisance parameters in the model.\n",
+ "Suppose we observe the invariant mass distribution and we are searching for a narrow resonance in the spectrum. Our signal + background model is (after many careful studies) an exponential plus a gaussian. We also know the expected number of events for the signal under a particular theory. We want to be able to also parametrize similar models, where the number of signal events is multiplied by the \"signal strength\", $\\mu$. So we will write the number of signal events as $\\mu \\times n_{exp}$ where $n_{exp}$ is the one from the nominal theory. In this way we can also write the background-only model as the special case for $\\mu=0$. The number of background events is not well known from theory, so we can estimate it from data, which means that $n_b$ is a nuisance parameter in the model.\n",
"\n",
"Consider systematic on the luminosity, scale and resolution of the signal (location and width)"
]
@@ -1876,7 +1876,7 @@
"\n",
"$$ t_\\mu = -2\\log\\lambda (\\mu) = -2\\log \\frac{L(\\mu, \\hat{\\hat{\\theta}}(\\mu))}{L(\\hat{\\mu}, \\hat{\\theta})}$$\n",
"\n",
- "High value means incompatiblity with data. If we want to test a specific $\\mu$ we can compute the p-value $= \\int_{t_{\\mu, obs}}^\\infty f(t_\\mu|\\mu) dt_\\mu$."
+ "High value means incompatibility with data. If we want to test a specific $\\mu$ we can compute the p-value $= \\int_{t_{\\mu, obs}}^\\infty f(t_\\mu|\\mu) dt_\\mu$."
]
},
{
@@ -1916,7 +1916,7 @@
"\\end{cases}\n",
"$$\n",
"\n",
- "If $\\hat \\mu<0$ it means that we are observing less events than the one predicted by the background-only model. Since we are truncating the definition of test statistics we are not considering downward fluctuation as discrepancies with the model. High value of $\\hat\\mu$ means high value of $q_0$ and large discrepancy with the background-only model."
+ "If $\\hat \\mu<0$ it means that we are observing fewer events than the ones predicted by the background-only model. Since we are truncating the definition of test statistics we are not considering downward fluctuations as discrepancies with the model. High value of $\\hat\\mu$ means high value of $q_0$ and large discrepancy with the background-only model."
]
},
{
@@ -1931,8 +1931,8 @@
"\n",
"$$ \\text{p-value} = p_0 = \\int_{q_{0, obs}}^\\infty f(q_0|\\mu=0)\\, dq_0$$\n",
"\n",
- "### Exercize\n",
- "If we assume that the background model is true, how many times we will get $\\hat \\mu<0$? If $f(t_0|\\mu=0)$ is a $\\chi^2$ distribution what about $f(q_0|\\mu=0)$?"
+ "### Exercise\n",
+ "If we assume that the background model is true, how often will we get $\\hat \\mu<0$? If $f(t_0|\\mu=0)$ is a $\\chi^2$ distribution what about $f(q_0|\\mu=0)$?"
]
},
{
@@ -1944,7 +1944,7 @@
},
"source": [
"## $q_\\mu$ statistic for exclusion\n",
- "Suppose we want to put an upper limit, so we define as null hypothesis to exclude the hypotesis signal+background with $\\mu$ as signal multiplier.\n",
+ "Suppose we want to put an upper limit, so the null hypothesis we want to exclude is signal+background, with $\\mu$ as signal multiplier.\n",
"\n",
"$$\n",
"q_\\mu=\\begin{cases}\n",
@@ -1953,7 +1953,7 @@
"\\end{cases}\n",
"$$\n",
"\n",
- "we set $q_\\mu=0$ when observing a value of $\\mu$ greater than the one we are observing since we don't want it to enter in the rejection region when doing an upper limit; we don't want that upper fluctuation count as bad agreement with data."
+ "we set $q_\\mu=0$ when observing a value of $\\mu$ greater than the one we are observing since we don't want it to enter in the rejection region when doing an upper limit; we don't want upward fluctuations to count as bad agreement with data."
]
},
{
@@ -1985,9 +1985,9 @@
"\n",
"$$\\frac{\\sup L(0, m_H, \\theta)}{\\sup L(\\mu, m_H, \\theta)}$$\n",
"\n",
- "Is $m_H$ playing a role in the numerator? No! The numerator is not a special case of the numerator! $m_H$ is not a nuisance parameter that can be profiled.\n",
+ "Is $m_H$ playing a role in the numerator? No! The numerator is not a special case of the denominator! $m_H$ is not a nuisance parameter that can be profiled.\n",
"\n",
- "The commont solution is to repeat the test for fixed value of $m_H$: in that case $m_H$ is considered to be a constant."
+ "The common solution is to repeat the test for fixed values of $m_H$: in that case $m_H$ is considered to be a constant."
]
},
{
@@ -2101,7 +2101,7 @@
"source": [
"### Look elsewhere effect\n",
"\n",
- "The maximum significance we have found is 5 around 125. Usually this is called \"local significace\", since it is computed for a particular $m_H$. The problem is that we have repeated the test many times and we have to consider that we are taking into account the maximum discrepancy. This is also know as \"problem of multiple comparisons\".\n",
+ "The maximum significance we have found is 5 around 125. Usually this is called \"local significance\", since it is computed for a particular $m_H$. The problem is that we have repeated the test many times and we have to consider that we are taking into account the maximum discrepancy. This is also known as \"problem of multiple comparisons\".\n",
"\n",
"One can solve this problem redefining the test statistic as:\n",
"\n",
@@ -2156,7 +2156,7 @@
"source": [
"## Exclusions\n",
"\n",
- "We are also interested to know what is the minimum $\\mu$ that we can exclude. This is done as hypothesis inversion. We have to find a $\\mu_{95}$ that is exluded at 5%. As before this is done with a simple scan. Let's do it for $m_H=110 GeV$."
+ "We are also interested in knowing what is the minimum $\\mu$ that we can exclude. This is done as hypothesis inversion. We have to find a $\\mu_{95}$ that is excluded at 5%. As before this is done with a simple scan. Let's do it for $m_H=110 GeV$."
]
},
{
@@ -2427,9 +2427,9 @@
}
},
"source": [
- "## Topic not coverted, but important if you want to do hypothesis test in HEP\n",
+ "## Topics not covered, but important if you want to do hypothesis tests in HEP\n",
"\n",
- " * Look elesewhere effect\n",
+ " * Look elsewhere effect\n",
" * CLs\n",
" * Asimov dataset"
]
@@ -2442,8 +2442,8 @@
}
},
"source": [
- "## Last (big) exercize\n",
- "Redo one of the example where we have use the frequentist calculator, without using it, reimplementing the toy generation and the computation of the test statistic by hand"
+ "## Last (big) exercise\n",
+ "Redo one of the examples where we have used the frequentist calculator, without using it, reimplementing the toy generation and the computation of the test statistic by hand"
]
}
],