observed values x

expectation value of x
expectation value of y
background
assume bins independent Poisson RV

Neyman-Pearson test statistic optimal,

linear but not always calculable: for RV (n-
=>determine to maximise separation dim) with M bins => bins total
Fisher separation between and =>idea: make reasonable Ansatz, then use MC Why?
define separation:
optimal for Gaussian with common covariance
10 Unfolding compare data and theory
-usually theory->MC(detector distortion)->compare
with data much simpler
supervised learning: minimize error function
Multivariate Analysis +compare experiments directly
+compare theory later developed

NN formulation of problem:
s: activation function (eg. logistic sigmoid)
single layer perceptron limited resolution
+hidden layer =>multilayer perceptron efficiency
many observables with little separation power each background
+>1hidden layers=>DeepNN more general parametrization:
structure
neighboring layers only->feed-forward network
BDT performance quantified by purity
DT:
define Impurity gain: 2p(1-p) Gini
physically motivated, all events retained priors depend on production cross sections classifier
but (-) sensitive to statistical fluctuation test statistic to classify

set S: "sample space":

: definition by Kolmogorov's axioms in set theory
P>=0 positive definition
if A,B disjunct: P(A)+P(B) additive fixed: Poisson
P(S)=1 normalized Binomial
subjective probability: degree of belief sum of Poisson again Poisson

frequentist: relative frequency of outcomes

Central Limit Theorem
: Gauss
RV
Bayes Theorem function of RV is RV

posterior - likelihood - prior Chi-Square:

total probability:

1 Probability 2 Probability Distributions Uniform

Log-normal: y Gaussian, , multiply random

Independence
errors => CLT

Covariance Exponential
Correlation
pseudorandom: randomness tests
(linear) Correlation coefficient
like true random, but reproducible and repeat after uniformity
period uncorrelated (at all levels)
test:
x,y independent => e.g independent equidistribution
Student t distribution: measure sampling from updown tests
measurement
Gaussian pairwise, triplewise correlation
! not other way around: e.g.
Central Limit Theorem application:
n independent continuous RV with and Breit-Wigner: energy loss of charged particle analytically not solvable
exceptions: => is Gaussian RV error propagation breaks down
malicious PD (mean not well-defined) many free parameters+unknown correlations
with and between
one dominant factor
Landau-distribution (energy loss of charged particle)

1 Uncertainties
because of long tail moments undefined
Breit-Wigner (Cauchy) distribution

precision: reproducability Statistical Methods transformation method: r<-uniform=>x<-f(x)

equal probability: g(r)dr=f(x)dx
reduces with
accuracy: closeness to actual value
statistical <=> systematical 3 Monte Carlo method => solve F(x(r))=G(r)=r for x
not unique: 1-r also <-uniform

simulation: num calc with solution related to PD enhanced: enclose f with g: with RV x according to
integration: (especially higher dimensions superior) can be generated
combining measurements
generate x
acceptance-rejection method: generate u from uniform
if => if u<f(x) accept

generate x in interval
generate f in interval
Error propagation if accept
RV function of with unknown PD holds if second derivative small
only know and error: 1.reject true Bayesian confidence intervals
=> determine E[y] and Var(y) 2.accept false "coverage" probability:
Neyman-Pearson lemma frequency of true value within confidence interval
Orthogonal transformation choosing critical region overcoverage not maximally powerful
transform to with diagonal highest power with t such: undercoverage serious flaw
always possible with linear transformation Frequentist
=> diagonalize normally test statistic t: lower dim->reduce data
construct test statistic: Neyman construction for

get from MC
but see MVA
9 Prediction
Linear test statistics , Fisher discriminant
intervals and limits
significance of signal
P(observe n or more events from background alone)
4 Statistical Tests
each bin Poisson

P-value: observed result equally or less compatible with

Feldman-Cousins
RV (unlike ) which is specified before test Goodness-of-fit test
report upperlimit+centrallimit near physical boundary
("observed significance/confidence level") construct t(x) reflecting level of agreement with wthout flip-flopping
use Neyman with x in CI with greatest likelihood ratio
not probability for true, ( no RV) follows chi2 distribution with N-1 degrees of freedom
to give probability for H0 prior needs to be given (distribution free) (in general N-m m:parameters
(Bayesian) estimated from data)
near physical boundaries
P-Value: problem: low sensitivity->stronger limit for 0 events
Pearsons Square test interpretation of P-Value: probability of observing n_obs
level of agreement of two histograms (eg. data,theory) chi2 statistic: or more events under H_0 report observed value sensitivity: separation of signal and background
for poisson data N with mean vb=0.5 n_obs=5 give v_s=4.5±2.2 misleading, error on pretend to 0 in limit setting
CLs method
depends on binning!: low sensitivity=>no exclusion of model
large binning->info loss
but at least 5 entries per bin otherwise not distribution
free usual CLs method:
=>for small data samples: Kolmogorov-Smirnov (no overcoverage, flip-flopping
binning)

Likelihood function: ML estimators:

efficient
asymptotically unbiased
use asymptotic normality (->Gaussian for infinite sample
size)
examples
Gaussian
simple
5 Parameter estimation
Exponential unbinned data (no information loss)
consistent
invariant under variable transformation

analytic method sample: set of n independent observations of x

=> Random vector sample mean normal
insert in sampling distribution variance:

if analytic difficult: MC method methods for estimating MVB (minimum variance bound)
simulate experiments efficient RCF inequality:
compute ML estimates =minimum variance
use e.g. sampling variance for estimating

use RCF lower bound 6 Maximum Likelihood method terminology statistic: consistent: (minimum requirement)
if MC toilsome: graphical method
estimator
taylor logL
=> MSE (mean square error)

=> unbiased
N Gaussian RV: ML=chi^2 why?
combination of different experiments
if b=0 for all n

minimize even for non-Gaussian asymptotically unbiased

connection to chi2?
Least Squares method b=0 for

if (1) (independent) Gaussian sampling distribution is PD of estimator

combining measurements, goodness of fit... (2) linear in
(3) functional form of hypothesis correct goodness-of-fit
=>chi2 distributed with N-m(#parameters) DoF as in chi2 =likelihood?
note: underestimated errors can worsen chi2

with binned data

Bayesian
extended ML:
knowledge contained in
if number of observations is Poisson with mean :
connection with Bayesian =>report mode
if prior uniform ML=Bayesian
1. : +info, reduce (Bayesian postulate) problem with uniform prior:
2. : same normalization (usually not a problem)
transformation of variables! Bayesian not invariant in
general

ML with binned data

very large data loglikelihood difficult to compute
histogram as measurement of N dim random vector
according to multinomial distribution