How to analyse a closure test
There are two
validphys reports which are used to analyse closure
tests. The report which compares two different closure tests is
generated using the
vp-comparefits tool. The individual report which
compares a closure test to the PDF used as the underlying law (from which
pseudodata is generated) can be found in the main
nnpdf repository under
Comparison to underlying PDF
This report is very similar to the full validphys fit report produced by
vp-comparefits, the main differences are: only the closure fit is
specified and it is compared to the PDF specified under the key
fakepdf in the fit runcard; and the report contains some additional
sections dedicated to the closure test statistical estimators.
The \(\chi^2\) between the central predictions of the closure test and the underlying PDF predictions, a measure of how well the underlying predictions are reproduced by the closure test
The mean across replicas of the \(\chi^2\) between predictions from a single replica and the central replica. Can be thought of as the variance of the replica predictions normalised by the covariance.
\(\chi^2\) between central predictions and level 1 replica minus the \(\chi^2\) between level 1 data and underlying PDF, gives some information on the direction the central predictions are being pulled with respect the the level 1 data
Detailed information on the closure test estimators can be found here.
Comparison to another closure test
To compare to another closure test the user can use a special command
line option with
vp-comparefits --closure <other options>
vp-comparefits to use a special closure comparison report
which compares to closures for the statistical estimators above.
care should be taken that both of the closure tests being compared were fitting
pseudodata which had the same level 1 shift applied to the same underlying law.
Clearly the closure test estimators have some underlying dependence on these.
To ensure this, check that both fits have a matching
fakepdf key and
filterseed, additionally the fits should have been ran during a time window
within which the data generation algorithm was not modified.