How to run a closure test
Closure tests are a way to validate methodology by fitting on pseudodata generated from pre-existing PDFs. There are different levels of closure tests which aim to validate different components of the fitting toolchain.
For more detailed information on the conception of closure tests, see the NNPDF3.0 paper.
Each closure test defines a
fakepdf in the runcard, which will be referred to
here as the underlying law. For the purpose of the closure test it can be thought
of as being a proxy for the true PDF.
There are three levels of closure test:
central pseudodata is given by central predictions of the underlying law
no MC noise is added on top of the central data, each replica is fitting the same set of data
central pseudodata is shifted by some noise η which is drawn from the experimental covariance matrix and represents ‘real’ central values provided by experimentalists which do not sit exactly on the underlying law but are consistent with it according to their own uncertainty
no MC noise is added, each replica fits a subset of the same shifted data. There is however a difference in the training/validation split used for stopping, the spread on replicas can be thought of as the spread due to this split in addition to any methodological uncertainty.
central pseudodata is shifted by level 1 noise η
MC noise is added on top of the level 1 shift
level 2 is a proxy of a real fit, where the underlying law is known
The advantage of knowing the underlying law is that we can see how well the methodology is extracting this from shifted data, using closure test estimators.
The main obvious disadvantage is that a pre-existing PDF may not be a suitable proxy for the underlying law.
Preparing the closure test runcard
To run a closure test we require a standard fit runcard. The main section
which controls closure test specific behaviour can be found under
Before you’ve made any changes, a typical
closuretest section will be as follows:
closuretest: filterseed : 0 # Random seed to be used in filtering data partitions fakedata : False fakepdf : MMHT2014nnlo68cl errorsize : 1.0 # uncertainties rescaling fakenoise : False rancutprob : 1.0 # Fraction of data to be included in the fit rancutmethod: 0 # Method to select rancutprob data fraction rancuttrnval: False # 0(1) to output training(valiation) chi2 in report printpdf4gen: False # To print info on PDFs during minimization
True will cause closure test pseudodata to be generated
and subsequently fitted. The PDf which the pseudodata will be generated from
is specified by the
fakepdf key. It is strongly advised to set the
t0pdfset, found under
datacuts to be the same PDF, unless specifically
testing the impact of the t0 procedure.
fakenoise key specifies whether or not the level 1 shift η will be
add to the pseudodata during the filtering step, this is require for
both level 1 and level 2 closure tests.
An example of a typical level 1 or level 2
closuretest specification is given
closuretest: filterseed : 0 # Random seed to be used in filtering data partitions fakedata : True fakepdf : MMHT2014nnlo68cl errorsize : 1.0 # uncertainties rescaling fakenoise : True rancutprob : 1.0 # Fraction of data to be included in the fit rancutmethod: 0 # Method to select rancutprob data fraction rancuttrnval: False # 0(1) to output training(valiation) chi2 in report printpdf4gen: False # To print info on PDFs during minimization
Note that it is critical that two closure tests which are to be compared have
filterseed. They should also both have been run during a time where
no major changes were made to data generation. This is because fits with
different level 1 noise produce different closure test estimators. See for
example a report
comparing two level 2 closure tests with identical settings apart from
filterseed. Note that setting
filterseed to 0 will use the default seed for
There are still some relevant settings to the closure test. For the above example we would choose that the t0 set was the same as the underlying law:
datacuts: t0pdfset : MMHT2014nnlo68cl # PDF set to generate t0 covmat ...
Finally we need to specify whether or not MC replicas will be generated in the
fit, differentiating between a level 1 and level 2 closure test. This can be achieved
fitting to be
fitting: ... genrep : True ...
Summary for each level of closure test
See below for the keys which specify each level of closure test, other keys can be chosen by the user.
fitting: ... genrep : False ... closuretest: ... fakedata : True fakenoise : False ...
fitting: ... genrep : False ... closuretest: ... fakedata : True fakenoise : True ...
fitting: ... genrep : True ... closuretest: ... fakedata : True fakenoise : True ...
Running a closure test with
Running a closure test with
n3fit will require a valid
n3fit runcard, with
the closure test settings modified as shown
above. The difference
between running a closure fit in
n3fit and a standard fit is that the user is
required to run
vp-setupfit on the runcard before running
n3fit. This is
because the filtering of the data is required to generate the pseudodata central
values. The filtered data should then be rebuilt before the fit, so there is no
risk of the fit crashing due to multiple replicas rebuilding the data
simultaneously. The workflow is as follows:
$ vp-setupfit fitname.yml $ vp-rebuild-data fitname $ n3fit fitname.yml <replica_number>
You will still need to evolve the fit and run
postfit as with a standard