Downloading resources
validphys is designed so that, by default, resources stored in known remote
locations are downloaded automatically and seamlessly used where necessary.
Available resources include PDF sets, completed fits, theories, and results of
past validphys runs that have been uploaded to the server.
The vp-get tool, described below,
can be used to download the same items manually.
Automatic operation
By default when some resource such as a PDF is required by validphys (or
derived tools such as vp-setupfit), the code will first look for it in some
local directory specified in the profile file. If it is not found
there, it will try to download it from some remote repository (also specified in
the profile).
For example a validphys runcard such as
pdf: NNPDF40_nnlo_as_01180
fit: NNPDF40_nlo_as_01180
theoryid: 208
use_cuts: "fromfit"
dataset_input:
dataset: ATLASWZRAP36PB
cfac: [EWK]
actions_:
- plot_fancy
- plot_chi2dist
Will download if necessary the fit called NNPDF40_nlo_as_01180, the
PDF set called NNPDF40_nnlo_as_01180 and the theory with ID 208, when validphys
is executed with the default settings. In practice one rarely has to worry about
installing resources by hand when working with NNPDF tools.
The behaviour of downloading automatically can be disabled by passing the
--no-net flag to supported tools. In that case, failure to find a given
resource locally will result in an error and exiting the program. The --net
flag makes the default behaviour explicit and has no effect otherwise.
What can be downloaded
The following resources are found automatically:
- Fits
Fits (specified by the
fitkey) can be downloaded if they have previously been uploaded with vp-upload. The corresponding PDF set will be installed as appropriate.- PDF sets
PDF sets (specified among others by the
pdfkey) are searched for in both NNPDF and LHAPDF repositories. If the PDF is not found and a fit with the same name exists, it will be downloaded and the corresponding PDF set will be installed and made available for usage.- Theories
Theories (specified by the
theoryidkey) are downloaded and uncompressed.validphysoutput filesFiles produced by
validphyscan be used as input to subsequent validphys analyses (for example χ² tables are used for αs fits). The user needs to have HTTP access to the repository, which is provided when installing using the bootstrap script. Output files are not specified by any top level config key, but instead actions can specify their own logic, for example for using an existing file instead of computing it.
The vp-get tool
The vp-get tool can be used to download resources manually, in the same way
validphys would do.
The basic syntax is
vp-get <resource_type> <resource_name>
The available options for <resource type> can be seen with vp-get --list.
They correspond to the resources described above.
$ vp-get --list
Available resource types:
- fit
- pdf
- theoryID
- vp_output_file
For example to download the fit NNPDF31_nlo_as_0118_1000 we would write
$ vp-get fit NNPDF31_nlo_as_0118_1000
If the resource is already installed locally, the tool will display some information on it and bail out:
$ vp-get fit NNPDF31_nlo_as_0118_1000
FitSpec(name='NNPDF31_nlo_as_0118_1000', path=PosixPath('/home/zah/anaconda3/envs/nnpdf-dev/share/NNPDF/results/NNPDF31_nlo_as_0118_1000'))
Downloading resources in code (validphys.loader)
The automatic download logic is implemented in the validphys.loader,
specifically by the validphys.loader.RemoteLoader and
validphys.loader.FallbackLoader classes.
The logic is as follows: Given a resource type <foo>, the normal
validphys.loader.Loader class would implement a check_<foo> method
returning an object containing the appropriate metadata (such as file paths), or
raise a LoaderError if the object cannot be found. The check_<foo> method
of FallbackLoader (which is generated dynamically) will intercept the
LoaderError and, if it happens, call the download_<foo> method of
RemoteLoader, if it exists. That method should cause the resource to be
installed in such a way that the subsequent call of the Loader.check_<foo>
method succeeds. That is it should downoad the resource to the relevant search
path, and uncompress it if needed.
In practice one can get a download aware loader by using a FallbackLoader
instance, which will try to obtain all the required resources from remote
locations.
from validphys.loader import FallbackLoader as Loader
l = Loader()
#Will download theory 151 if needed.
l.check_dataset('NMC', theoryid=151)
Conversely the Loader class will only search locally.
from validphys.loader import Loader
l = Loader()
l.check_dataset('NMC', theoryid=151)
---------------------------------------------------------------------------
TheoryNotFound Traceback (most recent call last)
<ipython-input-7-30e29a1539e8> in <module>
----> 1 l.check_dataset('NMC', theoryid=151)
~/nngit/nnpdf/validphys2/src/validphys/loader.py in check_dataset(self, name, rules, sysnum, theoryid, cfac, frac, cuts, use_fitcommondata, fit, weight)
416
417 if not isinstance(theoryid, TheoryIDSpec):
--> 418 theoryid = self.check_theoryID(theoryid)
419
420 theoryno, _ = theoryid
~/nngit/nnpdf/validphys2/src/validphys/loader.py in check_theoryID(self, theoryID)
288 if not theopath.exists():
289 raise TheoryNotFound(("Could not find theory %s. "
--> 290 "Folder '%s' not found") % (theoryID, theopath) )
291 return TheoryIDSpec(theoryID, theopath)
292
TheoryNotFound: Could not find theory 151. Folder '/home/zah/anaconda3/share/NNPDF/data/theory_151' not found
Output files uploaded to the validphys can be retrieved specifying their path
(starting from the report ID). They will be either downloaded (when using
FallbackLoader) or retrieved from the cache:
from validphys.loader import FallbackLoader as Loader
l = Loader()
l.check_vp_output_file('qTpvLZLwS924oAsmpMzhFw==/figures/f_ns0_fitunderlyinglaw_plot_closure_pdf_histograms_0.pdf')
PosixPath('/home/zah/anaconda3/share/NNPDF/vp-cache/qTpvLZLwS924oAsmpMzhFw==/figures/f_ns0_fitunderlyinglaw_plot_closure_pdf_histograms_0.pdf')