validphys is designed so that, by default, resources stored in known remote
locations are downloaded automatically and seamlessly used where necessary.
Available resources include PDF sets, completed fits, theories, and results of
validphys runs that have been uploaded to the server.
vp-get tool, described below,
can be used to download the same items manually.
By default when some resource such as a PDF is required by
derived tools such as
vp-setupfit), the code will first look for it in some
local directory specified in the profile file. If it is not found
there, it will try to download it from some remote repository (also specified in
For example a
validphys runcard such as
pdf: NNPDF40_nnlo_as_01180 fit: NNPDF40_nlo_as_01180 theoryid: 208 use_cuts: "fromfit" dataset_input: dataset: ATLASWZRAP36PB cfac: [EWK] actions_: - plot_fancy - plot_chi2dist
Will download if necessary the fit called
PDF set called
NNPDF40_nnlo_as_01180 and the theory with ID 208, when validphys
is executed with the default settings. In practice one rarely has to worry about
installing resources by hand when working with NNPDF tools.
The behaviour of downloading automatically can be disabled by passing the
--no-net flag to supported tools. In that case, failure to find a given
resource locally will result in an error and exiting the program. The
flag makes the default behaviour explicit and has no effect otherwise.
What can be downloaded
The following resources are found automatically:
Fits (specified by the
fitkey) can be downloaded if they have previously been uploaded with vp-upload. The corresponding PDF set will be installed as appropriate.
- PDF sets
PDF sets (specified among others by the
Theories (specified by the
theoryidkey) are downloaded and uncompressed.
Files produced by
validphyscan be used as input to subsequent validphys analyses (for example χ² tables are used for αs fits). The user needs to have HTTP access to the repository, which is provided when installing using the bootstrap script. Output files are not specified by any top level config key, but instead actions can specify their own logic, for example for using an existing file instead of computing it.
vp-get tool can be used to download resources manually, in the same way
validphys would do.
The basic syntax is
vp-get <resource_type> <resource_name>
The available options for
<resource type> can be seen with
They correspond to the resources described above.
$ vp-get --list Available resource types: - fit - pdf - theoryID - vp_output_file
For example to download the fit
NNPDF31_nlo_as_0118_1000 we would write
$ vp-get fit NNPDF31_nlo_as_0118_1000
If the resource is already installed locally, the tool will display some information on it and bail out:
$ vp-get fit NNPDF31_nlo_as_0118_1000 FitSpec(name='NNPDF31_nlo_as_0118_1000', path=PosixPath('/home/zah/anaconda3/envs/nnpdf-dev/share/NNPDF/results/NNPDF31_nlo_as_0118_1000'))
Downloading resources in code (
The logic is as follows: Given a resource type
<foo>, the normal
validphys.loader.Loader class would implement a
returning an object containing the appropriate metadata (such as file paths), or
LoaderError if the object cannot be found. The
FallbackLoader (which is generated dynamically) will intercept the
LoaderError and, if it happens, call the
download_<foo> method of
RemoteLoader, if it exists. That method should cause the resource to be
installed in such a way that the subsequent call of the
method succeeds. That is it should downoad the resource to the relevant search
path, and uncompress it if needed.
In practice one can get a download aware loader by using a
instance, which will try to obtain all the required resources from remote
from validphys.loader import FallbackLoader as Loader l = Loader() #Will download theory 151 if needed. l.check_dataset('NMC', theoryid=151)
Loader class will only search locally.
from validphys.loader import Loader l = Loader() l.check_dataset('NMC', theoryid=151) --------------------------------------------------------------------------- TheoryNotFound Traceback (most recent call last) <ipython-input-7-30e29a1539e8> in <module> ----> 1 l.check_dataset('NMC', theoryid=151) ~/nngit/nnpdf/validphys2/src/validphys/loader.py in check_dataset(self, name, rules, sysnum, theoryid, cfac, frac, cuts, use_fitcommondata, fit, weight) 416 417 if not isinstance(theoryid, TheoryIDSpec): --> 418 theoryid = self.check_theoryID(theoryid) 419 420 theoryno, _ = theoryid ~/nngit/nnpdf/validphys2/src/validphys/loader.py in check_theoryID(self, theoryID) 288 if not theopath.exists(): 289 raise TheoryNotFound(("Could not find theory %s. " --> 290 "Folder '%s' not found") % (theoryID, theopath) ) 291 return TheoryIDSpec(theoryID, theopath) 292 TheoryNotFound: Could not find theory 151. Folder '/home/zah/anaconda3/share/NNPDF/data/theory_151' not found
Output files uploaded to the
validphys can be retrieved specifying their path
(starting from the report ID). They will be either downloaded (when using
FallbackLoader) or retrieved from the cache:
from validphys.loader import FallbackLoader as Loader l = Loader() l.check_vp_output_file('qTpvLZLwS924oAsmpMzhFw==/figures/f_ns0_fitunderlyinglaw_plot_closure_pdf_histograms_0.pdf') PosixPath('/home/zah/anaconda3/share/NNPDF/vp-cache/qTpvLZLwS924oAsmpMzhFw==/figures/f_ns0_fitunderlyinglaw_plot_closure_pdf_histograms_0.pdf')