Uploading results to the validphys repository
The primary method to share results within the collaboration is by uploading the
corresponding files to the NNPDF server. Most commonly, results are
uploaded to the validphys repository, so that they are accessible from
The files in this repository are backed up to two locations, indexed and cross referenced with the mailing list. The HTTP access to the files is password protected.
The uploading system is designed to be integrated with validphys. Reports,
hopefully filled with the appropriate metadata in the runcard, can
be uploaded directly, or after they have
been completed using the vp-upload script. Arbitrary
files can be uploaded using the ``wiki-upload `` script,
which will interactively ask the user to fill in the metadata. In either case an
URL will be returned with the location of the resource accessible with a web
browser.
In order to be able to upload files, the user must have a valid SSH key
installed in the NNPDF server access, and the rsync
command must be present.
Several settings relevant to uploading files are configured in profile files.
Metadata
Currently the following information is used to index the results:
title(string)
author(string)
keywords(list of strings)
The first two are self explanatory, and keywords is a list of tags used to
categorize the result, such as ATLAS jets or nn31final. You can see more
examples in the webpage. Keywords are used in
various ways to aid the discoverability of the result, and so it is important to
set them properly. Some keywords may be used to display the report in a
prominent place of the index page.
For validphys runcards, this data is read from a meta mapping declared in
the runcard. For example
meta:
title: PDF comparisons
author: NNPDF Collaboration
keywords: [gallery]
validphys uses this mapping to write a meta.yaml file with the same
information to the output folder. This file is then used by the indexers.
Conventions for writing metadata
Reports endowed with the correct metadata can be retrieved from the index even several years after they are uploaded.
Always fill appropriately the metadata fields for anything you upload.
Fill the author field with a complete form of your name, e.g. Zahari Kassabov rather than ZK, and always use the same name.
Add keywords that are relevant to the result you are uploading. Use existing tags if possible.
Metadata from HTML fallback
An index.html file in the uploaded output folder will serve as a source of
metadata if the meta.yaml file is not present (e.g. because the meta mapping
was not defined in the runcard). To automatically generate an index.html file
from a report action, one may set the option main:True (alternatively there
is the out_filename option, which may be used to specify the filename). In the
template, use the pandoc-markdown syntax
to set the metadata at the top of the file. In the runcard you would write:
template: mytemplate.md
actions_:
- report(main=True)
and you would begin mytemplate.md, using YAML syntax, like:
---
title: Testing the fit {@fit@}
author: Zahari Kassabov
keywords: [nnpdf31, nolhc]
...
Note that you can use the report syntax to get the parameters from the
runcard. If you only want to set title or author, you can also
prefix the two first lines of the markdown templates with %:
% Template title
% Myself, the template author
Content...
This is mostly useful for sub-reports not at the top level, in more complicated documents.
Uploading directly from validphys
When the --upload flag is set in the invocation of the validphys command,
the contents of the output folder will be uploaded to the NNPDF data server,
after validphys is done. Use this if you have filled the meta mapping in the
runcard and already know that the output is going to be good enough
to share. Otherwise use vp-upload after checking the result.
validphys will check the SSH connection before doing any work, and
it will fail early if it cannot be established.
The vp-upload script
The vp-upload script uploads completed results to the NNPDF server, such as
reports and fits. To upload a completed validphys report, use:
vp-upload <output folder>
The output folder is expected to contain the metadata (e.g. in the
form of a meta.yaml file). If it doesn’t exist or you want to upload and index
arbitrary files, use the wiki-upload command.
The script automatically detects (validphys.uploadutils.check_input()) the type of the input.
A fit is defined to be any folder structure that contains a filter.yml file at its root, a PDF is any
folder containing a .info file at the root and a replica 0, and a report is any such structure containing an
index.html file at the root. The input folder is then placed in the correct location in the
server accordingly.
Note
If there is already a fit or PDF on the server with the same name as the fit or PDF
you wish to upload, then this command will not overwrite the resource that already
exists. To overwite such a resource on the server, use the --force option.
The code is documented at validphys.scripts.vp_upload.
Note that fits are indexed separately, and can be retrieved with the vp-get
command.
The wiki-upload script
The wiki-upload script is a more interactive counterpart to vp-upload. It
allows uploading arbitrary files that do not have metadata attached. It will
construct the metadata by asking the user to fill it in before uploading the
result. The usage is:
wiki-upload <file or folder>
This will cause the user to be prompted for the various metadata fields and the
file or folder to be uploaded to the server, together with a generated
meta.yaml file used for indexing.
The code is documented at validphys.scripts.wiki_upload.
The validphys index page
The source of the report index page is:
serverscripts/validphys-reports/index.html
inside the validphys2 directory in the main repository. This page can be
edited to reflect the current interests (the Makefile directly uploads to the
server). See the documentation on web scripts for more details.