nnpdf_data package

nnpdf_data.load_dataset_metadata(dataset_name, variant=None)[source]

Given a dataset name, return the metadata

Subpackages

Submodules

nnpdf_data.commondataparser module

This module implements parsers for commondata and its associated metadata and uncertainties files into useful structures that can be fed to the main nnpdf_data.coredata.CommonData class.

A CommonData file is completely defined by a dataset name (which defines the folder in which the information is) and observable name (which defines the specific data, fktables and plotting settings to read).

<experiment>_<process>_<energy>{_<extras>}_<observable>

Where the folder name is <experiment>_<process>_<energy>{_<extras>}

The definition of all information for a given dataset (and all its observable) is in the metadata.yaml file and its implemented_observables.

This module defines a number of parsers using the validobj library.

The full metadata.yaml is read as a SetMetaData object which contains a list of ObservableMetaData. These ObservableMetaData are the “datasets” of NNPDF for all intents and purposes. The parent SetMetaData collects some shared variables such as the version of the dataset, arxiv, inspire or hepdata ids, the folder in which the data is, etc.

The main class in this module is thus ObservableMetaData which holds _all_ information about the particular dataset-observable that we are interested in (and a reference to its parent).

Inside the ObservableMetaData we can find:
  • TheoryMeta: contains the necessary information to read the (new style) fktables

  • KinematicsMeta: containins metadata about the kinematics

  • Variant: variant to be used

  • PlottingOptions: plotting style and information for validphys, only utilized if

    validphys is also installed.

The CommonMetaData defines how the CommonData file is to be loaded, by modifying the CommonMetaData using one of the loaded Variants one can change the resulting nnpdf_data.coredata.CommonData object.

class nnpdf_data.commondataparser.ObservableMetaData(observable_name: str, observable: dict, ndata: int, plotting: validphys.plotoptions.plottingoptions.PlottingOptions, process_type: Annotated[Union[nnpdf_data.process_options._Process, str], InputType(Any), Validator(<function ValidProcess at 0x7fa3bf3f6980>)], kinematic_coverage: list[str], kinematics: nnpdf_data.commondataparser.ValidKinematics, data_uncertainties: list[typing.Annotated[pathlib.Path, InputType(<class 'str'>), Validator(<function ValidPath at 0x7fa3bf49cae0>)]], data_central: Optional[Annotated[pathlib.Path, InputType(<class 'str'>), Validator(<function ValidPath at 0x7fa3bf49cae0>)]] = None, theory: Optional[nnpdf_data.commondataparser.TheoryMeta] = None, tables: Optional[list] = <factory>, npoints: Optional[list] = <factory>, variants: Optional[dict[str, nnpdf_data.commondataparser.Variant]] = <factory>, applied_variant: Optional[str] = None, ported_from: Optional[str] = None, _parent: Optional[Any] = None)[source]

Bases: object

applied_variant: str | None = None
apply_variant(variant_name)[source]

Return a new instance of this class with the variant applied

This class also defines how the variant is applied to the commondata. If more than a variant is being used, this function will be called recursively until all variants are applied.

check()[source]

Various checks to apply manually to the observable before it is used anywhere These are not part of the __post_init__ call since they can only happen after the metadata has been read, the observable selected and (likely) variants applied.

property cm_energy
data_central: Path | None = None
data_uncertainties: list[pathlib.Path]
digest_plotting_variable(variable)[source]

Digest plotting variables in the line_by or figure_by fields and return the appropiate kX or other label such that the plotting functions of validphys can understand it.

These might be variables included as part of the kinematics or extra labels defined in the plotting dictionary.

property experiment
property is_integrability
property is_nnpdf_special

Is this an NNPDF special dataset used for e.g., Lagrange multipliers or QED fits

property is_ported_dataset

Return True if this is an automatically ported dataset that has not been updated

property is_positivity
kinematic_coverage: list[str]
kinematics: ValidKinematics
property kinlabels

Return the kinematic labels in the same order as they are set in kinematic_coverage (which in turns follow the key kinematic_coverage) If this is a ported dataset, rely on the process type using the legacy labels

load_data_central()[source]

Loads the data for this commondata returns a dataframe

Returns:

a dataframe containing the data

Return type:

pd.DataFrame

load_kinematics(fill_to_three=True, drop_minmax=True)[source]

Returns a dataframe with the kinematic information

Parameters:
  • fill_to_three (bool) – ensure that there are always three columns (repeat the last one) in the kinematics

  • drop_minmax (bool) – Drop the min and max value, necessary for legacy comparisons

Returns:

a dataframe containing the kinematics

Return type:

pd.DataFrame

load_uncertainties()[source]

Returns a dataframe with all appropiate uncertainties

Returns:

a dataframe containing the uncertainties

Return type:

pd.DataFrame

property name
ndata: int
property nnpdf_metadata
npoints: list | None
observable: dict
observable_name: str
property path_data_central
property path_kinematics
property paths_uncertainties
plotting: PlottingOptions
property plotting_options
ported_from: str | None = None
property process
process_type: _Process | str
property setname
tables: list | None
theory: TheoryMeta | None = None
variants: dict[str, nnpdf_data.commondataparser.Variant] | None
class nnpdf_data.commondataparser.SetMetaData(setname: str, version: int, version_comment: str, nnpdf_metadata: dict, implemented_observables: list[nnpdf_data.commondataparser.ObservableMetaData], arXiv: ValidReference | None = None, iNSPIRE: ValidReference | None = None, hepdata: ValidReference | None = None, metadata_file: Path | None = None)[source]

Bases: object

Metadata of the whole set

property allowed_datasets

Return the implemented datasets as a list <setname>_<observable>

property allowed_observables

observable} dictionary

Type:

Returns the implemented observables as a {observable_name.upper()

arXiv: ValidReference | None = None
property cm_energy

Return the center of mass energy as GeV if it can be understood from the name otherwise return None

property folder
hepdata: ValidReference | None = None
iNSPIRE: ValidReference | None = None
implemented_observables: list[nnpdf_data.commondataparser.ObservableMetaData]
metadata_file: Path | None = None
nnpdf_metadata: dict
select_observable(obs_name_raw)[source]

Check whether the observable is implemented and return said observable

setname: str
version: int
version_comment: str
class nnpdf_data.commondataparser.TheoryMeta(FK_tables: list[tuple], operation: str = 'NULL', conversion_factor: float = 1.0, shifts: dict | None = None, normalization: dict | None = None, comment: str | None = None)[source]

Bases: object

Contains the necessary information to load the associated fktables

The theory metadata must always contain a key FK_tables which defines the fktables to be loaded. The FK_tables is organized as a double list such that:

The inner list is concatenated In practice these are different fktables that might refer to the same observable but that are divided in subgrids for practical reasons. The outer list instead are the operands for whatever operation needs to be computed in order to match the experimental data.

In addition there are other flags that can affect how the fktables are read or used:

  • operation: defines the operation to apply to the outer list

  • shifts: mapping with the single fktables and their respective shifts

    useful to create “gaps” so that the fktables and the respective experimental data are ordered in the same way (for instance, when some points are missing from a grid)

This class is inmutable, what is read from the commondata metadata should be considered final

Example

>>> from nnpdf_data.commondataparser import TheoryMeta
>>> from validobj import parse_input
>>> from ruamel.yaml import YAML
>>> theory_raw = '''
... FK_tables:
...   - - fk1
...   - - fk2
...     - fk3
... operation: ratio
... '''
>>> theory = YAML(typ='safe').load(theory_raw)
>>> parse_input(theory, TheoryMeta)
TheoryMeta(FK_tables=[['fk1'], ['fk2', 'fk3']], operation='RATIO', shifts = None, conversion_factor=1.0, comment=None, normalization=None))
FK_tables: list[tuple]
comment: str | None = None
conversion_factor: float = 1.0
fktables_to_paths(grids_folder)[source]

Given a source for pineappl grids, constructs the lists of fktables to be loaded

normalization: dict | None = None
operation: str = 'NULL'
classmethod parser(yaml_file)[source]

The yaml databases in the server use “operands” as key instead of “FK_tables”

shifts: dict | None = None
class nnpdf_data.commondataparser.ValidKinematics(file: Path, variables: dict[str, nnpdf_data.commondataparser.ValidVariable])[source]

Bases: object

Contains the metadata necessary to load the kinematics of the dataset. The variables should be a dictionary with the key naming the variable and the content complying with the ValidVariable spec.

Only the kinematics defined by the key kinematic_coverage will be loaded, which must be three.

Three shall be the number of the counting and the number of the counting shall be three. Four shalt thou not count, neither shalt thou count two, excepting that thou then proceedeth to three. Once the number three, being the number of the counting, be reached, then the kinematics be loaded in the direction of thine validobject.

apply_label(var, value)[source]

For a given value for a given variable, return the labels as label = value (unit) If the variable is not included in the list of variables, returns None as the variable could’ve been transformed by a kinematic transformation

file: Path
get_label(var)[source]

For the given variable, return the label as label (unit) If the label is an “extra” return the last one

variables: dict[str, nnpdf_data.commondataparser.ValidVariable]
class nnpdf_data.commondataparser.ValidReference(url: str, version: int | None = None, journal: str | None = None, tables: list[int] = <factory>)[source]

Bases: object

Holds literature information for the dataset

journal: str | None = None
tables: list[int]
url: str
version: int | None = None
class nnpdf_data.commondataparser.ValidVariable(label: str, description: str = '', units: str = '')[source]

Bases: object

Defines the variables

apply_label(value)[source]

Return a string formatted as label = value (units)

description: str = ''
full_label()[source]
label: str
units: str = ''
class nnpdf_data.commondataparser.Variant(data_uncertainties: list[pathlib.Path] | None = None, theory: TheoryMeta | None = None, data_central: Path | None = None, experiment: str | None = None)[source]

Bases: object

The new commondata format allow the usage of variants A variant can overwrite a number of keys, as defined by this dataclass:

  • data_uncertainties

  • theory

  • data_central

This class may overwrite some other keys for the benefit of reproducibility of old NNPDF fits, but the usage of these features is undocumented and discouraged.

data_central: Path | None = None
data_uncertainties: list[pathlib.Path] | None = None
experiment: str | None = None
theory: TheoryMeta | None = None
nnpdf_data.commondataparser.load_commondata(metadata)[source]

TODO: update this docstring since now the load_commondata_new takes the information from the metadata, and the name -> split is done outside

In the current iteration of the commondata, each of the commondata (i.e., an observable from a data publication) correspond to one single observable inside a folder which is named as “<experiment>_<process>_<energy>_<extra>” The observable is defined by a last suffix of the form “_<obs>” so that the full name of the dataset is always:

"<experiment>_<process>_<energy>{_<extra>}_<obs>"

where <extra> is optional.

This function right now works under the assumotion that the folder/observable is separated in the last _ so that:

folder_name = <experiment>_<process>_<energy>{_<extra>}

but note that this convention is still not fully defined.

This function returns a commondata object constructed by parsing the metadata.

Once a variant is selected, it can no longer be changed

Note that this function reproduces parse_commondata below, which parses the _old_ file format

nnpdf_data.commondataparser.parse_new_metadata(metadata_file, observable_name, variant=None)[source]

Given a metadata file in the new format and the specific observable to be read load and parse the metadata and select the observable. If any variants are selected, apply them.

The triplet (metadata_file, observable_name, variant) define unequivocally the information to be parsed from the commondata library

nnpdf_data.commondataparser.parse_set_metadata(metadata_file)[source]

Read the metadata file

nnpdf_data.coredata module

The all-important CommonData object

class nnpdf_data.coredata.CommonData(setname: str, ndata: int, commondataproc: str, nkin: int, nsys: int, commondata_table: DataFrame, systype_table: DataFrame, legacy: bool = False, legacy_names: list | None = None, kin_variables: list | None = None)[source]

Bases: object

Data contained in Commondata files, relevant cuts applied.

Parameters:
  • setname (str) – Name of the dataset

  • ndata (int) – Number of data points

  • commondataproc (str) – Process type, one of 21 options

  • nkin (int) – Number of kinematics specified

  • nsys (int) – Number of systematics

  • commondata_table (pd.DataFrame) – Pandas dataframe containing the commondata

  • systype_table (pd.DataFrame) – Pandas dataframe containing the systype index for each systematic alongside the uncertainty type (ADD/MULT/RAND) and name (CORR/UNCORR/THEORYCORR/SKIP)

  • systematics_table (pd.DataFrame) – Panda dataframe containing the table of systematics

property additive_errors

Returns the systematics which are additive (systype is ADD) as absolute uncertainties (same units as data), with SKIP uncertainties removed.

property central_values
commondata_table: DataFrame
commondataproc: str
export(folder_path)[source]

Wrapper around export_data and export_uncertainties to write both uncertainties and data after filtering to a given folder

export_data(buffer)[source]

Exports the central data defined by this commondata instance to the given buffer

export_uncertainties(buffer)[source]

Exports the uncertainties defined by this commondata instance to the given buffer

get_cv()[source]
get_kintable()[source]
kin_variables: list | None = None
property kinematics
legacy: bool = False
legacy_names: list | None = None
property multiplicative_errors

Returns the systematics which are multiplicative (systype is MULT) in a percentage format, with SKIP uncertainties removed.

ndata: int
nkin: int
nsys: int
setname: str
property stat_errors
systematic_errors(central_values=None)[source]

Returns all systematic errors as absolute uncertainties, with a single column for each uncertainty. Converts multiplicative_errors to units of data and then appends onto additive_errors. By default uses the experimental central values to perform conversion, but the user can supply a 1-D array of central values, with length self.ndata, to use instead of the experimental central values to calculate the absolute contribution of the multiplicative systematics.

Parameters:

central_values (None, np.array) – 1-D array containing alternative central values to combine with multiplicative uncertainties. This array must have length equal to self.ndata. By default central_values is None, and the central values of the commondata are used.

Returns:

systematic_errors – Dataframe containing systematic errors.

Return type:

pd.DataFrame

systematics_table: DataFrame | None
systype_table: DataFrame
with_central_value(cv)[source]
with_cuts(cuts)[source]

A method to return a CommonData object where an integer mask has been applied, keeping only data points which pass cuts.

Note if the first data point passes cuts, the first entry of cuts should be 0.

Paramters

cuts: list or validphys.core.Cuts or None

nnpdf_data.coredata.generate_path_filtered_data(fit_path, setname)[source]

Utility to ensure that both the loader and tools like setupfit utilize the same convention to generate the names of generated pseudodata

nnpdf_data.disp_theory module

nnpdf_data.process_options module

Module to hold process dependent options

Only variables included in the _Vars enum and processes included in the Processes dictionary are allowed.

nnpdf_data.theory module

This module provides a unique source and definition for all the possible parameters that a theory card can contain.

class nnpdf_data.theory.TheoryCard(ID: int, PTO: Literal[0, 1, 2, 3], FNS: str, IC: int, ModEv: Literal['EXA', 'TRN'], XIR: float, XIF: float, QED: int, Q0: float, mc: float, Qmc: float, kcThr: float, mb: float, Qmb: float, kbThr: float, mt: float, Qmt: float, ktThr: float, CKM: list[float], MZ: float, MW: float, GF: float, SIN2TW: float, TMC: int, MP: float, Comments: str, alphas: float, alphaqed: float, Qref: float, XIA: float = 1.0, NfFF: Optional[int] = None, nfref: Optional[int] = None, MaxNfPdf: Optional[int] = 5, nf0: Optional[int] = None, HQ: Optional[Literal['POLE', 'MSBAR']] = 'POLE', IterEv: Optional[int] = None, ModSV: Optional[str] = None, DAMP: Optional[int] = 0, DAMPPOWERc: Optional[int] = None, DAMPPOWERb: Optional[int] = None, FONLLParts: Optional[str] = None, PTOEKO: Optional[int] = None, PTODIS: Optional[int] = None, n3lo_ad_variation: Optional[list] = <factory>, n3lo_cf_variation: Optional[int] = 0, use_fhmruvv: Optional[bool] = True, MaxNfAs: Optional[int] = None, SxRes: Optional[int] = None, SxOrd: Optional[str] = None, EScaleVar: Optional[int] = None, Qedref: Optional[float] = None, global_nx: Optional[int] = None)[source]

Bases: object

CKM: list[float]
Comments: str
DAMP: int | None = 0
DAMPPOWERb: int | None = None
DAMPPOWERc: int | None = None
EScaleVar: int | None = None
FNS: str
FONLLParts: str | None = None
GF: float
HQ: Literal['POLE', 'MSBAR'] | None = 'POLE'
IC: int
ID: int
IterEv: int | None = None
MP: float
MW: float
MZ: float
MaxNfAs: int | None = None
MaxNfPdf: int | None = 5
ModEv: Literal['EXA', 'TRN']
ModSV: str | None = None
NfFF: int | None = None
PTO: Literal[0, 1, 2, 3]
PTODIS: int | None = None
PTOEKO: int | None = None
Q0: float
QED: int
Qedref: float | None = None
Qmb: float
Qmc: float
Qmt: float
Qref: float
SIN2TW: float
SxOrd: str | None = None
SxRes: int | None = None
TMC: int
XIA: float = 1.0
XIF: float
XIR: float
alphaqed: float
alphas: float
asdict()[source]
find_nf(mu)[source]

Given a value of q, find the corresponding number of flavours for this theory.

global_nx: int | None = None
kbThr: float
kcThr: float
ktThr: float
mb: float
mc: float
mt: float
n3lo_ad_variation: list | None
n3lo_cf_variation: int | None = 0
nf0: int | None = None
nfref: int | None = None
use_fhmruvv: bool | None = True
exception nnpdf_data.theory.TheoryCardError[source]

Bases: Exception

exception nnpdf_data.theory.TheoryNotFoundInDatabase[source]

Bases: FileNotFoundError

nnpdf_data.theorydbutils module

theorydbutils.py

low level utilities for querying the theory database file and representing the data as a python object.

exception nnpdf_data.theorydbutils.TheoryNotFoundInDatabase[source]

Bases: Exception

nnpdf_data.theorydbutils.fetch_all(theory_database: Path)[source]

Looks in the theory database and returns a dataframe with theory info for all theories that can be found in the folder theory_database. Every .yaml file found in the folder will be considered a possible theory.

Parameters:

theory_database (Path) – pathlib.Path pointing to the folder with theory cards

Returns:

theory_info_dataframe – dataframe filled with all entries in theorydb file

Return type:

pd.Dataframe

Example

>>> from nnpdf_data import theory_cards
>>> from nnpdf_data.theorydbutils import fetch_all
>>> theory_df = fetch_all(theory_cards)
nnpdf_data.theorydbutils.fetch_theory(theory_database: Path, theoryID: int)[source]

Looks in the theory card folder and returns a dictionary of theory info for the theory number specified by theoryID.

Parameters:
  • theory_database (Path) – pathlib.Path pointing to the folder with the theory cards

  • theoryID (int) – numeric identifier of theory to query info

Returns:

theory_info_dict – dictionary filled with relevant entry from theory database

Return type:

dict

Example

>>> from nnpdf_data import theory_cards
>>> from nnpdf_data.theorydbutils import fetch_theory
>>> theory = fetch_theory(theory_cards, 700)

nnpdf_data.utils module

nnpdf_data.utils.get_nnpdf_profile(profile_path=None)[source]

Returns the NNPDF profile as a dictionary

If no profile_path is provided it will be autodiscovered in the following order:

  1. Environment variable $NNPDF_PROFILE_PATH

  2. ${XDG_CONFIG_HOME}/NNPDF/nnprofile.yaml (usually ~/.config/nnprofile)

Any value not filled by 1 or 2 will then be filled by the default values found within the validphys python package nnporfile_default.yaml

If nnpdf_share is set to the special key RELATIVE_TO_PYTHON the python prefix (Path(sys.prefix)/"share"/"NNPDF") will be used.

The data_path key is special in that it will _always_ be filled with the location of the nnpdf_data package (albeit with lower priority if there is an entry).

nnpdf_data.utils.parse_yaml_inp(input_yaml, spec)[source]

Helper function to parse yaml using the validobj library and print useful error messages in case of a parsing error.

https://validobj.readthedocs.io/en/latest/examples.html#yaml-line-numbers

nnpdf_data.utils.quick_yaml_load(filepath)[source]

If libyaml is available, use the C loader to speed up some of the read https://pyyaml.org/wiki/LibYAML libyaml is available for most linux distributions

nnpdf_data.validphys_compatibility module

This file exists solely for me to be able to upload a package to PyPI for the nnpdf data which does not depend on the rest of the NNPDF code. This file should not be modified. Everything in this file is deprecated and should be removed, and the only reason this is needed is because we are still mixing new and old data.

This also means that _things_ that would’ve been loaded as a kinematic transformation or result transformation are loaded as boring strings that cannot do anything, as it should be.

class nnpdf_data.validphys_compatibility.PlottingOptions(func_labels: dict = <factory>, dataset_label: Optional[str] = None, experiment: Optional[str] = None, nnpdf31_process: Optional[str] = None, data_reference: Optional[str] = None, theory_reference: Optional[str] = None, process_description: Optional[str] = None, y_label: Optional[str] = None, x_label: Optional[str] = None, kinematics_override: Optional[str] = None, result_transform: Optional[str] = None, x: Optional[str] = None, plot_x: Optional[str] = None, x_scale: Optional[str] = None, y_scale: Optional[str] = None, line_by: Optional[list] = None, figure_by: Optional[list] = None, extra_labels: Optional[Mapping[str, List]] = None, normalize: Optional[dict] = None, already_digested: Optional[bool] = True)[source]

Bases: object

already_digested: bool | None = True
data_reference: str | None = None
dataset_label: str | None = None
experiment: str | None = None
extra_labels: Mapping[str, List] | None = None
figure_by: list | None = None
func_labels: dict
kinematics_override: str | None = None
line_by: list | None = None
nnpdf31_process: str | None = None
normalize: dict | None = None
plot_x: str | None = None
process_description: str | None = None
result_transform: str | None = None
theory_reference: str | None = None
x: str | None = None
x_label: str | None = None
x_scale: str | None = None
y_label: str | None = None
y_scale: str | None = None
nnpdf_data.validphys_compatibility.legacy_to_new_map(dataset_name, sys=None)[source]

Find the new dataset name and variant corresponding to an old dataset and systematics choice

nnpdf_data.validphys_compatibility.new_to_legacy_map(dataset_name, variant_used)[source]

Loop over the dictionary and find the right dataset.

Since it is posible to have more than 1 dataset mapped to the same new one, returns a list of everything that matches.

This function will loop over the entire dictionary of mappings and selects 1. All datasets that match exactly what’s in the runcard (dataset & variant): exact_matches 2. All datasets that match the dataset name: matches If there are any exact_matches, it will return only those; otherwise, return all matches if there are no matches at all, return None