nnpdf_data package
- nnpdf_data.load_dataset_metadata(dataset_name, variant=None)[source]
Given a dataset name, return the metadata
Subpackages
- nnpdf_data.filter_utils package
- Submodules
- nnpdf_data.filter_utils.correlations module
- nnpdf_data.filter_utils.hera_utils module
- nnpdf_data.filter_utils.legacy_jets_utils module
HEP_table_to_df_ATLAS_1JET_7TEV()HEP_table_to_df_ATLAS_1JET_8TEV_R06()HEP_table_to_df_ATLAS_2JET_7TEV_R06()JEC_error_matrix_CMS_2JET_7TEV()bin_by_bin_covmat_CMS_2JET_7TEV()block_diagonal_corr_CMS_1JET_8TEV()dat_file_to_df_CMS_2JET_7TEV()fill_df_ATLAS_1JET_7TEV()fill_df_ATLAS_1JET_8TEV_R06()fill_df_ATLAS_2JET_7TEV_R06()get_corr_dat_file_CMS_2JET_7TEV()get_data_values_ATLAS_1JET_8TEV_R06()get_data_values_CMS_1JET_8TEV()get_data_values_CMS_2JET_7TEV()get_kinematics_ATLAS_1JET_8TEV_R06()get_kinematics_CMS_1JET_8TEV()get_kinematics_CMS_2JET_7TEV()get_stat_correlations_CMS_1JET_8TEV()get_stat_uncertainties_CMS_1JET_8TEV()get_stat_uncertainties_CMS_2JET_7TEV()get_uncertainties_df_CMS_1JET_8TEV()lumi_covmat_CMS_2JET_7TEV()process_err_ATLAS_1JET_8TEV_R06()process_err_ATLAS_2JET_7TEV_R06()process_err_CMS_1JET_8TEV()range_str_to_floats()uncertainties_df_CMS_1JET_8TEV()unfolding_error_matrix_CMS_2JET_7TEV()
- nnpdf_data.filter_utils.nmc_hepdata_utils module
- nnpdf_data.filter_utils.poldata_utils module
- nnpdf_data.filter_utils.uncertainties module
- nnpdf_data.filter_utils.utils module
Submodules
nnpdf_data.commondataparser module
This module implements parsers for commondata and its associated metadata and uncertainties files
into useful structures that can be fed to the main nnpdf_data.coredata.CommonData class.
A CommonData file is completely defined by a dataset name (which defines the folder in which the information is) and observable name (which defines the specific data, fktables and plotting settings to read).
<experiment>_<process>_<energy>{_<extras>}_<observable>
Where the folder name is <experiment>_<process>_<energy>{_<extras>}
The definition of all information for a given dataset (and all its observable) is in the
metadata.yaml file and its implemented_observables.
This module defines a number of parsers using the validobj library.
The full metadata.yaml is read as a SetMetaData object
which contains a list of ObservableMetaData.
These ObservableMetaData are the “datasets” of NNPDF for all intents and purposes.
The parent SetMetaData collects some shared variables such as the version of the dataset,
arxiv, inspire or hepdata ids, the folder in which the data is, etc.
The main class in this module is thus ObservableMetaData which holds _all_ information
about the particular dataset-observable that we are interested in (and a reference to its parent).
- Inside the
ObservableMetaDatawe can find: TheoryMeta: contains the necessary information to read the (new style) fktablesKinematicsMeta: containins metadata about the kinematicsVariant: variant to be usedPlottingOptions: plotting style and information for validphys, only utilized ifvalidphys is also installed.
The CommonMetaData defines how the CommonData file is to be loaded,
by modifying the CommonMetaData using one of the loaded Variants one can change the resulting
nnpdf_data.coredata.CommonData object.
- class nnpdf_data.commondataparser.ObservableMetaData(observable_name: str, observable: dict, ndata: int, plotting: validphys.plotoptions.plottingoptions.PlottingOptions, process_type: Annotated[Union[nnpdf_data.process_options._Process, str], InputType(Any), Validator(<function ValidProcess at 0x7fa3bf3f6980>)], kinematic_coverage: list[str], kinematics: nnpdf_data.commondataparser.ValidKinematics, data_uncertainties: list[typing.Annotated[pathlib.Path, InputType(<class 'str'>), Validator(<function ValidPath at 0x7fa3bf49cae0>)]], data_central: Optional[Annotated[pathlib.Path, InputType(<class 'str'>), Validator(<function ValidPath at 0x7fa3bf49cae0>)]] = None, theory: Optional[nnpdf_data.commondataparser.TheoryMeta] = None, tables: Optional[list] = <factory>, npoints: Optional[list] = <factory>, variants: Optional[dict[str, nnpdf_data.commondataparser.Variant]] = <factory>, applied_variant: Optional[str] = None, ported_from: Optional[str] = None, _parent: Optional[Any] = None)[source]
Bases:
object- apply_variant(variant_name)[source]
Return a new instance of this class with the variant applied
This class also defines how the variant is applied to the commondata. If more than a variant is being used, this function will be called recursively until all variants are applied.
- check()[source]
Various checks to apply manually to the observable before it is used anywhere These are not part of the __post_init__ call since they can only happen after the metadata has been read, the observable selected and (likely) variants applied.
- property cm_energy
- data_uncertainties: list[pathlib.Path]
- digest_plotting_variable(variable)[source]
Digest plotting variables in the
line_byorfigure_byfields and return the appropiatekXor other label such that the plotting functions of validphys can understand it.These might be variables included as part of the kinematics or extra labels defined in the plotting dictionary.
- property experiment
- property is_integrability
- property is_nnpdf_special
Is this an NNPDF special dataset used for e.g., Lagrange multipliers or QED fits
- property is_ported_dataset
Return True if this is an automatically ported dataset that has not been updated
- property is_positivity
- kinematics: ValidKinematics
- property kinlabels
Return the kinematic labels in the same order as they are set in
kinematic_coverage(which in turns follow the key kinematic_coverage) If this is a ported dataset, rely on the process type using the legacy labels
- load_data_central()[source]
Loads the data for this commondata returns a dataframe
- Returns:
a dataframe containing the data
- Return type:
pd.DataFrame
- load_kinematics(fill_to_three=True, drop_minmax=True)[source]
Returns a dataframe with the kinematic information
- load_uncertainties()[source]
Returns a dataframe with all appropiate uncertainties
- Returns:
a dataframe containing the uncertainties
- Return type:
pd.DataFrame
- property name
- property nnpdf_metadata
- property path_data_central
- property path_kinematics
- property paths_uncertainties
- plotting: PlottingOptions
- property plotting_options
- property process
- property setname
- theory: TheoryMeta | None = None
- variants: dict[str, nnpdf_data.commondataparser.Variant] | None
- class nnpdf_data.commondataparser.SetMetaData(setname: str, version: int, version_comment: str, nnpdf_metadata: dict, implemented_observables: list[nnpdf_data.commondataparser.ObservableMetaData], arXiv: ValidReference | None = None, iNSPIRE: ValidReference | None = None, hepdata: ValidReference | None = None, metadata_file: Path | None = None)[source]
Bases:
objectMetadata of the whole set
- property allowed_datasets
Return the implemented datasets as a list <setname>_<observable>
- property allowed_observables
observable} dictionary
- Type:
Returns the implemented observables as a {observable_name.upper()
- arXiv: ValidReference | None = None
- property cm_energy
Return the center of mass energy as GeV if it can be understood from the name otherwise return None
- property folder
- hepdata: ValidReference | None = None
- iNSPIRE: ValidReference | None = None
- implemented_observables: list[nnpdf_data.commondataparser.ObservableMetaData]
- class nnpdf_data.commondataparser.TheoryMeta(FK_tables: list[tuple], operation: str = 'NULL', conversion_factor: float = 1.0, shifts: dict | None = None, normalization: dict | None = None, comment: str | None = None)[source]
Bases:
objectContains the necessary information to load the associated fktables
The theory metadata must always contain a key
FK_tableswhich defines the fktables to be loaded. TheFK_tablesis organized as a double list such that:The inner list is concatenated In practice these are different fktables that might refer to the same observable but that are divided in subgrids for practical reasons. The outer list instead are the operands for whatever operation needs to be computed in order to match the experimental data.
In addition there are other flags that can affect how the fktables are read or used:
operation: defines the operation to apply to the outer list
- shifts: mapping with the single fktables and their respective shifts
useful to create “gaps” so that the fktables and the respective experimental data are ordered in the same way (for instance, when some points are missing from a grid)
This class is inmutable, what is read from the commondata metadata should be considered final
Example
>>> from nnpdf_data.commondataparser import TheoryMeta >>> from validobj import parse_input >>> from ruamel.yaml import YAML >>> theory_raw = ''' ... FK_tables: ... - - fk1 ... - - fk2 ... - fk3 ... operation: ratio ... ''' >>> theory = YAML(typ='safe').load(theory_raw) >>> parse_input(theory, TheoryMeta) TheoryMeta(FK_tables=[['fk1'], ['fk2', 'fk3']], operation='RATIO', shifts = None, conversion_factor=1.0, comment=None, normalization=None))
- fktables_to_paths(grids_folder)[source]
Given a source for pineappl grids, constructs the lists of fktables to be loaded
- class nnpdf_data.commondataparser.ValidKinematics(file: Path, variables: dict[str, nnpdf_data.commondataparser.ValidVariable])[source]
Bases:
objectContains the metadata necessary to load the kinematics of the dataset. The variables should be a dictionary with the key naming the variable and the content complying with the
ValidVariablespec.Only the kinematics defined by the key
kinematic_coveragewill be loaded, which must be three.Three shall be the number of the counting and the number of the counting shall be three. Four shalt thou not count, neither shalt thou count two, excepting that thou then proceedeth to three. Once the number three, being the number of the counting, be reached, then the kinematics be loaded in the direction of thine validobject.
- apply_label(var, value)[source]
For a given value for a given variable, return the labels as label = value (unit) If the variable is not included in the list of variables, returns None as the variable could’ve been transformed by a kinematic transformation
- get_label(var)[source]
For the given variable, return the label as label (unit) If the label is an “extra” return the last one
- variables: dict[str, nnpdf_data.commondataparser.ValidVariable]
- class nnpdf_data.commondataparser.ValidReference(url: str, version: int | None = None, journal: str | None = None, tables: list[int] = <factory>)[source]
Bases:
objectHolds literature information for the dataset
- class nnpdf_data.commondataparser.ValidVariable(label: str, description: str = '', units: str = '')[source]
Bases:
objectDefines the variables
- class nnpdf_data.commondataparser.Variant(data_uncertainties: list[pathlib.Path] | None = None, theory: TheoryMeta | None = None, data_central: Path | None = None, experiment: str | None = None)[source]
Bases:
objectThe new commondata format allow the usage of variants A variant can overwrite a number of keys, as defined by this dataclass:
data_uncertainties
theory
data_central
This class may overwrite some other keys for the benefit of reproducibility of old NNPDF fits, but the usage of these features is undocumented and discouraged.
- data_uncertainties: list[pathlib.Path] | None = None
- theory: TheoryMeta | None = None
- nnpdf_data.commondataparser.load_commondata(metadata)[source]
TODO: update this docstring since now the load_commondata_new takes the information from the metadata, and the name -> split is done outside
In the current iteration of the commondata, each of the commondata (i.e., an observable from a data publication) correspond to one single observable inside a folder which is named as “<experiment>_<process>_<energy>_<extra>” The observable is defined by a last suffix of the form “_<obs>” so that the full name of the dataset is always:
"<experiment>_<process>_<energy>{_<extra>}_<obs>"where <extra> is optional.
This function right now works under the assumotion that the folder/observable is separated in the last _ so that:
folder_name = <experiment>_<process>_<energy>{_<extra>}
but note that this convention is still not fully defined.
This function returns a commondata object constructed by parsing the metadata.
Once a variant is selected, it can no longer be changed
Note that this function reproduces
parse_commondatabelow, which parses the _old_ file format
- nnpdf_data.commondataparser.parse_new_metadata(metadata_file, observable_name, variant=None)[source]
Given a metadata file in the new format and the specific observable to be read load and parse the metadata and select the observable. If any variants are selected, apply them.
The triplet (metadata_file, observable_name, variant) define unequivocally the information to be parsed from the commondata library
nnpdf_data.coredata module
The all-important CommonData object
- class nnpdf_data.coredata.CommonData(setname: str, ndata: int, commondataproc: str, nkin: int, nsys: int, commondata_table: DataFrame, systype_table: DataFrame, legacy: bool = False, legacy_names: list | None = None, kin_variables: list | None = None)[source]
Bases:
objectData contained in Commondata files, relevant cuts applied.
- Parameters:
setname (str) – Name of the dataset
ndata (int) – Number of data points
commondataproc (str) – Process type, one of 21 options
nkin (int) – Number of kinematics specified
nsys (int) – Number of systematics
commondata_table (pd.DataFrame) – Pandas dataframe containing the commondata
systype_table (pd.DataFrame) – Pandas dataframe containing the systype index for each systematic alongside the uncertainty type (ADD/MULT/RAND) and name (CORR/UNCORR/THEORYCORR/SKIP)
systematics_table (pd.DataFrame) – Panda dataframe containing the table of systematics
- property additive_errors
Returns the systematics which are additive (systype is ADD) as absolute uncertainties (same units as data), with SKIP uncertainties removed.
- property central_values
- commondata_table: DataFrame
- export(folder_path)[source]
Wrapper around export_data and export_uncertainties to write both uncertainties and data after filtering to a given folder
- export_data(buffer)[source]
Exports the central data defined by this commondata instance to the given buffer
- export_uncertainties(buffer)[source]
Exports the uncertainties defined by this commondata instance to the given buffer
- property kinematics
- property multiplicative_errors
Returns the systematics which are multiplicative (systype is MULT) in a percentage format, with SKIP uncertainties removed.
- property stat_errors
- systematic_errors(central_values=None)[source]
Returns all systematic errors as absolute uncertainties, with a single column for each uncertainty. Converts
multiplicative_errorsto units of data and then appends ontoadditive_errors. By default uses the experimental central values to perform conversion, but the user can supply a 1-D array of central values, with lengthself.ndata, to use instead of the experimental central values to calculate the absolute contribution of the multiplicative systematics.- Parameters:
central_values (None, np.array) – 1-D array containing alternative central values to combine with multiplicative uncertainties. This array must have length equal to
self.ndata. By defaultcentral_valuesis None, and the central values of the commondata are used.- Returns:
systematic_errors – Dataframe containing systematic errors.
- Return type:
pd.DataFrame
- systype_table: DataFrame
nnpdf_data.disp_theory module
nnpdf_data.process_options module
Module to hold process dependent options
Only variables included in the _Vars enum and processes included in the Processes dictionary are allowed.
nnpdf_data.theory module
This module provides a unique source and definition for all the possible parameters that a theory card can contain.
- class nnpdf_data.theory.TheoryCard(ID: int, PTO: Literal[0, 1, 2, 3], FNS: str, IC: int, ModEv: Literal['EXA', 'TRN'], XIR: float, XIF: float, QED: int, Q0: float, mc: float, Qmc: float, kcThr: float, mb: float, Qmb: float, kbThr: float, mt: float, Qmt: float, ktThr: float, CKM: list[float], MZ: float, MW: float, GF: float, SIN2TW: float, TMC: int, MP: float, Comments: str, alphas: float, alphaqed: float, Qref: float, XIA: float = 1.0, NfFF: Optional[int] = None, nfref: Optional[int] = None, MaxNfPdf: Optional[int] = 5, nf0: Optional[int] = None, HQ: Optional[Literal['POLE', 'MSBAR']] = 'POLE', IterEv: Optional[int] = None, ModSV: Optional[str] = None, DAMP: Optional[int] = 0, DAMPPOWERc: Optional[int] = None, DAMPPOWERb: Optional[int] = None, FONLLParts: Optional[str] = None, PTOEKO: Optional[int] = None, PTODIS: Optional[int] = None, n3lo_ad_variation: Optional[list] = <factory>, n3lo_cf_variation: Optional[int] = 0, use_fhmruvv: Optional[bool] = True, MaxNfAs: Optional[int] = None, SxRes: Optional[int] = None, SxOrd: Optional[str] = None, EScaleVar: Optional[int] = None, Qedref: Optional[float] = None, global_nx: Optional[int] = None)[source]
Bases:
object
- exception nnpdf_data.theory.TheoryNotFoundInDatabase[source]
Bases:
FileNotFoundError
nnpdf_data.theorydbutils module
theorydbutils.py
low level utilities for querying the theory database file and representing the data as a python object.
- nnpdf_data.theorydbutils.fetch_all(theory_database: Path)[source]
Looks in the theory database and returns a dataframe with theory info for all theories that can be found in the folder
theory_database. Every.yamlfile found in the folder will be considered a possible theory.- Parameters:
theory_database (Path) – pathlib.Path pointing to the folder with theory cards
- Returns:
theory_info_dataframe – dataframe filled with all entries in theorydb file
- Return type:
pd.Dataframe
Example
>>> from nnpdf_data import theory_cards >>> from nnpdf_data.theorydbutils import fetch_all >>> theory_df = fetch_all(theory_cards)
- nnpdf_data.theorydbutils.fetch_theory(theory_database: Path, theoryID: int)[source]
Looks in the theory card folder and returns a dictionary of theory info for the theory number specified by theoryID.
- Parameters:
theory_database (Path) – pathlib.Path pointing to the folder with the theory cards
theoryID (int) – numeric identifier of theory to query info
- Returns:
theory_info_dict – dictionary filled with relevant entry from theory database
- Return type:
Example
>>> from nnpdf_data import theory_cards >>> from nnpdf_data.theorydbutils import fetch_theory >>> theory = fetch_theory(theory_cards, 700)
nnpdf_data.utils module
- nnpdf_data.utils.get_nnpdf_profile(profile_path=None)[source]
Returns the NNPDF profile as a dictionary
If no
profile_pathis provided it will be autodiscovered in the following order:Environment variable $NNPDF_PROFILE_PATH
${XDG_CONFIG_HOME}/NNPDF/nnprofile.yaml (usually ~/.config/nnprofile)
Any value not filled by 1 or 2 will then be filled by the default values found within the validphys python package nnporfile_default.yaml
If
nnpdf_shareis set to the special keyRELATIVE_TO_PYTHONthe python prefix (Path(sys.prefix)/"share"/"NNPDF") will be used.The
data_pathkey is special in that it will _always_ be filled with the location of the nnpdf_data package (albeit with lower priority if there is an entry).
- nnpdf_data.utils.parse_yaml_inp(input_yaml, spec)[source]
Helper function to parse yaml using the validobj library and print useful error messages in case of a parsing error.
https://validobj.readthedocs.io/en/latest/examples.html#yaml-line-numbers
- nnpdf_data.utils.quick_yaml_load(filepath)[source]
If libyaml is available, use the C loader to speed up some of the read https://pyyaml.org/wiki/LibYAML libyaml is available for most linux distributions
nnpdf_data.validphys_compatibility module
This file exists solely for me to be able to upload a package to PyPI for the nnpdf data which does not depend on the rest of the NNPDF code. This file should not be modified. Everything in this file is deprecated and should be removed, and the only reason this is needed is because we are still mixing new and old data.
This also means that _things_ that would’ve been loaded as a kinematic transformation or result transformation are loaded as boring strings that cannot do anything, as it should be.
- class nnpdf_data.validphys_compatibility.PlottingOptions(func_labels: dict = <factory>, dataset_label: Optional[str] = None, experiment: Optional[str] = None, nnpdf31_process: Optional[str] = None, data_reference: Optional[str] = None, theory_reference: Optional[str] = None, process_description: Optional[str] = None, y_label: Optional[str] = None, x_label: Optional[str] = None, kinematics_override: Optional[str] = None, result_transform: Optional[str] = None, x: Optional[str] = None, plot_x: Optional[str] = None, x_scale: Optional[str] = None, y_scale: Optional[str] = None, line_by: Optional[list] = None, figure_by: Optional[list] = None, extra_labels: Optional[Mapping[str, List]] = None, normalize: Optional[dict] = None, already_digested: Optional[bool] = True)[source]
Bases:
object
- nnpdf_data.validphys_compatibility.legacy_to_new_map(dataset_name, sys=None)[source]
Find the new dataset name and variant corresponding to an old dataset and systematics choice
- nnpdf_data.validphys_compatibility.new_to_legacy_map(dataset_name, variant_used)[source]
Loop over the dictionary and find the right dataset.
Since it is posible to have more than 1 dataset mapped to the same new one, returns a list of everything that matches.
This function will loop over the entire dictionary of mappings and selects 1. All datasets that match exactly what’s in the runcard (dataset & variant): exact_matches 2. All datasets that match the dataset name: matches If there are any exact_matches, it will return only those; otherwise, return all matches if there are no matches at all, return None