nnpdf_data.filter_utils package
Submodules
nnpdf_data.filter_utils.correlations module
nnpdf_data.filter_utils.hera_utils module
- class nnpdf_data.filter_utils.hera_utils.commondata(central_values: numpy.ndarray, kinematics: numpy.ndarray, statistical_uncertainties: numpy.ndarray, systematic_uncertainties: numpy.ndarray, systypes: List[tuple[str, str]], process: str, dataset_name: str, kinematic_quantities: List[str])[source]
Bases:
object- central_values: ndarray
- kinematics: ndarray
- statistical_uncertainties: ndarray
- systematic_uncertainties: ndarray
nnpdf_data.filter_utils.legacy_jets_utils module
Utils specifically needed to ease the implementation of the 7 and 8 TeV jet datasets, these are: CMS_2JET_7TEV, CMS_1JET_8TEV, ATLAS_1JET_8TEV_R06, ATLAS_2JET_7TEV_R06
- nnpdf_data.filter_utils.legacy_jets_utils.HEP_table_to_df_ATLAS_1JET_7TEV(table)[source]
Given hep data table, version, and variant, initialize a pandas DataFrame with index given by Ndata, columns by the uncertainties and np.nan entries.
Note that for this dataset, the same type of systematic JES uncertainty can be either symmetric or asymmetric depending on the bin in pT. Hence, all JES uncertainties are treated as asymmetric here.
- nnpdf_data.filter_utils.legacy_jets_utils.HEP_table_to_df_ATLAS_1JET_8TEV_R06(table, version, variant='nominal')[source]
Given hep data table return a pandas DataFrame with index given by Ndata, columns by the uncertainties and np.nan entries
- nnpdf_data.filter_utils.legacy_jets_utils.HEP_table_to_df_ATLAS_2JET_7TEV_R06(heptable, scenario='nominal')[source]
Given hep data table return a pandas DataFrame with index given by Ndata, columns by the uncertainties and np.nan entries
- Parameters:
heptable (str) – path to hepdata table
scenario (0, 1, 2) – 0 = nominal, 1 = stronger, 2 = weaker
- nnpdf_data.filter_utils.legacy_jets_utils.JEC_error_matrix_CMS_2JET_7TEV()[source]
Jet Energy Scale (JET): 14 Asymmetric uncertainties correlated across all rapidity and mass bins (CORR). This uncertainty is not always presented as (left<0 and right>0), e.g. [-delta_left,+delta_right] Hence the D’Agostini prescription for symmetrising errors is not valid here because it works with the only case displayed above. Instead, we use here the experimentalists prescription, where we take every subpart of the uncertainty left and right as independent source of uncertainty. This is motivated by taking the average of the left and right uncertainty, hence the origin of the sqrt(2) that we divide by.
- Returns:
covariance matrix for JET energy scale uncertainty
- Return type:
np.array
- nnpdf_data.filter_utils.legacy_jets_utils.bin_by_bin_covmat_CMS_2JET_7TEV()[source]
Bin-by-Bin uncertainty: this is a symmetric uncertainty fully uncorrelated accross bins of mass and rapidity (UNCORR)
NOTE: this function is needed to test only whether the full covmat coincides with the old implementation and can be removed at some point
- Returns:
covariance matrix for bin by bin uncertainty
- Return type:
np.array
- nnpdf_data.filter_utils.legacy_jets_utils.block_diagonal_corr_CMS_1JET_8TEV(tables)[source]
forms block diagonal correlation matrix for stat uncertainties. Each block corresponds to a rapidity bin.
- Parameters:
tables (list) – list of integers numbering the tables
- Returns:
block diagonal matrix of dim ndata x ndata
- Return type:
np.array
- nnpdf_data.filter_utils.legacy_jets_utils.dat_file_to_df_CMS_2JET_7TEV()[source]
from dijet_sys.dat table return a pandas DataFrame with index given by Ndata, columns by the uncertainties and np.nan entries
- Returns:
list of dataframes
- Return type:
- nnpdf_data.filter_utils.legacy_jets_utils.fill_df_ATLAS_1JET_7TEV(table, version, variant='nominal')[source]
Fill a data frame with index corresponding to measured datapoints and columns to different uncertainties Each df is for a fixed rapidity bin.
- nnpdf_data.filter_utils.legacy_jets_utils.fill_df_ATLAS_1JET_8TEV_R06(table, version, variant='nominal')[source]
Fill a data frame with index corresponding to measured datapoints and columns to different uncertainties Each df is for a fixed rapidity bin.
- nnpdf_data.filter_utils.legacy_jets_utils.fill_df_ATLAS_2JET_7TEV_R06(heptable, scenario='nominal')[source]
Fill a data frame with index corresponding to dijet mass bins and columns to different uncertainties Each df is for a fixed rapidity bin.
- nnpdf_data.filter_utils.legacy_jets_utils.get_corr_dat_file_CMS_2JET_7TEV(filename)[source]
read out correlation matrices from the dijet_corr.dat file
- nnpdf_data.filter_utils.legacy_jets_utils.get_data_values_ATLAS_1JET_8TEV_R06(tables, version)[source]
returns the central data. Note: central data is the same for both correlation scenarios.
- nnpdf_data.filter_utils.legacy_jets_utils.get_data_values_CMS_1JET_8TEV(tables, version)[source]
returns the central data
- nnpdf_data.filter_utils.legacy_jets_utils.get_data_values_CMS_2JET_7TEV(tables, version)[source]
returns the central data
- nnpdf_data.filter_utils.legacy_jets_utils.get_kinematics_ATLAS_1JET_8TEV_R06(tables, version)[source]
returns the relevant kinematics values. Note: kinematics are the same for both correlation scenarios.
- nnpdf_data.filter_utils.legacy_jets_utils.get_kinematics_CMS_1JET_8TEV(tables, version)[source]
returns the relevant kinematics values
- nnpdf_data.filter_utils.legacy_jets_utils.get_kinematics_CMS_2JET_7TEV(tables, version)[source]
returns the relevant kinematics values
- nnpdf_data.filter_utils.legacy_jets_utils.get_stat_correlations_CMS_1JET_8TEV(table)[source]
Read a CMS_8TeV_jets_Ybin#___CMS_8TeV_jets_Ybin#.dat file and output the statistical correlation matrix.
- Parameters:
table (int) – number of the table
- Returns:
2-D array
- Return type:
np.array
- nnpdf_data.filter_utils.legacy_jets_utils.get_stat_uncertainties_CMS_1JET_8TEV()[source]
function used to get the statistical uncertainty from the HEPdata tables.
- Returns:
array with ordered stat errors for all tables
- Return type:
np.array
- nnpdf_data.filter_utils.legacy_jets_utils.get_stat_uncertainties_CMS_2JET_7TEV()[source]
function used to get the statistical uncertainty from the HEPdata tables.
- Returns:
dictionary with keys = number of table value = list of statistical uncertainties
- Return type:
- nnpdf_data.filter_utils.legacy_jets_utils.lumi_covmat_CMS_2JET_7TEV()[source]
Luminosity uncertainty: this is a symmetric uncertainty of 2.2% correlated accross all mass and rapidity bins and all CMS datasets at 7 TeV, hence the keyword (CMSLUMI11).
NOTE: this function is needed to test only whether the full covmat coincides with the old implementation and can be removed at some point
- Returns:
covariance matrix for luminosity uncertainty
- Return type:
np.array
- nnpdf_data.filter_utils.legacy_jets_utils.process_err_ATLAS_1JET_8TEV_R06(error)[source]
Note: the d’Agostini prescription for the symmetrization of the error does not hold here. We follow the experimentalists prescription.
- Parameters:
error (dictionary) – e.g. {‘label’: ‘sys’, ‘symerror’: 0.1%}
- Returns:
tuple containing two floats
- Return type:
- nnpdf_data.filter_utils.legacy_jets_utils.process_err_ATLAS_2JET_7TEV_R06(error, cv)[source]
Converts an error given in percentage of the central data value in absolute value.
Note: the d’Agostini prescription for the symmetrization of the error does not hold here. We follow here the experimental prescription
- nnpdf_data.filter_utils.legacy_jets_utils.process_err_CMS_1JET_8TEV(df)[source]
Given the uncertainties dataframe, if the two variations in the pair (of uncertainties) have the same sign, only the largest (in absolute value) is retained, while the other is set to zero
nnpdf_data.filter_utils.nmc_hepdata_utils module
“Common functions to parse NMC data from Hepdata.
nnpdf_data.filter_utils.poldata_utils module
- nnpdf_data.filter_utils.poldata_utils.fluctuate_data(central: ndarray, abserr: ndarray) ndarray[source]
Fluctuate the central values according to the uncertainties. This is done because when the measurements are not real data but rather pseudodata projections, the central values are the ones computed from theory predictions and fluctuaded according to the projected uncertainties.
- Parameters:
central (np.ndarray) – array of central values
abserr (np.ndarray) – array containing the values of the errors
- Returns:
fluctuated central values according to a normal distribution
- Return type:
np.ndarray
- nnpdf_data.filter_utils.poldata_utils.read_central_values(path: Path) ndarray[source]
Read the central values from the theory predictions.
- Parameters:
path (Path) – path to the file containing the central values
- Returns:
array containing the central predictions
- Return type:
np.ndarray
- nnpdf_data.filter_utils.poldata_utils.read_excel(path_xlsx: Path, beams: tuple) DataFrame[source]
Parse the xlsx file containing all the information regarding the projections and returns the ones corresponding to the chosen beam energies.
- Parameters:
path_xlsx (Path) – path to the xlsx file
beams (tuple) – tuple specifying the beam energies of the lepton & proton
- Returns:
returns a panda table corresponding to the chosen beams
- Return type:
pd.DataFrame
- nnpdf_data.filter_utils.poldata_utils.read_txt_data(path_txt: Path) DataFrame[source]
Reads a rawdata .txt file and returns a pandas dataframe in which the columns are ordered as [x, Q2, abs]
- Parameters:
path_txt (Path) – path to the .txt file
- Returns:
a panda table containing the values
- Return type:
pd.DataFrame
- nnpdf_data.filter_utils.poldata_utils.write_data(df: DataFrame, asym: bool = True, abserr: ndarray | None = None, add_fluctuate: bool = False) None[source]
Write the input kinematics, central values, and uncertainties into the new commondata format.
- Parameters:
df (pd.DataFrame) – a pandas table containing the information required to generate the commondata
asym (bool) – boolean that states whether or not the observable is an Asymmetry, as is the case for ATHENA
abserr (Optional[Union[np.ndarray, None]]) – if not None contains the fluctuated centra values
add_fluctuate (bool) – whether or not to fluctuate the central values
nnpdf_data.filter_utils.uncertainties module
nnpdf_data.filter_utils.utils module
General Python utilities for commondata implementation.
This module provides helpful functions that automate a few tasks that are regularly needed for the implementation of experimental data to the commondata format. If there are any additional functions that could be added here as they could simplify some repetitve tasks, please do suggest.
Before the usage of any functions, it is recommended to read the docstrings of the function to understand the inputs and outputs.
- nnpdf_data.filter_utils.utils.check_xq2_degenearcy(Q2, x)[source]
Check is the pair of (x,Q2) is unique.
- nnpdf_data.filter_utils.utils.concat_matrices(rows, columns, list_of_matrices)[source]
Join smaller matrices into a large matrix.
This function aims to simplify the process of joining multiple smaller matrices into one large matrix. Such a need could arise, for instance, when cross variable covariance matrices are provided but the user needs to join all the matrices to generate the full covariance matrix corresponding to the entire dataset.
- Parameters:
rows (int) – No. of rows of matrices to be concatenated. E.g., if 6 matrices: A, B, C, D, E, F need to be joined as [[A, B, C], [D, E, F]], the number of rows would be 2.
columns (int) – No. of columns of matrices to be concatenated. In the above example, this would be 3.
list_of_matrices (list) – A list of the matrices that have to concatenated row by row. In the above example, this would be [A, B, C, D, E, F]. The matrices themselves need to be provided as a list of lists, or a numpy 2d array. If the user has the matrix in a 1d row by row form, use matList_to_matrix() to convert it. It is assumed the user verifies that all the input matrices have the correct dimensions. Matrices with incompatible dimensions will lead to undesired behavior.
- Returns:
final_mat_list – A one dimensional list which contains the elements of the final, fully concatenated matrix row by row.
- Return type:
- nnpdf_data.filter_utils.utils.cormat_to_covmat(err_list, cormat_list)[source]
Convert correlation matrix elements to covariance matrix elements.
- Parameters:
err_list (list) – A one dimensional list which contains the uncertainty associated to each data point in order.
cormat_list (list) – A one dimensional list which contains the elements of the correlation matrix row by row. Since experimental datasets provide these matrices in a list form, this simplifies the implementation for the user.
- Returns:
covmat_list – A one dimensional list which contains the elements of the covariance matrix row by row.
- Return type:
- nnpdf_data.filter_utils.utils.correlation_to_covariance(correlation, uncertainties)[source]
Converts a correlation matrix into a covariance matrix using a list of uncertainties.
Parameters:
- correlationnp.ndarray
A square matrix of correlations.
- uncertaintiesnp.ndarray
A 1D array of uncertainties.
Returns:
- np.ndarray
The corresponding covariance matrix.
- nnpdf_data.filter_utils.utils.covmat_to_artunc(ndata, covmat_list, no_of_norm_mat=0, full_precision=False)[source]
Convert the covariance matrix to a matrix of artificial uncertainties.
- Parameters:
ndata (integer) – Number of data points
covmat_list (list) – A one dimensional list which contains the elements of the covariance matrix row by row. Since experimental datasets provide these matrices in a list form, this simplifies the implementation for the user.
no_of_norm_mat (int) – Normalized covariance matrices may have an eigenvalue of 0 due to the last data point not being linearly independent. To allow for this, the user should input the number of normalized matrices that are being treated in an instance. For example, if a single covariance matrix of a normalized distribution is being processed, the input would be 1. If a covariance matrix contains pertains to 3 normalized datasets (i.e. cross covmat for 3 distributions), the input would be 3. The default value is 0 for when the covariance matrix pertains to an absolute distribution.
full_precision (bool) – Use mpm to keep a larger number of decimals for numerical stability when computing the eigenvalues/eigenvectors
- Returns:
artunc – A two dimensional matrix (given as a list of lists) which contains artificial uncertainties to be added to the commondata. i^th row (or list) contains the artificial uncertainties of the i^th data point.
- Return type:
- nnpdf_data.filter_utils.utils.cross_cormat_to_covmat(row_err_list, col_err_list, cormat_list)[source]
Convert cross correlation matrix elements (i.e. those between different different variables or observables) to covariance matrix elements.
- Parameters:
row_err_list (list) – A one dimensional list which contains the uncertainty associated to each data point of the variable that is given on the vertical axis.
col_err_list (list) – A one dimensional list which contains the uncertainty associated to each data point of the variable that is given on the horizontal axis.
cormat_list (list) – A one dimensional list which contains the elements of the correlation matrix row by row. Since experimental datasets provide these matrices in a list form, this simplifies the implementation for the user.
- Returns:
covmat_list – A one dimensional list which contains the elements of the covariance matrix row by row.
- Return type:
- nnpdf_data.filter_utils.utils.decompose_covmat(covmat, full_precision=False)[source]
Given a covmat it return an array sys with shape (ndat,ndat) giving ndat correlated systematics for each of the ndat point. The original covmat is obtained by doing sys@sys.T
- nnpdf_data.filter_utils.utils.matlist_to_matrix(rows, columns, mat_list)[source]
Convert a 1d list to a 2d matrix.
Note: This utils function is not strictly needed for data implementation, however, it is provided for the aid of the user due to how matrices are treated throughout all the other functions. This function allows the user to convert a list that contains the elemnets of matrix row by row to a proper matrix, if need be for any reason.
- nnpdf_data.filter_utils.utils.percentage_to_absolute(percentage, value)[source]
Compute the absolute value of uncertainty from percentage.
- Parameters:
percentage (string/float) – Experimental datasets can provide the percentage uncertainties with a % sign or without one. The function will autostrip % sign and convert to a float type in case the percentage uncertainty comes with a % sign. Else, it will directly perform the computation.
value (float) – The data point
- Returns:
absolute – The absolute value of the uncertainty
- Return type:
- nnpdf_data.filter_utils.utils.prettify_float(dumper, value)[source]
Override the default yaml representer: https://github.com/yaml/pyyaml/blob/48838a3c768e3d1bcab44197d800145cfd0719d6/lib/yaml/representer.py#L189
This function is used to prettify the float representation in the yaml file. If the float has more than 8 digits, it will be represented in scientific notation with 8 digits.
Note:
When importing yaml in a module,
yaml.add_representer(float, prettify_float)
must be called to use this function.
- nnpdf_data.filter_utils.utils.sort_eigenvalues(evals, evecs)[source]
Defines ordering of the eigenvalues and eigenvectors such that eigenvalues are given in decreasing order and the first non-zero entry of each eigenvector is positive. :param evals: :type evals: output of np.linalg.eig :param evecs: :type evecs: output of np.linalg.eig
- Returns:
evacs_sorted, evecs_sorted (ordered eigen- values and vectors,)
as defined above
- nnpdf_data.filter_utils.utils.symmetrize_errors(delta_plus, delta_minus)[source]
Compute the symmterized uncertainty and the shift in data point.
- nnpdf_data.filter_utils.utils.trimat_to_fullmat(mode, tri_mat_list)[source]
Convert a list of values of a triangular matrix to a symmetric matrix.
Experimental datasets can provide the entries of correlation or covariance matrices as a triangular matrix, as these matrices are symmetric by their very nature. This function can convert these list to a complete symmetric matrix, that can be used for the dataset implementation.
- modebool
Enter 0 or 1 based on the following scenarios: Use mode 0 if matrix entries are given row by row such as:
0 1 2 3 4 5 6 7 8 9
Use mode 1 if the matrix entries are given column by column such as:
0 1 3 6 2 4 7 5 8 9
Please note that the numbers above (0-9) are not entries of the matrix but rather the index of the entries of the list which contains the elements of the triangular matrix.
- tri_mat_listlist
A list containing the elements of the triangular matrix, for example, for a 4*4 matrix, the list of triangular matrix entries could be: [a, b, c, d, e, f, g, h, i, j]
- Returns:
mat_list – A one dimensional list which contains the elements of the fully populated, symmetric matrix row by row.
- Return type:
- nnpdf_data.filter_utils.utils.uncert_skip_variant(source_file, skip_file, uncert_file, uncert_name, remove_source=True)[source]
Create two new uncertainty files, one where the specified uncertainty is skipped, and one with only the specified uncertainty.
This function should necessarily be used in the filter.py file only after the main uncertainty file has been created and saved. To ensure that the filter.py file works for everyone, the source_file and destination_file should be just the file names and not the full path.
- Parameters:
source_file (str) – The name of the original uncertainty file
skip_file (str) – The name of the uncertainty file where the uncertainty is skipped
uncert_file (str) – The name of the uncertainty file where only the specified uncertainty is present
uncert_name (str) – The name of the uncertainty to be skipped
remove_source (bool) – If True, the source_file will be removed after the operation is complete