validphys.theorycovariance package

Submodules

validphys.theorycovariance.construction module

construction.py Tools for constructing theory covariance matrices and computing their chi2s.

class validphys.theorycovariance.construction.ProcessInfo(preds, namelist, sizes)

Bases: tuple

namelist

Alias for field number 1

preds

Alias for field number 0

sizes

Alias for field number 2

validphys.theorycovariance.construction.combine_by_type(each_dataset_results_central_bytheory)

Groups the datasets bu process and returns an instance of the ProcessInfo class

Parameters:

each_dataset_results_central_bytheory (list[list[(DataResult,ThPredictionsResult)]]) – Tuples of DataResult and ThPredictionsResult (where only the second is used for the construction of the theory covariance matrix), wrapped in a list such that there is a tuple per theoryid, wrapped in another list per dataset.

Returns:

Class with info needed to construct the theory covmat.

Return type:

ProcesInfo validphys.theorycovariance.construction.ProcessInfo

validphys.theorycovariance.construction.compute_covs_pt_prescrip(point_prescription, name1, deltas1, name2=None, deltas2=None)

Utility to compute the covariance matrix by prescription given the shifts with respect to the central value for a pair of processes.

The processes are defined by the variables name1 and name2 with deltas1 and deltas2 the associated shifts wrt the central prediction.

This utility also allows for the computation of the theory covmat for a single process or dataset if names2=deltas2=None.

Parameters:

• point_prescription (str) – defines the point prescription to be utilized

• name1 (str) – Process name of the first set of shifts

• deltas1 (list(np.ndarray)) – list of shifts for each of the non-central theories

• name2 (str) – Process name of the second set of shifts

• deltas2 (list(np.ndarray)) – list of shifts for each of the non-central theories

validphys.theorycovariance.construction.covmat_3fpt(deltas1, deltas2)

Returns theory covariance sub-matrix for 3pt factorisation scale variation only, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_3pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 3pt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_5barpt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 5barpt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_5pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 5pt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_7pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 7pt prescription (Gavin), given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_9pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 9pt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_n3lo_ad(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for each of the singlet splitting function variations.

Normalization is given by:

(n_pt - 1)

where:

• n_pt = number of point presctiption

validphys.theorycovariance.construction.covmat_n3lo_fhmv(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for all the FHMV splitting function variations.

validphys.theorycovariance.construction.covmat_n3lo_singlet(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for all the singlet splitting function variations.

validphys.theorycovariance.construction.covs_pt_prescrip(combine_by_type, point_prescription)

Produces the sub-matrices of the theory covariance matrix according to a point prescription which matches the number of input theories. If 5 theories are provided, a scheme ‘bar’ or ‘nobar’ must be chosen in the runcard in order to specify the prescription. Sub-matrices correspond to applying the scale variation prescription to each pair of processes in turn, using a different procedure for the case where the processes are the same relative to when they are different.

validphys.theorycovariance.construction.experimentplustheory_corrmat_custom(procs_covmat, theory_covmat_custom)

Calculates the correlation matrix for the experimental plus theory covariance matrices, correlations by prescription.

validphys.theorycovariance.construction.fromfile_covmat(covmatpath, procs_data, procs_index)

Reads a general theory covariance matrix from file. Then 1: Applies cuts to match experiment covariance matrix 2: Expands dimensions to match experiment covariance matrix

by filling additional entries with 0.

validphys.theorycovariance.construction.procs_index_matched(groups_index, procs_index)

procs_index but matched to the dataset order given by groups_index.

validphys.theorycovariance.construction.theory_corrmat_custom(theory_covmat_custom)

Calculates the theory correlation matrix for scale variations with variations by process type

validphys.theorycovariance.construction.theory_covmat_custom(theory_covmats_fitting)

Sum all the theory covmat listed in point_prescriptions.

validphys.theorycovariance.construction.theory_covmat_custom_fitting(theory_covmat_custom_per_prescription, procs_index_matched)

theory_covmat_custom_per_prescription but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.

validphys.theorycovariance.construction.theory_covmat_custom_per_prescription(covs_pt_prescrip, procs_index, combine_by_type)

Takes the individual sub-covmats between each two processes and assembles them into a full covmat. Then reshuffles the order from ordering by process to ordering by experiment as listed in the runcard

validphys.theorycovariance.construction.theory_covmat_dataset(results, results_central_bytheoryids, point_prescription, use_theorycovmat)

Compute the theory covmat for a collection of theoryids for a single dataset.

In general this will come from some point prescription and it could be guessed from the input however, it takes as input all relevant variables for generality

validphys.theorycovariance.construction.theory_normcovmat_custom(theory_covmat_custom, procs_data_values)

Calculates the theory covariance matrix for scale variations normalised to data, with variations according to the relevant prescription.

validphys.theorycovariance.construction.total_theory_covmat(theory_covmat_custom, user_covmat)

Sum of scale variation and user covmat, where both are used.

validphys.theorycovariance.construction.total_theory_covmat_fitting(total_theory_covmat, procs_index_matched)

total_theory_covmat but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.

validphys.theorycovariance.construction.user_covmat(procs_data, procs_index, loaded_user_covmat_path)

General theory covariance matrix provided by the user. Useful for testing the impact of externally produced covariance matrices. Matrices must be produced as a csv of pandas DataFrame, and uploaded to the validphys server. The server path is then provided via user_covmat_path in theorycovmatconfig in the runcard. For more information see documentation.

validphys.theorycovariance.construction.user_covmat_fitting(user_covmat, procs_index_matched)

user_covmat but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.

validphys.theorycovariance.output module

output.py Basic tools for plotting theory covariance matrices and their properties.

validphys.theorycovariance.output.matrix_plot_labels(df)

Returns the tick locations and labels, and the starting point values for each category, based on a dataframe to be plotted. The dataframe is assumed to be multiindexed by (process, dataset, points) or else (dataset, points). The tick location is in the centre of the dataset, and labelling is by the outermost index of the multiindex.

validphys.theorycovariance.output.plot_corrmat_heatmap(corrmat, title)

Matrix plot of a correlation matrix

validphys.theorycovariance.output.plot_covmat_heatmap(covmat, title)

Matrix plot of a covariance matrix.

validphys.theorycovariance.output.plot_diag_cov_comparison(theory_covmat_custom, procs_covmat, procs_data_values, theoryids, point_prescription)

Plot of sqrt(cov_ii)/|data_i| for cov = exp, theory, exp+theory

validphys.theorycovariance.output.plot_diag_cov_comparison_by_experiment(theory_covmat_custom_dataspecs, experiments_covmat_no_table, procs_data_values_experiment, dataspecs)

Plot of sqrt(cov_ii)/|data_i| for cov = exp, theory, exp+theory, by experiment

validphys.theorycovariance.output.plot_diag_cov_comparison_by_process(theory_covmat_custom_dataspecs, procs_covmat, procs_data_values, dataspecs)

Plot of sqrt(cov_ii)/|data_i| for cov = exp, theory, exp+theory, by process

dataspecs here is used also to specify whether to plot or not the total uncertainty, defined as the experimental uncertainty plus the uncertainty coming from the theory covariance matrix for the relevant dataspec entry.

validphys.theorycovariance.output.plot_expcorrmat_heatmap(procs_corrmat)

Matrix plot of the experiment correlation matrix

validphys.theorycovariance.output.plot_expplusthcorrmat_heatmap_custom(experimentplustheory_corrmat_custom, theoryids, point_prescription)

Matrix plot of the exp + theory correlation matrix

validphys.theorycovariance.output.plot_normexpcovmat_heatmap(procs_normcovmat)

Matrix plot of the experiment covariance matrix normalised to data.

validphys.theorycovariance.output.plot_normthcovmat_heatmap_custom(theory_normcovmat_custom, theoryids, point_prescription)

Matrix plot for block diagonal theory covariance matrix by process type

validphys.theorycovariance.output.plot_thcorrmat_heatmap_custom(theory_corrmat_custom, theoryids, point_prescription)

Matrix plot of the theory correlation matrix, correlations by process type

validphys.theorycovariance.tests module

tests.py Tools for testing theory covariance matrices and their properties.

class validphys.theorycovariance.tests.LabeledShifts(process, dataset_name, shifts)

Bases: tuple

dataset_name

Alias for field number 1

process

Alias for field number 0

shifts

Alias for field number 2

validphys.theorycovariance.tests.alltheory_vector(dataset_alltheory)

Returns a DataFrame with the theory vectors for matched dataspecs for the scale-varied theories, including the central

validphys.theorycovariance.tests.concatenated_shx_vector(shx_vector)

Single DataFrame for all the datasets of the shift vectors.

validphys.theorycovariance.tests.dataset_alltheory(each_dataset_results_central_bytheory)

Returns a LabeledShifts tuple corresponding to the theory vectors for all the scale varied theories, including the central.

validphys.theorycovariance.tests.deltamiss_plot(theory_shift_test, ordered_alltheory_vector, fnorm_shifts_ordered, tripleindex_thcovmat_complete, dataspecs)

Produces a plot of the missing component of the shift vector, transformed back to the data space.

validphys.theorycovariance.tests.diagdf_theory_covmat(theory_covmat_custom)

Return a Dataframe indexed with groups and dataset of the diagonal entry of the theory covmat.

validphys.theorycovariance.tests.doubleindex_set_byprocess(group_dataset_inputs_by_process)

The (group, dataset) index ordered by process.

validphys.theorycovariance.tests.doubleindex_thcovmat(theory_covmat_custom)

The unique set of (group, dataset) index of the theory covmat.

validphys.theorycovariance.tests.efficiency(theory_shift_test)

Returns (efficiency = 1 - fmiss/f) with which the theory covariance matrix encapsulates the NNLO-NLO shift.

validphys.theorycovariance.tests.eigenvector_plot(evals_nonzero_basis, fnorm_shifts_ordered, tripleindex_thcovmat_complete, dataspecs)

Produces a plot of the eigenvectors for the projected matrix, transformed back to the data space.

validphys.theorycovariance.tests.evals_nonzero_basis(ordered_alltheory_vector, theory_covmat_custom, point_prescription, orthonormalisation: (<class 'str'>, <class 'NoneType'>) = None)

validphys.theorycovariance.tests.fnorm_shifts_byprocess(concatenated_shx_vector, doubleindex_set_byprocess)

Shift vector ordered by process.

validphys.theorycovariance.tests.fnorm_shifts_ordered(concatenated_shx_vector, doubleindex_thcovmat)

Shifts vectors ordered as the theory covmat.

validphys.theorycovariance.tests.ordered_alltheory_vector(alltheory_vector, doubleindex_thcovmat)

Order the group and the dataset of all the theory vectors of the scale-varied predictions in the same way the theory covmat is ordered.

validphys.theorycovariance.tests.projected_condition_num(evals_nonzero_basis)

validphys.theorycovariance.tests.projector_eigenvalue_ratio(theory_shift_test)

Produces a plot of the ratio between the projectors and the square roots of the corresponding eigenvalues.

validphys.theorycovariance.tests.shift_diag_cov_comparison(sqrtdiags_thcovmat_byprocess, fnorm_shifts_byprocess, point_prescription, ticklocs_thcovmat, dataspecs)

Plot of the comparison of a shift between two pertubative order and the diagonal entries of the theory covmat, both normalized to the first of the two perturbative orders.

validphys.theorycovariance.tests.shift_vector(matched_dataspecs_results, process, dataset_name)

Returns a DataFrame of normalised shift vectors and normalization vector for matched dataspecs.

validphys.theorycovariance.tests.sqrtdiags_thcovmat_byprocess(doubleindex_set_byprocess, diagdf_theory_covmat, concatenated_shx_vector)

Ratio of the sqrts of the diagonal entries of the theory covmat to the normalization entries of the shift vectors, ordered by process.

validphys.theorycovariance.tests.theory_covmat_eigenvalues(theory_shift_test)

Returns a table of s = sqrt(eigenvalue), the projector and the ratio of the two, ordered by largest eigenvalue.

validphys.theorycovariance.tests.theory_shift_test(fnorm_shifts_ordered, evals_nonzero_basis)

Compares the NNLO-NLO shift, f, with the eigenvectors and eigenvalues of the theory covariance matrix, and returns the component of the NNLO-NLO shift space which is missed by the covariance matrix space: fmiss, as well as the projections of the shift vector onto each of the eigenvectors: projectors.

validphys.theorycovariance.tests.theta(theory_shift_test)

Returns the angle between the NNLO-NLO shift vector and the component of this which is captured by the theory covariance matrix

validphys.theorycovariance.tests.ticklocs_thcovmat(theory_covmat_custom)

validphys.theorycovariance.tests.tripleindex_thcovmat_complete(theory_covmat_custom)

The complete tripleindex of the theory covmat.

validphys.theorycovariance.tests.validation_theory_chi2(theory_shift_test)

Returns the theory chi2 for comparing NNLO-NLO shift with theory covariance matrix.

validphys.theorycovariance.tests.vectors_3pt(splitdiffs)

Returns the linearly independent vectors for 3pt prescription

validphys.theorycovariance.tests.vectors_5barpt(splitdiffs)

Returns the linearly independent vectors for 5barpt prescription

validphys.theorycovariance.tests.vectors_5pt(splitdiffs)

Returns the linearly independent vectors for 5pt prescription

validphys.theorycovariance.tests.vectors_7pt(splitdiffs)

Returns the linearly independent vectors for 7pt prescription

validphys.theorycovariance.tests.vectors_9pt(splitdiffs)

Returns the linearly independent vectors for 9pt prescription

validphys.theorycovariance.theorycovarianceutils module

theorycovarianceutils.py

Low level utilities for theorycovariance module

validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination(theoryids, point_prescription: (<class 'str'>, <class 'NoneType'>) = None)

Checks that a valid theory combination corresponding to an existing prescription has been inputted

validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_internal(theoryids, point_prescription: (<class 'str'>, <class 'NoneType'>) = None)

Checks that a valid theory combination corresponding to an existing prescription has been inputted

validphys.theorycovariance.theorycovarianceutils.check_fit_dataset_order_matches_grouped(group_dataset_inputs_by_metadata, data_input, processed_metadata_group)

Check for use with theory covmat generation.

Makes sure that the order of datasets listed in the fit runcard is the same as that specified by the metadata grouping. Otherwise there can be a misalignment between the experiment covmat and theory covmat.

validphys.theorycovariance.theorycovarianceutils.process_lookup(name)

Returns the nnpdf31_process of the corresponding dataset.

Module contents

theorycovariance

Tools for constructing and studying theory covariance matrices.

This package contains tools for constructing covariance matrices and computing their chi2s in construction.py, tools for plotting covariance matrices and tabulating their properties in output.py, and tools for testing their properties in tests.py.