validphys.theorycovariance package

Submodules

validphys.theorycovariance.construction module

construction.py Tools for constructing theory covariance matrices and computing their chi2s.

class validphys.theorycovariance.construction.ProcessInfo(theory, namelist, sizes)

Bases: tuple

namelist

Alias for field number 1

sizes

Alias for field number 2

theory

Alias for field number 0

validphys.theorycovariance.construction.abs_chi2_data_diagtheory_dataset(each_dataset_results, total_covmat_diagtheory_datasets)

For a diagonal theory covmat

validphys.theorycovariance.construction.abs_chi2_data_diagtheory_proc(procs_results, total_covmat_diagtheory_procs)

For a diagonal theory covmat

validphys.theorycovariance.construction.abs_chi2_data_theory_dataset(each_dataset_results, total_covmat_datasets)

Returns an array of tuples (member_chi², central_chi², numpoints) corresponding to each data set, where theory errors are included

validphys.theorycovariance.construction.abs_chi2_data_theory_proc(procs_results, total_covmat_procs)

Like abs_chi2_data_theory_dataset but for procs not datasets

validphys.theorycovariance.construction.chi2_block_impact(theory_block_diag_covmat, procs_covmat, procs_results)

Returns total chi2 including theory cov mat

validphys.theorycovariance.construction.chi2_diag_only(theory_diagcovmat, procs_covmat, data_theory_diff)

Returns total chi2 including only diags of theory cov mat

validphys.theorycovariance.construction.chi2_impact(theory_covmat_singleprocess, procs_covmat, procs_results)

Returns total chi2 including theory cov mat

validphys.theorycovariance.construction.chi2_impact_custom(theory_covmat_custom, procs_covmat, procs_results)

Returns total chi2 including theory cov mat

validphys.theorycovariance.construction.combine_by_type(each_dataset_results_bytheory, dataset_names)

Groups the datasets according to processes and returns three objects: theories_by_process: the relevant theories grouped by process type ordered_names: dictionary with keys of process type and values being the

corresponding list of names of datasets, in the order they are appended to theories_by_process

dataset_size: dictionary with keys of dataset name and values being the

number of points in that dataset

validphys.theorycovariance.construction.covmap(combine_by_type, dataset_names)

Creates a map between the covmat indices from matrices ordered by process to matrices ordered by experiment as listed in the runcard

validphys.theorycovariance.construction.covmat_3fpt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 3pt factorisation scale variation only, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_3pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 3pt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_3rpt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 3pt renormalisation scale variation only, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_5barpt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 5barpt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_5pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 5pt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_7pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 7pt prescription (Gavin), given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_7pt_orig(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for original 7pt prescription, now deprecated but kept for posterity, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covmat_9pt(name1, name2, deltas1, deltas2)

Returns theory covariance sub-matrix for 9pt prescription, given two dataset names and collections of scale variation shifts

validphys.theorycovariance.construction.covs_pt_prescrip(combine_by_type, process_starting_points, theoryids, point_prescription, fivetheories, seventheories)

Produces the sub-matrices of the theory covariance matrix according to a point prescription which matches the number of input theories. If 5 theories are provided, a scheme ‘bar’ or ‘nobar’ must be chosen in the runcard in order to specify the prescription. Sub-matrices correspond to applying the scale variation prescription to each pair of processes in turn, using a different procedure for the case where the processes are the same relative to when they are different.

validphys.theorycovariance.construction.data_theory_diff(procs_results)

Returns (D-T) for central theory, for use in chi2 calculations

validphys.theorycovariance.construction.dataset_names(data_input)

Returns a list of the names of the datasets, in the same order as they are inputted in the runcard

validphys.theorycovariance.construction.experimentplusblocktheory_corrmat(procs_covmat, theory_block_diag_covmat)

Calculates the correlation matrix for the experimental plus theory covariance matrices, block diagonal by dataset.

validphys.theorycovariance.construction.experimentplusblocktheory_covmat(procs_covmat, theory_block_diag_covmat)

Calculates the experiment + theory covariance matrix for scale variations.

validphys.theorycovariance.construction.experimentplusblocktheory_normcovmat(procs_covmat, theory_block_diag_covmat, procs_data_values, experimentplustheory_normcovmat)

Calculates the experiment + theory covariance matrix for scale variations normalised to data, block diagonal by data set.

validphys.theorycovariance.construction.experimentplustheory_corrmat_custom(procs_covmat, theory_covmat_custom)

Calculates the correlation matrix for the experimental plus theory covariance matrices, correlations by prescription.

validphys.theorycovariance.construction.experimentplustheory_corrmat_singleprocess(procs_covmat, theory_covmat_singleprocess)

Calculates the correlation matrix for the experimental plus theory covariance matrices.

validphys.theorycovariance.construction.experimentplustheory_covmat_custom(procs_covmat, theory_covmat_custom)

Calculates the experiment + theory covariance matrix for scale variations correlated according to the relevant prescription.

validphys.theorycovariance.construction.experimentplustheory_covmat_singleprocess(procs_covmat_no_table, theory_covmat_singleprocess_no_table)

Calculates the experiment + theory covariance matrix for scale variations.

validphys.theorycovariance.construction.experimentplustheory_normcovmat_custom(procs_covmat, theory_covmat_custom, procs_data_values, experimentplustheory_normcovmat)

Calculates the experiment + theory covariance matrix for scale variations normalised to data, correlations by process type.

validphys.theorycovariance.construction.experimentplustheory_normcovmat_singleprocess(procs_covmat, theory_covmat_singleprocess, procs_data)

Calculates the experiment + theory covariance matrix for scale variations normalised to data.

validphys.theorycovariance.construction.fromfile_covmat(covmatpath, procs_data, procs_index)

Reads a general theory covariance matrix from file. Then 1: Applies cuts to match experiment covariance matrix 2: Expands dimensions to match experiment covariance matrix

by filling additional entries with 0.

validphys.theorycovariance.construction.make_scale_var_covmat(predictions)

Takes N theory predictions at different scales and applies N-pt scale variations to produce a covariance matrix.

validphys.theorycovariance.construction.process_starting_points(combine_by_type)

Returns a dictionary of indices in the covariance matrix corresponding to the starting point of each process.

validphys.theorycovariance.construction.procs_index_matched(groups_index, procs_index)

procs_index but matched to the dataset order given by groups_index.

validphys.theorycovariance.construction.theory_block_diag_covmat(theory_covmat_datasets, procs_index)

Takes the theory covariance matrices for individual datasets and returns a data frame with a block diagonal theory covariance matrix by dataset

validphys.theorycovariance.construction.theory_blockcorrmat(theory_block_diag_covmat)

Calculates the theory correlation matrix for scale variations with block diagonal entries by dataset only

validphys.theorycovariance.construction.theory_corrmat_custom(theory_covmat_custom)

Calculates the theory correlation matrix for scale variations with variations by process type

validphys.theorycovariance.construction.theory_corrmat_singleprocess(theory_covmat_singleprocess)

Calculates the theory correlation matrix for scale variations.

validphys.theorycovariance.construction.theory_covmat_custom(covs_pt_prescrip, covmap, procs_index)

Takes the individual sub-covmats between each two processes and assembles them into a full covmat. Then reshuffles the order from ordering by process to ordering by experiment as listed in the runcard

validphys.theorycovariance.construction.theory_covmat_custom_fitting(theory_covmat_custom, procs_index_matched)

theory_covmat_custom but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.

validphys.theorycovariance.construction.theory_covmat_datasets(each_dataset_results_bytheory, fivetheories)

Produces an array of theory covariance matrices. Each matrix corresponds to a different dataset, which must be specified in the runcard.

validphys.theorycovariance.construction.theory_covmat_singleprocess(theory_covmat_singleprocess_no_table, fivetheories)

Duplicate of theory_covmat_singleprocess_no_table but with a table decorator.

validphys.theorycovariance.construction.theory_covmat_singleprocess_no_table(theoryids_procs_central_values_no_table, procs_index, theoryids, fivetheories)

Calculates the theory covariance matrix for scale variations. The matrix is a dataframe indexed by procs_index.

validphys.theorycovariance.construction.theory_diagcovmat(theory_covmat_singleprocess)

Returns theory covmat with only diagonal values

validphys.theorycovariance.construction.theory_diagonal_covmat(theory_covmat_singleprocess_no_table, procs_index)

Returns theory covmat with only diagonal values

validphys.theorycovariance.construction.theory_normblockcovmat(theory_block_diag_covmat, procs_data_values)

Calculates the theory covariance matrix for scale variations normalised to data, block diagonal by dataset.

validphys.theorycovariance.construction.theory_normcovmat_custom(theory_covmat_custom, procs_data_values)

Calculates the theory covariance matrix for scale variations normalised to data, with variations according to the relevant prescription.

validphys.theorycovariance.construction.theory_normcovmat_singleprocess(theory_covmat_singleprocess, procs_data_values)

Calculates the theory covariance matrix for scale variations normalised to data.

validphys.theorycovariance.construction.total_covmat_datasets(each_dataset_results_bytheory, fivetheories)

Produces an array of total covariance matrices; the sum of experimental and scale-varied theory covariance matrices. Each matrix corresponds to a different dataset, which must be specified in the runcard. These are needed for calculation of chi2 per dataset.

validphys.theorycovariance.construction.total_covmat_diagtheory_datasets(each_dataset_results_bytheory, fivetheories)

Same as total_covmat_theory_datasets but for diagonal theory only

validphys.theorycovariance.construction.total_covmat_diagtheory_procs(procs_results_theory, fivetheories)

Same as total_covmat_datasets but per proc rather than per dataset. Needed for calculation of chi2 per proc.

validphys.theorycovariance.construction.total_covmat_procs(procs_results_theory, fivetheories)

Same as total_covmat_datasets but per experiment rather than per dataset. Needed for calculation of chi2 per experiment.

validphys.theorycovariance.construction.total_theory_covmat(theory_covmat_custom, user_covmat)

Sum of scale variation and user covmat, where both are used.

validphys.theorycovariance.construction.total_theory_covmat_fitting(total_theory_covmat, procs_index_matched)

total_theory_covmat but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.

validphys.theorycovariance.construction.user_covmat(procs_data, procs_index, loaded_user_covmat_path)

General theory covariance matrix provided by the user. Useful for testing the impact of externally produced covariance matrices. Matrices must be produced as a csv of pandas DataFrame, and uploaded to the validphys server. The server path is then provided via user_covmat_path in theorycovmatconfig in the runcard. For more information see documentation.

validphys.theorycovariance.construction.user_covmat_fitting(user_covmat, procs_index_matched)

user_covmat but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.

validphys.theorycovariance.output module

output.py Basic tools for plotting theory covariance matrices and their properties.

validphys.theorycovariance.output.matrix_plot_labels(df)

Returns the tick locations and labels, and the starting point values for each category, based on a dataframe to be plotted. The dataframe is assumed to be multiindexed by (process, dataset, points) or else (dataset, points). The tick location is in the centre of the dataset, and labelling is by the outermost index of the multiindex.

validphys.theorycovariance.output.plot_blockcovdiff_heatmap(theory_block_diag_covmat, procs_covmat)

Matrix plot (thcov + expcov)/expcov

validphys.theorycovariance.output.plot_corrmat_heatmap(corrmat, title)

Matrix plot of a correlation matrix

validphys.theorycovariance.output.plot_covdiff_heatmap_custom(theory_covmat_custom, procs_covmat, theoryids, fivetheories)

Matrix plot (thcov + expcov)/expcov

validphys.theorycovariance.output.plot_covmat_heatmap(covmat, title)

Matrix plot of a covariance matrix.

validphys.theorycovariance.output.plot_datasets_chi2_theory(procs_data, each_dataset_chi2, abs_chi2_data_theory_dataset)

Plot the chi² of all datasets, before and after adding theory errors, with bars.

validphys.theorycovariance.output.plot_diag_cov_comparison(theory_covmat_custom, procs_covmat, procs_data_values, theoryids, fivetheories)

Plot of sqrt(cov_ii)/|data_i| for cov = exp, theory, exp+theory

validphys.theorycovariance.output.plot_diag_cov_impact(theory_covmat_custom, procs_covmat, procs_data_values, theoryids, fivetheories)

Plot ((expcov)^-1_ii)^-0.5 versus ((expcov + thcov)^-1_ii)^-0.5

validphys.theorycovariance.output.plot_expcorrmat_heatmap(procs_corrmat)

Matrix plot of the experiment correlation matrix

validphys.theorycovariance.output.plot_expplusblockthcorrmat_heatmap(experimentplusblocktheory_corrmat)

Matrix plot of the exp + theory correlation matrix

validphys.theorycovariance.output.plot_expplusthcorrmat_heatmap_custom(experimentplustheory_corrmat_custom, theoryids, fivetheories)

Matrix plot of the exp + theory correlation matrix

validphys.theorycovariance.output.plot_normexpcovmat_heatmap(procs_normcovmat)

Matrix plot of the experiment covariance matrix normalised to data.

validphys.theorycovariance.output.plot_normexpplusblockthcovmat_heatmap(experimentplusblocktheory_normcovmat)

Matrix plot of the exp + theory covariance matrix normalised to data

validphys.theorycovariance.output.plot_normexpplusthcovmat_heatmap_custom(experimentplustheory_normcovmat_custom, theoryids, fivetheories)

Matrix plot of the exp + theory covariance matrix normalised to data

validphys.theorycovariance.output.plot_normthblockcovmat_heatmap(theory_normblockcovmat)

Matrix plot for block diagonal theory covariance matrix

validphys.theorycovariance.output.plot_normthcovmat_heatmap_custom(theory_normcovmat_custom, theoryids, fivetheories)

Matrix plot for block diagonal theory covariance matrix by process type

validphys.theorycovariance.output.plot_thblockcorrmat_heatmap(theory_blockcorrmat)

Matrix plot of the theory correlation matrix

validphys.theorycovariance.output.plot_thcorrmat_heatmap_custom(theory_corrmat_custom, theoryids, fivetheories)

Matrix plot of the theory correlation matrix, correlations by process type

validphys.theorycovariance.output.procs_chi2_table_diagtheory(procs_data, pdf, abs_chi2_data_diagtheory_proc, abs_chi2_data_theory_dataset_by_process)

Same as groups_chi2_table but including diagonal theory covariance matrix. Note: we use groups_chi2_table here but provide data grouped by process.

validphys.theorycovariance.output.procs_chi2_table_theory(procs_data, pdf, abs_chi2_data_theory_proc, abs_chi2_data_theory_dataset_by_process)

Same as groups_chi2_table but including theory covariance matrix. Note: we use groups_chi2_table here but provide data grouped by process.

validphys.theorycovariance.tests module

tests.py Tools for testing theory covariance matrices and their properties.

class validphys.theorycovariance.tests.LabeledShifts(process, dataset_name, shifts)

Bases: tuple

dataset_name

Alias for field number 1

process

Alias for field number 0

shifts

Alias for field number 2

validphys.theorycovariance.tests.all_matched_data_lengths(all_matched_datasets)

Returns a list of the data sets lengths.

validphys.theorycovariance.tests.alltheory_vector(matched_dataspecs_dataset_alltheory, matched_dataspecs_dataset_theory)

Returns a DataFrame with the theory vectors for matched dataspecs for the scale-varied theories (not the central one).

validphys.theorycovariance.tests.combine_by_type_dataspecs(all_matched_results, matched_dataspecs_dataset_name)

Like combine_by_type but for matched dataspecs

validphys.theorycovariance.tests.covmap_dataspecs(combine_by_type_dataspecs, matched_dataspecs_dataset_name)

Like covmap but for matched dataspecs.

validphys.theorycovariance.tests.covs_pt_prescrip_dataspecs(combine_by_type_dataspecs, process_starting_points_dataspecs, dataspecs_theoryids, point_prescription, fivetheories, seventheories)

Like covs_pt_prescrip but for matched dataspecs.

validphys.theorycovariance.tests.dataspecs_dataset_alltheory(matched_dataspecs_results, process, dataset_name)

Returns a LabeledShifts tuple corresponding to the theory vectors for all the scale varied theories (not the central one), processed by data set and experiment for matched dataspecs.

validphys.theorycovariance.tests.dataspecs_dataset_prediction_shift(matched_dataspecs_results, process, dataset_name)

Compute the difference in theory predictions between two dataspecs. This can be used in combination with matched_datasets_from_dataspecs It returns a LabeledShifts containing dataset_name, process and shifts.

validphys.theorycovariance.tests.dataspecs_dataset_theory(matched_dataspecs_results, process, dataset_name)

Returns a tuple of shifts processed by data set and experiment for matched dataspecs.

validphys.theorycovariance.tests.deltamiss_plot(theory_shift_test, allthx_vector, evals_nonzero_basis, shx_vector)

Produces a plot of the missing component of the shift vector, transformed back to the data space.

validphys.theorycovariance.tests.efficiency(theory_shift_test)

Returns (efficiency = 1 - fmiss/f) with which the theory covariance matrix encapsulates the NNLO-NLO shift.

validphys.theorycovariance.tests.eigenvector_plot(evals_nonzero_basis, shx_vector)

Produces a plot of the eigenvectors for the projected matrix, transformed back to the data space.

validphys.theorycovariance.tests.evals_nonzero_basis(allthx_vector, thx_covmat, thx_vector, collected_theoryids, fivetheories, seventheories: (<class 'str'>, <class 'NoneType'>) = None, orthonormalisation: (<class 'str'>, <class 'NoneType'>) = None)

Projects the theory covariance matrix from the data space into the basis of non-zero eigenvalues, dependent on point-prescription. Then returns the eigenvalues (w) and eigenvectors (v) in the data space. There are three methods to linearly orthonormalise the basis vectors for the covariance matrix, and the choice must be specified using the “orthonormalisation” flag in the runcard. The choices are: gs, the Gram-Schmidt method; qr, QR decomposition; svd, singular value decomposition. QR is the method which should be used as standard; the others exist for testing purposes.

validphys.theorycovariance.tests.matched_experiments_index(matched_dataspecs_dataset_name, all_matched_data_lengths)

Returns MultiIndex composed of data set name and starting point of data set.

validphys.theorycovariance.tests.process_starting_points_dataspecs(combine_by_type_dataspecs)

Like process_starting_points but for matched dataspecs.

validphys.theorycovariance.tests.projected_condition_num(evals_nonzero_basis)

validphys.theorycovariance.tests.projector_eigenvalue_ratio(theory_shift_test)

Produces a plot of the ratio between the projectors and the square roots of the corresponding eigenvalues.

validphys.theorycovariance.tests.shift_diag_cov_comparison(allthx_vector, shx_vector, thx_covmat, thx_vector)

Produces a plot of a comparison between the NNLO-NLO shift and the envelope given by the diagonal elements of the theory covariance matrix.

validphys.theorycovariance.tests.shift_vector(matched_dataspecs_dataset_prediction_shift, matched_dataspecs_dataset_theory)

Returns a DataFrame of normalised shift vectors for matched dataspecs.

validphys.theorycovariance.tests.theory_corrmat_custom_dataspecs(theory_covmat_custom_dataspecs)

Calculates the theory correlation matrix for scale variations with variations by process type

validphys.theorycovariance.tests.theory_covmat_custom_dataspecs(covs_pt_prescrip_dataspecs, covmap_dataspecs, matched_experiments_index)

Like theory_covmat_custom but for matched dataspecs.

validphys.theorycovariance.tests.theory_covmat_eigenvalues(theory_shift_test)

Returns a table of s = sqrt(eigenvalue), the projector and the ratio of the two, ordered by largest eigenvalue.

validphys.theorycovariance.tests.theory_matrix_threshold(theory_threshold: (<class 'int'>, <class 'float'>) = 0)

Returns the threshold below which theory correlation elements are set to zero when comparing to shift correlation matrix

validphys.theorycovariance.tests.theory_shift_test(shx_vector, evals_nonzero_basis)

Compares the NNLO-NLO shift, f, with the eigenvectors and eigenvalues of the theory covariance matrix, and returns the component of the NNLO-NLO shift space which is missed by the covariance matrix space: fmiss, as well as the projections of the shift vector onto each of the eigenvectors: projectors.

validphys.theorycovariance.tests.theory_vector(matched_dataspecs_dataset_theory)

Returns a DataFrame of the central theory vector for matched dataspecs.

validphys.theorycovariance.tests.theta(theory_shift_test)

Returns the angle between the NNLO-NLO shift vector and the component of this which is captured by the theory covariance matrix

validphys.theorycovariance.tests.validation_theory_chi2(theory_shift_test)

Returns the theory chi2 for comparing NNLO-NLO shift with theory covariance matrix.

validphys.theorycovariance.tests.vectors_3pt(splitdiffs)

Returns the linearly independent vectors for 3pt prescription

validphys.theorycovariance.tests.vectors_5barpt(splitdiffs)

Returns the linearly independent vectors for 5barpt prescription

validphys.theorycovariance.tests.vectors_5pt(splitdiffs)

Returns the linearly independent vectors for 5pt prescription

validphys.theorycovariance.tests.vectors_7pt(splitdiffs)

Returns the linearly independent vectors for 7pt prescription

validphys.theorycovariance.tests.vectors_9pt(splitdiffs)

Returns the linearly independent vectors for 9pt prescription

validphys.theorycovariance.theorycovarianceutils module

theorycovarianceutils.py

Low level utilities for theorycovariance module

validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination(theoryids, fivetheories, point_prescription: (<class 'str'>, <class 'NoneType'>) = None)

Checks that a valid theory combination corresponding to an existing prescription has been inputted

validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_dataspecs(dataspecs_theoryids, fivetheories)

Like check_correct_theory_combination but for matched dataspecs.

validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_internal(theoryids, fivetheories, point_prescription: (<class 'str'>, <class 'NoneType'>) = None)

Checks that a valid theory combination corresponding to an existing prescription has been inputted

validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_theoryconfig(collected_theoryids, fivetheories)

validphys.theorycovariance.theorycovarianceutils.check_fit_dataset_order_matches_grouped(group_dataset_inputs_by_metadata, data_input, processed_metadata_group)

Check for use with theory covmat generation.

Makes sure that the order of datasets listed in the fit runcard is the same as that specified by the metadata grouping. Otherwise there can be a misalignment between the experiment covmat and theory covmat.

validphys.theorycovariance.theorycovarianceutils.process_lookup(name)

Returns the nnpdf31_process of the corresponding dataset.

Module contents

theorycovariance

Tools for constructing and studying theory covariance matrices.

This package contains tools for constructing covariance matrices and computing their chi2s in construction.py, tools for plotting covariance matrices and tabulating their properties in output.py, and tools for testing their properties in tests.py.