validphys.theorycovariance package
Submodules
validphys.theorycovariance.construction module
construction.py Tools for constructing theory covariance matrices and computing their chi2s.
- class validphys.theorycovariance.construction.ProcessInfo(theory, namelist, sizes)
Bases:
tuple
- namelist
Alias for field number 1
- sizes
Alias for field number 2
- theory
Alias for field number 0
- validphys.theorycovariance.construction.abs_chi2_data_diagtheory_dataset(each_dataset_results, total_covmat_diagtheory_datasets)[source]
For a diagonal theory covmat
- validphys.theorycovariance.construction.abs_chi2_data_diagtheory_proc(procs_results, total_covmat_diagtheory_procs)[source]
For a diagonal theory covmat
- validphys.theorycovariance.construction.abs_chi2_data_theory_dataset(each_dataset_results, total_covmat_datasets)[source]
Returns an array of tuples (member_chi², central_chi², numpoints) corresponding to each data set, where theory errors are included
- validphys.theorycovariance.construction.abs_chi2_data_theory_proc(procs_results, total_covmat_procs)[source]
Like abs_chi2_data_theory_dataset but for procs not datasets
- validphys.theorycovariance.construction.chi2_block_impact(theory_block_diag_covmat, procs_covmat, procs_results)[source]
Returns total chi2 including theory cov mat
- validphys.theorycovariance.construction.chi2_diag_only(theory_diagcovmat, procs_covmat, data_theory_diff)[source]
Returns total chi2 including only diags of theory cov mat
- validphys.theorycovariance.construction.chi2_impact(theory_covmat_singleprocess, procs_covmat, procs_results)[source]
Returns total chi2 including theory cov mat
- validphys.theorycovariance.construction.chi2_impact_custom(theory_covmat_custom, procs_covmat, procs_results)[source]
Returns total chi2 including theory cov mat
- validphys.theorycovariance.construction.combine_by_type(each_dataset_results_bytheory, dataset_names)[source]
Groups the datasets according to processes and returns three objects: theories_by_process: the relevant theories grouped by process type ordered_names: dictionary with keys of process type and values being the
corresponding list of names of datasets, in the order they are appended to theories_by_process
- dataset_size: dictionary with keys of dataset name and values being the
number of points in that dataset
- validphys.theorycovariance.construction.covmap(combine_by_type, dataset_names)[source]
Creates a map between the covmat indices from matrices ordered by process to matrices ordered by experiment as listed in the runcard
- validphys.theorycovariance.construction.covmat_3fpt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 3pt factorisation scale variation only, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_3pt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 3pt prescription, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_3rpt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 3pt renormalisation scale variation only, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_5barpt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 5barpt prescription, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_5pt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 5pt prescription, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_7pt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 7pt prescription (Gavin), given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_7pt_orig(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for original 7pt prescription, now deprecated but kept for posterity, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covmat_9pt(name1, name2, deltas1, deltas2)[source]
Returns theory covariance sub-matrix for 9pt prescription, given two dataset names and collections of scale variation shifts
- validphys.theorycovariance.construction.covs_pt_prescrip(combine_by_type, process_starting_points, theoryids, point_prescription, fivetheories, seventheories)[source]
Produces the sub-matrices of the theory covariance matrix according to a point prescription which matches the number of input theories. If 5 theories are provided, a scheme ‘bar’ or ‘nobar’ must be chosen in the runcard in order to specify the prescription. Sub-matrices correspond to applying the scale variation prescription to each pair of processes in turn, using a different procedure for the case where the processes are the same relative to when they are different.
- validphys.theorycovariance.construction.data_theory_diff(procs_results)[source]
Returns (D-T) for central theory, for use in chi2 calculations
- validphys.theorycovariance.construction.dataset_names(data_input)[source]
Returns a list of the names of the datasets, in the same order as they are inputted in the runcard
- validphys.theorycovariance.construction.experimentplusblocktheory_corrmat(procs_covmat, theory_block_diag_covmat)[source]
Calculates the correlation matrix for the experimental plus theory covariance matrices, block diagonal by dataset.
- validphys.theorycovariance.construction.experimentplusblocktheory_covmat(procs_covmat, theory_block_diag_covmat)[source]
Calculates the experiment + theory covariance matrix for scale variations.
- validphys.theorycovariance.construction.experimentplusblocktheory_normcovmat(procs_covmat, theory_block_diag_covmat, procs_data_values, experimentplustheory_normcovmat)[source]
Calculates the experiment + theory covariance matrix for scale variations normalised to data, block diagonal by data set.
- validphys.theorycovariance.construction.experimentplustheory_corrmat_custom(procs_covmat, theory_covmat_custom)[source]
Calculates the correlation matrix for the experimental plus theory covariance matrices, correlations by prescription.
- validphys.theorycovariance.construction.experimentplustheory_corrmat_singleprocess(procs_covmat, theory_covmat_singleprocess)[source]
Calculates the correlation matrix for the experimental plus theory covariance matrices.
- validphys.theorycovariance.construction.experimentplustheory_covmat_custom(procs_covmat, theory_covmat_custom)[source]
Calculates the experiment + theory covariance matrix for scale variations correlated according to the relevant prescription.
- validphys.theorycovariance.construction.experimentplustheory_covmat_singleprocess(procs_covmat_no_table, theory_covmat_singleprocess_no_table)[source]
Calculates the experiment + theory covariance matrix for scale variations.
- validphys.theorycovariance.construction.experimentplustheory_normcovmat_custom(procs_covmat, theory_covmat_custom, procs_data_values, experimentplustheory_normcovmat)[source]
Calculates the experiment + theory covariance matrix for scale variations normalised to data, correlations by process type.
- validphys.theorycovariance.construction.experimentplustheory_normcovmat_singleprocess(procs_covmat, theory_covmat_singleprocess, procs_data)[source]
Calculates the experiment + theory covariance matrix for scale variations normalised to data.
- validphys.theorycovariance.construction.fromfile_covmat(covmatpath, procs_data, procs_index)[source]
Reads a general theory covariance matrix from file. Then 1: Applies cuts to match experiment covariance matrix 2: Expands dimensions to match experiment covariance matrix
by filling additional entries with 0.
- validphys.theorycovariance.construction.make_scale_var_covmat(predictions)[source]
Takes N theory predictions at different scales and applies N-pt scale variations to produce a covariance matrix.
- validphys.theorycovariance.construction.process_starting_points(combine_by_type)[source]
Returns a dictionary of indices in the covariance matrix corresponding to the starting point of each process.
- validphys.theorycovariance.construction.procs_index_matched(groups_index, procs_index)[source]
procs_index but matched to the dataset order given by groups_index.
- validphys.theorycovariance.construction.theory_block_diag_covmat(theory_covmat_datasets, procs_index)[source]
Takes the theory covariance matrices for individual datasets and returns a data frame with a block diagonal theory covariance matrix by dataset
- validphys.theorycovariance.construction.theory_blockcorrmat(theory_block_diag_covmat)[source]
Calculates the theory correlation matrix for scale variations with block diagonal entries by dataset only
- validphys.theorycovariance.construction.theory_corrmat_custom(theory_covmat_custom)[source]
Calculates the theory correlation matrix for scale variations with variations by process type
- validphys.theorycovariance.construction.theory_corrmat_singleprocess(theory_covmat_singleprocess)[source]
Calculates the theory correlation matrix for scale variations.
- validphys.theorycovariance.construction.theory_covmat_custom(covs_pt_prescrip, covmap, procs_index)[source]
Takes the individual sub-covmats between each two processes and assembles them into a full covmat. Then reshuffles the order from ordering by process to ordering by experiment as listed in the runcard
- validphys.theorycovariance.construction.theory_covmat_custom_fitting(theory_covmat_custom, procs_index_matched)[source]
theory_covmat_custom but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.
- validphys.theorycovariance.construction.theory_covmat_datasets(each_dataset_results_bytheory, fivetheories)[source]
Produces an array of theory covariance matrices. Each matrix corresponds to a different dataset, which must be specified in the runcard.
- validphys.theorycovariance.construction.theory_covmat_singleprocess(theory_covmat_singleprocess_no_table, fivetheories)[source]
Duplicate of theory_covmat_singleprocess_no_table but with a table decorator.
- validphys.theorycovariance.construction.theory_covmat_singleprocess_no_table(theoryids_procs_central_values_no_table, procs_index, theoryids, fivetheories)[source]
Calculates the theory covariance matrix for scale variations. The matrix is a dataframe indexed by procs_index.
- validphys.theorycovariance.construction.theory_diagcovmat(theory_covmat_singleprocess)[source]
Returns theory covmat with only diagonal values
- validphys.theorycovariance.construction.theory_diagonal_covmat(theory_covmat_singleprocess_no_table, procs_index)[source]
Returns theory covmat with only diagonal values
- validphys.theorycovariance.construction.theory_normblockcovmat(theory_block_diag_covmat, procs_data_values)[source]
Calculates the theory covariance matrix for scale variations normalised to data, block diagonal by dataset.
- validphys.theorycovariance.construction.theory_normcovmat_custom(theory_covmat_custom, procs_data_values)[source]
Calculates the theory covariance matrix for scale variations normalised to data, with variations according to the relevant prescription.
- validphys.theorycovariance.construction.theory_normcovmat_singleprocess(theory_covmat_singleprocess, procs_data_values)[source]
Calculates the theory covariance matrix for scale variations normalised to data.
- validphys.theorycovariance.construction.total_covmat_datasets(each_dataset_results_bytheory, fivetheories)[source]
Produces an array of total covariance matrices; the sum of experimental and scale-varied theory covariance matrices. Each matrix corresponds to a different dataset, which must be specified in the runcard. These are needed for calculation of chi2 per dataset.
- validphys.theorycovariance.construction.total_covmat_diagtheory_datasets(each_dataset_results_bytheory, fivetheories)[source]
Same as total_covmat_theory_datasets but for diagonal theory only
- validphys.theorycovariance.construction.total_covmat_diagtheory_procs(procs_results_theory, fivetheories)[source]
Same as total_covmat_datasets but per proc rather than per dataset. Needed for calculation of chi2 per proc.
- validphys.theorycovariance.construction.total_covmat_procs(procs_results_theory, fivetheories)[source]
Same as total_covmat_datasets but per experiment rather than per dataset. Needed for calculation of chi2 per experiment.
- validphys.theorycovariance.construction.total_theory_covmat(theory_covmat_custom, user_covmat)[source]
Sum of scale variation and user covmat, where both are used.
- validphys.theorycovariance.construction.total_theory_covmat_fitting(total_theory_covmat, procs_index_matched)[source]
total_theory_covmat but reindexed so the order of the datasets matches those in the experiment covmat so they are aligned when fitting.
- validphys.theorycovariance.construction.user_covmat(procs_data, procs_index, loaded_user_covmat_path)[source]
General theory covariance matrix provided by the user. Useful for testing the impact of externally produced covariance matrices. Matrices must be produced as a csv of pandas DataFrame, and uploaded to the validphys server. The server path is then provided via
user_covmat_path
intheorycovmatconfig
in the runcard. For more information see documentation.
validphys.theorycovariance.output module
output.py Basic tools for plotting theory covariance matrices and their properties.
- validphys.theorycovariance.output.matrix_plot_labels(df)[source]
Returns the tick locations and labels, and the starting point values for each category, based on a dataframe to be plotted. The dataframe is assumed to be multiindexed by (process, dataset, points) or else (dataset, points). The tick location is in the centre of the dataset, and labelling is by the outermost index of the multiindex.
- validphys.theorycovariance.output.plot_blockcovdiff_heatmap(theory_block_diag_covmat, procs_covmat)[source]
Matrix plot (thcov + expcov)/expcov
- validphys.theorycovariance.output.plot_corrmat_heatmap(corrmat, title)[source]
Matrix plot of a correlation matrix
- validphys.theorycovariance.output.plot_covdiff_heatmap_custom(theory_covmat_custom, procs_covmat, theoryids, fivetheories)[source]
Matrix plot (thcov + expcov)/expcov
- validphys.theorycovariance.output.plot_covmat_heatmap(covmat, title)[source]
Matrix plot of a covariance matrix.
- validphys.theorycovariance.output.plot_datasets_chi2_theory(procs_data, each_dataset_chi2, abs_chi2_data_theory_dataset)[source]
Plot the chi² of all datasets, before and after adding theory errors, with bars.
- validphys.theorycovariance.output.plot_diag_cov_comparison(theory_covmat_custom, procs_covmat, procs_data_values, theoryids, fivetheories)[source]
Plot of sqrt(cov_ii)/|data_i| for cov = exp, theory, exp+theory
- validphys.theorycovariance.output.plot_diag_cov_impact(theory_covmat_custom, procs_covmat, procs_data_values, theoryids, fivetheories)[source]
Plot ((expcov)^-1_ii)^-0.5 versus ((expcov + thcov)^-1_ii)^-0.5
- validphys.theorycovariance.output.plot_expcorrmat_heatmap(procs_corrmat)[source]
Matrix plot of the experiment correlation matrix
- validphys.theorycovariance.output.plot_expplusblockthcorrmat_heatmap(experimentplusblocktheory_corrmat)[source]
Matrix plot of the exp + theory correlation matrix
- validphys.theorycovariance.output.plot_expplusthcorrmat_heatmap_custom(experimentplustheory_corrmat_custom, theoryids, fivetheories)[source]
Matrix plot of the exp + theory correlation matrix
- validphys.theorycovariance.output.plot_normexpcovmat_heatmap(procs_normcovmat)[source]
Matrix plot of the experiment covariance matrix normalised to data.
- validphys.theorycovariance.output.plot_normexpplusblockthcovmat_heatmap(experimentplusblocktheory_normcovmat)[source]
Matrix plot of the exp + theory covariance matrix normalised to data
- validphys.theorycovariance.output.plot_normexpplusthcovmat_heatmap_custom(experimentplustheory_normcovmat_custom, theoryids, fivetheories)[source]
Matrix plot of the exp + theory covariance matrix normalised to data
- validphys.theorycovariance.output.plot_normthblockcovmat_heatmap(theory_normblockcovmat)[source]
Matrix plot for block diagonal theory covariance matrix
- validphys.theorycovariance.output.plot_normthcovmat_heatmap_custom(theory_normcovmat_custom, theoryids, fivetheories)[source]
Matrix plot for block diagonal theory covariance matrix by process type
- validphys.theorycovariance.output.plot_thblockcorrmat_heatmap(theory_blockcorrmat)[source]
Matrix plot of the theory correlation matrix
- validphys.theorycovariance.output.plot_thcorrmat_heatmap_custom(theory_corrmat_custom, theoryids, fivetheories)[source]
Matrix plot of the theory correlation matrix, correlations by process type
- validphys.theorycovariance.output.procs_chi2_table_diagtheory(procs_data, pdf, abs_chi2_data_diagtheory_proc, abs_chi2_data_theory_dataset_by_process)[source]
Same as groups_chi2_table but including diagonal theory covariance matrix. Note: we use groups_chi2_table here but provide data grouped by process.
validphys.theorycovariance.tests module
tests.py Tools for testing theory covariance matrices and their properties.
- class validphys.theorycovariance.tests.LabeledShifts(process, dataset_name, shifts)
Bases:
tuple
- dataset_name
Alias for field number 1
- process
Alias for field number 0
- shifts
Alias for field number 2
- validphys.theorycovariance.tests.all_matched_data_lengths(all_matched_datasets)[source]
Returns a list of the data sets lengths.
- validphys.theorycovariance.tests.alltheory_vector(matched_dataspecs_dataset_alltheory, matched_dataspecs_dataset_theory)[source]
Returns a DataFrame with the theory vectors for matched dataspecs for the scale-varied theories (not the central one).
- validphys.theorycovariance.tests.combine_by_type_dataspecs(all_matched_results, matched_dataspecs_dataset_name)[source]
Like combine_by_type but for matched dataspecs
- validphys.theorycovariance.tests.covmap_dataspecs(combine_by_type_dataspecs, matched_dataspecs_dataset_name)[source]
Like covmap but for matched dataspecs.
- validphys.theorycovariance.tests.covs_pt_prescrip_dataspecs(combine_by_type_dataspecs, process_starting_points_dataspecs, dataspecs_theoryids, point_prescription, fivetheories, seventheories)[source]
Like covs_pt_prescrip but for matched dataspecs.
- validphys.theorycovariance.tests.dataspecs_dataset_alltheory(matched_dataspecs_results, process, dataset_name)[source]
Returns a LabeledShifts tuple corresponding to the theory vectors for all the scale varied theories (not the central one), processed by data set and experiment for matched dataspecs.
- validphys.theorycovariance.tests.dataspecs_dataset_prediction_shift(matched_dataspecs_results, process, dataset_name)[source]
Compute the difference in theory predictions between two dataspecs. This can be used in combination with matched_datasets_from_dataspecs It returns a
LabeledShifts
containingdataset_name
,process
andshifts
.
- validphys.theorycovariance.tests.dataspecs_dataset_theory(matched_dataspecs_results, process, dataset_name)[source]
Returns a tuple of shifts processed by data set and experiment for matched dataspecs.
- validphys.theorycovariance.tests.deltamiss_plot(theory_shift_test, allthx_vector, evals_nonzero_basis, shx_vector)[source]
Produces a plot of the missing component of the shift vector, transformed back to the data space.
- validphys.theorycovariance.tests.efficiency(theory_shift_test)[source]
Returns (efficiency = 1 - fmiss/f) with which the theory covariance matrix encapsulates the NNLO-NLO shift.
- validphys.theorycovariance.tests.eigenvector_plot(evals_nonzero_basis, shx_vector)[source]
Produces a plot of the eigenvectors for the projected matrix, transformed back to the data space.
- validphys.theorycovariance.tests.evals_nonzero_basis(allthx_vector, thx_covmat, thx_vector, collected_theoryids, fivetheories, seventheories: (<class 'str'>, <class 'NoneType'>) = None, orthonormalisation: (<class 'str'>, <class 'NoneType'>) = None)[source]
Projects the theory covariance matrix from the data space into the basis of non-zero eigenvalues, dependent on point-prescription. Then returns the eigenvalues (w) and eigenvectors (v) in the data space. There are three methods to linearly orthonormalise the basis vectors for the covariance matrix, and the choice must be specified using the “orthonormalisation” flag in the runcard. The choices are: gs, the Gram-Schmidt method; qr, QR decomposition; svd, singular value decomposition. QR is the method which should be used as standard; the others exist for testing purposes.
- validphys.theorycovariance.tests.matched_experiments_index(matched_dataspecs_dataset_name, all_matched_data_lengths)[source]
Returns MultiIndex composed of data set name and starting point of data set.
- validphys.theorycovariance.tests.process_starting_points_dataspecs(combine_by_type_dataspecs)[source]
Like process_starting_points but for matched dataspecs.
- validphys.theorycovariance.tests.projector_eigenvalue_ratio(theory_shift_test)[source]
Produces a plot of the ratio between the projectors and the square roots of the corresponding eigenvalues.
- validphys.theorycovariance.tests.shift_diag_cov_comparison(allthx_vector, shx_vector, thx_covmat, thx_vector)[source]
Produces a plot of a comparison between the NNLO-NLO shift and the envelope given by the diagonal elements of the theory covariance matrix.
- validphys.theorycovariance.tests.shift_vector(matched_dataspecs_dataset_prediction_shift, matched_dataspecs_dataset_theory)[source]
Returns a DataFrame of normalised shift vectors for matched dataspecs.
- validphys.theorycovariance.tests.theory_corrmat_custom_dataspecs(theory_covmat_custom_dataspecs)[source]
Calculates the theory correlation matrix for scale variations with variations by process type
- validphys.theorycovariance.tests.theory_covmat_custom_dataspecs(covs_pt_prescrip_dataspecs, covmap_dataspecs, matched_experiments_index)[source]
Like theory_covmat_custom but for matched dataspecs.
- validphys.theorycovariance.tests.theory_covmat_eigenvalues(theory_shift_test)[source]
Returns a table of s = sqrt(eigenvalue), the projector and the ratio of the two, ordered by largest eigenvalue.
- validphys.theorycovariance.tests.theory_matrix_threshold(theory_threshold: (<class 'int'>, <class 'float'>) = 0)[source]
Returns the threshold below which theory correlation elements are set to zero when comparing to shift correlation matrix
- validphys.theorycovariance.tests.theory_shift_test(shx_vector, evals_nonzero_basis)[source]
Compares the NNLO-NLO shift, f, with the eigenvectors and eigenvalues of the theory covariance matrix, and returns the component of the NNLO-NLO shift space which is missed by the covariance matrix space: fmiss, as well as the projections of the shift vector onto each of the eigenvectors: projectors.
- validphys.theorycovariance.tests.theory_vector(matched_dataspecs_dataset_theory)[source]
Returns a DataFrame of the central theory vector for matched dataspecs.
- validphys.theorycovariance.tests.theta(theory_shift_test)[source]
Returns the angle between the NNLO-NLO shift vector and the component of this which is captured by the theory covariance matrix
- validphys.theorycovariance.tests.validation_theory_chi2(theory_shift_test)[source]
Returns the theory chi2 for comparing NNLO-NLO shift with theory covariance matrix.
- validphys.theorycovariance.tests.vectors_3pt(splitdiffs)[source]
Returns the linearly independent vectors for 3pt prescription
- validphys.theorycovariance.tests.vectors_5barpt(splitdiffs)[source]
Returns the linearly independent vectors for 5barpt prescription
- validphys.theorycovariance.tests.vectors_5pt(splitdiffs)[source]
Returns the linearly independent vectors for 5pt prescription
validphys.theorycovariance.theorycovarianceutils module
theorycovarianceutils.py
Low level utilities for theorycovariance module
- validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination(theoryids, fivetheories, point_prescription: (<class 'str'>, <class 'NoneType'>) = None)
Checks that a valid theory combination corresponding to an existing prescription has been inputted
- validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_dataspecs(dataspecs_theoryids, fivetheories)[source]
Like check_correct_theory_combination but for matched dataspecs.
- validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_internal(theoryids, fivetheories, point_prescription: (<class 'str'>, <class 'NoneType'>) = None)[source]
Checks that a valid theory combination corresponding to an existing prescription has been inputted
- validphys.theorycovariance.theorycovarianceutils.check_correct_theory_combination_theoryconfig(collected_theoryids, fivetheories)[source]
- validphys.theorycovariance.theorycovarianceutils.check_fit_dataset_order_matches_grouped(group_dataset_inputs_by_metadata, data_input, processed_metadata_group)[source]
Check for use with theory covmat generation.
Makes sure that the order of datasets listed in the fit runcard is the same as that specified by the metadata grouping. Otherwise there can be a misalignment between the experiment covmat and theory covmat.
Module contents
theorycovariance
Tools for constructing and studying theory covariance matrices.
This package contains tools for constructing covariance matrices and computing their chi2s in construction.py, tools for plotting covariance matrices and tabulating their properties in output.py, and tools for testing their properties in tests.py.