n3fit.hyper_optimization package

Submodules

n3fit.hyper_optimization.filetrials module

Custom hyperopt trial object for persistent file storage in the form of json and pickle files within the nnfit folder

class n3fit.hyper_optimization.filetrials.FileTrials(replica_path, parameters=None, **kwargs)

Bases: Trials

Stores trial results on the fly inside the nnfit replica folder

Parameters

• replica_path (path) – Replica folder as generated by n3fit

• parameters (dict) – Dictionary of parameters on which we are doing hyperoptimization

classmethod from_pkl(pickle_filepath)

Load and return an instance of FileTrials from a pickle file.

If a pickle file from previous run is present this method can be used

to instantiate an initial FileTrials object to restart.

new_trial_docs(tids, specs, results, miscs)

new_trial_ids(n)

refresh()

This is the “flushing” method which is called at the end of every trial to save things in the database. We are are overloading it in order to also write to a json file with every single trial.

property rstate

Returns the rstate attribute.

Notes

rstate() stores a numpy.random.Generator which is important to make hyperopt restarts reproducible in the hyperparameter space. It can be passed later as the rstate parameters of hyperopt.fmin.

to_pkl()

Dump FileTrials object into a pickle file.

n3fit.hyper_optimization.filetrials.space_eval_trial(space, trial)

This function is a wrapper around hyperopt’s space eval in order to add to the json a dictionary containing the human-readable values. i.e., the standard json would say: “optimizer = [5]” and we want it to say optimizer = “Adam” But all this function does before calling hyperopt’s space_eval is to “unlist” the items. If you think space_eval should do that by itself, you are not alone https://github.com/hyperopt/hyperopt/issues/383#issuecomment-378561408

# Arguments:

• space: the dictionary containing the hyperopt space samplers we pass

  to the hyperparametrizable function

• trial: trial dictionary. This is a dictionary containing (among other things)

  the list of parameters that were tried for this iteration of hyperopt

# Returns:

A dictionary containing the values of all the parameters in a human-readable format

n3fit.hyper_optimization.hyper_scan module

The HyperScanner class is basically a dictionary containing all parameters, the functions defined as hp_ (from hyperspace)

The goal of this module is to read all parameters in the hyperopt section of the runcard and modify the parameter dictionary so that now it is filled with the hyperop sampler objects

The idea behind the wrappers if that if you ever want to use another hyperoptimization library, assuming that it also takes just

• a function

• a dictionary of spaces of parameters

you can do so by simply modifying the wrappers to point somewhere else (and, of course the function in the fitting action that calls the minimization).

class n3fit.hyper_optimization.hyper_scan.ActivationStr(fun_name)

Bases: object

Upon call this class returns an array where the activation function fun_name repeated as many times as hidden layers we have

# Arguments:

• fun_name: name of the activation function

class n3fit.hyper_optimization.hyper_scan.HyperScanner(parameters, sampling_dict, steps=5)

Bases: object

The HyperScanner generates a dictionary of parameters for scanning It takes cares of known correlation between parameters by tying them together It also provides methods for updating the parameter dictionaries after using hyperopt

It takes as inpujt the dictionaries defining the NN/fit and the hyperparameter scan from the NNPDF runcard and substitutes in parameters samplers according to the hyper_scan dictionary.

# Arguments:

• parameters: the fitting[parameters] dictionary of the NNPDF runcard

• sampling_dict: the hyperscan dictionary of the NNPDF runcard defining

  the search space of the scan

• steps: when taking discrete steps between two parameters, number of steps

  to take

# Parameters accepted by sampling_dict:

• stopping:

  • min_epochs, max_epochs

  • min_patience, max_patience

architecture(initializers=None, activations=None, max_drop=None, n_layers=None, min_units=15, max_units=25, layer_types=None, output_size=None)

Modifies the following entries of the parameters dictionary:

• initializer

• dropout

• nodes_per_layer

• activation_per_layer

• layer_type

as_dict()

optimizer(optimizers)

This function look at the optimizers implemented in MetaModel Since each optimizer can take different parameters, the input to this function, optimizer is a list of dictionaries, each defining the name of the optimizer (which needs to be implemented in n3fit) and the options to modify.

The accepted options are:

• learning_rate

• clipnorm

but for hyperopt it will look as a list of dictionaries

[ { optimizer_name: optimizer_name, learning_rate: sampler },

{ optimizer_name: optimizer_name, learning_rate: sampler }, …

]

and will sample one from this list.

Note that the keys within the dictionary (optimizer_name and learning_rate) should be named as the keys used by the compiler of the model as they are used as they come.

positivity(min_multiplier=None, max_multiplier=None, min_initial=None, max_initial=None)

Modifies the following entries of the parameters dictionary:

• pos_multiplier

• pos_initial

Sampling between max and min is uniform for the multiplier and loguniform for the initial

space_eval(trial)

Evaluate a trial using the original parameters dictionary

stopping(min_epochs=None, max_epochs=None, min_patience=None, max_patience=None)

Modifies the following entries of the parameters dictionary:

• epochs

• stopping_patience

Takes self.steps between the min and maximum values given

n3fit.hyper_optimization.hyper_scan.hp_choice(key, choices)

Sample from the list or array choices

n3fit.hyper_optimization.hyper_scan.hp_loguniform(key, lower_end, higher_end)

Sample from lower_end to higher_end lograithmically. Note that it is different from numpy’s logspace in that it takes the lower and higher boundaries, not the value of the exponent

n3fit.hyper_optimization.hyper_scan.hp_quniform(key, lower_end, higher_end, step_size=None, steps=None, make_int=False)

Like uniform but admits a step_size

n3fit.hyper_optimization.hyper_scan.hp_uniform(key, lower_end, higher_end)

Sample uniformly between lower_end and higher_end

n3fit.hyper_optimization.hyper_scan.hyper_scan_wrapper(replica_path_set, model_trainer, hyperscanner, max_evals=1)

This function receives a ModelTrainer object as well as the definition of the hyperparameter scan (hyperscanner) and performs max_evals evaluations of the hyperparametrizable function of model_trainer.

A tries.json file will be saved in the replica_path_set folder with the information of all trials. An additional tries.pkl file will also be generated in the same folder that stores the previous states of FileTrials, this file can be used for restarting purposes.

Parameters

• replica_path_set (path) – folder where to create the tries.json and tries.pkl files

• model_trainer (n3fit.ModelTrainer.ModelTrainer) – a ModelTrainer object with the hyperparametrizable method

• hyperscanner (n3fit.hyper_optimization.hyper_scan.HyperScanner) – a HyperScanner object defining the scan

• max_evals (int) – Number of trials to run

Returns

• dict

• parameters of the best trial as found by hyperopt

n3fit.hyper_optimization.hyper_scan.optimizer_arg_wrapper(hp_key, option_dict)

n3fit.hyper_optimization.mongofiletrials module

n3fit.hyper_optimization.penalties module

Penalties that can be applied to the hyperopt loss

Penalties in this module usually take as signature the positional arguments:

pdf_model: n3fit.backends.keras_backend.MetaModel

model taking a (1, xgrid_size, 1) array as input and returning a (1, xgrid_size, 14, replicas) pdf.

stopping_object: n3fit.stopping.Stopping

object holding the information about the validation model and the stopping parameters

although not all penalties use both.

And return a float to be added to the hyperscan loss.

New penalties can be added directly in this module. The name in the runcard must match the name used in this module.

n3fit.hyper_optimization.penalties.integrability(pdf_model=None, **_kwargs)

Adds a penalty proportional to the value of the integrability integration It adds a 0-penalty when the value of the integrability is equal or less than the value of the threshold defined in validphys::fitveto

The penalty increases exponentially with the growth of the integrability number

Returns

array of integrability penalties for each replica

Return type

NDArray

Example

>>> from n3fit.hyper_optimization.penalties import integrability
>>> from n3fit.model_gen import pdfNN_layer_generator
>>> fake_fl = [{'fl' : i, 'largex' : [0,1], 'smallx': [1,2]} for i in ['u', 'ubar', 'd', 'dbar', 'c', 'g', 's', 'sbar']]
>>> pdf_model = pdfNN_layer_generator(nodes=[8], activations=['linear'], seed=0, flav_info=fake_fl, fitbasis="FLAVOUR")
>>> isinstance(integrability(pdf_model), float)
True

n3fit.hyper_optimization.penalties.patience(stopping_object, alpha: float = 0.0001, **_kwargs)

Adds a penalty for fits that have finished too soon, which means the number of epochs or its patience is not optimal. The penalty is proportional to the validation loss and will be 0 when the best epoch is exactly at max_epoch - patience The alpha factor is chosen so that at 10k epochs distance the penalty is 2.7 * val_loss

Parameters

alpha (float) – dumping factor for the exponent

Returns

patience penalty for each replica

Return type

NDArray

Example

>>> from n3fit.hyper_optimization.penalties import patience
>>> from types import SimpleNamespace
>>> fake_stopping = SimpleNamespace(e_best_chi2=1000, stopping_patience=500, total_epochs=5000, vl_loss=2.42)
>>> patience(fake_stopping, alpha=1e-4)
3.434143467595683

n3fit.hyper_optimization.penalties.saturation(pdf_model=None, n=100, min_x=1e-06, max_x=0.0001, flavors=None, **_kwargs)

Checks the pdf models for saturation at small x by checking the slope from min_x to max_x. Sum the saturation loss of all pdf models

Parameters

• n (int) – Number of point to evaluate the saturation

• min_x (float) – Initial point for checking the slope

• max_x (float) – Final point for checking the slope

• flavors (list(int)) – indices of the flavors to inspect

Returns

array of saturation penalties for each replica

Return type

NDArray

Example

>>> from n3fit.hyper_optimization.penalties import saturation
>>> from n3fit.model_gen import pdfNN_layer_generator
>>> fake_fl = [{'fl' : i, 'largex' : [0,1], 'smallx': [1,2]} for i in ['u', 'ubar', 'd', 'dbar', 'c', 'g', 's', 'sbar']]
>>> pdf_model = pdfNN_layer_generator(nodes=[8], activations=['linear'], seed=0, flav_info=fake_fl, fitbasis="FLAVOUR")
>>> isinstance(saturation(pdf_model, 5), float)
True

n3fit.hyper_optimization.rewards module

Target functions to minimize during hyperparameter scan

These are implemented in the HyperLoss class which incorporates various statistics (average, standard deviation, best/worst case) both across multiple replicas of a model and across different folds.

Key functionalities include: - Support for different loss types such as Chi-square (chi2) and phi-square (phi2). - Calculation of statistical measures (average, best_worst, std) over replicas and folds. - Incorporation of penalties into the loss computation. - Detailed tracking and storage of loss metrics for further analysis.

New statistics can be added directly in this class as staticmethods and via IMPLEMENTED_STATS; their name in the runcard must match the name in the module

Example

>>> import numpy as np
>>> from n3fit.hyper_optimization.rewards import HyperLoss
>>> losses = np.array([1.0, 2.0, 3.0])
>>> loss_average = HyperLoss(fold_statistic="average")
>>> loss_best_worst = HyperLoss(fold_statistic="best_worst")
>>> loss_std = HyperLoss(fold_statistic="std")
>>> print(f"{loss_average.reduce_over_folds.__name__} {loss_average.reduce_over_folds(losses)}")
>>> print(f"{loss_best_worst.reduce_over_folds.__name__} {loss_best_worst.reduce_over_folds(losses)}")
>>> print(f"{loss_std.reduce_over_folds.__name__} {loss_std.reduce_over_folds(losses)}")
_average 2.0
_best_worst 3.0
_std 0.816496580927726

class n3fit.hyper_optimization.rewards.HyperLoss(loss_type: str = None, replica_statistic: str = None, fold_statistic: str = None, reduce_proportion: float = 0.85, penalties_in_loss: bool = False)

Bases: object

Class to compute the hyper_loss based on the individual replica losses.

Computes the statistic over the replicas and then over the folds, both statistics default to the average.

The compute_loss method saves intermediate metrics such as the chi2 of the folds or the phi regardless of the loss type that has been selected. These metrics are saved in the properties

phi2_vector: list of phi per fold chi2_matrix: list of chi2 per fold, per replica

Parameters

• loss_type (str) – the type of loss over the replicas to use. Options are “chi2” and “phi2”.

• replica_statistic (str) – the statistic over the replicas to use, for per replica losses. Options are average, best_worst and _std.

• fold_statistic (str) – the statistic over the folds to use. Options are average, best_worst and _std.

• reduce_proportion (float (default 0.85)) – Proportion of replicas to select when computing statistics.

• penalties_in_loss (bool) – whether the penalties should be included in the output of compute_loss

compute_loss(penalties: dict[str, numpy.ndarray], validation_loss: ndarray, kfold_loss: ndarray, pdf_object: N3PDF, experimental_data: list[validphys.core.DataGroupSpec], fold_idx: int = 0) → float

Compute the loss, including added penalties, for a single fold.

Save the phi of the assemble and the chi2 of the separate replicas, and the penalties into the phi2_vector, chi2_matrix and penalties attributes.

Parameters

• penalties (Dict[str, NDArray(replicas)]) – Dict of penalties for each replica. Possible keys are ‘saturation’, ‘patience’ and ‘integrability’ as defined in ‘penalties.py’ and instantiated within ModelTrainer.

• experimental_loss (NDArray(replicas)) – Experimental loss for each replica.

• pdf_object (n3fit.vpinterface.N3PDF) – N3fitted PDF

• experimental_data (List[validphys.core.DataGroupSpec]) – List of tuples containing validphys.core.DataGroupSpec instances for each group data set

• fold_idx (int) – k-fold index. Defaults to 0.

• include_penalties (float) – Whether to include the penalties in the returned loss value

Returns

loss – The computed loss over the replicas.

Return type

float

Example

>>> import numpy as np
>>> from n3fit.hyper_optimization.rewards import HyperLoss
>>> from n3fit.model_gen import generate_pdf_model
>>> from n3fit.vpinterface import N3PDF
>>> from validphys.loader import Loader
>>> hyper = HyperLoss(loss_type="chi2", replica_statistic="average", fold_statistic="average")
>>> penalties = {'saturation': np.array([1.0, 2.0]), 'patience': np.array([3.0, 4.0]), 'integrability': np.array([5.0, 6.0]),}
>>> experimental_loss = np.array([0.1, 0.2])
>>> ds = Loader().check_dataset("NMC_NC_NOTFIXED_P_EM-SIGMARED", variant="legacy", theoryid=399, cuts="internal")
>>> experimental_data = [Loader().check_experiment("My DataGroupSpec", [ds])]
>>> fake_fl = [{'fl' : i, 'largex' : [0,1], 'smallx': [1,2]} for i in ['u', 'ubar', 'd', 'dbar', 'c', 'g', 's', 'sbar']]
>>> pdf_model = generate_pdf_model(nodes=[8], activations=['linear'], seed=0, num_replicas=2, flav_info=fake_fl, fitbasis="FLAVOUR")
>>> pdf = N3PDF(pdf_model.split_replicas())
>>> loss = hyper.compute_loss(penalties, experimental_loss, pdf, experimental_data)

n3fit.hyper_optimization.rewards.fit_distance(pdfs_per_fold=None, **_kwargs)

Loss function for hyperoptimization based on the distance of the fits of all folds to the first fold

n3fit.hyper_optimization.rewards.fit_future_tests(n3pdfs=None, experimental_models=None, **_kwargs)

Use the future tests as a metric for hyperopt

NOTE: this function should only be called once at the end of every hyperopt iteration, as it is destructive for the models

Module contents