n3fit.hyper_optimization package
Submodules
n3fit.hyper_optimization.filetrials module
Custom hyperopt trial object for persistent file storage in the form of json and pickle files within the nnfit folder
- class n3fit.hyper_optimization.filetrials.FileTrials(replica_path, parameters=None, **kwargs)[source]
Bases:
Trials
Stores trial results on the fly inside the nnfit replica folder
- Parameters
replica_path (path) – Replica folder as generated by n3fit
parameters (dict) – Dictionary of parameters on which we are doing hyperoptimization
- classmethod from_pkl(pickle_filepath)[source]
Load and return an instance of FileTrials from a pickle file.
- If a pickle file from previous run is present this method can be used
to instantiate an initial FileTrials object to restart.
- refresh()[source]
This is the “flushing” method which is called at the end of every trial to save things in the database. We are are overloading it in order to also write to a json file with every single trial.
- n3fit.hyper_optimization.filetrials.space_eval_trial(space, trial)[source]
This function is a wrapper around hyperopt’s space eval in order to add to the json a dictionary containing the human-readable values. i.e., the standard json would say: “optimizer = [5]” and we want it to say optimizer = “Adam” But all this function does before calling hyperopt’s space_eval is to “unlist” the items. If you think space_eval should do that by itself, you are not alone https://github.com/hyperopt/hyperopt/issues/383#issuecomment-378561408
- # Arguments:
- space: the dictionary containing the hyperopt space samplers we pass
to the hyperparametrizable function
- trial: trial dictionary. This is a dictionary containing (among other things)
the list of parameters that were tried for this iteration of hyperopt
- # Returns:
A dictionary containing the values of all the parameters in a human-readable format
n3fit.hyper_optimization.hyper_scan module
The HyperScanner class is basically a dictionary containing all parameters, the functions defined as hp_ (from hyperspace)
The goal of this module is to read all parameters in the hyperopt section of the runcard and modify the parameter dictionary so that now it is filled with the hyperop sampler objects
The idea behind the wrappers if that if you ever want to use another hyperoptimization library, assuming that it also takes just
a function
a dictionary of spaces of parameters
you can do so by simply modifying the wrappers to point somewhere else (and, of course the function in the fitting action that calls the minimization).
- class n3fit.hyper_optimization.hyper_scan.ActivationStr(fun_name)[source]
Bases:
object
Upon call this class returns an array where the activation function fun_name repeated as many times as hidden layers we have
- # Arguments:
fun_name: name of the activation function
- class n3fit.hyper_optimization.hyper_scan.HyperScanner(parameters, sampling_dict, steps=5)[source]
Bases:
object
The HyperScanner generates a dictionary of parameters for scanning It takes cares of known correlation between parameters by tying them together It also provides methods for updating the parameter dictionaries after using hyperopt
It takes as inpujt the dictionaries defining the NN/fit and the hyperparameter scan from the NNPDF runcard and substitutes in parameters samplers according to the hyper_scan dictionary.
- # Arguments:
parameters: the fitting[parameters] dictionary of the NNPDF runcard
- sampling_dict: the hyperscan dictionary of the NNPDF runcard defining
the search space of the scan
- steps: when taking discrete steps between two parameters, number of steps
to take
- # Parameters accepted by sampling_dict:
- stopping:
min_epochs, max_epochs
min_patience, max_patience
- architecture(initializers=None, activations=None, max_drop=None, n_layers=None, min_units=15, max_units=25, layer_types=None, output_size=None)[source]
- Modifies the following entries of the parameters dictionary:
initializer
dropout
nodes_per_layer
activation_per_layer
layer_type
- optimizer(optimizers)[source]
This function look at the optimizers implemented in MetaModel Since each optimizer can take different parameters, the input to this function, optimizer is a list of dictionaries, each defining the name of the optimizer (which needs to be implemented in n3fit) and the options to modify.
- The accepted options are:
learning_rate
clipnorm
- but for hyperopt it will look as a list of dictionaries
- [ { optimizer_name: optimizer_name, learning_rate: sampler },
{ optimizer_name: optimizer_name, learning_rate: sampler }, …
]
and will sample one from this list.
Note that the keys within the dictionary (optimizer_name and learning_rate) should be named as the keys used by the compiler of the model as they are used as they come.
- n3fit.hyper_optimization.hyper_scan.hp_choice(key, choices)[source]
Sample from the list or array
choices
- n3fit.hyper_optimization.hyper_scan.hp_loguniform(key, lower_end, higher_end)[source]
Sample from lower_end to higher_end lograithmically. Note that it is different from numpy’s logspace in that it takes the lower and higher boundaries, not the value of the exponent
- n3fit.hyper_optimization.hyper_scan.hp_quniform(key, lower_end, higher_end, step_size=None, steps=None, make_int=False)[source]
Like uniform but admits a step_size
- n3fit.hyper_optimization.hyper_scan.hp_uniform(key, lower_end, higher_end)[source]
Sample uniformly between lower_end and higher_end
- n3fit.hyper_optimization.hyper_scan.hyper_scan_wrapper(replica_path_set, model_trainer, hyperscanner, max_evals=1)[source]
This function receives a
ModelTrainer
object as well as the definition of the hyperparameter scan (hyperscanner
) and performsmax_evals
evaluations of the hyperparametrizable function ofmodel_trainer
.A
tries.json
file will be saved in thereplica_path_set
folder with the information of all trials. An additionaltries.pkl
file will also be generated in the same folder that stores the previous states of FileTrials, this file can be used for restarting purposes.- Parameters
replica_path_set (path) – folder where to create the
tries.json
andtries.pkl
filesmodel_trainer (
n3fit.ModelTrainer.ModelTrainer
) – aModelTrainer
object with thehyperparametrizable
methodhyperscanner (
n3fit.hyper_optimization.hyper_scan.HyperScanner
) – aHyperScanner
object defining the scanmax_evals (int) – Number of trials to run
- Returns
dict
parameters of the best trial as found by
hyperopt
n3fit.hyper_optimization.mongofiletrials module
Hyperopt trial object for parallel hyperoptimization with MongoDB. Data are fetched from MongoDB databases and stored in the form of json and tar.gz files within the nnfit folder.
- class n3fit.hyper_optimization.mongofiletrials.MongoFileTrials(replica_path, db_host='localhost', db_port=27017, db_name='hyperopt-db', num_workers=1, parameters=None, *args, **kwargs)[source]
Bases:
MongoTrials
MongoDB implementation of
n3fit.hyper_optimization.filetrials.FileTrials
.- Parameters
replica_path (path) – Replica folder as generated by n3fit.
db_host (str) – MongoDB database connection host. Defaults to “localhost”.
db_port (int) – MongoDB database connection port. Defaults to 27017.
db_name (str) – MongoDB database name. Defaults to “hyperopt-db”.
num_workers (int) – Number of MongoDB workers to be initiated concurrently. Defaults to 1.
parameters (dict) – Dictionary of parameters on which we are doing hyperoptimization. Default to None.
store_trial (bool) – If True, store data into json file. Default to True.
- static extract_mongodb_database(database_tar_file, path='/home/runner/work/nnpdf/nnpdf/doc/sphinx')[source]
Untar MongoDB database for use in restarts.
- property rstate
Returns the rstate attribute; see
n3fit.hyper_optimization.filetrials.FileTrials
.
- class n3fit.hyper_optimization.mongofiletrials.MongodRunner(db_name='hyperopt-db', db_port=27017)[source]
Bases:
object
Class to manage a MongoDB instance.
This class is responsible for automatically creating and managing a MongoDB database using the mongod command. It allows for starting and stopping a MongoDB instance programmatically.
- Parameters
- n3fit.hyper_optimization.mongofiletrials.convert_bson_to_dict(obj)[source]
Recursively convert a BSON object to a standard Python dictionary.
This function is particularly useful for converting MongoDB query results, which may contain BSON types like ObjectId and SON, into a more manageable dictionary format.
- Parameters
obj (dict or bson.SON or list or any) – The object to convert. Can be a BSON object (like SON), a dictionary containing BSON types, a list of such objects, or any other type.
- Returns
A Python dictionary with all BSON types converted to standard Python types (e.g., ObjectId converted to string). If the input is a list, returns a list of converted elements. For other types, returns the object as is.
- Return type
Examples
>>> from bson import ObjectId, SON >>> sample_son = SON([('_id', ObjectId('507f1f77bcf86cd799439011')), ('name', 'John Doe')]) >>> convert_bson_to_dict(sample_son) {'_id': '507f1f77bcf86cd799439011', 'name': 'John Doe'}
>>> sample_list = [SON([('_id', ObjectId('507f1f77bcf86cd799439011')), ('name', 'John Doe')]), {'age': 30}] >>> convert_bson_to_dict(sample_list) [{'_id': '507f1f77bcf86cd799439011', 'name': 'John Doe'}, {'age': 30}]
n3fit.hyper_optimization.penalties module
Penalties that can be applied to the hyperopt loss
Penalties in this module usually take as signature the positional arguments:
- pdf_model:
n3fit.backends.keras_backend.MetaModel
model taking a
(1, xgrid_size, 1)
array as input and returning a(1, xgrid_size, 14, replicas)
pdf.- stopping_object:
n3fit.stopping.Stopping
object holding the information about the validation model and the stopping parameters
although not all penalties use both.
And return a float to be added to the hyperscan loss.
New penalties can be added directly in this module. The name in the runcard must match the name used in this module.
- n3fit.hyper_optimization.penalties.integrability(pdf_model=None, **_kwargs)[source]
Adds a penalty proportional to the value of the integrability integration It adds a 0-penalty when the value of the integrability is equal or less than the value of the threshold defined in validphys::fitveto
The penalty increases exponentially with the growth of the integrability number
- Returns
array of integrability penalties for each replica
- Return type
NDArray
Example
>>> from n3fit.hyper_optimization.penalties import integrability >>> from n3fit.model_gen import pdfNN_layer_generator >>> fake_fl = [{'fl' : i, 'largex' : [0,1], 'smallx': [1,2]} for i in ['u', 'ubar', 'd', 'dbar', 'c', 'g', 's', 'sbar']] >>> pdf_model = pdfNN_layer_generator(nodes=[8], activations=['linear'], seed=0, flav_info=fake_fl, fitbasis="FLAVOUR") >>> isinstance(integrability(pdf_model), float) True
- n3fit.hyper_optimization.penalties.patience(stopping_object, alpha: float = 0.0001, **_kwargs)[source]
Adds a penalty for fits that have finished too soon, which means the number of epochs or its patience is not optimal. The penalty is proportional to the validation loss and will be 0 when the best epoch is exactly at max_epoch - patience The
alpha
factor is chosen so that at 10k epochs distance the penalty is 2.7 * val_loss- Parameters
alpha (float) – dumping factor for the exponent
- Returns
patience penalty for each replica
- Return type
NDArray
Example
>>> from n3fit.hyper_optimization.penalties import patience >>> from types import SimpleNamespace >>> fake_stopping = SimpleNamespace(e_best_chi2=1000, stopping_patience=500, total_epochs=5000, vl_loss=2.42) >>> patience(fake_stopping, alpha=1e-4) 3.434143467595683
- n3fit.hyper_optimization.penalties.saturation(pdf_model=None, n=100, min_x=1e-06, max_x=0.0001, flavors=None, **_kwargs)[source]
Checks the pdf models for saturation at small x by checking the slope from
min_x
tomax_x
. Sum the saturation loss of all pdf models- Parameters
- Returns
array of saturation penalties for each replica
- Return type
NDArray
Example
>>> from n3fit.hyper_optimization.penalties import saturation >>> from n3fit.model_gen import pdfNN_layer_generator >>> fake_fl = [{'fl' : i, 'largex' : [0,1], 'smallx': [1,2]} for i in ['u', 'ubar', 'd', 'dbar', 'c', 'g', 's', 'sbar']] >>> pdf_model = pdfNN_layer_generator(nodes=[8], activations=['linear'], seed=0, flav_info=fake_fl, fitbasis="FLAVOUR") >>> isinstance(saturation(pdf_model, 5), float) True
n3fit.hyper_optimization.rewards module
Target functions to minimize during hyperparameter scan
These are implemented in the HyperLoss class which incorporates various statistics (average, standard deviation, best/worst case) both across multiple replicas of a model and across different folds.
Key functionalities include: - Support for different loss types such as Chi-square (chi2) and phi-square (phi2). - Calculation of statistical measures (average, best_worst, std) over replicas and folds. - Incorporation of penalties into the loss computation. - Detailed tracking and storage of loss metrics for further analysis.
New statistics can be added directly in this class as staticmethods and via IMPLEMENTED_STATS; their name in the runcard must match the name in the module
Example
>>> import numpy as np
>>> from n3fit.hyper_optimization.rewards import HyperLoss
>>> losses = np.array([1.0, 2.0, 3.0])
>>> loss_average = HyperLoss(fold_statistic="average")
>>> loss_best_worst = HyperLoss(fold_statistic="best_worst")
>>> loss_std = HyperLoss(fold_statistic="std")
>>> print(f"{loss_average.reduce_over_folds.__name__} {loss_average.reduce_over_folds(losses)}")
>>> print(f"{loss_best_worst.reduce_over_folds.__name__} {loss_best_worst.reduce_over_folds(losses)}")
>>> print(f"{loss_std.reduce_over_folds.__name__} {loss_std.reduce_over_folds(losses)}")
_average 2.0
_best_worst 3.0
_std 0.816496580927726
- class n3fit.hyper_optimization.rewards.HyperLoss(loss_type: Optional[str] = None, replica_statistic: Optional[str] = None, fold_statistic: Optional[str] = None, penalties_in_loss: bool = False)[source]
Bases:
object
Class to compute the hyper_loss based on the individual replica losses.
Computes the statistic over the replicas and then over the folds, both statistics default to the average.
The
compute_loss
method saves intermediate metrics such as the chi2 of the folds or the phi regardless of the loss type that has been selected. These metrics are saved in the propertiesphi_vector
: list of phi per foldchi2_matrix
: list of chi2 per fold, per replica- Parameters
loss_type (str) – the type of loss over the replicas to use. Options are “chi2” and “phi2”.
replica_statistic (str) – the statistic over the replicas to use, for per replica losses. Options are “average”, “best_worst”, and “std”.
fold_statistic (str) – the statistic over the folds to use. Options are “average”, “best_worst”, and “std”.
penalties_in_loss (bool) – whether the penalties should be included in the output of
compute_loss
- compute_loss(penalties: dict[str, numpy.ndarray], experimental_loss: ndarray, pdf_model: MetaModel, experimental_data: list[validphys.core.DataGroupSpec], fold_idx: int = 0) float [source]
Compute the loss, including added penalties, for a single fold.
Save the phi of the assemble and the chi2 of the separate replicas, and the penalties into the
phi_vector
,chi2_matrix
andpenalties
attributes.- Parameters
penalties (Dict[str, NDArray(replicas)]) – Dict of penalties for each replica. Possible keys are ‘saturation’, ‘patience’ and ‘integrability’ as defined in ‘penalties.py’ and instantiated within
ModelTrainer
.experimental_loss (NDArray(replicas)) – Experimental loss for each replica.
pdf_model (
n3fit.backends.MetaModel
) – N3fitted meta-model.experimental_data (List[validphys.core.DataGroupSpec]) – List of tuples containing validphys.core.DataGroupSpec instances for each group data set
fold_idx (int) – k-fold index. Defaults to 0.
include_penalties (float) – Whether to include the penalties in the returned loss value
- Returns
loss – The computed loss over the replicas.
- Return type
Example
>>> import numpy as np >>> from n3fit.hyper_optimization.rewards import HyperLoss >>> from n3fit.model_gen import generate_pdf_model >>> from validphys.loader import Loader >>> hyper = HyperLoss(loss_type="chi2", replica_statistic="average", fold_statistic="average") >>> penalties = {'saturation': np.array([1.0, 2.0]), 'patience': np.array([3.0, 4.0]), 'integrability': np.array([5.0, 6.0]),} >>> experimental_loss = np.array([0.1, 0.2]) >>> ds = Loader().check_dataset("NMC_NC_NOTFIXED_P_EM-SIGMARED", theoryid=399, cuts="internal") >>> experimental_data = [Loader().check_experiment("My DataGroupSpec", [ds])] >>> fake_fl = [{'fl' : i, 'largex' : [0,1], 'smallx': [1,2]} for i in ['u', 'ubar', 'd', 'dbar', 'c', 'g', 's', 'sbar']] >>> pdf_model = generate_pdf_model(nodes=[8], activations=['linear'], seed=0, num_replicas=2, flav_info=fake_fl, fitbasis="FLAVOUR") >>> loss = hyper.compute_loss(penalties, experimental_loss, pdf_model, experimental_data)