OmniSafe Experiment Grid#

ExperimentGrid([exp_name])

Tool for running many experiments given hyper-parameters ranges.

Experiment Grid#

Documentation

class omnisafe.common.experiment_grid.ExperimentGrid(exp_name='')[source]#

Tool for running many experiments given hyper-parameters ranges.

Initialize the ExperimentGrid.

Parameters:

exp_name (str) – Name of the experiment grid.

__init__(exp_name='')[source]#

Initialize the ExperimentGrid.

Parameters:

exp_name (str) – Name of the experiment grid.

_default_shorthand(key)[source]#

Default shorthand. Create a default shorthand for the key, built from the first three letters of each colon-separated part. But if the first three letters contains something which isn’t alphanumeric, shear that off.

Example

>>> _default_shorthand('env_name:SafetyPointGoal1-v0')
'env'
Parameters:

key (string) – Name of parameter.

_variants(keys, vals)[source]#

Recursively builds list of valid variants.

Parameters:

  • keys (list) – List of keys.

  • vals (list) – List of values.

add(key, vals, shorthand=None, in_name=False)[source]#

Add a parameter (key) to the grid config, with potential values (vals).

By default, if a shorthand isn’t given, one is automatically generated from the key using the first three letters of each colon-separated term.

Hint

This function is called in omnisafe/examples/benchmarks/run_experiment_grid.py.

Example

>>> add('env_id', ['SafetyPointGoal1-v0', 'MountainCar-v0', 'Acrobot-v1'])
>>> add('algo', ['SAC', 'DDPG', 'TD3'])
>>> add('seed', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Parameters:

  • key (string) – Name of parameter.

  • vals (value or list of values) – Allowed values of parameter.

  • shorthand (string) – Optional, shortened name of parameter. For example, maybe the parameter steps_per_epoch is shortened to steps.

  • in_name (bool) – When constructing variant names, force the inclusion of this parameter into the name.

check_variant_vaild(variant)[source]#

Check if the variant is valid.

print()[source]#

Print a helpful report about the experiment grid.

This function prints a helpful report about the experiment grid, including the name of the experiment grid, the parameters being varied, and the possible values for each parameter.

In Command Line:

===================== ExperimentGrid [test] runs over parameters: =====================
env_name                                [env]
['SafetyPointGoal1-v0', 'MountainCar-v0', 'Acrobot-v1']
algo                               [algo]
['SAC', 'DDPG', 'TD3']
seed                                    [seed]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Return type:

None

run(thunk, num_pool=1, parent_dir=None, is_test=False, gpu_id=None)[source]#

Run each variant in the grid with function ‘thunk’.

Note: ‘thunk’ must be either a callable function, or a string. If it is a string, it must be the name of a parameter whose values are all callable functions.

Uses call_experiment to actually launch each experiment, and gives each variant a name using self.variant_name().

Maintenance note: the args for ExperimentGrid.run should track closely to the args for call_experiment. However, seed is omitted because we presume the user may add it as a parameter in the grid.

save_grid_config()[source]#

Save experiment grid configurations as json.

save_results(exp_names, variants, results)[source]#

Save results to a file.

save_same_exps_config(exps_log_dir, variant)[source]#

Save experiment grid configurations as json.

update_dic(total_dic, item_dic)[source]#

Updater of multi-level dictionary.

This function is used to update the total dictionary with the item dictionary.

Parameters:

  • total_dic (dict) – Total dictionary.

  • item_dic (dict) – Item dictionary.

variant_name(variant)[source]#

Given a variant (dict of valid param/value pairs), make an exp_name.

A variant’s name is constructed as the grid name (if you’ve given it one), plus param names (or shorthands if available) and values separated by underscores.

..warning::

if seed is a parameter, it is not included in the name.

Example

>>> variant_name({'env_id': 'SafetyPointGoal1-v0', 'algo': 'SAC', 'seed': 0})
exp_name = 'SafetyPointGoal1-v0_SAC_0'
Parameters:

variant (dict) – Variant dictionary.

variants()[source]#

Makes a list of dict, where each dict is a valid config in the grid.

There is special handling for variant parameters whose names take the form 'full:param:name'.

The colons are taken to indicate that these parameters should have a nested dict structure. eg, if there are two params,

Hint

Key

Val

'base:param:a'

1

'base:param:b'

2

'base:param:c'

3

'special:d'

4

'special:e'

5

the variant dict will have the structure

{
    'base': {
        'param': {
            'a': 1,
            'b': 2,
            'c': 3
        }
    },
    'special': {
        'd': 4,
        'e': 5
    }
}