Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decide whether to keep model kwargs override, add model kwargs, etc. #86

Open
sgbaird opened this issue Jan 25, 2025 · 0 comments
Open

Comments

@sgbaird
Copy link
Owner

sgbaird commented Jan 25, 2025

Currently embedded in (somewhat complex) Jinja logic. Maybe just requires a dummy var passed to Jinja template.

Current:

model_kwargs={
{% if task == "Multi" %}
{% if model == "Fully Bayesian" %}
"num_samples": 512,
"warmup_steps": 512,
"transforms": Specified_Task_ST_MTGP_trans
{% elif model == "Custom" %}
"botorch_acqf_class": UpperConfidenceBound,
"transforms": Specified_Task_ST_MTGP_trans
{% else %}
"transforms": Specified_Task_ST_MTGP_trans
{% endif %}
{% elif model == "Fully Bayesian" %}
"num_samples": 512,
"warmup_steps": 512
{% elif model == "Custom" %}
"botorch_acqf_class": UpperConfidenceBound
{% else %}
{}
{% endif %}
},

Not using anymore:

def add_model_specific_keys(option_names, opt):
"""Add model-specific keys to the options dictionary (in-place).
This function adds model-specific keys to the options dictionary `opt`. For
example, if use_custom_gen is a hidden variable, and the model is
FULLYBAYESIAN, then use_custom_gen should be True.
It also sets the value of the key `model_kwargs` based on the value of
`MODEL_OPT_KEY` in `opt`.
Parameters
----------
option_names : list
A list of option names.
opt : dict
The options dictionary.
Examples
--------
The following example is demonstrative, the range of cases may be expanded
later.
>>> option_names = [
... "objective",
... "model",
... "custom_gen",
... "existing_data",
... "sum_constraint",
... "order_constraint",
... "linear_constraint",
... "composition_constraint",
... "categorical",
... "custom_threshold",
... "fidelity",
... "synchrony",
... ]
>>> opt = {
... "objective": "single",
... "model": "Default",
... "existing_data": False,
... "sum_constraint": False,
... "order_constraint": False,
... "linear_constraint": False,
... "composition_constraint": False,
... "categorical": False,
... "custom_threshold": False,
... "fidelity": "single",
... "synchrony": "single",
... }
>>> add_model_specific_keys(option_names, opt)
{
"objective": "single",
"model": "Default",
"existing_data": False,
"sum_constraint": False,
"order_constraint": False,
"linear_constraint": False,
"composition_constraint": False,
"categorical": False,
"custom_threshold": False,
"fidelity": "single",
"synchrony": "single",
"custom_gen": False,
"model_kwargs": {},
}
"""
# opt.setdefault(cst.CUSTOM_GEN_KEY, opt[cst.MODEL_OPT_KEY] ==
# cst.FULLYBAYESIAN_KEY) NOTE: setdefault was conflicting with
# create_model_options, which already was setting defaults Now it's simply
# overriding whatever was there
# opt[cst.CUSTOM_GEN_KEY] = opt[cst.MODEL_OPT_KEY] == cst.FULLYBAYESIAN_KEY
if opt[cst.TASK_OPT_KEY] == "Multi":
opt[cst.MODEL_OPT_KEY] == cst.CUSTOM_KEY
opt[cst.CUSTOM_GEN_KEY] = (
(opt[cst.MODEL_OPT_KEY] == cst.FULLYBAYESIAN_KEY)
or (opt[cst.MODEL_OPT_KEY] == cst.CUSTOM_KEY)
or (opt[cst.TASK_OPT_KEY] == "Multi")
)
# log_fn(f"opt: {opt}")
# increased from the default in Ax tutorials for quality/robustness
opt["model_kwargs"] = (
{"num_samples": 1024, "warmup_steps": 1024}
if opt[cst.MODEL_OPT_KEY] == cst.FULLYBAYESIAN_KEY
else {}
) # override later to 16 and 32 later on, but only for test script
# verify that all variables (hidden and visible) are represented
assert all(
[opt.get(option_name, None) is not None for option_name in option_names]
), f"option_names {option_names} not in opt {opt}"

Nor really the following since removal of brute force testing (replacement not implemented yet):

def model_kwargs_test_override(render_datum):
"""make sure tests run faster for SAASBO"""
model_kwargs = render_datum[cst.MODEL_KWARGS_KEY]
if "num_samples" in model_kwargs and "warmup_steps" in model_kwargs:
model_kwargs["num_samples"] = 16
model_kwargs["warmup_steps"] = 32
return render_datum

And in general these may require some renaming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant