Repository navigation
[Bug]: fixed_parameters causing BotorchError #5254
Description
What happened?
Hello.
I've encountered the bug that some experiment configurations cause a mysterious BotorchError on trial generations when fixing parameters. To demonstrate this, I've created this minimal example below that causes this problem.
(Please be aware that this is a simplified example. I'm aware that I could use parameter_type="int" instead of "float" etc. I'm merely forcing this bug for the simplest case.)
I thank you already for taking time to help with this issue <3
Please provide a minimal, reproducible example of the unexpected behavior.
from ax.api.client import Client
from ax.api.configs import RangeParameterConfig
parameter_configs = [
RangeParameterConfig(name="p1", bounds=(-1.0, 0.0), parameter_type="float"),
RangeParameterConfig(name="p2", bounds=(75.0, 150.0), parameter_type="float", step_size=1.0),
RangeParameterConfig(name="p3", bounds=(-150.0, -75.0), parameter_type="float", step_size=1.0),
RangeParameterConfig(name="p4", bounds=(0.0, 0.1), parameter_type="float", step_size=0.01),
]
client = Client()
client.configure_experiment(parameters=parameter_configs)
client.configure_optimization(objective="score")
client.configure_generation_strategy(initialize_with_center=False)
dataset = [
{"p1": -1.0, "p2": 130.0, "p3": -120.0, "p4": 0.1, "score": 0.01},
{"p1": -1.0, "p2": 90.0, "p3": -140.0, "p4": 0.08, "score": 0.08},
{"p1": -1.0, "p2": 150.0, "p3": -75.0, "p4": 0.06, "score": 0.09},
{"p1": 0.0, "p2": 140.0, "p3": -150.0, "p4": 0.09, "score": 0.12},
{"p1": -1.0, "p2": 120.0, "p3": -120.0, "p4": 0.09, "score": 0.70},
]
### Load data
for data in dataset:
parameters = {k: v for k, v in data.items() if k not in ["score"]}
raw_data = {k: v for k, v in data.items() if k in ["score"]}
trial_index = client.attach_trial(parameters=parameters)
client.complete_trial(trial_index=trial_index, raw_data=raw_data)
client.get_next_trials(max_trials=1, fixed_parameters={"p1": 0.0})Please paste any relevant traceback/logs produced by the example provided.
---------------------------------------------------------------------------
BotorchError Traceback (most recent call last)
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generators/torch/botorch_modular/utils.py:937, in validate_candidates(candidates, bounds, discrete_choices, inequality_constraints, feature_names, task_features, equality_constraints)
936 try:
--> 937 columnwise_clamp(
938 candidates, lower=bounds[0], upper=bounds[1], raise_on_violation=True
939 )
940 except BotorchError as e:
File ~/miniconda3/envs/default/lib/python3.12/site-packages/botorch/optim/utils/acquisition_utils.py:61, in columnwise_clamp(X, lower, upper, raise_on_violation)
60 if raise_on_violation and not X.allclose(out):
---> 61 raise BotorchError(
62 f"Original value(s) are out of bounds: {out=}, {X=}, {lower=}, {upper=}."
63 )
65 return out
BotorchError: Original value(s) are out of bounds: out=tensor([[ 1., 96., 75., 0.]], dtype=torch.float64), X=tensor([[1.5000e+02, 9.6000e+01, 5.0000e-02, 0.0000e+00]], dtype=torch.float64), lower=tensor([[-1., 50., 75., 0.]], dtype=torch.float64), upper=tensor([[1.0000e+00, 2.0000e+02, 1.5000e+02, 1.0000e-01]], dtype=torch.float64).
During handling of the above exception, another exception occurred:
CandidateGenerationError Traceback (most recent call last)
Cell In[7], line 1
----> 1 client.get_next_trials(max_trials=1, fixed_parameters={"p1": 0.0})
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/api/client.py:469, in Client.get_next_trials(self, max_trials, fixed_parameters)
467 trials: list[Trial] = []
468 with with_rng_seed(seed=self._random_seed):
--> 469 grs_for_trials = self._generation_strategy_or_choose().gen(
470 experiment=self._experiment,
471 n=1,
472 fixed_features=(
473 ObservationFeatures(
474 parameters=cast(CoreTParameterization, fixed_parameters)
475 )
476 if fixed_parameters is not None
477 else None
478 ),
479 num_trials=max_trials,
480 )
482 for trial_grs in grs_for_trials:
483 assert len(trial_grs) == 1
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_strategy.py:316, in GenerationStrategy.gen(self, experiment, data, n, fixed_features, num_trials)
313 num_trials = max(num_trials, 1) # Ensure at least 1 trial
314 for _ in range(num_trials):
315 grs_for_multiple_trials.append(
--> 316 self._gen_with_multiple_nodes(
317 experiment=experiment,
318 data=data,
319 n=n,
320 pending_observations=pending_observations,
321 fixed_features=fixed_features,
322 first_generation_in_multi=len(grs_for_multiple_trials) < 1,
323 )
324 )
325 return grs_for_multiple_trials
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_strategy.py:565, in GenerationStrategy._gen_with_multiple_nodes(self, experiment, n, pending_observations, data, fixed_features, first_generation_in_multi)
563 transitioned = self._transition_to_next_node()
564 try:
--> 565 gr = self._curr.gen(
566 experiment=experiment,
567 data=data,
568 pending_observations=pending_observations,
569 skip_fit=not (first_generation_in_multi or transitioned),
570 n=n,
571 **pack_gs_gen_kwargs,
572 )
573 except DataRequiredError as err:
574 # Generator needs more data, so we log the error and return
575 # as many generator runs as we were able to produce, unless
576 # no trials were produced at all (in which case its safe to raise).
577 if len(grs_this_gen) == 0:
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_node.py:533, in GenerationNode.gen(self, experiment, pending_observations, skip_fit, data, n, **gs_gen_kwargs)
526 gr = self._gen_maybe_deduplicate(
527 experiment=experiment,
528 data=data,
529 pending_observations=pending_observations,
530 **generator_gen_kwargs,
531 )
532 except Exception as e:
--> 533 gr = self._try_gen_with_fallback(
534 exception=e,
535 experiment=experiment,
536 data=data,
537 pending_observations=pending_observations,
538 **generator_gen_kwargs,
539 )
541 gr._generation_node_name = self.name
542 gr._suggested_experiment_status = self.suggested_experiment_status
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_node.py:656, in GenerationNode._try_gen_with_fallback(self, exception, experiment, data, pending_observations, **generator_gen_kwargs)
654 error_type = type(exception)
655 if error_type not in self.fallback_specs:
--> 656 raise exception
658 # identify fallback model to use
659 fallback_model = self.fallback_specs[error_type]
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_node.py:526, in GenerationNode.gen(self, experiment, pending_observations, skip_fit, data, n, **gs_gen_kwargs)
520 # Step 3: Generate from the underlying adapter and generator, with fallback.
521 try:
522 # Generate from the main generator on this node. If deduplicating,
523 # keep generating until each of `generator_run.arms` is not a
524 # duplicate of a previous active arm (e.g. not from a failed trial)
525 # on the experiment.
--> 526 gr = self._gen_maybe_deduplicate(
527 experiment=experiment,
528 data=data,
529 pending_observations=pending_observations,
530 **generator_gen_kwargs,
531 )
532 except Exception as e:
533 gr = self._try_gen_with_fallback(
534 exception=e,
535 experiment=experiment,
(...) 538 **generator_gen_kwargs,
539 )
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_node.py:622, in GenerationNode._gen_maybe_deduplicate(self, experiment, pending_observations, data, **generator_gen_kwargs)
620 while n_gen_draws < MAX_GEN_ATTEMPTS:
621 n_gen_draws += 1
--> 622 gr = self._gen(
623 experiment=experiment,
624 data=data,
625 pending_observations=pending_observations,
626 **generator_gen_kwargs,
627 )
628 if not self.should_deduplicate or not dedup_against_arms:
629 return gr # Not deduplicating.
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generation_node.py:588, in GenerationNode._gen(self, experiment, pending_observations, data, n, **generator_gen_kwargs)
584 if n is None and generator_spec.generator_gen_kwargs:
585 # If `n` is not specified, ensure that the `None` value does not
586 # override the one set in `generator_spec.generator_gen_kwargs`.
587 n = generator_spec.generator_gen_kwargs.get("n", None)
--> 588 return generator_spec.gen(
589 experiment=experiment,
590 data=data,
591 # `n` cannot be `None` after this point. If it is, the adapter will not
592 # know how many arms to generate. This is the lowest common ancestor
593 # of all `gen`-s in GS and GN, so we apply the default here.
594 n=n if n is not None else 1,
595 # For `pending_observations`, prefer the input to this function, as
596 # `pending_observations` are dynamic throughout the experiment and thus
597 # unlikely to be specified in `generator_spec.generator_gen_kwargs`.
598 pending_observations=pending_observations,
599 **generator_gen_kwargs,
600 )
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generation_strategy/generator_spec.py:291, in GeneratorSpec.gen(self, **generator_gen_kwargs)
289 # copy to ensure there is no in-place modification
290 generator_gen_kwargs = deepcopy(generator_gen_kwargs)
--> 291 generator_run = fitted_adapter.gen(**generator_gen_kwargs)
293 generator_run._gen_metadata = (
294 {} if generator_run.gen_metadata is None else generator_run.gen_metadata
295 )
297 return generator_run
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/adapter/base.py:908, in Adapter.gen(self, n, search_space, optimization_config, pending_observations, fixed_features, model_gen_options)
901 base_gen_args = self._get_transformed_gen_args(
902 search_space=search_space,
903 optimization_config=optimization_config,
904 pending_observations=pending_observations,
905 fixed_features=fixed_features,
906 )
907 # Apply terminal transform and gen
--> 908 gen_results = self._gen(
909 n=n,
910 search_space=base_gen_args.search_space,
911 optimization_config=base_gen_args.optimization_config,
912 pending_observations=base_gen_args.pending_observations,
913 fixed_features=base_gen_args.fixed_features,
914 model_gen_options=model_gen_options,
915 )
917 observation_features = gen_results.observation_features
918 best_obsf = gen_results.best_observation_features
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/adapter/torch.py:927, in TorchAdapter._gen(self, n, search_space, pending_observations, fixed_features, model_gen_options, optimization_config)
918 search_space_digest, torch_opt_config = self._get_transformed_model_gen_args(
919 search_space=search_space,
920 pending_observations=pending_observations,
(...) 923 optimization_config=optimization_config,
924 )
926 # Generate the candidates
--> 927 gen_results = self.generator.gen(
928 n=n,
929 search_space_digest=search_space_digest,
930 torch_opt_config=torch_opt_config,
931 )
933 gen_metadata = dict(gen_results.gen_metadata)
934 if (
935 isinstance(optimization_config, MultiObjectiveOptimizationConfig)
936 and gen_metadata.get("objective_thresholds", None) is not None
(...) 940 # if using a hypervolume based acquisition function, then
941 # the inferred objective thresholds are in gen_metadata.
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generators/torch/botorch_modular/generator.py:399, in BoTorchGenerator.gen(self, n, search_space_digest, torch_opt_config)
391 model_gen_options = torch_opt_config.model_gen_options or {}
392 candidate_set, sampling_strategy = _build_candidate_generation_options(
393 model_gen_options=model_gen_options,
394 torch_opt_config=torch_opt_config,
395 surrogate=self.surrogate,
396 acqf=acqf,
397 )
--> 399 candidates, expected_acquisition_value, weights = acqf.optimize(
400 n=n,
401 search_space_digest=search_space_digest,
402 inequality_constraints=_to_inequality_constraints(
403 linear_constraints=torch_opt_config.linear_constraints
404 ),
405 equality_constraints=_to_equality_constraints(
406 equality_constraints=torch_opt_config.equality_constraints
407 ),
408 fixed_features=torch_opt_config.fixed_features,
409 rounding_func=botorch_rounding_func,
410 optimizer_options=assert_is_instance(
411 opt_options,
412 dict,
413 ),
414 candidate_set=candidate_set,
415 sampling_strategy=sampling_strategy,
416 )
417 gen_metadata = self._get_gen_metadata_from_acqf(
418 acqf=acqf,
419 torch_opt_config=torch_opt_config,
420 expected_acquisition_value=expected_acquisition_value,
421 )
422 return TorchGenResults(
423 points=candidates.detach().cpu(),
424 weights=weights,
425 gen_metadata=gen_metadata,
426 )
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generators/torch/botorch_modular/acquisition.py:769, in Acquisition.optimize(self, n, search_space_digest, inequality_constraints, equality_constraints, fixed_features, rounding_func, optimizer_options, candidate_set, sampling_strategy)
760 candidates, acqf_values = optimize_acqf_discrete_local_search(
761 acq_function=self.acqf,
762 q=n,
(...) 766 **optimizer_options_with_defaults,
767 )
768 # Validate candidates before returning
--> 769 validate_candidates(
770 candidates=candidates,
771 bounds=bounds,
772 discrete_choices=ssd.discrete_choices
773 if ssd.discrete_choices
774 else None,
775 inequality_constraints=inequality_constraints,
776 feature_names=ssd.feature_names,
777 task_features=ssd.task_features,
778 )
779 n_candidates = candidates.shape[0]
780 return (
781 candidates,
782 acqf_values,
783 arm_weights[:n_candidates] * n_candidates / n,
784 )
File ~/miniconda3/envs/default/lib/python3.12/site-packages/ax/generators/torch/botorch_modular/utils.py:941, in validate_candidates(candidates, bounds, discrete_choices, inequality_constraints, feature_names, task_features, equality_constraints)
937 columnwise_clamp(
938 candidates, lower=bounds[0], upper=bounds[1], raise_on_violation=True
939 )
940 except BotorchError as e:
--> 941 raise CandidateGenerationError(f"Candidate violates bounds: {e}")
943 # 2. Discrete value validation
944 task_features_set = set(task_features) if task_features else set()
CandidateGenerationError: Candidate violates bounds: Original value(s) are out of bounds: out=tensor([[ 1., 96., 75., 0.]], dtype=torch.float64), X=tensor([[1.5000e+02, 9.6000e+01, 5.0000e-02, 0.0000e+00]], dtype=torch.float64), lower=tensor([[-1., 50., 75., 0.]], dtype=torch.float64), upper=tensor([[1.0000e+00, 2.0000e+02, 1.5000e+02, 1.0000e-01]], dtype=torch.float64).Ax Version
1.3.1
Python Version
3.12.13
Operating System
Ubuntu 26
(Optional) Describe any potential fixes you've considered to the issue outlined above.
Root cause
Acquisition.optimize() in ax/generators/torch/botorch_modular/acquisition.py picks the optimize_acqf_discrete_local_search branch (lines ~755-759) when the search space becomes fully discrete after merging in fixed_parameters. That branch converts the discrete_choices mapping to a positional list via:
discrete_choices = [
torch.tensor(c, device=self.device, dtype=self.dtype)
for c in discrete_choices.values()
]This assumes .values() iterates in ascending feature-index order (dimension i → discrete_choices[i]). But mk_discrete_choices() (ax/generators/utils.py:634-646) builds this dict via:
discrete_choices = {**discrete_choices, **{k: [v] for k, v in fixed_features.items()}}Dict **-merge preserves insertion order, not key order — any fixed parameter whose index wasn't already a key in ssd.discrete_choices (e.g. a continuous parameter that only becomes "discrete" because it was fixed) gets appended at the end instead of inserted in numeric position.
Repro case: p1 (continuous, index 0) is fixed via fixed_parameters; p2, p3, p4 (indices 1-3) are already discretized via step_size. The merged dict ends up ordered {1, 2, 3, 0} instead of {0, 1, 2, 3}. optimize_acqf_discrete_local_search treats the resulting list purely positionally, so the returned candidate's columns are actually [p2, p3, p4, p1] while validate_candidates checks it against bounds ordered [p1, p2, p3, p4] → spurious CandidateGenerationError: ... out of bounds.
Proposed fix
1. acquisition.py (minimal, targeted): index by key instead of relying on .values() order, matching the pattern already used by the neighboring optimize_acqf_discrete branch (line 788):
if optimizer == "optimize_acqf_discrete_local_search":
discrete_choices = [
- torch.tensor(c, device=self.device, dtype=self.dtype)
- for c in discrete_choices.values()
+ torch.tensor(discrete_choices[i], device=self.device, dtype=self.dtype)
+ for i in range(len(discrete_choices))
]This is safe here specifically because determine_optimizer only returns "optimize_acqf_discrete_local_search" when fully_discrete holds (len(discrete_choices) == len(ssd.feature_names)), and every key is a validated feature index in [0, len(feature_names)) — so the key set is guaranteed to be exactly {0, ..., d-1} at this point.
2. ax/generators/utils.py, mk_discrete_choices() (root-cause, defense-in-depth):
if fixed_features is not None:
discrete_choices = {
**discrete_choices,
**{k: [v] for k, v in fixed_features.items()},
}
- return discrete_choices
+ # Some callers rely on iteration order matching ascending feature index
+ # (dimension i -> value); the `**` merge above can append fixed-feature
+ # keys out of order, so restore key order here.
+ return dict(sorted(discrete_choices.items()))Sorting by key here is unconditionally safe (doesn't require the density assumption fix #1 relies on) and protects any current/future .values()-style consumer of this mapping, not just the one branch.
I'd recommend applying both: #1 fixes the crash with a minimal diff consistent with existing code style; #2 closes the underlying footgun in the shared utility.
Pull Request
None
Code of Conduct
- I agree to follow Ax's Code of Conduct
Activity
MahendraMula commentedon Sep 15, 2026
Hi @benrabe93 ! I was able to reproduce the error and am working on a fix for this. I'll open a PR shorty.
bodapatisaikrishna commentedon Sep 17, 2026
Hi — checking in on this. @MahendraMula mentioned opening a PR a couple of days ago; is that still in progress? If not, I'm happy to take the ordering fix in the local-search path plus a regression test.
Fix discrete_choices ordering with fixed_parameters (#5254) (#5293)