Skip to content

Core Components

Benchmark

Bases: BenchmarkEntity

Benchmark class represents a benchmark in the LunaBench system.

This class is responsible for managing benchmark-related operations, including creating and deleting benchmarks. It provides methods for interacting with the benchmark data and executing benchmark runs.

create(name: str) -> Benchmark staticmethod

Create a new benchmark with the given name.

The name for a benchmark must be unique. The returned Benchmark object can be used to interact and configure the new benchmark.

Parameters:

  • name (str) –

    The name of the new benchmark.

Returns:

  • Benchmark –

    The newly created Benchmark object.

open(name: str) -> Benchmark staticmethod

Load a benchmark if it exists, otherwise create a new one.

Parameters:

  • name (str) –

    The name of the benchmark.

Returns:

  • Benchmark –

    The loaded or newly created Benchmark object.

load(name: str) -> Benchmark staticmethod

Load a benchmark from the database by its name.

Parameters:

  • name (str) –

    The name of the benchmark to load.

Returns:

load_all() -> list[Benchmark] staticmethod

Load all benchmarks from the database.

Loading all benchmarks from the database can be a slow operation and should be used sparingly.

Returns:

  • list[Benchmark] –

    A list of Benchmark objects representing all benchmarks in the database. If no benchmarks are found, an empty list is returned.

reset(*, mode: ResetLevel | Literal['All', 'Unfinished', 'Failed']) -> None

Clear results for the benchmark.

Removes algorithm, metric, and feature results from the database. Metric results are cascaded (cleared along with algorithms). After the operation, the entity is reloaded from the database so its in-memory state is fully consistent.

Parameters:

  • mode (ResetLevel | Literal['All', 'Unfinished', 'Failed']) –

    ResetLevel.ALL or "All" clears all results unconditionally. ResetLevel.UNFINISHED or "Unfinished" clears only non-DONE results (includes failed). ResetLevel.FAILED or "Failed" clears only FAILED results. No default — must be explicitly provided.

set_modelset(modelset: str | ModelSet) -> None

Set the modelset for the benchmark.

This method sets the modelset for the benchmark. Changing the modelset can affect the results of the benchmark. Therfore its recommended to not change the modelset after the benchmark has been created. If it is necessary, the results of the benchmark should be deleted and the benchmark itself should be re-run.

Parameters:

  • modelset (str | ModelSet) –

    Set the modelset for the benchmark to this modelset. It can be the name of the modelset or the modelset itself.

remove_modelset() -> None

Remove the modelset from the benchmark.

This method removes the modelset from the benchmark. If the modelset is not set, this method does nothing. After removing the modelset, the results of the benchmark may be invalid.j

add_model(model: ModelSource, *, suffixes: Iterable[str] = MODEL_FILE_SUFFIXES) -> None

Add a model to this benchmark's modelset.

Convenience wrapper around ModelSet.add. If the benchmark has no modelset yet, one named after the benchmark is created and attached, so Benchmark.create("my_bench").add_model(m) puts m into the modelset "my_bench". This modelset is created even if model is an empty iterable, so add_model([]) on a benchmark with no modelset still creates and attaches an empty one named after the benchmark.

Models already in the modelset are skipped with a warning rather than duplicated, so the same script can be run again safely.

Parameters:

  • model (ModelSource) –

    The model to add. It can be

    • a Model,
    • a path (str or Path) to a model file. .lp and .mps are read as such; any other suffix is read as a model encoded with Model.encode,
    • a path to a directory, in which case every file whose suffix is in suffixes is added, in file-name order,
    • an iterable mixing any of the above, nested to any depth, in which case all of them are added.

    Models loaded from a file are named after the file stem, not after the name recorded inside the file.

  • suffixes (Iterable[str], default: MODEL_FILE_SUFFIXES ) –

    Which file suffixes a directory scan picks up. Defaults to (".lp", ".mps"). A leading dot is optional, so "mps" works. Matching is case-sensitive, as it is in Model.from_. This never filters a file named outright.

Raises:

  • FileNotFoundError –

    Raised if a given path does not exist, or is a directory holding no file with one of suffixes.

  • ValueError –

    Raised if suffixes is empty, or if a given file is neither an .lp/.mps file nor an encoded model.

  • ModelSetNotLoadedError –

    Raised if this benchmark's modelset cannot be loaded from the database.

  • ModelNameAlreadyUsedError –

    Raised if a different model already uses the same name.

remove_model(model: ModelSource, *, suffixes: Iterable[str] = MODEL_FILE_SUFFIXES) -> None

Remove a model from this benchmark's modelset.

Convenience wrapper around ModelSet.remove_model. Unlike add_model, this never creates a modelset.

Parameters:

  • model (ModelSource) –

    The model to remove. It accepts everything add_model accepts: a Model, a path to a model file, a path to a directory of such files, or an iterable of those.

  • suffixes (Iterable[str], default: MODEL_FILE_SUFFIXES ) –

    Which file suffixes a directory scan picks up, as in add_model. Pass the same value here that was passed to add_model, or the models added from a directory will not be found again.

Raises:

  • DataNotExistError –

    Raised if no modelset is configured for this benchmark.

  • FileNotFoundError –

    Raised if a given path does not exist, or is a directory holding no file with one of suffixes.

  • ValueError –

    Raised if suffixes is empty, or if a given file is neither an .lp/.mps file nor an encoded model.

  • ModelSetNotLoadedError –

    Raised if this benchmark's modelset cannot be loaded from the database.

  • RuntimeError –

    Raised if the model (or one of the models, for an iterable) is not part of the modelset.

get_feature(name: str) -> FeatureEntity

Get a feature by its name from a benchmark.

If the feature is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the feature to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the feature.

add_feature(name: str, feature: BaseFeature[Any]) -> FeatureEntity

Add a feature to the benchmark with a given name.

This method adds a feature to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the feature will be used to calculate the metrics for each algorithm result.

Also, the feature must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

  • name (str) –

    Name of the feature to add.

  • feature (BaseFeature[Any]) –

    The feature to add.

Returns:

  • Feature –

    The added feature.

remove_feature(feature: str | FeatureEntity) -> None

Remove a feature from the benchmark.

Parameters:

  • feature (str | FeatureEntity) –

    The name of the feature to remove or the feature object itself. Make sure to use the FeatureUserModel object and not only an IFeature object. This is important because the feature name is used to identify the feature.

get_metric(name: str) -> MetricEntity

Get a metric by its name from a benchmark.

If the metric is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the metric to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the metric.

add_metric(name: str, metric: BaseMetric) -> MetricEntity

Add a metric to the benchmark with a given name.

This method adds a metric to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the metric will be calculated for each algorithm result.

Also, the metric must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

  • name (str) –

    The name of the metric to add.

  • metric (BaseMetric) –

    An instance of the metric to add.

Returns:

  • Metric –

    The added metric.

remove_metric(metric: str | MetricEntity) -> None

Remove a metric from the benchmark.

Parameters:

  • metric (str | MetricEntity) –

    The name of the metric to remove or the metric object itself. Make sure to use the MetricUserModel object and not only an IMetric object. This is important because the metric name is used to identify the metric.

get_algorithm(name: str) -> AlgorithmEntity

Get an algorithm by its name from a benchmark.

If the algorithm is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the algorithm to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the feature.

add_algorithm(name: str, algorithm: IAlgorithm[Any] | BaseAlgorithmSync | BaseAlgorithmAsync[Any]) -> AlgorithmEntity

Add an algorithm to the benchmark with a given name.

This method adds an algorithm to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the results for this algorithm will be calculated.

Also, the algorithm must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

Returns:

  • AlgorithmEntity –

    The added algorithm.

remove_algorithm(algorithm: str | AlgorithmEntity) -> None

Remove an algorithm from the benchmark.

Parameters:

  • algorithm (str | AlgorithmEntity) –

    The name of the algorithm to remove or the algorithm object itself. Make sure to use the AlgorithmUserModel object and not only an IAlgorithm object. This is important because the algorithm name is used to identify the algorithm.

get_plot(name: str) -> PlotEntity

Get a plot by its name from a benchmark.

If the plot is not present, an error will be raised.

Parameters:

  • name (str) –

    The name of the algorithm to be retrieved.

Raises:

  • DataNotExistError –

    Raised if its name couldn't retrieve the plot.

add_plot(name: str, plot: BasePlot) -> PlotEntity

Add a plot to the benchmark with a given name.

This method adds a plot to the benchmark. The name must be unique within the benchmark. When the benchmark is rerun, the results for this plot will be calculated.

Also, the plot must be defined in the registry. If this isn't the case, an error will be received. To fix this, please check the documentation on how to do this.

Parameters:

  • name (str) –

    The name of the plot to add.

  • plot (BasePlot) –

    The plot to add.

Returns:

  • Plot –

    The added plot.

remove_plot(plot: str | PlotEntity) -> None

Remove a plot from the benchmark.

Parameters:

  • plot (str | Plot) –

    The name of the plot to remove or the plot object itself. Make sure to use the Plot object and not only an IPlot object. This is important because the plot name is used to identify the plot.

run_features() -> None

Calculate all configured features for all models of this benchmark.

run_algorithms() -> None

Calculate all configured features for all models of this benchmark.

run_plots() -> None

Execute all plots registered in the benchmark.

Iterates through all plots in the benchmark, validates each plot against the benchmark data, and executes the plot generation. Each plot is validated before execution to ensure required data (metrics, features, etc.) is available. Plot execution is sequential and follows the order defined in the benchmark configuration.

Raises:

  • RuntimeError –

    If plot validation or execution fails. The RuntimeError wraps the underlying error, which may be PlotRunError (for validation failures) or UnknownLunaBenchError (for unexpected execution errors). Only raised in FAIL_ON_ERROR mode; in CONTINUE_ON_ERROR mode, errors are logged as warnings instead.

Notes

In FAIL_ON_ERROR mode, the method stops at the first validation or execution error. In CONTINUE_ON_ERROR mode, errors are logged and execution continues with remaining plots.

add_dependencies() -> None

Add any required dependencies for the benchmark execution.

run(*, retry_uncompleted: bool = False) -> None

Execute the benchmark.

Parameters:

  • retry_uncompleted (bool, default: False ) –

    If True, clear only non-DONE (uncompleted) results before running so that failed or incomplete components are retried while DONE results are preserved. Defaults to False.

export(exporter: Exporter[T]) -> T

Export all benchmark results using the given exporter strategy.

Builds a full BenchmarkResultContainer (all feature, metric, and algorithm results of this benchmark) and hands it to the exporter. Any object implementing the Exporter protocol can be used, so custom export formats do not require changes to this class::

benchmark.export(CsvExporter(delimiter=";"))  # built-in exporter
benchmark.export(MyArrowExporter())  # custom exporter

Parameters:

  • exporter (Exporter[T]) –

    The exporter strategy that converts benchmark results into the target format.

Returns:

  • T –

    The exported payload, e.g. str for CSV/JSON or a pd.DataFrame for the DataFrame exporter.

to_csv(path: str | Path | None = None, *, delimiter: str = ',', quoting: CsvQuoting = 'minimal', include_solution: bool = False) -> str | None

Render all benchmark results as CSV.

Convenience wrapper for self.export(CsvExporter(...)). Mirrors pandas.DataFrame.to_csv: if path is given, the CSV is written to that file and None is returned; otherwise the CSV is returned as a string.

Parameters:

  • path (str | Path | None, default: None ) –

    File path to write the CSV to. If None, the CSV is returned as a string instead. Defaults to None.

  • delimiter (str, default: ',' ) –

    Field delimiter. Defaults to ",".

  • quoting (CsvQuoting, default: 'minimal' ) –

    Quoting style: "minimal", "all", "nonnumeric", or "none". Defaults to "minimal".

  • include_solution (bool, default: False ) –

    Whether to include the serialized solution column (base64-encoded). Defaults to False.

Returns:

  • str | None –

    The benchmark results rendered as CSV, or None if written to path.

to_json(path: str | Path | None = None, *, indent: int | None = None, orient: JsonOrient = 'records', include_solution: bool = False) -> str | None

Render all benchmark results as JSON.

Convenience wrapper for self.export(JsonExporter(...)). Mirrors pandas.DataFrame.to_json: if path is given, the JSON is written to that file and None is returned; otherwise the JSON is returned as a string.

Parameters:

  • path (str | Path | None, default: None ) –

    File path to write the JSON to. If None, the JSON is returned as a string instead. Defaults to None.

  • indent (int | None, default: None ) –

    Indentation width for pretty-printing; None for compact output. Defaults to None.

  • orient (JsonOrient, default: 'records' ) –

    JSON layout passed to DataFrame.to_json. Defaults to "records".

  • include_solution (bool, default: False ) –

    Whether to include the serialized solution column (base64-encoded). Defaults to False.

Returns:

  • str | None –

    The benchmark results rendered as JSON, or None if written to path.

plot_summary(*, columns: int | None = None, rows: int | None = None, save_dir: str | None = None, figure_filename: str = 'summary', file_formats: tuple[str, ...] | str = ('png',), show: bool = True, title: str | None = None) -> list[Path]

Draw every plot of this benchmark into a single figure, one panel each.

Convenience wrapper for luna_bench.plots.plot_summary(self, ...), which is also usable on its own. Run the benchmark first: the panels are drawn from the results it holds, and a plot that cannot be drawn is logged and left out rather than stopping the figure.

Parameters:

  • columns (int | None, default: None ) –

    Number of columns of the grid. By default derived from rows, or from the number of plots so the grid stays roughly square.

  • rows (int | None, default: None ) –

    Number of rows of the grid. By default as many as columns needs.

  • save_dir (str | None, default: None ) –

    Directory to write the figure into. Defaults to the benchmark's own plots directory, and to nowhere if the benchmark has none.

  • figure_filename (str, default: 'summary' ) –

    Stem of the written files. Defaults to "summary".

  • file_formats (tuple[str, ...] | str, default: ('png',) ) –

    Output formats written to save_dir, one file each. Defaults to ("png",).

  • show (bool, default: True ) –

    Whether to open the figure in a window. Defaults to True.

  • title (str | None, default: None ) –

    Title above the grid. Defaults to the name of the benchmark.

Returns:

  • list[Path] –

    The files that were written.

Examples:

>>> bench.run()
>>> bench.plot_summary(columns=3, show=False)

to_dataframe(*, include_solution: bool = False) -> pd.DataFrame

Return all benchmark results as a single DataFrame.

Convenience wrapper for self.export(DataFrameExporter(...)): algorithm results form the row spine (one row per (algorithm, model)), metrics merge on (algorithm, model), and features merge on model. Feature values are repeated across algorithms for the same model since features are model-level.

Parameters:

  • include_solution (bool, default: False ) –

    Whether to include the serialized solution as a solution column. Defaults to False.

Returns:

  • DataFrame –

    A DataFrame with columns algorithm, model, plus one column per result field of each feature and metric.

list_feature_classes() -> list[type[BaseFeature]]

Return the feature classes registered on this benchmark.

list_metrics_classes() -> list[type[BaseMetric]]

Return the metric classes registered on this benchmark.

list_plots_classes() -> list[type[BasePlot]]

Return the plot classes registered on this benchmark.

list_algorithms() -> list[tuple[type[BaseAlgorithmSync | BaseAlgorithmAsync[Any]], dict[str, Any]]]

Return the algorithm classes registered on this benchmark.

ModelSet

Bases: ModelSetEntity

Set of models.

Represents a collection of models with operations for creating, loading, adding, removing, and deleting models.

Attributes:

  • id (int) –

    The unique identifier for the model set.

  • name (str) –

    The name of the model set.

  • models (list[ModelMetadata]) –

    A list of ModelData objects representing the models in this set.

create(modelset_name: str) -> ModelSet staticmethod

Create a new model set with the given dataset name.

Creates a new model set using the provided dataset name and a model set creation use case.

Parameters:

  • modelset_name (str) –

    The name of the dataset.

Returns:

  • ModelSet –

    An instance of ModelSet representing the successfully created model set.

load(name: str) -> ModelSet staticmethod

Load a model set by its ID.

Retrieves a model set from the database using its unique identifier.

Parameters:

  • name (str) –

    The unique name of the model set to load.

Returns:

load_all() -> list[ModelSet] staticmethod

Load all model sets from the database.

Retrieves all model sets stored in the database.

Returns:

load_all_models() -> list[ModelMetadata] staticmethod

Load all models from the database.

Retrieves all models stored in the database, regardless of which model set they belong to.

Returns:

  • list[ModelMetadata] –

    A list of ModelData objects representing all models in the database.

add(model: ModelSource, *, suffixes: Iterable[str] = MODEL_FILE_SUFFIXES) -> None

Add a model to this model set.

Adds the specified model to this model set and updates the model set's state.

Models already present in this model set are skipped with a warning instead of being duplicated, so re-running the same script is safe.

Models are added one at a time and each one is committed on its own, so a failure part-way through an iterable leaves the earlier models added.

Parameters:

  • model (ModelSource) –

    The model to add to this model set. It can be

    • a Model,
    • a path (str or Path) to a model file. .lp and .mps are read as such; any other suffix is read as a model encoded with Model.encode,
    • a path to a directory, in which case every file whose suffix is in suffixes is added, in file-name order,
    • an iterable mixing any of the above, nested to any depth, in which case all of them are added.

    Models loaded from a file are named after the file stem, not after the name recorded inside the file.

  • suffixes (Iterable[str], default: MODEL_FILE_SUFFIXES ) –

    Which file suffixes a directory scan picks up. Defaults to (".lp", ".mps"). A leading dot is optional, so "mps" works. Matching is case-sensitive, as it is in Model.from_. This never filters a file named outright.

Raises:

  • FileNotFoundError –

    Raised if a given path does not exist, or is a directory holding no file with one of suffixes.

  • ValueError –

    Raised if suffixes is empty, or if a given file is neither an .lp/.mps file nor an encoded model.

  • ModelNameAlreadyUsedError –

    Raised if a different model already uses the same name.

remove_model(model: ModelSource, *, suffixes: Iterable[str] = MODEL_FILE_SUFFIXES) -> None

Remove a model from this model set.

Removes the specified model from this model set and updates the model set's state.

Parameters:

  • model (ModelSource) –

    The model to remove from this model set. It accepts everything add accepts: a Model, a path to a model file, a path to a directory of such files, or an iterable of those.

    Models are removed one at a time and each removal is committed on its own. If one of them fails, the earlier ones stay removed and the raised RuntimeError names them.

  • suffixes (Iterable[str], default: MODEL_FILE_SUFFIXES ) –

    Which file suffixes a directory scan picks up, as in add. Pass the same value here that was passed to add, or the models added from a directory will not be found again.

Raises:

  • FileNotFoundError –

    Raised if a given path does not exist, or is a directory holding no file with one of suffixes.

  • ValueError –

    Raised if suffixes is empty, or if a given file is neither an .lp/.mps file nor an encoded model.

delete() -> None

Delete this model set from the database.

Permanently removes this model set from the database.