Built-in Metrics

Bases: BaseMetric[RuntimeResult]

Metric that captures the total runtime of a solution computation.

This metric provides a simple way to track and compare the computational time required by different algorithms to produce their solutions.

Examples:

>>> from luna_bench.metrics import Runtime
>>> metric = Runtime()
>>> result = metric.run(solution, feature_results)
>>> print(f"Runtime: {result.runtime_seconds} seconds")

Bases: BaseMetric[ApproximationRatioResult]

Metric that calculates the approximation ratio of a solution against the optimal.

The approximation ratio measures how close a found expectation value of a given solution is to the optimal solution. It uses the expectation value (average energy) of all samples in the solution rather than just the best sample. A value of 1.0 indicates an optimal solution, while values > 1.0 indicate progressively worse solution quality.

For minimization problems: AR = expectation_value / optimal_value

For maximization problems: AR = optimal_value / expectation_value

Attributes:

• abt_diff (float) –

  Absolute tolerance for considering a value as zero. Used to prevent division by zero errors. Default is 1e-3.

• value_source (ValueSource) –

  The value_source defines if the objective values or the raw energies values form the algorithm should be used to calulate the approximation ratio. Default is ValueSource.OBJ.

Examples:

>>> from luna_bench.metrics import ApproximationRatio
>>> metric = ApproximationRatio()
>>> result = metric.run(solution, feature_results)
>>> print(f"Approximation Ratio: {result.approximation_ratio}")

Bases: BaseMetric[FeasibilityRatioResult]

Metric that calculates the ratio of feasible solutions.

The Feasibility Ratio measures what proportion of the samples returned by a solver satisfy all constraints of the optimization problem. This is particularly important for constrained optimization problems where not all samples may be valid solutions.

A feasibility ratio of 1.0 indicates all samples are feasible, while 0.0 indicates none are feasible.

Examples:

>>> from luna_bench.metrics import FeasibilityRatio
>>> metric = FeasibilityRatio()
>>> result = metric.run(solution, feature_results)
>>> print(f"Feasibility Ratio: {result.feasibility_ratio}")

Bases: BaseMetric[BestSolutionFoundResult]

Metric that extracts the Best Solution Found (BSF).

The Best Solution Found metric extracts the best feasible objective value from the solution set. The best value is the lowest objective value for minimization problems and the highest for maximization problems, considering only feasible samples.

Unlike :class:BestSolutionFoundRatio, this metric reports the raw best feasible value and does not require an optimal solution for comparison.

Attributes:

• value_source (ValueSource) –

  Defines whether the objective values or the raw energy values from the algorithm should be used to determine the best solution. Default is ValueSource.OBJ.

Examples:

>>> from luna_bench.metrics import BestSolutionFound
>>> metric = BestSolutionFound()
>>> result = metric.run(solution, feature_results)
>>> print(f"Best Solution Found: {result.best_solution_found}")

Bases: BaseMetric[TimeToSolutionResult]

Metric that calculates the Time-to-Solution (TTS) for finding optimal solutions.

The Time-to-Solution metric measures how long it takes for a solver to find the optimal solution with a high probability (99%). This metric is particularly useful for comparing heuristic solvers and quantum optimization algorithms.

For heuristic solvers, TTS is calculated based on the probability of finding the optimal solution within a set of samples:

.. math::

\text{TTS}(X) = \left( \frac{t_{\text{solve}}}{M} \right)
\left( \lceil \frac{\log(1 - 0.99)}{\log(1 - p^*)} \rceil \right)

Where: - :math:t_{\text{solve}} is the total time taken to solve the problem, - :math:M is the number of samples, - :math:p^* is the probability of measuring the optimal solution.

Special cases: - If :math:p^* = 0 (no optimal solutions found): TTS = infinity - If :math:p^* = 1 (all solutions optimal): TTS = time per sample

Attributes:

• target_probability (float) –

  The target probability of finding the optimal solution. Default is 0.99.

• abs_tol (float) –

  Absolute tolerance for comparing objective values. Default is 1e-6.

Examples:

>>> from luna_bench.metrics import TimeToSolution
>>> metric = TimeToSolution()
>>> result = metric.run(solution, feature_results)
>>> print(f"TTS: {result.time_to_solution} seconds")
>>> print(f"Probability of optimal: {result.probability_optimal}")

Bases: BaseMetric[FractionOfOverallBestSolutionResult]

Metric that calculates the Fraction of Overall Best Solution (FOB).

This metric evaluates how often a solver finds the overall best solution. It calculates the fraction of samples where the solver's solution matches the best known solution.

The Fraction of Overall Best solution found (FOB) is defined as:

.. math::

\text{FOB}({X_i}) = \frac{|\{i \mid \hat{c}(X_i) = 1, \forall i \in I\}|}{|I|}

Where: - :math:X_i represents the sample set obtained for instance :math:i, - :math:\hat{c}(X_i) = 1 indicates that the solution for instance :math:i is the best solution found for that instance, - :math:I is the set of all instances.

This metric is beneficial for comparing solvers based on their ability to find the best solution for each instance. It provides insight into how effective a solver is at consistently finding the optimal solution.

Attributes:

• abs_tol (float) –

  Absolute tolerance for considering two values as equal. Default is 1e-6.

Examples:

>>> from luna_bench.metrics import (
...     FractionOfOverallBestSolution,
... )
>>> metric = FractionOfOverallBestSolution()
>>> result = metric.run(solution, feature_results)
>>> print(f"FOB: {result.fraction_of_overall_best_solution}")