Luna Bench

Luna Bench is a benchmarking framework for comparing optimization algorithms across quantum, hybrid, and classical domains. Built on top of Luna Quantum, it provides a standardized, reproducible, and extensible pipeline for evaluating algorithm performance.

Key Features

• Reproducible Benchmarks: All benchmark configurations, model sets, and results are persisted in a SQLite database, ensuring full reproducibility across runs.
• Standardized Evaluation: Compare algorithms side-by-side using built-in metrics like approximation ratio, runtime, feasibility, and time-to-solution.
• Extensible Components: Register custom algorithms, features, metrics, and plots using a simple decorator-based system.
• Automated Pipeline: Run the full benchmarking workflow (features, algorithms, metrics, plots) with a single method call.
• Quantum and Classical: Support for both synchronous solvers (e.g., SCIP) and asynchronous cloud/quantum backends.

How It Works

Luna Bench operates as a four-stage pipeline that executes in order:

graph LR
    A[Features] --> B[Algorithms]
    B --> C[Metrics]
    C --> D[Plots]

1. Features: Extract model-level properties (e.g., variable count, problem size, constraint structure).
2. Algorithms: Run each algorithm on every model in the model set (cartesian product).
3. Metrics: Evaluate each algorithm's solutions using configured metrics.
4. Plots: Generate visualizations from the collected metrics and features data.

Quick Example

from luna_quantum import Model, Variable, Vtype, Bounds, Unbounded
from luna_bench import Benchmark, ModelSet
from luna_bench.algorithms import ScipAlgorithm
from luna_bench.features import VarNumberFeature
from luna_bench.metrics import Runtime, ApproximationRatio

# Build a model and add it to a model set
model = Model("problem1")
with model.environment:
    x = Variable("x", vtype=Vtype.REAL, bounds=Bounds(lower=0, upper=Unbounded))
    y = Variable("y", vtype=Vtype.REAL, bounds=Bounds(lower=0, upper=Unbounded))
model.objective = 3 * x + 2 * y
model.constraints += x + y <= 10

model_set = ModelSet.create("my-models")
model_set.add(model)

# Create and configure a benchmark
bench = Benchmark.create("my-benchmark")
bench.set_modelset(model_set)

# Add components
bench.add_algorithm("scip", ScipAlgorithm())
bench.add_feature("num-vars", VarNumberFeature())
bench.add_metric("runtime", Runtime())
bench.add_metric("approx-ratio", ApproximationRatio())

# Run the full pipeline
bench.run()

Installation

Luna Bench requires Python 3.13+. Install with uv or pip:

uv add luna-bench

pip install luna-bench

What's Next