Skip to content

Getting Started

This guide walks you through setting up your first benchmark with Luna Bench, from installation to running a full evaluation pipeline.

Installation

Luna Bench requires Python 3.13 or later.

uv add luna-bench

poetry add luna-bench

pip install luna-bench

Note

Luna Bench depends on Luna Model for model handling. It will be installed automatically as a dependency.

Creating a Model Set

A ModelSet is a named collection of optimization models that your benchmark will evaluate. Models are created using the Model class from Luna Quantum.

Python
from luna_quantum import Model, Variable, Vtype, Bounds, Unbounded
from luna_bench import ModelSet

# Build a model: maximize 3x + 2y subject to a couple of constraints
model = Model("problem1")
with model.environment:
    x = Variable("x", vtype=Vtype.REAL, bounds=Bounds(lower=0, upper=Unbounded))
    y = Variable("y", vtype=Vtype.REAL, bounds=Bounds(lower=0, upper=Unbounded))
model.objective = 3 * x + 2 * y
model.constraints += x + y <= 10
model.constraints += 2 * x + y >= 5

# Create a model set and add the model
model_set = ModelSet.create("my-models")
model_set.add(model)

Persistence

Model sets are automatically persisted in a local SQLite database. You can reload them in any future session using ModelSet.load("my-models") without needing access to the original files.

Setting Up a Benchmark

A Benchmark ties together a model set, one or more algorithms, features, metrics, and plots into a single evaluation pipeline.

Create the benchmark and assign a model set

Python
from luna_bench import Benchmark

bench = Benchmark.create("my-benchmark")
bench.set_modelset(model_set)

Add algorithms

Algorithms are the solvers you want to compare. Each algorithm is registered with a unique name. Here we compare two QAOA configurations with different circuit depths (reps controls the number of QAOA layers).

Python
from luna_quantum.solve.parameters.algorithms import QAOA

bench.add_algorithm("qaoa-2", QAOA(reps=2))
bench.add_algorithm("qaoa-4", QAOA(reps=4))

Add features

Features extract properties from each model before the algorithms run. These properties can be used later in plots and analysis.

Python
from luna_bench.features import VarNumberFeature, OptSolFeature

bench.add_feature("num-vars", VarNumberFeature())
bench.add_feature("optimal-solution", OptSolFeature())

Add metrics

Metrics evaluate the quality of each algorithm's results.

Python
from luna_bench.metrics import (
    Runtime,
    ApproximationRatio,
    FeasibilityRatio,
    BestSolutionFound,
    TimeToSolution,
    FractionOfOverallBestSolution,
)

bench.add_metric("runtime", Runtime())
bench.add_metric("approx-ratio", ApproximationRatio())
bench.add_metric("feasibility", FeasibilityRatio())
bench.add_metric("best-solution", BestSolutionFound())
bench.add_metric("time-to-solution", TimeToSolution())
bench.add_metric("fraction-best", FractionOfOverallBestSolution())

Add plots (optional)

Plots generate visualizations from the collected metrics and features data.

Python
from luna_bench.plots import AverageRuntimePlot

bench.add_plot("avg-runtime", AverageRuntimePlot())

Running the Pipeline

Luna Bench executes a four-stage pipeline in order:

graph LR
    A[Features] --> B[Algorithms]
    B --> C[Metrics]
    C --> D[Plots]

Run the full pipeline

The simplest way to run a benchmark is to execute all stages at once:

Python
bench.run()

Run individual stages

If you need finer control, you can run each stage independently:

Python
# Stage 1: Extract features from all models
bench.run_features()

# Stage 2: Execute all algorithms on all models
bench.run_algorithms()

# Stage 3: Compute metrics from algorithm results
bench.run_metrics()

# Stage 4: Generate plots
bench.run_plots()

Tip

Running individual stages is useful when you want to add new metrics or plots to an existing benchmark without re-running the algorithms, which is typically the most time-consuming stage.

Loading Previous Results

All benchmark data: configurations, model sets, algorithm results, and metrics: is automatically persisted in SQLite databases. You can reload any previous benchmark or model set by name.

Python
# Load an existing model set
model_set = ModelSet.load("my-models")

# Load an existing benchmark (with all its results)
bench = Benchmark.load("my-benchmark")

# Or open it: load if it exists, otherwise create a fresh one
bench = Benchmark.open("my-benchmark")

This makes it straightforward to:

  • Resume an interrupted benchmark run.
  • Add new algorithms or metrics to a previously completed benchmark.
  • Share benchmark results across team members by distributing the database file.

Complete Example

Here is a full working example that ties everything together:

Python
from luna_quantum import Model, Variable, Vtype, Bounds, Unbounded
from luna_quantum.solve.parameters.algorithms import QAOA
from luna_bench import Benchmark, ModelSet
from luna_bench.features import VarNumberFeature, OptSolFeature
from luna_bench.metrics import Runtime, ApproximationRatio, FeasibilityRatio
from luna_bench.plots import AverageRuntimePlot

# --- Model ---
model = Model("problem1")
with model.environment:
    x = Variable("x", vtype=Vtype.REAL, bounds=Bounds(lower=0, upper=Unbounded))
    y = Variable("y", vtype=Vtype.REAL, bounds=Bounds(lower=0, upper=Unbounded))
model.objective = 3 * x + 2 * y
model.constraints += x + y <= 10
model.constraints += 2 * x + y >= 5

# --- Model Set ---
model_set = ModelSet.create("getting-started-models")
model_set.add(model)

# --- Benchmark ---
bench = Benchmark.create("getting-started-benchmark")
bench.set_modelset(model_set)

# Algorithms: compare two QAOA configurations with different circuit depths
bench.add_algorithm("qaoa-2", QAOA(reps=2))
bench.add_algorithm("qaoa-4", QAOA(reps=4))

# Features
bench.add_feature("num-vars", VarNumberFeature())
bench.add_feature("optimal-solution", OptSolFeature())

# Metrics
bench.add_metric("runtime", Runtime())
bench.add_metric("approx-ratio", ApproximationRatio())
bench.add_metric("feasibility", FeasibilityRatio())

# Plots
bench.add_plot("avg-runtime", AverageRuntimePlot())

# Run the full pipeline: features -> algorithms -> metrics -> plots
bench.run()

Next Steps

  • Architecture

    Understand the layered design and pipeline execution model.

    Architecture

  • Model Sets

    Learn how to create, manage, and share model sets.

    Model Sets

  • Algorithms

    Explore built-in solvers and learn how to register custom algorithms.

    Algorithms

  • Metrics

    Dive into the available metrics and how to create your own.

    Metrics

  • Features

    Extract and use model properties in your analysis.

    Features

  • Plots

    Visualize benchmark results with built-in and custom plots.

    Plots