Architecture

Luna Bench follows a layered architecture that separates the public API from internal implementation details. This page describes the key architectural decisions, the registration system, the pipeline execution model, and the persistence layer.

Package Structure

The codebase is organized into two distinct layers: a public API layer that users interact with directly, and an internal layer that contains implementation details.

graph TB
    subgraph "Public API Layer"
        core["Benchmark, ModelSet\n(top-level)"]
        builtins["algorithms/ features/ metrics/ plots/\nBuilt-in components"]
        custom["custom/\nDecorators, base classes, result types"]
        entities["entities/\nData transfer objects"]
    end

    subgraph "Internal Layer (_internal/)"
        usecases["usecases/\nBusiness logic orchestration"]
        mappers["mappers/\nEntity-to-domain and domain-to-entity conversion"]
        registries["registries/\nThread-safe component registries"]
        dao["dao/\nData access objects (Peewee ORM)"]
        domain_models["domain_models/\nInternal domain representations"]
        wrappers["wrappers/\nAdapter wrappers for external libraries"]
    end

    core --> usecases
    builtins --> custom
    custom --> registries
    usecases --> mappers
    usecases --> dao
    usecases --> registries
    mappers --> domain_models
    dao --> domain_models

Public API Layer

Internal Layer

Registration System

Luna Bench uses a decorator-based registration system backed by metaclasses. When you define a custom component, you decorate it with one of the provided decorators, and it is automatically registered in a thread-safe registry.

sequenceDiagram
    participant User as User Code
    participant Dec as @feature / @algorithm / ...
    participant Meta as RegisteredClassMeta / MetricClassMeta
    participant Reg as Registry

    User->>Dec: Decorate class with @feature
    Dec->>Meta: Metaclass __init_subclass__ triggers
    Meta->>Reg: Register class in thread-safe registry
    Note over Reg: Class is now discoverable<br/>via RegistryInfo
    User->>Reg: RegistryInfo.log_registered_features()
    Reg-->>User: Returns registered feature IDs

Decorators

The four registration decorators are:

• @feature: Registers a BaseFeature subclass
• @algorithm: Registers a BaseAlgorithmSync or BaseAlgorithmAsync subclass
• @metric: Registers a BaseMetric subclass
• @plot: Registers a BasePlot subclass

Metaclasses

Two metaclasses power the registration:

• RegisteredClassMeta: Used by features, algorithms, and plots. Intercepts class creation to insert the class into the appropriate registry.
• MetricClassMeta: A specialized variant for metrics that handles additional metric-specific metadata.

Inspecting the registries

RegistryInfo (in luna_bench.custom) reports what is currently registered. Each method logs the IDs and returns them as a list:

Pipeline Execution

The benchmark runs a four-stage pipeline. Each stage feeds its results into the next. The stages always execute in order: Features, Algorithms, Metrics, Plots.

flowchart TD
    Start([bench.run]) --> FeaturesStage

    subgraph FeaturesStage ["Stage 1: Features"]
        direction LR
        F1[For each model in ModelSet]
        F2[Run all registered features]
        F3[Store FeatureResult per model]
        F1 --> F2 --> F3
    end

    FeaturesStage --> AlgorithmsStage

    subgraph AlgorithmsStage ["Stage 2: Algorithms"]
        direction LR
        A1["Cartesian product:\nalgorithm x model"]
        A2[Run algorithm on model]
        A3[Store AlgorithmResult per pair]
        A1 --> A2 --> A3
    end

    AlgorithmsStage --> MetricsStage

    subgraph MetricsStage ["Stage 3: Metrics"]
        direction LR
        M1["For each (algorithm, model) pair"]
        M2[Evaluate all registered metrics]
        M3[Store MetricResult per triple]
        M1 --> M2 --> M3
    end

    MetricsStage --> PlotsStage

    subgraph PlotsStage ["Stage 4: Plots"]
        direction LR
        P1[Collect all feature and metric data]
        P2[Run all registered plots]
        P3[Generate and store visualizations]
        P1 --> P2 --> P3
    end

    PlotsStage --> End([Pipeline complete])

Stage Details

Stage 1: Features. Feature extractors run once per model in the model set. They compute properties such as variable count, constraint count, or problem structure. Feature results are persisted and available to later stages.

Stage 2: Algorithms. Every registered algorithm runs on every model (cartesian product). Synchronous algorithms (BaseAlgorithmSync) execute in-process. Asynchronous algorithms (BaseAlgorithmAsync) submit jobs to a background task queue and poll for results.

Stage 3: Metrics. For each (algorithm, model) result pair, all registered metrics are evaluated. Metrics compute values like approximation ratio, runtime, feasibility rate, or time-to-solution.

Stage 4: Plots. Plot generators receive the full set of feature and metric results. They produce visualizations (e.g., comparison charts, scaling plots) and store them alongside the benchmark data.

Dependency Injection

Luna Bench uses a container-based dependency injection system. Internal components receive their dependencies through containers rather than constructing them directly. This makes the system testable and allows swapping implementations.

The framework defines the following containers:

Dependencies are wired using the @inject decorator, which resolves dependencies from the appropriate container at call time.

Database

Luna Bench persists all benchmark configurations and results in SQLite databases managed through the Peewee ORM. Two separate database files are used:

Schema Overview

The main database contains the following tables:

• Benchmark: Benchmark configuration and metadata
• ModelSet: Named collections of models
• Model: Individual optimization models belonging to a model set
• Algorithm: Registered algorithm instances and their configurations
• Feature: Registered feature instances
• Metric: Registered metric instances
• Plot: Registered plot instances
• Result tables: Store outputs from each pipeline stage (feature results, algorithm results, metric results, plot artifacts)

Async Algorithm Support

For algorithms that call cloud services or quantum hardware, Luna Bench provides BaseAlgorithmAsync. Async algorithms submit work to a Huey-based job queue rather than blocking the main thread.

Key characteristics:

• Subclass BaseAlgorithmAsync instead of BaseAlgorithmSync to define an async algorithm.
• Jobs are submitted to a background task queue backed by Huey and the luna_bench-jobs.db SQLite database.
• A configurable pool of workers (default: 10) processes jobs concurrently.
• The pipeline automatically waits for all async jobs to complete before proceeding to the Metrics stage.
• Worker count can be adjusted based on API rate limits or hardware constraints.