Smart Routing

Smart routing lets the server pick the best harness and model for a session automatically, based on the first message. Instead of committing to one model up front, the server asks a router, "given this task, which harness and model fit?", and applies the answer for the rest of the session.

Two routers are available:

Built-in judge: an LLM call using the server's own llm: block. No extra infrastructure.
External routing API: the server delegates the decision to an external routes:select service that you point it at.

Enable smart routing

Smart routing is enabled by configuration alone:

Configure a server llm: block and the server uses the built-in judge backed by it, with no routing: block required.
Or add a routing: block with provider: external to delegate to an external routing API.

For example, this enables the built-in judge with an OpenAI model:

llm:
  model: openai/gpt-5.5
  connection:
    api_key: ${OPENAI_API_KEY}

The API key may also come from the provider's usual environment variables. See built-in routing settings for custom endpoints, Databricks profiles, and fallback models.

Start the server with your config file:

omni server -c path/to/config.yaml

With a router configured, users see an Auto option in the harness picker; picking it defers harness + model selection to the router on the first message. With neither an llm: block nor an external routing: block, routing stays off and the Auto option is hidden.

Configure the external routing API

To delegate routing to an external service, add a top-level routing: block to the server config with provider: external:

routing:
  provider: external
  base_url: https://gateway.example.com/ai-gateway/routing/v1
  router_name: task_v0
  model_prefix: databricks-
  api_key: ${ROUTING_API_KEY}

See external routing settings for all fields and validation behavior, and authentication precedence for choosing between an API key and a Databricks profile.

Implement an external router

The server calls POST <base_url>/routes:select with the candidate harness and model combinations, the task, and the requested routing strategy. A typical request looks like this:

{
  "route_options": [
    { "model": "claude-opus-4-8", "harness": "claude-sdk" },
    { "model": "gpt-5-5", "harness": "codex" },
    { "model": "gpt-5-4-mini", "harness": "pi" }
  ],
  "task": { "prompt": "Refactor the auth module and add tests" },
  "route_selector": { "router_name": "task_v0" }
}

Return a response with one of the offered model and harness combinations:

{
  "route_selection": [
    {
      "route_option": { "model": "claude-opus-4-8", "harness": "claude-sdk" }
    }
  ],
  "rationale": "Multi-file refactor with tests: favor the most capable model."
}

Implement the external routing API contract for the complete schema, matching rules, and error behavior.

If the service returns an error or an invalid selection, the server surfaces the reason and falls back to the session's default harness rather than blocking it.