Building OpenFusion: How I Built a Local Multi-Model AI Fusion Engine

If you ask three different senior engineers how to solve a complex systems problem, you will not get one answer. You will get three distinct perspectives, each highlighting trade-offs the others missed. One notices an edge case in memory consumption; another flags a database lock contention; the third spots an operational security hole.

When you put those three perspectives in a room and have a facilitator extract the consensus, cross-check the contradictions, and synthesize a single plan, the final decision is almost always superior to what any single person proposed alone.

Earlier this year, OpenRouter published benchmark data proving the exact same dynamic holds true for artificial intelligence. By fanning out a prompt to a panel of diverse models and having a judge model synthesize their responses, a panel of budget models outperformed solo frontier models on deep research and complex reasoning tasks—landing within striking distance of the best models on earth at a fraction of the cost.

The catch? You had to route everything through OpenRouter's cloud infrastructure.

I wanted that exact council-of-models superpower running directly on my own machine, accessible to any coding agent I use (Claude Code, Cursor, Cline, Codex, Zed), compatible with any model provider I choose, and completely private. So I built OpenFusion.

Here is how the architecture works, why it is designed this way, and the hard engineering problems I had to solve to make multi-model fusion work reliably across desktop agent tools.

The core premise: a fusion engine, not an agent

The first architectural decision I locked in was defining what OpenFusion is not.

OpenFusion is not an autonomous agent. It does not browse the web, execute terminal commands, or edit project files. Autonomous agents are great at doing legwork, but embedding tool execution inside every candidate model during a multi-model debate would cause cost, latency, and token consumption to explode exponentially.

Instead, OpenFusion is a pure fusion engine. Your primary coding agent does the investigative legwork first—gathering files, reproduction steps, and constraints into a concise dossier. It then hands that dossier to OpenFusion via a single tool call. OpenFusion runs the debate and hands back a battle-tested synthesis.

~75% of the performance lift in model fusion comes from the synthesis step itself, not just raw candidate diversity. Splitting analysis from writing is what makes the output reliably superior.

The two-step judge: why one pass is never enough

When people try building multi-model pipelines, the rookie mistake is asking a judge model to "read these answers and write the best combined response."

In practice, single-pass judging fails. The judge model suffers from position bias (favoring the first or longest response), gets overwhelmed by subtle contradictions, and often ends up copying one candidate while ignoring crucial edge cases found by the others.

To fix this, OpenFusion enforces a strict two-step judging process on the same provider and model:

Step 1: Structured analysis via forced function calling

The judge is first invoked with a mandatory function call: record_analysis. The model is explicitly forbidden from answering the user's prompt directly. Instead, it acts as an impartial analyst extracting five structured dimensions:

• Consensus: Points where candidate models substantially agree.
• Contradictions: Points where candidate models disagree or provide conflicting solutions.
• Partial coverage: Aspects of the problem that only a subset of models addressed.
• Unique insights: High-value observations or edge cases caught by only one model.
• Blind spots: Critical considerations that none of the candidates handled adequately.

Step 2: Dedicated synthesis

Once the engine receives this structured critique, it feeds both the raw candidate answers and the structured analysis into the second judge invocation. Because the hard analytical work of identifying flaws and agreements has already been done, the judge model can focus 100% of its attention on producing a clear, comprehensive, and cohesive answer.

How the engine runs: single process, dual interface

Running local developer tooling requires zero friction. I did not want users managing background daemon services, configuring reverse proxies, or juggling multiple terminal windows.

OpenFusion runs as a single Node.js process exposing two simultaneous communication channels:

• Standard I/O (stdio) for MCP: Coding agents communicate with OpenFusion over standard input/output using the Model Context Protocol JSON-RPC standard.
• Local HTTP Server (127.0.0.1:9077): In the exact same process, an Express server hosts a fast REST API and serves a glass-morphic React dashboard and interactive Playground.
• Standalone Mode (openfusion-ui): Because stdio servers shut down when the parent agent exits, OpenFusion includes a standalone binary that starts the dashboard independently, reading and writing to the same shared database.

Breaking through the 60-second MCP client timeout wall

This was the hardest real-world problem I encountered while building OpenFusion.

In the MCP specification, tool calls are synchronous request-response round-trips. However, popular agent clients (like OpenAI Codex or ZCode) enforce rigid client-side timeouts—often hardcoded to 60 seconds.

If you fan out a prompt to four models and run a two-step judge, cloud API latency might total 75 seconds. In sequential mode on local hardware, it can take several minutes. When the client's 60-second timer expires, the agent forcibly aborts the connection—discarding the computed answer even though the server completed the work successfully.

To solve this across all clients without requiring complex client-side protocol upgrades, I designed a deferred retrieval protocol:

• Immediate Kickoff Return: When a fusion starts, OpenFusion detaches the execution into a background worker and returns in under one second with a lightweight ticket containing a reference ID and an explicit instruction.
• Bounded Long-Polling: When the agent follows the instruction and calls fusion({ _resume_from: "<id>" }), OpenFusion holds the connection open for up to 40 seconds (safely below the 60-second ceiling). If the computation finishes in that window, it returns the final answer immediately.
• Crash-Resilient State: Every job's lifecycle is written to a local SQLite database (~/.openfusion/openfusion.db). If an agent restarts or drops connection mid-flight, the work is never lost.

Supporting both cloud and local Apple Silicon models

Different developers have different hardware constraints. A developer on a laptop running on battery power might want cloud models; a developer working on proprietary code might want strictly local execution.

OpenFusion supports both through two specialized execution modes:

For local inference, OpenFusion includes built-in adapters for rapid-mlx (Apple Silicon unified memory execution) and ollama-cloud. These endpoints require no API keys and automatically query the provider's /v1/models route to discover whatever local models you have downloaded.

Secure key management and transparent accounting

When building developer tools, security cannot be an afterthought:

• Encrypted local credentials: API keys are never stored in plaintext JSON. They are encrypted using AES-256-GCM in ~/.openfusion/secrets.enc, keyed by a machine-bound master.key file with strict operating-system permissions (chmod 600).
• Granular token accounting: Every single fusion logs its exact token counts and costs to SQLite. You can open the dashboard at any moment to see the exact latency, token consumption, and cost breakdown per model across both candidates and the judge.

What I learned shipping OpenFusion

1. Prompt synthesis is a distinct discipline. Simply merging text produces disjointed summaries. You have to force the judge model to identify disagreements and resolve them explicitly before drafting.
2. Never trust client-side timeouts in agent protocols. Building for the real world means assuming the agent client will cut your connection after 60 seconds. Designing asynchronous, durable polling mechanisms from day one saves weeks of debugging.
3. Local-first tooling creates trust. Developers love multi-model intelligence, but they hate sending their entire codebase to a single proprietary intermediary. Giving them complete ownership of their keys, database, and local models creates true confidence.

OpenFusion is fully open source. If you want to give your coding agents access to an on-demand council of models, check out the GitHub repository.

If you are looking to design resilient, multi-model AI architectures or custom agent workflows for your team, book a discovery consultation—let's build an automation pipeline that delivers consistent, verified results.