I put the same agent to fix real bugs with three local engines: 284B, 27B and 27B in two bits

Every piece of coverage of the DeepSeek Harness I found describes it with DeepSeek’s official API. None runs it with a local model. And that is exactly the question I care about, because I have a 16 GB card and a swarm of machines where things get tested for real.

So I wired it to three very different engines and gave them the same job: fixing real bugs in real projects, verified by the projects’ own tests.

What the harness is and why it matters

An agent harness is the scaffolding around the model: the loop that decides when to call it, which tools to offer it, how to log the session and when to stop. DeepSeek released theirs under the MIT license, with a clear design idea: everything is a plugin, including the model adapter, the tool registry and the agent loop itself.

That last part is what makes this article possible. If the adapter is replaceable, so is the model.

I installed it from the repository in 28 seconds and it compiled in 147. It has 2,587 TypeScript and JavaScript files, so it is not exactly small.

And here comes the first finding: it ships no adapter for local models. Out of the box there are two, DeepSeek’s and their gateway’s. But its “custom provider” path speaks the OpenAI protocol, which is exactly what ollama, llama.cpp and Colibrì expose. There is not a line of code to write, only a declaration:

llm-pi-ai:
  providers:
    local:
      api: openai-completions
      baseURL: http://127.0.0.1:8090/v1
      apiKeyEnv: API_KEY
      models:
        - id: my-model

agent-default-model:
  provider: local
  model: my-model

How I measured

I used SWE-bench Verified, which is the standard the labs report: 500 real bugs pulled from large repositories, each with its environment prepared and its tests.

The protocol, which matters more than the result:

The agent sees only the text of the bug report. Never the reference patch, never the tests it is evaluated with. It works on the repository at the commit before the fix. Afterwards I extract the diff it left, apply the official test patch and run the tests that must pass. Solved means they pass, not that the agent says it managed.

I picked three instances from the easiest category, from different repositories: astropy, django and matplotlib. Three is not a statistical measurement and the easy ones are the favourable scenario. It is a thermometer, not a ranking, and I say so before giving a single number.

Engine 1: Qwen3.8-27B, the one that does not fit

I started with the model I already measured as not fitting in 16 GB: a 17 GB file against a 16 GB card, with a third of the model on the CPU.

Before anything worked, I hit the most instructive failure of the day.

The agent was answering about a Three.js scene with a rotating cube, something with no relation to the task. It looked like a model hallucinating out of control. The real cause was in the server log:

truncating input prompt  limit=2050  prompt=8195  keep=4  new=2050

ollama was silently cutting the 8,195-token prompt down to 2,050, keeping four tokens from the start. It discarded 75%, the task included. The model was answering coherently to the little that reached it, and the warning only appears in a log nobody reads.

Raising the window to 32,768, the same tasks ran correctly. There is a good part of the explanation for why so many people conclude that “local models are no good for agents”.

With that sorted, the thermometer:

Two out of three, in two and a half hours.

The matplotlib one deserves a pause: the agent found the same line the project’s maintainers did and applied the same fix with different syntax. Where the official fix writes hist_kwargs['density'] = density, it wrote hist_kwargs.update(density=density). Semantically identical, and it never saw the patch.

Astropy did not fail through incapacity. It ran out of time: its last output was “The checkout imports now. Reproduce the bug:”, which is the right path. Thirty model calls in an hour, with the GPU at 15% because it spent most of the time waiting on the CPU.

Engine 2: DeepSeek V4 Flash, 284 billion parameters from disk

The second engine is the most spectacular and gives the worst result.

Colibrì is an inference engine in pure C, with no dependencies, that treats disk, memory and card as a single hierarchy. Its promise is running enormous models on ordinary hardware by streaming the experts from disk instead of loading them.

I compiled the engine in 7.3 seconds and pulled down DeepSeek V4 Flash: 284 billion parameters, 167 GB on disk, 48 shards. The documentation warns that a download can end with a truncated shard and still report success, so I compared the size of all 74 files against the repository before touching anything. All 74 exact.

And it works. I asked it what 17 times 23 is and it answered correctly, with these numbers:

Six gigabytes of memory for a 167 GB model. It does not load it into memory: it streams it from disk. And that 61% cache hit rate is the only thing keeping it from being unusable.

But as an agent’s engine, the verdict is clear: 0.27 tokens per second. A trivial 16-token call took 60 seconds. A second, slightly larger call died with a server 500. And the server handles one request at a time: no parallelism is possible, every reasoning step waits on the previous one.

With the astropy instance needing thirty calls, this is days, not hours. I did not take it to SWE-bench because the outcome was already decided.

A technical note: this engine does not use the card. Its source file has zero hooks into CUDA or Vulkan, while the project’s main engine has 112 and 42. So the 35 seconds and the 30 GB read were achieved with processor and disk alone.

Engine 3: Bonsai-27B in two bits, the one that does fit

The third is the direct answer to the first one’s problem.

Bonsai is a derivative of Qwen3.6-27B with ternary quantization: the same model size compressed to 7.06 GB, nine times less than at full precision. Its authors claim it keeps 95% of the intelligence and, in their own words, agentic behaviour below four bits.

That is exactly what can be verified.

I compiled llama.cpp with Vulkan and found the card’s ceiling by measurement, raising the window until it blew up:

The real ceiling is 131,072, but with 492 MiB of headroom it is too tight to leave fixed. I settled at 65,536, which is already four times what Qwen3.8 could support, and with the whole model inside the card.

And here I hit the day’s second silent failure, sibling to the first.

All three instances were finishing in under a minute with an empty diff. At first glance it looked like prodigious speed. It was the agent crashing:

CONTEXT_WINDOW_EXCEEDED: request (18180 tokens)
exceeds the available context size (16384 tokens)

In llama.cpp, the context is the server’s total and gets split across the slots for parallel attention. Declaring 65,536 with four slots does not give one window of 65,536: it gives four of 16,384. For serving several users that makes sense; for an agent, which is a single conversation accumulating context, it is the opposite of what you need. With a single slot, the request gets the full window.

If I had kept the timings without looking at the diffs, I would have published that Bonsai is twenty times faster. And it would have been false.

1. matplotlib-139892,101 sSolved with a 13-line patch.
2. django-132973,377 sSolved with 57 lines, the longest of the three.
3. astropy-143093,581 sRan out of budget without writing a single line of diff.

Bonsai against the same three instances. The per-instance budget was one hour.

Now with the correct window:

Two out of three, in thirteen minutes.

The comparison that matters

Both models are the same size and from the same family. One single variable changes: whether they fit entirely on the card.

They tie on fixes. Bonsai is eleven times faster.

1. Bonsai-27B ternary52.3 tok/sFits entirely on the card, with a 65,536 window and headroom to spare.
2. Qwen3.8-27B in 4 bits28.9 tok/sA third of the layers stay on the CPU: the file weighs 17 GB and the card has 16.
3. DeepSeek V4 Flash, 284B from disk0.27 tok/sTechnically beautiful, with 6.27 GB resident for 167 on disk. For an agent chaining dozens of calls, unusable.

Measured on an RTX 4070 Ti SUPER. The DeepSeek V4 Flash bar is at the minimum drawable size: its real value is 194 times smaller than Bonsai's and at exact scale it would be invisible.

And the most interesting detail: they solve different instances. Bonsai fixed astropy, where Qwen ran out of time. Qwen fixed django, where Bonsai chose a defensive patch, well reasoned but not the one the test expected: it understood that lazy objects break when used in queries and decided to unwrap them, instead of removing the whole deprecation mechanism as Qwen did.

It is not that one is better. They have different profiles.

My take

What I take from all of this is not a ranking, it is three things.

The first: for local agents, the model fitting entirely on the card weighs more than its numeric precision. A 27B at two bits tied with one from the same family carrying four times the bits, because the second spent most of its time waiting on the CPU. Aggressive quantization has a real cost in fine reasoning, and it shows in the django case, but the trade came out in favour.

The second, and the most useful if you are going to attempt this: the two failures that cost me the most time were neither the model’s nor the agent’s, but badly configured context windows that fail silently. One truncated the prompt and only said so in a server log; the other split the window across slots and returned an error that looked like the agent’s. In both cases the visible symptom was “the local model is no good for this”. It is good; what was no good was my configuration.

The third: streaming a 284-billion-parameter model from disk genuinely works and is technically beautiful, with 6.27 GB resident for 167 on disk. But at 0.27 tokens per second it belongs to a different category of use. For one-off questions where quality matters and waiting does not, it makes sense. For an agent, it does not.

When I would use each one

• Ternary Bonsai for local agents, no argument. It fits, it is fast and it leaves window to spare.
• Qwen3.8 if the task depends on fine reasoning over code and time is not tight, or if you have a 24 GB card and there is no spill.
• DeepSeek V4 through Colibrì for specific queries where you want an enormous model’s capability and can wait. Not for agents, at least while the engine does not use the card.

And above all: before blaming the model, look at how much window it is actually seeing.

---

The raw data from the three tests, with times, diffs and per-test verdicts, is in the blog’s measurement dataset.