Claude Code, Codex, OpenCode, OpenClaw, Hermes, mini-swe-agent and Raven on MiniMax-M3: 7 agent harnesses measured on Terminal-Bench 2.0

A cheap model is not used on its own: it runs inside a harness, the program that gives it tools, builds its context, executes what it asks for and decides when it is done. Claude Code, Codex, OpenCode and the rest are exactly that, harnesses, and almost all of them accept any compatible model. The question I kept asking was which of them gets the most out of a model that is not top tier. What I found as answers were opinions, not measurements with the same model, the same tasks and the same limits.

So I measured it. Seven harnesses, the same MiniMax-M3, 40 tasks from Terminal-Bench 2.0, 280 runs, every request to the model saved in full and every passing task reviewed by a judge that did not know which harness it was looking at.

What I measured, and under which rules

Before the first run I wrote down a protocol with the question, the harnesses, the tasks and the decision rule, and I amended it in writing every time something changed: nine amendments, each with its reason and date, all committed before looking at the data they affect.

• Harnesses: Claude Code, Codex CLI, OpenCode, OpenClaw, Hermes Agent, mini-swe-agent and Raven. The first five are the most popular today; mini-swe-agent is the minimal reference; Raven is a manager harness with sub-agents, the opposite idea.
• Model: MiniMax-M3 in all seven, on MiniMax’s flat-rate plan. MiniMax has two dialects: Anthropic-compatible and OpenAI-compatible; each harness speaks its own.
• Tasks: 40 drawn (seed 17) from the 79 Terminal-Bench 2.0 tasks whose reference solution passes on my machine. Ten tasks were left out before starting because not even the official solution passes them here (interrupted downloads, emulation).
• How each one runs: with Harbor 0.24, the official Terminal-Bench runner. One run at a time, the same time limit each task sets, the same machine, the harness order rotated per task.
• What is measured: whether the task’s own tests pass (reward 1). Those tests are mounted after the agent finishes, so the agent never sees them.
• Rule: tasks solved per harness with a 95 % confidence interval (Wilson); paired comparison between each pair of harnesses with exact McNemar and Holm correction for the 21 comparisons.

Between each harness and MiniMax I put my own proxy that saves the full body of every request and every response. With that I recomputed the real token counts (the harnesses’ counters do not always include the cache) and reconstructed what each agent did in order to audit it.

Results

The 39 are the 40 minus fix-code-vulnerability, which I explain below; with it included, the order is the same. Cost is the equivalent at MiniMax-M3’s public rate (0.30 USD per million input tokens, 0.06 for cache reads, 1.20 for output), with tokens recounted from the saved responses.

• solved
• not solved
• tarea left out: the repo shows the fix

	mini-swe-agent	Codex CLI	OpenCode	Claude Code	OpenClaw	Hermes Agent	Raven
cobol-modernization
code-from-image
fix-code-vulnerability
headless-terminal
log-summary-date-ranges
modernize-scientific-stack
multi-source-data-merger
openssl-selfsigned-cert
prove-plus-comm
regex-log
reshard-c4-data
build-pov-ray
cancel-async-tasks
custom-memory-heap-crash
feal-differential-cryptanalysis
hf-model-inference
password-recovery
pypi-server
torch-tensor-parallelism
compile-compcert
crack-7z-hash
mteb-retrieve
count-dataset-tokens
overfull-hbox
path-tracing-reverse
query-optimize
sqlite-db-truncate
torch-pipeline-parallelism
extract-elf
feal-linear-cryptanalysis
large-scale-text-editing
sanitize-git-repo
llm-inference-batching-scheduler
regex-chess
video-processing
kv-store-grpc
make-mips-interpreter
polyglot-c-py
polyglot-rust-c
winning-avg-corewars
solved (39)	29	26	25	23	22	22	20

Ten tasks were solved by all seven and five by none. The comparison lives in the middle band. There, the largest difference is between mini-swe-agent and Raven: on the tasks where they disagree, mini-swe-agent won 10 and Raven 1. On its own, that difference would give p = 0.012; corrected for the 21 comparisons, as the protocol says, it rises to 0.25 and no longer holds. No other pair comes close. With 39 tasks, the honest conclusion is that no harness is provably better than another, and that the simplest one did no worse than any of them.

Cost is where the gap shows: Raven spent almost three times what OpenCode did for the same 40 tasks, and solved 5 fewer. OpenCode came out best on solved tasks relative to cost: third in tasks, first in price and in time.

The same task, seven ways

To see the difference in style I took a task all seven solved, openssl-selfsigned-cert (create a key, a self-signed certificate, a combined PEM and a script that verifies it), and replayed it with the real timings: each mark is a request to the model, at the second it was made.

Same model, same task, and Claude Code finished in 100 seconds with 11 requests while Raven took 276 with 33. The screenshots of each harness, further down, come from these same runs.

Was there cheating?

The pass rates looked high to me for Terminal-Bench 2.0, so before trusting them I reviewed each of the 174 passing tasks. First, an automated sweep identical for all seven: reading or editing the tests, touching the verifier, downloading solutions, writing the expected output without computing it. Then a blind judge (Opus) that read the transcript with harness and tool names removed and classified each pass as legitimate, cheating or doubtful, using a rubric written before starting. I tested it first on two control transcripts, one cheating and one clean, and it got both right.

Result: 171 legitimate. The other 3 all come from fix-code-vulnerability. That task’s repository ships the vulnerability fix as an uncommitted change, and a git diff shows it. All seven harnesses ran that git diff and all seven passed. The judge flagged three as cheating and let through four that did the same thing, so the judge was not consistent either. Since the defect belongs to the task and everyone saw it, I removed it from the comparison for all seven equally.

The seven, one by one

Each harness, with what it solved, how I connected it to MiniMax, what it tripped over, how it works according to what the proxy saved, and what I would use it for. The screenshots are each one’s real console on the same task. The TUI shown in each reel comes from a separate smoke run: same task, same model and same version, but in interactive mode and sped up, because the measured runs were headless. All seven solved the task in that run too.

mini-swe-agent: the simplest, and the one that solved the most

29 of 39 tasks, 8.79 USD, 9.3 minutes median. It is the reference harness from the SWE-bench team, designed to be minimal.

How I used it. It installs with uv tool install mini-swe-agent (version 2.4.6) and talks to MiniMax in the OpenAI dialect through LiteLLM: --model=openai/MiniMax-M3, with OPENAI_BASE_URL and OPENAI_API_BASE pointing at the OpenAI-compatible URL, and MSWEA_COST_TRACKING=ignore_errors because LiteLLM does not know MiniMax-M3’s price.

Gotchas. The current LiteLLM version imports mcp at startup and Harbor’s installer does not include it: mini-swe-agent kept retrying every 4 seconds. The fix is adding --with mcp to the install. On two tasks the install failed because of a half-synced apt mirror, and those two runs were counted as unsolved.

How it works. Its system prompt is 64 characters long. It offers a single tool, bash, and does every edit through the shell: heredocs, sed, Python scripts. It does not use streaming or set a temperature. Across the 40 runs it made 1,744 calls, all to bash.

What I would use it for. It is my first choice for a cheap model that knows how to use a terminal: fewer layers, less context spent on instructions, fewer things that can break. It has no memory, no sub-agents and no interface; if you need those, it is not for you.

Codex CLI: second, despite its stumbles

26 of 39 tasks, 7.33 USD, 9.3 minutes median.

How I used it. npm install -g @openai/codex (0.163 and later 0.164) and the OpenAI dialect, but through its Responses API (/v1/responses): it is the only one of the seven that uses it. You have to set OPENAI_BASE_URL and also openai_base_url in config.toml, because since 0.118 Codex only reads the URL from that file.

Gotchas. In 39 runs it warns that it cannot find metadata for MiniMax-M3 and will use generic values. In 39 it first tries to connect over WebSocket, gets 404 five times and only then falls back to HTTPS. On three long tasks its context compaction died: it waits for a compaction item that MiniMax does not return. And it does not offer the apply_patch tool to this model, so it edits by writing files through the shell.

How it works. Its own 17,587-character system prompt, 1,246 calls to exec_command, and some calls to tools that are not in its list (read_file, update_plan): the model invents them and Codex rejects them.

What I would use it for. Even with all that, it came second. If you already use Codex, it works with MiniMax without patches; expect very long tasks to die during compaction.

OpenCode: the cheapest and the fastest

25 of 39 tasks, 4.13 USD, 6.4 minutes median. Half the cost of Codex and the lowest time of the seven.

How I used it. npm i -g opencode-ai (1.19.3 and later 1.19.4) with the anthropic provider and the Anthropic-compatible dialect. The base URL has to end in /v1 (ANTHROPIC_BASE_URL=…/anthropic/v1), the opposite of Claude Code; without the /v1 it returns 404.

Gotchas. With version 1.19.4, 11 runs exited with code 1 because of a Node EPIPE on exit, after the work was done: 9 of those 11 still passed the tests. To title each session it asks a small model, claude-haiku-5-5; MiniMax accepted that name and answered with MiniMax-M3.1-Flash. It is one request per run and cost less than a cent in total.

How it works. A prompt of about 9,700 characters, 876 calls to bash, a todo list (todowrite) on long tasks, and a git snapshot per step. It never used sub-agents.

What I would use it for. If you pay per token, it is the one I would pick: almost the same result as the top ones at half the cost. Pin the version, because it changed mid-week and the EPIPE came with it.

Claude Code: the only one that worked without patches

23 of 39 tasks, 7.23 USD, 12.6 minutes median.

How I used it. The official installer (versions 2.1.305 to 2.1.310 over the week) and the Anthropic-compatible dialect: ANTHROPIC_BASE_URL=…/anthropic, ANTHROPIC_MODEL=MiniMax-M3 and the three aliases ANTHROPIC_DEFAULT_HAIKU_MODEL, …_SONNET_MODEL and …_OPUS_MODEL also pointing at MiniMax-M3. With those aliases it makes no calls to any other model.

Gotchas. No configuration ones: it is the only one of the seven that needed no patch. On one long task the model went over the 32,000 output-token cap for a single turn and the run died; you can raise it with CLAUDE_CODE_MAX_OUTPUT_TOKENS, but I left the defaults for everyone.

How it works. It offers twelve tools, including Agent for sub-agents, but with MiniMax-M3 it never used it. 79 % of its calls went to Bash; it edits with Edit (exact text replacement). It was the fastest on the certificate task.

What I would use it for. It is the easiest to connect to MiniMax and the one with the fewest surprises. With this model it got no advantage from what makes it strong with Anthropic’s models.

OpenClaw: the personal assistant that also codes

22 of 39 tasks, 5.85 USD, 10.7 minutes median.

How I used it. openclaw@2026.10.5, pinned, with the OpenAI dialect (/v1/chat/completions) and the configuration in ~/.openclaw/openclaw.json.

Gotchas. It was the hardest to get working. It requires Node 24 and Harbor’s adapter installs Node 22, both for installing and for running: I fixed it with a subclass of the adapter. Without declaring maxTokens: 32768 and contextWindow: 512000 on the model, MiniMax-M3 spent the whole output cap reasoning and returned empty turns. Its semantic memory requested 378 embeddings from text-embedding-3-small; MiniMax answered 200 with an error inside and OpenClaw carried on without memory and without saying so.

How it works. It is a personal assistant first and a coding agent second: its system prompt is about 50,000 characters, the largest of the seven, and it ends by asking what to call it. Even so, three out of four of its calls went to exec.

What I would use it for. If you already live in OpenClaw for other things, it codes well with a cheap model. As a dedicated coding agent, the setup cost is not repaid by what it solved.

Hermes Agent: many tools, many retries

22 of 39 tasks, 6.51 USD, 12.5 minutes median.

How I used it. hermes-agent==0.19.0 with uv, pinned, provider openai-api. The key has to be in the environment of the process that launches Harbor, not only passed to the agent.

Gotchas. The official install.sh ends with an error because of a desktop component that does not apply in a container, and it also installed a version without the hermes version command that Harbor uses to verify: I replaced it with the uv install and a pinned version. On every run it probes local servers such as Ollama or llama.cpp: 956 404 responses over the week, which cost no tokens but do cost time. Two runs hit its cap of 90 iterations and one died after three retries due to MiniMax overload.

How it works. Seventeen tools, among them terminal, execute_code and patch; it called delegate_task in only two runs. Together with Raven, it had the most runs end in a timeout or error: 17 of 40.

What I would use it for. It is a general agent with memory, skills and scheduled tasks; for coding with a cheap model, that breadth did not turn into more solved tasks.

Raven: the manager that delegates, and the most expensive

20 of 39 tasks, 12.22 USD, 12.0 minutes median.

How I used it. Harbor ships no adapter for Raven, so I wrote one, with its own tests. Raven 0.2.5 pinned by commit, installed with uv in a separate environment, with the OpenAI-compatible custom provider and protocol: "chat" forced.

Gotchas. The first version of the adapter glued the export onto the command and bash read the flags as variables. Its install is the slowest of the seven. Its long-term memory was off in every run because the package that provides it is distributed separately. One run was taken down by a DNS outage on my network.

How it works. By default it is a manager that splits the work among sub-agents: it delegated in 34 of 40 runs, the only one that did so systematically. It also makes internal calls to the same model to check permissions, title the session, judge its sub-agents and classify the request. All of that adds up: a median of 33.5 requests per run, the highest, and the highest cost.

What I would use it for. Delegation makes sense with a strong model and long jobs that can be split up. With a cheap model and single-sitting tasks, every management layer is context and time that does not turn into solved tasks.

What it cost

The 280 runs would cost about 52 USD at MiniMax-M3’s public rate. Most of the tokens are cache re-reads of the same context, which is what makes a long agent cheap: the median per run goes from 330 thousand tokens (OpenCode) to 750 thousand (Raven). Some harnesses’ counters do not match the wire: OpenCode, for example, logged 252 thousand output tokens over the 40 runs, when the saved responses add up to one million. That is why every number in this post comes from the saved responses, not from what each harness says it spent.

Threats to validity

• A single model. Everything here holds for MiniMax-M3. With an Anthropic or OpenAI model the order may change, and it is reasonable to expect Claude Code or Codex to do better with the model they were built for.
• 39 tasks. Enough to see trends, not to separate harnesses that differ by three or four tasks.
• Versions that changed. The protocol said “pinned versions”, but Harbor’s adapters for Claude Code, Codex, OpenCode and mini-swe-agent install the latest version, and the first three changed during the week. OpenClaw, Hermes and Raven did stay pinned.
• Task selection. I excluded the ten that the reference solution does not pass here; that may leave tasks somewhat easier than the benchmark average.
• Infrastructure errors inside the analysis. Two failed mini-swe-agent installs and a DNS outage that affected Raven were counted as unsolved.
• Auxiliary calls. OpenCode asked a small model for titles and OpenClaw attempted embeddings; they weigh less than a cent, but they are not strictly the same model.
• The judge. It was inconsistent on the task with the solution in plain sight; on the rest it agreed with the automated sweep.

Verdict

With a cheap model like MiniMax-M3, the harness matters less than it seems, and a more elaborate one does not help: the simplest one, mini-swe-agent, solved more than any other, though no difference reaches significance with 39 tasks. If you want the best balance between what it solves and what it costs, OpenCode. If you want to connect without a fight, Claude Code. Codex performs well despite its warnings. OpenClaw and Hermes are general agents that also code. Raven, with its sub-agent management, was the most expensive and solved the fewest with this model. SWE-bench Verified and Aider Polyglot with the same seven come next, and I will update this table.

Frequently asked questions

OpenCode vs Claude Code vs Codex: which solves more tasks with the same model?

With MiniMax-M3 on 39 Terminal-Bench 2.0 tasks, Codex CLI solved 26, OpenCode 25 and Claude Code 23. The differences between the three fit within the noise: with 39 tasks, no comparison between two harnesses was significant after correcting for the 21 possible comparisons.

Best coding agent harness for a cheap model like MiniMax M3?

In this measurement, the simplest one: mini-swe-agent, with a single tool (bash) and a 64-character system prompt, solved 29 of 39 tasks, the most of any. It did no worse than any other. But the gap to the rest is not significant with 39 tasks: what can be claimed is that a more elaborate harness did not help this model.

Hermes Agent vs Claude Code benchmark?

With MiniMax-M3, Hermes Agent solved 22 of 39 tasks and Claude Code 23, at similar cost (6.51 and 7.23 USD equivalent for the 40 runs). Hermes had 17 runs end in a timeout or error and probed local model servers on every run.

How much does it cost to run an agent benchmark with MiniMax M3?

The 280 runs would cost about 52 USD at MiniMax-M3's public rate, recounting tokens from the saved responses: from 4.13 USD (OpenCode) to 12.22 USD (Raven) per harness for 40 tasks. Most of it is tokens read from cache. I ran it on MiniMax's flat-rate plan.

Did the agents cheat on Terminal-Bench?

I reviewed all 174 passing tasks: an automated sweep identical for everyone and a blind judge that did not know which harness it was looking at. 171 were legitimate. The other 3 come from a single task, fix-code-vulnerability, whose repository leaves the solution in plain sight via git diff; all seven harnesses looked at it, so I removed it from the comparison for everyone.

How do I connect Claude Code, Codex or OpenCode to MiniMax?

MiniMax has two dialects. Claude Code and OpenCode use the Anthropic-compatible one (ANTHROPIC_BASE_URL; OpenCode requires the path to end in /v1). Codex uses the OpenAI-compatible one through its Responses API (OPENAI_BASE_URL and openai_base_url in config.toml). The full method for each, including the stumbles, is in the post.

Can I reproduce it?

Yes. The protocol with the decision rule written before measuring and its nine amendments, the Raven harness adapter, the logging proxy, the analysis scripts, the audit and the per-run results are in the repository and in a gist linked in the sources.

Sources

• Terminal-Bench and Harbor, the official runner.
• The harnesses: Claude Code (repo), Codex CLI, OpenCode (repo), OpenClaw (repo), Hermes Agent (repo), mini-swe-agent (repo) and Raven.
• MiniMax: Anthropic-compatible API.
• The protocol, the proxy, the Raven adapter, the analysis scripts, the audit and the per-run results.

Measured October 20 to 24, 2026 on a NixOS machine, one run at a time. Harbor 0.24.0, Terminal-Bench 2.0 (40 tasks, seed 17), MiniMax-M3. Claude Code 2.1.305–2.1.310, Codex CLI 0.163–0.164, OpenCode 1.19.3–1.19.4, OpenClaw 2026.10.5, Hermes Agent 0.19.0, mini-swe-agent 2.4.6, Raven 0.2.5 (05cf9c35). 280 runs; 174 passes reviewed by a blind judge.