Mage-VL-Cowork on 16 GB: it fits, it answers in 2.5 s, but it won't click

Microsoft released Mage-VL-Cowork on Hugging Face on October 30, 2026 and pushed the code to the microsoft/Mage repository on November 2, with a big headline number: 66.0% on OSWorld-Verified with only 4 billion parameters. In the chart on its model card it sits above every open agent of its size, and 3.6 points short of the GPT-5.4-mini in the reference paper (69.6%).

If that holds, half a tower of lab machines moves: an agent that understands screenshots and decides to click or open a terminal, fitting on a home GPU.

The model weighs 9.48 GB in bfloat16. On paper, slack. A 16 GB RTX 4070 Ti SUPER should swallow it without breaking a sweat. I pulled it onto fedora, brought it up with the SGLang image Microsoft publishes in its repository, and asked it to click a button.

It didn’t click once.

What Mage-VL-Cowork is

Two parts glued together. The left one is Mage-ViT, the team’s own visual encoder, with 16×16 patches and spatial_merge_size=2, projecting a 1,024-dimension embedding. The right one is Qwen3-4B-Instruct-2507, the language model, with a hidden size of 2,560. Between them lives the adapter that lifts the visual output into the text space.

What makes it different from any Qwen2.5-VL is the hybrid route. At every turn the model picks between two channels: computer_use for clicking, typing, dragging, scrolling or taking a screenshot; or bash for running a command on a persistent terminal. The promise is that for something as simple as renaming a hundred files, instead of dragging them through a graphical interface, the agent writes mv and measures the result.

Training was five stages on 318,837 verified trajectories, and the last one is an online GRPO phase with verifiable rewards.

I pulled it with a Hugging Face token. 25 files, 8.9 GB once the assets are stripped. The safetensors come split in two shards, 4.56 GB and 4.92 GB.

The official harness is a Docker container

The model’s README points to the mage_vl_cowork/osworld_eval/scripts/ folder in the GitHub repository. In there sits a script, serve_sglang_docker.sh, which in a single block makes clear how they expect it to run:

docker run --rm --name mage-vl-cowork \
  --gpus device=0 --shm-size=8g \
  -p 127.0.0.1:30001:30001 \
  -e SERVED_MODEL_NAME=Mage-VL-Cowork \
  -e SGLANG_MAX_PIXELS=4000000 \
  --mount type=bind,src="$MODEL_DIR",dst=/models/Mage-VL-Cowork,readonly \
  xiaoyizhang/mage-vl-cowork:0.1.0

The xiaoyizhang/mage-vl-cowork:0.1.0 image is 19.3 GB on disk. Inside is an SGLang patched by the Mage team so that llava_onevision2 runs with its tokenizer and its video RoPE.

On fedora (NixOS, Docker 29 with the nvidia CDI runtime) --gpus device=0 fails with AMD CDI spec not found. The replacement is --device nvidia.com/gpu=0. Without that change, the container doesn’t start.

Two minutes later the server exposes an OpenAI-compatible endpoint on 127.0.0.1:30001.

What I could run before measuring

Before building the bench I checked the text channel responds:

curl -s -X POST http://127.0.0.1:30001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Mage-VL-Cowork","messages":[{"role":"user","content":"Reply with exactly the word: OK"}],"max_tokens":32,"temperature":0}'

The model returned:

This is the initial state of the task. I’m considering how to respond to the user’s request, and I’m leaning toward a simple confirmation rather than adding

With no image, no system prompt, with the simplest possible question, the model returned an internal agent narration. It never reaches “OK”: the answer is cut off at 32 tokens still inside the reasoning. It’s trained to think before acting, and to it every prompt is the start of a task.

That was the first data point: the agent format is structural. It cannot be switched off from the client.

The measurement sheet

Before the first run, I committed lab/mage-vl-16gb/ficha.json with the hypothesis, the metrics, the decision rule and the threats to validity. The commit is earlier than the one for the results, so the rules were not tuned to the data.

The hypothesis: Mage-VL-Cowork 4B fits in 16 GB and runs at useful speed; the fixed agent format is a structural limitation.

The decision rule: publishable if it loads in under 180 s, if a 256-token answer takes under 5 s, if peak VRAM stays below 15 GB, and if across thirty clicks on labeled buttons it beats a trivial baseline that always clicks the center of the screen by at least fifteen points.

The trivial baseline: two rules that don’t look at the image. One always returns (500, 500). Another returns a uniformly random coordinate. If the model ties the first, that’s the headline.

The original sheet asked for a third baseline: Tesseract OCR on the screenshot, extracting the center of the detected text. The fedora host on NixOS doesn’t ship Tesseract by default and, given that the model collapses before it can compete against it, I replaced it with the two geometric rules above. The decision is recorded in the sheet as a deviation.

The bench: ten synthetic panels 1,000×1,000 px generated with Pillow. Each one has three or five colored buttons with a label rendered in DejaVu Sans Bold at 32 px. The size and position of each button are known in advance. For every panel, one button is picked at random as the target. Thirty runs in total: ten panels times three repetitions, with the order shuffled so they don’t cluster.

The sheet called for twenty prompts times three runs, sixty observations. The bench stopped at thirty because the result separated from zero on the first pass and scaling to sixty would not have changed the conclusion. Declared and left without reinterpretation.

The scoring metric: a click counts if the returned coordinate falls inside the target button’s rectangle. For the model, the coordinate is extracted from the response’s tool_call. For the baseline, it’s the fixed center or the uniform draw.

What I measured

Thirty runs, same endpoint, same machine, same Python-client clock.

The model’s format is partial for a concrete reason: the opening <think> tag never shows up. All thirty answers open with the reasoning narration at the first character, close with </think> at the end, and continue with the Action: line and a well-formed <tool_call> block. The model fulfills the response contract but drops the opening of the thought, likely because in the official harness the server itself injects it.

The first three rows meet the budget I set. Load is quick, VRAM fits with a 2 GB margin, and latency is comfortable for a decision loop. The agent format also holds up: thirty out of thirty answers brought the </think> close, the Action: line and a valid <tool_call>.

The last row is what sinks everything else.

What happens when I ask it to click

The question is always the same, paraphrased per button: “The user asked you to click the Save button in the panel. Do it.” The model receives the official Mage-VL-Cowork system prompt, with the full tool list, and the panel image as a base64 image_url.

Here is one real response, on a panel asking it to click the “Print” button:

I’m at the point where the only supported operation for this step is a screen capture, and the screenshot is useful here because it can confirm the current desktop state before I try to act on the Print button. The earlier reasoning already narrowed the likely path to the top-right system area, where Ubuntu 22.04 commonly places the quick settings and the Print command, but that was still tentative and depended on what was actually visible.

Since this step does not interact with the document or the panel, it lets me inspect the menu path instead of guessing blindly. Capturing the status area is the right next move before clicking anything else, especially because the task is specifically about the Print button in the panel and not about opening a shell or changing files.

</think>

Action: Take a click-free screenshot to inspect the current screen state.

<tool_call>{"name": "computer_use", "arguments": {"action": "screenshot"}}</tool_call>

Two things happen there. The first: the model ignores the image I just gave it and talks about an “Ubuntu 22.04” with a “quick settings area at the top right”. There is no Ubuntu in my panel: there is a violet rectangle with the word “Print” inside. The model prefers the desktop in its head to the one in front of it.

The second: when it finally picks an action, it picks screenshot. Not click. Across all thirty runs, 30 out of 30 tool calls were action: screenshot. Zero clicks.

The pattern is consistent with the training. Mage-VL-Cowork was trained inside a loop: the harness shows it a capture, the model answers, the harness executes the action, the harness returns the capture of the new state. The screenshot action exists because inside that loop it is free: the harness handles it, returns the same thing it already had, and the agent reasons with a freshly minted image.

Outside the loop, with no harness to serve the request, screenshot is a dead end. I am not going to send it a new capture: the one I sent is the only one. The model does not know there is no harness and asks, and asks, and asks for one more capture.

What the baseline says

A trivial rule that always returns (500, 500) hits nine of thirty buttons. Not because it is clever: my buttons tend to live near the center of the panel, and the point (500, 500) falls inside the target rectangle in three of the ten scenes (the benchmark seed). Replaying each scene three times gives nine hits out of thirty.

That 30% is geometric noise. I include it because a serious model would have to crush it: a network that looks at the button label and returns that rectangle’s coordinate should be close to 100%. Mage-VL-Cowork, outside its harness, is at 0.

What this exercise is worth

Three things I did measure and that are useful for anyone wanting to run it:

It fits in 16 GB. With the official SGLang image, the model leaves roughly 2 GB of headroom. An equivalent Qwen3-4B without the visual encoder would fit with twice the margin; with Mage-ViT and the KV cache SGLang reserves at startup (24,093 tokens, 3.3 GB for K and 3.3 GB for V, according to the server’s own log), the real limit sits near the ceiling.

The latency is honest. 72.86 tokens per second on a home GPU is exactly what you would expect from a 4B with flash attention and CUDA graphs. At 2.47 s per response, an agent loop of twenty calls (the average the paper reports for OSWorld after the reinforcement learning stage) would add up to a long minute of pure inference per task. That is a calculation, not a measurement: I did not run the full loop.

The server is the cleanest piece of the package. Microsoft shipped the whole runtime inside a container, and the container works on the first try, with the single --device flag change for NixOS’s idiosyncrasy.

What’s not worth trying

Reproducing the 66% on OSWorld at home needs the full harness: an Ubuntu inside a VM, the desktop_env from the repository, 50 interaction steps per task, 361 tasks, two runs for the average. Hours of compute for a number already in the paper.

What’s interesting is not whether it hits the 66% it claims. It’s whether without the harness it’s useful for anything. The answer from this bench is no: not as a general VLM, not as a one-shot spatial classifier.

When to use it and when not to

If you’re building a desktop agent with a full loop, pluggable language models and a harness that actually serves captures and clicks, Mage-VL-Cowork is a reasonable candidate by cost: a 4B that runs on a consumer card, a well-defined format, with the bash channel included for non-visual operations.

If you’re looking for a VLM to look at a screen once and answer, this isn’t it. For that task, Qwen2.5-VL-7B or InternVL2.5-8B remain the obvious candidates.

And if anyone wants to reproduce this, the server comes up in ninety seconds on a 4070 Ti SUPER. All the raw data lives in the blog repository under lab/mage-vl-16gb/.

What I did not measure

Robustness to variation. Temperature 0 across all runs. I don’t know what happens at temperature=0.7; probably nothing good, but I didn’t measure it.

The bash channel. The model also offers to open a terminal. I never asked it for a terminal task, so I don’t know whether at least that channel responds.

Real images. The ten panels are synthetic with Pillow. The original sheet committed to including ten real captures alongside the synthetic ones; I did not fulfill that mitigation. A real Ubuntu desktop screenshot, the environment the model was trained for, probably moves the result. What did come through clearly: facing the synthetic panel the model does not even try to read the buttons, it prefers to ask for another capture. To measure whether it would recognize “Save” at 1,000×1,000 you would have to force it to return the label, and it never got that far.

Latency without the system prompt. The sheet asked to also measure the response to the same prompt without the 4.2 kB MAGE_HYBRID_SYSTEM_PROMPT, to separate the agent-format cost from the model cost. I did not measure it on this bench; the only probe without a system prompt was the validation curl, which answered with the same internal narration. The exact overhead stays isolated.

Qwen2.5-VL-7B as a reference. It would have been useful to run the same bench against a general VLM. I did not pull it down because the experiment was about Mage-VL-Cowork and the GPU budget is tight. It stays on the queue.

Sources

• Model card on Hugging Face: microsoft/Mage-VL-Cowork
• Official repository with OSWorld harness: microsoft/Mage, mage_vl_cowork folder
• Server Docker image: xiaoyizhang/mage-vl-cowork on Docker Hub
• OSWorld-Verified benchmark: xlang-ai/OSWorld
• Language backbone: Qwen/Qwen3-4B-Instruct-2507
• Raw data and scripts: lab/mage-vl-16gb/ in this blog’s repository

November 5, 2026 · RTX 4070 Ti SUPER 16 GB on NixOS 26.05 · SGLang xiaoyizhang/mage-vl-cowork:0.1.0 · docker 29.8.1 with the NVIDIA CDI runtime · 30 runs, temperature 0, max_tokens 256.