Models on my own hardware

All topics

Almost everything published about open models is measured on datacenter cards. The question here is a different one: if the file weighs more than the memory available, does it still run, and at what cost? These pieces share a bench — the same 16 GB GPU, the same frozen inputs — so the numbers can be placed side by side instead of read in isolation.

24 pieces

Article2026-11-07An agent that reads my own posts and verifies the numbers aren't made upGoogle released version 2.14 of its agent kit on November 4, 2026, and among other things graduated its local execution sandboxes out of experimental. I wrote an agent that reads a technical blog post, extracts every number that appears in the text, and opens the raw JSON of the experiment to check that the number is actually there. On the post I published yesterday, it detected 46 numbers, verified the 9 that came from the measurements, flagged 1 that wasn't in the JSON because of a field I forgot to record, and classified the other 36 as context or configuration. It took one minute and sixteen seconds. A trivial regex gets it wrong: it claims 45 figures 'verified' because 50 == 50 by chance.#Agents#SDK#VerificationArticle2026-11-06Cola2 on 16 GB: it fits with offload, scores 56% on ARC-Easy, but the paper's harness won't runByteDance Seed released Cola2 on November 6, 2026: a language model with continuous latents and a diffusion transformer in place of the autoregressive one. I pulled it onto an RTX 4070 Ti SUPER, brought it up with the official example, and measured it on fifty ARC-Easy questions. It fits in 14.5 GB with the diffusion transformer offloaded to the CPU, lands 56% against the 22% of a trivial baseline that always picks the first option, and takes ten and a half seconds per question. The whole paper harness, with its official scorer, OOMs with every config I tried.#Models#GPU#BenchmarksArticle2026-11-05Mage-VL-Cowork on 16 GB: it fits, it answers in 2.5 s, but it won't clickMicrosoft released Mage-VL-Cowork, a 4-billion-parameter computer-use agent that scores 66% on OSWorld on paper. I pulled it onto an RTX 4070 Ti SUPER, brought it up with the official SGLang image, and measured it: loads in ninety to one hundred twenty seconds, holds 13,985 MB of VRAM, answers in 2.47 s. And across thirty prompts that asked it to click a button, it clicked zero times: every answer was to ask for another screenshot.#Agents#Models#GPUArticle2026-10-25Claude Code, Codex, OpenCode, OpenClaw, Hermes, mini-swe-agent and Raven on MiniMax-M3: 7 agent harnesses measured on Terminal-Bench 2.0I had seven coding agent harnesses drive the same cheap model, MiniMax-M3, on 40 Terminal-Bench 2.0 tasks: 280 runs with every request saved and every pass reviewed by a blind judge. The simplest harness, mini-swe-agent, solved 29 of 39; Codex 26, OpenCode 25, Claude Code 23, OpenClaw and Hermes 22, Raven 20. No difference between two harnesses survives the multiple-comparison correction. OpenCode was the cheapest and fastest; Raven, the most expensive.#Agents#Benchmarks#Claude CodeArticle2026-10-23Isolating a coding agent with gVisor or Kata: what it costs compared with Docker, measuredI measured what it costs to run a coding agent's work isolated in gVisor and in Kata Containers 4.3 (with QEMU and with Cloud Hypervisor) instead of a plain Docker container. Kata takes 6 to 7 s to start against 0.22 s for runc and leaves 199 to 302 MiB per container against 11.5; gVisor starts in 0.33 s, uses 48.6 MiB and ties with runc on a workload of thousands of small files.#Containers#Security#AgentsArticle2026-10-18ComfyUI vs Forge on a 16 GB GPU: they tie on SDXL, and ComfyUI is twice as fast with Z-Image when the models do not fitI measured ComfyUI, Forge Neo and the original Forge with the same files, seeds and settings on an RTX 4070 Ti SUPER, with diffusers as a baseline. With SDXL they tie at 6.7 s per image. With Z-Image Turbo, which does not fit together with its text encoder, ComfyUI takes 6.75 s and Forge Neo 13.4 s; with Qwen-Image fp8, 75.7 s against 96.0 s. The logs point to how each one manages memory between RAM and the GPU.#AI#Benchmarks#Image generationArticle2026-10-17NVFP4 vs FP8 vs GGUF on a 16 GB GPU without Blackwell: NVFP4 generates 2.3 times faster than BF16 and ties with BF16 on the full 1319-question GSM8KI measured the same NVIDIA model in BF16, NVFP4, MXFP8, FP8 and GGUF Q5_K_M on an RTX 4070 Ti SUPER. NVFP4 generates at 117 tokens per second against 51 for BF16, and FP8 is the fastest at reading a long prompt. On all 1319 GSM8K questions the three vLLM formats tie with BF16 within ±3 points, although token by token NVFP4 changes BF16's choice one time in ten.#AI#Benchmarks#Local inferenceArticle2026-10-16NVFP4 without Blackwell: I ran NVIDIA's Qwen-Image on an RTX 4070 Ti SUPER and the NF4 grain was in 8 matricesNVIDIA released Qwen-Image in NVFP4 for Blackwell only. On an RTX 4070 Ti SUPER the official route does not load, and the weights expanded to bf16 run slower than NF4 (145 s against 102 s per image) without writing better. What I did find is the source of the NF4 grain: not the layers with the worst error, but 8 matrices in the edge blocks that NVIDIA left unquantized. Keeping them in bf16 removes it at less than half a second more per image.#AI#Benchmarks#Local inferenceArticle2026-10-14Kandinsky 6 on a 16 GB card: the video works, and only the big model passed my Spanish testKandinsky 6.0 generates five seconds of video with audio. On my 16 GB RTX 4080 the official recipe ran out of memory with the distilled Lite model; with a two-phase recipe it ran, at about 2 minutes 21 seconds per vertical clip. I measured how faithful its Spanish is with Whisper across three models of the family: Lite distilled and Lite undistilled miss the rule I set before running; Pro distilled meets it, but its process reached 120 GiB of system RAM.#Models#Video#GPUArticle2026-10-14Why I moved my whole home lab to NixOS: four machines, one repository and what it costI had four machines with four different operating systems, services running in tmux, passwords inside unit files and several agent sessions touching everything at once. In nine days I moved them to NixOS with a single flake. I measured what it gives: a deploy takes 11 to 17 seconds, the same commit produces the same system hash on two different machines, and going back to the previous version takes 1.6 seconds. And I measured what it cost: 35 of 111 commits are fixes, every Python environment was lost and the router's DNS repointed 1,185 files on its own.#NixOS#Infrastructure#AgentsArticle2026-10-10Eight decision models against my data: none beat what I already use, and the index's second place ranked fraud backwardsI ran eight decision models on the Decision Index, which I reproduced, and on three banks of my own with real labels: video shot descriptions, card fraud and rain. None clearly beat each bank's classic reference. On fraud, the only bank where they separate clearly, d1-3B, second on the index, ranked backwards and JEV-9B was the clear best; on shots and rain the top ones tied.#IA#Benchmarks#Inferencia localArticle2026-10-10Qwen-Image 2.1 replaces FLUX for my covers: 7 times faster on the same 16 GB card, and what I loseI ran the same prompts and seeds through the FLUX that makes my covers and through Qwen-Image-2.1-Turbo quantized to NF4, on the same 16 GB card. Qwen takes 8.4 s per image against 61 s, writes 13 of 18 texts exactly against 7, and draws the requested scene in 11 of 15 covers against none. I lose the LoRA's atmosphere, and the version that fits 16 GB writes 4 fewer texts than unquantized Qwen.#IA#Benchmarks#Inferencia localArticle2026-10-07Strands Decider 2B as the judge of my shots: 37 ms, and a keyword rule ties itI measured Strands Decider 2B as a judge on 75 shots: with choice it hits AUC 0.91 in 34 ms per judgment, vs 0.99 for Claude Opus; a simple rule got 0.90. With its default primitive, noul, it caught none of the 31 generic shots; the v21 checkpoint does not improve it, and repetition across shots, my real failure, is caught by every judge. Data, labels and scripts are published.#AI#Benchmarks#Local inferenceArticle2026-09-26A model that does not write, it only decides: I had Intern-Decision predict the rain without training itIntern-Decision came out on 26 September 2026: 0.8B to 4B models that do not generate text and return one probability per question in a single forward pass. I installed it the same day, ran into three failures along the way, one of which made it between 2 and 3.2 times slower, and had it predict tomorrow's rain over 27,256 days in seven Chilean cities without training it. The 2B reaches an AUC of 0.823, almost the same as a Naive Bayes trained on 74,145 rows, and stays below logistic regression.#Models#Benchmarks#GPUArticle2026-09-23AWS says its agent harness saves 28% of tokens. I measured 1.82x more expensive locally, 88% cheaper in the cloudI tested Strands Harness, AWS's agent runtime, against a bare LangGraph agent: same 3 tools, same gatekeeper, same 20-turn conversation. Against a local Qwen3.8-27B with a 16k context window it spent 1.82 times more tokens. Against the same job on a cloud model, with temperature finally matched between the two, it spent 88.8% less. And three rounds of adversarial audit had to correct my own mistakes along the way: 11 extra tools, a turn limit that lied, and a cache double-count that inflated the first number 4x.#Agents#LangGraph#AWSArticle2026-09-18CUDA in Rust vs CUDA C++: Rust wins by 3.2% until you check that they do not compute the same thingI built cuda-oxide, NVIDIA's backend for writing CUDA kernels in Rust, and timed the same kernel against CUDA C++. On Mandelbrot Rust comes out 3.2% faster, but the two programs differ in 27,510 pixels; with the arithmetic matched bit for bit, Rust ends up 7% slower.#Rust#CUDA#GPUArticle2026-09-14ExLlamaV3 vs llama.cpp with Qwen3.8-27B on 16 GB: faster decoding, but it does not always winI gave both engines exactly the same input tokens inside a container with pod limits. ExLlamaV3 decodes faster and handled 131k tokens of context; with 13 GB files it loses less quality, with 15 GB files it loses to GGUF. In an agentic turn, prefill eats almost all of the lead.#ExLlamaV3#llama.cpp#QuantizationArticle2026-09-13A chat on Amazon Bedrock AgentCore for my blog: I broke it with 64 messages and rebuilt it with rules outside the promptTutorial: LangGraph chat on Amazon Bedrock AgentCore with an S3 Knowledge Base and guardrails, tested on a 64-message chat. Costs and a local option.#AI#AWS#SecurityArticle2026-09-02More bits where it hurts: I quantized a Qwen3.8-27B to fit my 4070 TiQ4 overflows 16 GB and Q3 gives up quality. A llama.cpp patch spreads the bits layer by layer: I measured a Qwen3.8-27B at 94% agreement with an 8-bit reference that does fit on a 4070 Ti.#Quantization#llama.cpp#GGUFArticle2026-08-17TabPFN and TabICL against tuned XGBoost: the model that does not train won on fourteen tables out of fourteenThe claim behind TabPFN and TabICL is that they predict on a table without ever training on it and still beat tuned boosting. I measured it on fourteen datasets from the Grinsztajn benchmark, with the same split and the same clock for everyone. The one that does not train wins, the advantage holds up to 32,000 rows instead of breaking, and the most-cited model can no longer be downloaded without an account.#Models#Benchmarks#GPUTutorial2026-08-16I ran AIFS 2.0, ECMWF's weather model, on my desktop card and landed 0.45 °C from the official forecastECMWF published the weights of its AI forecast model. It weighs under a gigabyte and runs on a desktop card: 48 hours of forecast in 108 seconds and 5.77 GB of memory, or 33 minutes if you have no GPU. I installed it step by step, compared it against the same centre's operational physical model, and wrote down the two stumbles that nearly made me publish nonsense.#Models#GPU#WeatherArticle2026-08-15I put the same agent to fix real bugs with three local engines: 284B, 27B and 27B in two bitsThe DeepSeek Harness has only been tested against the official API. I wired it to three local engines and gave them real SWE-bench Verified instances. Two tie on fixes, one is eleven times faster, and the one with 284 billion parameters barely leaves the starting line.#Agents#Models#GPUTutorial2026-08-15Pocket TTS: I cloned my voice on a 2019 CPU, no GPU and no cloudA 209 MB voice model that runs on CPU, clones a voice from 25 seconds of reference audio and allows commercial use with attribution. Step-by-step install, metrics measured on a 2019 i5, and my honest take next to ElevenLabs and friends.#AI#AudioArticle2026-08-15Does Qwen3.8-27B fit in 16 GB of VRAM? I measured it, and found out my own benchmark was lyingEveryone repeats that Alibaba's new model needs about 15 GB in 4 bits. My GPU has 16 and the file weighs 16.5. I measured tokens per second, real VRAM and the CPU/GPU split across three context sizes, and the model ended up correcting me.#Models#GPU#Benchmarks

All writing