Go 1.27 brings portable SIMD: it ties with NumPy out of cache and loses inside it

I was building a voice search engine. Every recording goes through a speaker encoder and out comes a vector of 192 numbers representing the voice, not what it says. Finding someone means comparing their vector against all the others and keeping the closest ones.

With 346 thousand recordings that is 66 million multiplications per query. The practical question was what to write it in, and I had no idea. So I measured it, and this article came out of that measurement.

A note about this article’s numbers. Every figure below is measured with a harness that pins cores, interleaves the implementations and discards the run if the machine is not idle. I arrived at that harness after two earlier batches came out wrong, and that story is at the end, in a separate section, so good figures do not get mixed with discarded ones. No figure in the body comes from those batches.

What Go 1.27 brings

Go 1.27 shipped with an experimental simd package in the standard library. It gives portable, width-independent vector types: you write Float32s, LoadFloat32s and MulAdd, and the same source code uses whatever wide instructions the machine has.

It sits behind a flag. Without it the compiler does not even see it:

package probe
	imports simd: build constraints exclude all Go files in .../src/simd

With GOEXPERIMENT=simd it compiles. Asking the package itself what it had chosen produced the first surprise:

width 16 float32 = 512 bits · emulated false

Sixteen lanes, meaning AVX-512, on a Ryzen 7 7800X3D. The code does not mention AVX anywhere.

Plain loopAVX-512What the Intel chose

Go without vectorizing · 1 lane

Go does not auto-vectorize, so the loop moves one number per turn. It is the starting point everything else is measured against.

Ryzen 7 7800X3D · 16 lanes

The package chose 16 float32 lanes without the code mentioning AVX. It is the widest that exists on x86 today.

Core i7-9750H · 4 lanes

Four lanes, that is 128 bits, on a machine that has AVX2 and could use eight. Portable yes; optimal, not always.

How much it gains, and where it stops gaining

Before the real task I ran a synthetic benchmark with two operations and three sizes: one that fits in L1, one in L3 and one that fits nowhere. That choice is what decides what a benchmark measures. Only small arrays measure the compute unit; only enormous arrays measure memory.

1. 8 KB, lives in L115.2×2,281 ns against 150 ns.
2. 4 MB, lives in L311.4×1.17 ms against 102 µs.
3. 64 MB, fits in no cache4.4×18.8 ms against 4.29 ms. Here the memory bus rules, not the compute unit.

Ryzen 7 7800X3D, Go 1.27.0, median of five single-threaded runs. Both paths were verified to give the same result, including sizes 1, 7, 15, 16, 17, 1000 and 10,001 that do not fill a vector.

With saxpy, which is y = a·x + y, the staircase is the same but steeper: 22.4× in L1, 20.3× in L3 and 6.7× in RAM.

That drop is the benchmark’s most useful result. A measurement announcing “twenty-two times faster” without saying the array size is measuring the cache and calling it SIMD.

A necessary caveat: Go does not auto-vectorize, so the scalar loop is literally one element per iteration. Part of the advantage is going from one lane to sixteen, not credit due to the package’s design.

Why the result depends on size

Before the tables it helps to have the reason there are two of them at hand. The advantage of vectorizing is not a property of the code: it is a property of where the array falls in the memory hierarchy.

1. L132 KB per core22.4×8 KB
2. L21 MB per coresin datonot measured
3. L396 MB shared20.3×4 MB · 31 MB
4. RAM32 GB DDR56.7×64 MB · 253 MB

The array fits whole in the cache closest to the core. Here the processor never waits for data, so the only thing that decides is how many operations fit per instruction: sixteen lanes against one.

This is where SIMD delivers everything it can.

The middle floor. I did not measure it separately: the benchmark sizes fall to one side or the other, and publishing an interpolation as if it were a measurement is exactly what this blog does not do.

No figure of my own. It gets said, not filled in.

The 7800X3D’s big cache, the one that makes this processor special. The 40,000-embedding corpus weighs 31 MB and fits whole. Computation still rules, and that is why OpenBLAS takes 2.4 times over the Go package here: when the bottleneck is the arithmetic unit, code quality decides.

The small corpus fits. The better-tuned implementation wins.

The array no longer fits in any cache and has to come over the bus on every pass. The processor spends most of its time waiting, so it barely matters how the loop is written. Here Go’s portable package and OpenBLAS’s tuned assembly tie.

Memory rules. The tuning stops showing.

With that in view, the two measurements that follow stop looking contradictory.

Large corpus: a tie

1. NumPy BLAS, 8 threads3.24 msSpread 3.2%, the most stable measurement of all.
2. Go SIMD, 8 goroutines3.29 msSpread 16%. The difference from NumPy fits entirely inside the noise.
3. Go SIMD, one thread9.54 msSpread 14%.
4. NumPy BLAS, one thread9.76 msSpread 10.4%. 2% slower than Go, which is to say a tie.
5. Go scalar61.2 msThe same computation without vectorizing.

253 MB of corpus against 96 MB of L3 cache: it does not fit. Ryzen 7800X3D with cores 0-7 pinned, seven interleaved rounds, median. Python 3.12.10, NumPy 2.5.2 on scipy-openblas.

It is a tie, not a win. Go comes out 2% faster than NumPy on one thread, 9.54 against 9.76, but that measurement’s spread is 14%: the difference fits entirely inside the noise and another run could invert it. Same with eight threads, 3.29 against 3.24.

That is already a result. Twenty lines of a portable package fresh out of the oven match OpenBLAS, which carries years of assembly hand-tuned per microarchitecture.

The reason for the tie is in another number: parallelizing gave 2.9 times in Go and 3.0 in NumPy, across eight physical cores. If the work were pure computation it would give close to eight, and both stopping at three is the clearest sign that the ceiling is not the code. It does not deliver because the corpus weighs 253 MB and fits in no cache, so both programs end up waiting on memory. When two implementations hit that ceiling, code quality stops mattering.

Small corpus: NumPy wins

I swapped the random vectors for real embeddings: 40,000 Spanish recordings from Common Voice run through ECAPA-TDNN, at 292 clips per second on the card.

The vectors are real and it shows: mean norm 1.000, mean similarity of 0.563 between clips of the same speaker against 0.119 between different speakers. The most similar pair of all, at 0.921, did turn out to be the same speaker.

1. NumPy BLAS, 8 threads0.060 ms3.2 times faster than Go on the same eight cores.
2. Go SIMD, 8 goroutines0.191 msSpread 45%. At this scale I suspect goroutine startup itself weighs, but I did not measure it.
3. NumPy BLAS, one thread0.393 ms2.4 times faster than Go. This time, outside the noise.
4. Go SIMD, one thread0.943 ms
5. Go scalar4.62 msSpread 78%, the worst of all.

31 MB of corpus against 96 MB of L3. Same harness: cores 0-7 pinned, nine interleaved rounds, median.

When the corpus fits in cache, the bottleneck goes back to being computation, and there OpenBLAS’s years of tuning show: 2.4 times on one thread and 3.2 on eight.

The control that was needed

Between the two tables two things changed at once: the corpus size and the nature of the data. Attributing the reversal to cache without separating them would have been a conclusion with nothing under it, so I ran the control: 40,000 random vectors, exactly the same size as the real ones.

1. NumPy BLAS, random0.398 msAgainst 0.393 ms with real embeddings: the same figure.
2. Go SIMD, random1.009 msAgainst 0.943 ms with real ones: 7% difference, which is the size of the spread itself.

40,000 random 192-dimension vectors, 31 MB. Nine interleaved rounds.

NumPy’s advantage holds at 2.5 times with random data and 2.4 with real, and each implementation’s two figures land inside the other’s noise. The nature of the data does not matter; the size does. The reversal between the two tables is about cache, not embeddings.

The second machine

Everything above is one processor. I repeated it on the other one I have, a 2019 Intel i7-9750H with AVX2 and no AVX-512:

corpus: 346059 vectors of 192 · SIMD width 4 (emulated false)

Four lanes, meaning 128 bits, on a machine that has AVX2 and could use eight. The portable package worked, but it did not choose the widest available. SIMD’s advantage over scalar fell to 1.8 times, against 6.4 on the Ryzen with that same corpus.

Part of that drop is four lanes against sixteen, and part is that this is a 2019 machine with slower memory that was also at load 4.4 during the measurement. Those numbers are a floor, not a ceiling, and they did not go through the harness: I did not pin cores or interleave, so they are good for the chosen vector width and little else. I did not measure NumPy on that machine.

What is established is that the same source code compiled and ran unchanged on Linux and macOS, choosing different widths. Portable yes; optimal, not always.

What I take away

The package delivers and is comfortable. Twenty lines, no assembly, no per-architecture build tags, and in the memory-bound regime it matches a library with years of manual tuning. For Go code that today writes numeric loops by hand, it is a clear improvement.

It does not replace NumPy for data work. When the corpus fits in cache, BLAS is 2.4 times ahead on one thread and 3.2 on eight, and it parallelizes without being asked.

The number that decides is not the speedup but where the data lives. The same comparison gives a tie or a defeat depending on whether the corpus fits in the third-level cache.

When I would use it and when not

I would use it in an existing Go service doing numeric work in loops, where bringing in Python would be absurd, and for processing audio or images element by element. I would not use it to replace NumPy for data work, nor counting on it to choose the maximum width, nor in production while it remains behind an experimental flag.

How I arrived at that harness

This section contains none of the article’s figures. It is here because the two ways I got it wrong are easier to repeat than to detect.

The first time I compared eight cores against one. The initial reading gave NumPy twice Go’s speed, and I was on the verge of writing it that way. OpenBLAS spreads the work across every core without announcing it, while my Go program ran on one thread. Setting OMP_NUM_THREADS and OPENBLAS_NUM_THREADS to 1 made NumPy three times slower and flipped the comparison.

The second time I measured a busy machine without knowing. With the above corrected, I drew conclusions and took them as good. Repeating them out of habit, the same numbers came out ten times worse. The machine had thirteen foreign processes at 100% CPU: the first batch had been taken with the machine idle and the check batch with it saturated, and neither of them knew.

That is when I wrote the harness. It pins the cores with taskset to the eight physical ones. It interleaves the implementations round by round instead of measuring all of A and then all of B, so that a degradation halfway through spreads the damage rather than punishing the second one. It checks the load before and after and aborts if it exceeds 2.0; it does not demand absolute stillness, and in the large-corpus run it went from 0.50 to 0.61 and was accepted. And it reports median and spread instead of the best time, because the best measures the best case and hides precisely the instability that matters here.

Its first attempt gave NumPy 0.000 ms. That was not a record: perf_counter returns seconds and I was printing them as milliseconds. A suspicious zero is always a units bug before it is a marvel.

What this measurement does not prove

Two home machines, not a server bench, and only one of the two went through the harness. One embedding model, one vector dimension and two corpus sizes. My Go implementation walks vector by vector while OpenBLAS solves the whole matrix-vector multiplication with blocking and unrolling: I am not measuring the package’s ceiling but a naive version against an expert one, and that is precisely the scenario for anyone picking it up fresh.

I also did not measure whether a C compiler would auto-vectorize the scalar loop. I assumed it in a draft of this article and removed it when I could not back it up: for floating-point reductions, compilers usually need explicit permission to reassociate.

The exact numbers are this hardware’s. The shape of the curve repeated on both machines, with synthetic and real data, and that is the only thing I would call general.

Sources

• Go 1.27 release notes, section “New experimental simd package”, where it is declared experimental and enabled with GOEXPERIMENT=simd.
• simd package documentation. The full API, including the Emulated method that says whether there is hardware behind it or not.
• OpenBLAS, the implementation NumPy competes with here, with per-microarchitecture assembly kernels.
• Common Voice in Spanish, where the 40,000 recordings came from.
• ECAPA-TDNN on SpeechBrain, the encoder that produces the 192-dimension vectors.