Hard rules against Random Cut Forest on the same stream: the model adds 13 points of recall and costs 7 times the throughput
I ran Apache Flink with Kafka and measured three deterministic rules against Amazon's Random Cut Forest over exactly the same payment stream, on a four-core box with no GPU. The rules reach 0.814 recall at 173 thousand events per second; the model reaches 0.943 but drops the pipeline to 23 thousand; together they hit 0.982.
Risk rules age badly. Amount over a threshold, so many operations in so many seconds, a country different from the previous one: they all work, they are all easy to defend in an audit, and they all fail in exactly the same way, against the pattern nobody wrote down.
The usual answer is to add an anomaly model. Amazon published its own, Random Cut Forest, and offered it for years as a SQL function in Kinesis. That function is gone; the library is not. It is open source and it drops into any Java process.
So I set up the honest experiment: the same payment stream, the same injected and labeled anomalies, and three ways of looking at it. Rules alone, model alone, and both in one pipeline.
The model adds 12.9 points of recall over the rules and drops throughput from 173,297 to 23,533 events per second. Together they reach 0.982, and 17% of the anomalies are seen by the model alone.
What I set up
Apache Flink 2.3.0 and Kafka 4.3.1 in containers, on a server with 4 virtual cores, 16 GB of RAM and no GPU. The generator produces synthetic card transactions with three kinds of injected, labeled anomaly: high amount, burst of operations and impossible country change.
The three deterministic rules, with per-card state:
- amount over 500,000 pesos,
- four or more operations within 60 seconds,
- a country different from the previous operation within 10 minutes.
The model is ThresholdedRandomCutForest 4.4.0, with 30 trees, a sample of 256 and three variables per event: amount, time since that card’s previous operation, and whether the country changed.
First, make sure the rules do not lie
Before comparing against anything I wrote the same three rules twice: once in the Flink job and once in twenty lines of Python. The Python version is the referee. If the two do not produce exactly the same set of alerts over the same file, the problem is mine and not the framework’s.
Over 1,016,212 events the job produced 113,800 alerts: the same 113,800, not one more, not one less. Only then did I start measuring.
Installation
docker compose up -d # kafka 4.3.1 in KRaft mode, jobmanager and taskmanager
docker run --rm -v $PWD/job:/w -w /w maven:3.9-eclipse-temurin-17 mvn -B package
docker exec srk-jm flink run -d -p 1 -c cl.efg.streamrisk.HybridJob /jars/stream-risk-0.1.0.jarWhat broke
The Kafka connector does not compile on its own. flink-connector-kafka 5.0.0-2.2 uses DeliveryGuarantee, which lives in flink-connector-base, and that artifact does not arrive as a transitive dependency: the connector’s own POM declares it with provided scope. The error is package org.apache.flink.connector.base does not exist and it goes away by adding the dependency by hand at the Flink version, also as provided, because at runtime the distribution supplies it.
The order of flink run arguments decides which job runs. I wrote flink run -d -p 1 /jars/app.jar -c cl.efg.streamrisk.RcfJob and Flink accepted it without a single warning: everything after the jar is an argument to the program, so it ran the manifest’s main class, which was the rules. I spent several minutes analysing results from the wrong job until I looked at the name in the web interface. The class goes before the jar.
Kafka 4 moved the tools. kafka.tools.GetOffsetShell no longer exists and returns ClassNotFoundException. The replacement is the kafka-get-offsets.sh script.
The forest does not fit in Flink state. Random Cut Forest objects are not Flink-serializable, so they cannot be stored in a ValueState. I ended up holding them in a map inside the operator. In this setup that makes no difference, because I did not enable checkpointing either, so a restart wipes rules and model alike; the difference shows up the moment checkpointing is on, because the rules’ state comes back on its own and the model’s does not. For production you would serialize the forest with the mapper the library itself provides and pay that cost per event, or accept the warm-up period after every restart.
The metrics, measured by me
A 4 vCPU server, 16 GB of RAM, no GPU. The same input file for everything: 101,668 events, 2,000 cards, 2,506 labeled anomalies.
| Metric | Rules | Random Cut Forest | Both |
|---|---|---|---|
| Recall | 0.814 | 0.943 | 0.982 |
| Precision | 0.182 | 0.339 | 0.162 |
| Flagged events | 11,198 | 6,970 | 15,176 |
| Operator cost, median | 1.8 µs | 21 µs | 23 µs |
| Operator cost, mean | 3.4 µs | 41.5 µs | 44.9 µs |
| Operator cost, 99th percentile | 14.5 µs | 528 µs | 540 µs |
| Throughput at parallelism 1 | 173,297 ev/s | — | 23,533 ev/s |
Three thresholds with per-card state. They see every high amount and every country jump with a recent previous operation, and miss 455 burst events: the ones that happen before the fourth operation, when a counting rule has nothing to fire on.
A single forest for the whole stream, fed with per-card variables. It recovers 320 of those events, almost all of them the first or second of their burst, and raises half the false positives the rules do.
The union of both views reaches 0.982 recall. The price is the sum of the false positives: 12,714, or 12.8% of the normal traffic.
End-to-end latency, from the instant the event is born to the instant the decision leaves, with the full pipeline:
| Input rate | Median | 95th percentile | 99th percentile | Max |
|---|---|---|---|---|
| 5,000 ev/s | 47 ms | 76 ms | 86 ms | 134 ms |
| 20,000 ev/s, last 30 s of a 2-minute run | 5,834 ms | 7,894 ms | 8,137 ms | 8,205 ms |
The second row is not a stable latency, and that is the finding. At 20,000 events per second the pipeline moves faster than the input, but barely: the run began 19 seconds behind and, two minutes later, decisions were still coming out 4.6 seconds after the event. The lag drains at about 0.12 seconds per second, so digesting that start takes more than two and a half minutes. The two headroom figures are not the same: against the 23,533 events per second measured while draining, an input of 20,000 leaves 15%; measured live from how fast the lag drains, the pipeline does about 22,400 and the real headroom is 12%, because the same box also runs the generator, the producer and the broker. At 12%, nothing is forgiven.
21% of the measured capacity. The pipeline keeps up.
85% of capacity. Last 30 s of a 2-minute run: the lag drains 0.12 s per second and is not digested yet.
Measured pipeline capacity: 23,533 ev/s
TaskManager resident memory: 672 MiB idle, 1.607 GiB running the job with one forest, 2.02 GiB with two thousand. That first jump is mostly the virtual machine’s heap growing under load, not the model. What I do attribute to the model is the difference between the last two figures: about 217 KB per forest, and even that is approximate because there is a garbage collector in the middle.
How to read this
The model sees what the rules cannot see yet. Of the 2,506 anomalies, 422 were caught by the forest alone, and looking at which ones gives the exact reason: 283 are the first operation of a burst and 135 the second. The rule counts four operations in 60 seconds, so by construction it cannot fire before the fourth; the model flags the first one because the gap since the previous operation is a hundred times shorter than usual for that card: a median of 125 milliseconds against 39,554 in ordinary traffic. The rule is not worse, it arrives three operations later. It cuts the other way too: 44 anomalies were seen by nobody.
Each one covers a different hole. The country jump is the clear case: the rule detects 440 out of 451 and the forest 444, but only 4 of those are seen by the model alone, so removing the rule changes nothing and removing the model changes nothing either. Where the split does matter is bursts: over 417 bursts of five operations each, the rule sees 1,213 of the 1,668 labeled events and the model 1,533, because it gets to the start of each one sooner.
The same model performs very differently depending on what you feed it. With the three variables computed globally, the forest reached 0.769 recall. With the same three variables computed per card in the previous stage of the pipeline, it went to 0.943. Same trees, same seed, same parameters: 17.4 points of recall came out of where the gap between operations is computed.
One model per customer sounds right and does not work. I tried 2,000 forests, one per card. They flagged 9 events in total, a recall of 0.001, and on top of that ran 3.8 times slower per event. The cause is a specific number: the library emits no score until it has seen 64 points, a value I confirmed by reading getOutputAfter() on a forest built with the job’s parameters. The median number of operations per card in my file is 51, and only 4.8% of the cards reach 64. Ninety-five percent of the models never got to speak.
- Required warm-up
- 64 operations
- Median per card
- 51 operations
- Cards that get there
- 96 of 2,000 · 4.8%
Same trees, same seed, same variables. The only thing that changes is how many operations each model sees.
Three stateful thresholds. The ceiling of the pipeline with no model.
The forest costs 7.36 times the throughput.
Four cores available, three watching: the forest stage depends on a single key, and the topic has a single partition.
Bars are to real scale: the rules bar is 7.36 times the one with the model inside.
The global forest kills parallelism. Being a single model, its stage depends on a single key, so it runs in one thread no matter what. Going from parallelism 1 to 4 moved throughput from 23,533 to 23,972 events per second: 1.9%. To be fair, the model is not the only ceiling here: the topic has a single partition, so reading does not spread either. Even fixing that, the forest stage would still be one thread, and with 4 cores available, three watch.
What this measurement does not prove
The data is synthetic and the anomalies are the ones I injected, so the numbers measure the ability to separate known signals, not real fraud.
The low precision has a concrete cause worth facing, because I first blamed the labeling and the data says otherwise. The velocity rule produces 8,755 false positives, and only 437 of those have an injected burst from the same card in the previous 60 seconds: the other 8,318 are normal traffic. With 2,000 cards spread over 2,917 seconds, each card gets about 1.1 operations per minute, and “four in 60 seconds” happens by chance at that density. The threshold is miscalibrated for the traffic I generated myself. That does not invalidate the comparison, because all three columns eat exactly the same file, but it means the precision column measures my threshold and not the method: what compares here is recall.
The model has an artifact of its own: 323 of its alerts are the first event of a card, where the gap since the previous operation is zero because there is no previous operation. That zero sits orders of magnitude away from the rest of the distribution and the forest reads it as strangeness. Encoding “no history” as its own thing, instead of as a zero, removes that group.
I left it running on real traffic
An experiment on synthetic data ends when the file ends. This one does not: I left the same rules and the same forest watching this site’s real traffic, the server’s authentications and public mentions of the name, and the result publishes itself on the lab board.
The difference with everything above is that there are no labels there. Nobody marks which request was a probe, so recall cannot be computed. What can be counted, and what the board counts, is how often each rule fires and how often the model flags something no rule named. That number is the one that decides whether the forest earns its place outside the lab.
The rules watching the site are not the card ones, because the traffic is different: a path nobody browses by hand (/.env, /wp-login.php, /.git/config), a burst of 120 requests a minute from one subnet, three ssh failures in ten minutes, a 5xx on a site that is static, and a source that had never named the domain. The forest, on the other hand, is the same one, with three variables: response size, gap since that key’s previous event, and whether the key is new.
Two things I learned in the first hours, useful to anyone building this. First: my own tests counted as attacks. Asking for /.env with curl from the server itself enters through the docker bridge, so it arrives with the gateway address and the pipeline flags it as a third party; traffic originating on the box had to be discarded. Second: addresses arrive clipped to /24 because Caddy masks them before writing its log, so the unit of all this is the visitor’s subnet, never their exact address.
The board rebuilds every six hours, which is late for something happening now. That is what the live stream is for: a page the server holds open and keeps writing, one row per event the moment the pipeline decides. No JavaScript, no WebSocket and no EventSource: an HTTP/2 response that never closes, with flush_interval -1 in Caddy so nothing buffers it on the way. The same trick the GEO detector’s report uses on this site.
One rule can never fire in this setup, and I would rather say it than leave the zero unexplained: brute force. Port 22 is not open to the world, it only listens on the private network, so the only one authenticating there is me. That feed is a baseline for the model, not a signal.
My take
The good:
- Random Cut Forest works outside AWS with no paperwork: two jars adding up to 421 KB, Apache 2.0 license, and 21 microseconds per event on a cheap server.
- Flink carries the model without breaking a sweat. At 5,000 events per second, 86 milliseconds at the 99th percentile is plenty to decide on a transaction before authorising it.
- The combination is where the real value sits: 0.982 recall, with the explainable rules intact for the audit and the model covering the gap.
The bad:
- The model does not survive a restart. While the forest lives outside Flink state, every deployment starts blind, and turning checkpointing on saves the rules but not the forest.
- It does not scale horizontally while it is a single forest. To go past 23 thousand events per second you have to split the model per segment, and then the warm-up problem comes back.
- The automatic threshold the library ships is too generous: 6,970 alerts over 101,668 events is 6.9% of the stream, impossible to review by hand.
- At 20,000 events per second the real headroom measured live is 12%, and with that any few-second hiccup takes minutes to digest.
Conclusion: the model does not replace the rules, it arrives before them: 418 of the 422 anomalies only it detects are the first two operations of a burst, exactly where a counting rule has nothing to fire on yet.
When I would use it
- In a stream of up to about 20,000 events per second where mature rules already exist and the new pattern none of them covers shows up. That is where the global forest earns its cost.
- I would not deploy it alone, without rules, in anything that has to be explained to a regulator: an anomaly score is not a justification.
- I would not build a model per customer without checking first how many events each one has. Under a few hundred per entity, it is memory thrown away.
The whole setup is four pieces: a docker-compose with Kafka and Flink, an event generator with a fixed seed, the twenty-line Python referee and three Java classes. The exact parameters of every run are in the tables above, which is what it takes to repeat it.
Primary sources, all checked against the exact version I ran:
- Apache Flink 2.3 release notes and the 2.3.0 artifact in the Apache archive.
- Random Cut Forest 4.4.0-java, Apache 2.0 license.
- flink-connector-kafka 5.0.0-2.2 on Maven Central, the one that requires declaring
flink-connector-baseby hand. - Apache Kafka 4.3.1 release notes.
Comments
No comments yet. The first one is yours.