An executable classifier reports 99% accuracy on its held-out set. Deployed, it misses obvious samples and raises alerts on perfectly legitimate internal binaries.
This is almost never a model problem. It is that the score measured something other than what you thought. Three biases recur, and none shows up in a confusion matrix — they live in the dataset, so the evaluation reproduces them faithfully instead of exposing them.
Bias 1: the corpus encodes provenance, not maliciousness
The most insidious, because the correlation it creates is entirely real — it just is not the one you meant to learn.
How does a corpus get built, in practice? Malicious samples come from sharing platforms, honeypots, customer submissions. Benign samples come from clean systems, vendor repositories, well-known software suites.
The mechanical result: almost all the malware is unsigned, and a large majority of the benign set is signed by a recognised publisher. The model learns, very efficiently, to read the signature. It reaches 99% because the signature is a nearly perfect predictor within that corpus.
In production the population changes. A locally compiled binary, a small company's internal tool, a utility from a minor vendor, a .NET compilation cache: all unsigned, all benign. The model treats them as malware, and the false-positive rate measured in-house bears no resemblance to the field.
Diagnosing it is not done on the headline metric but on a stratified evaluation. Split the evaluation 2 × 2 — benign/malicious × signed/unsigned — and look at performance inside each cell. If the model is excellent overall but mediocre among the unsigned, it did not learn maliciousness: it learned provenance.
The remedy is not to drop the signature features. Signing is a legitimate signal, and amputating it blinds the model to real information. The remedy is to break the correlation at the source, by collecting unsigned benign and signed malicious samples — that is, to pay the cost of a co-distributed corpus.
Bias 2: a random split measures memory, not generalisation
A random train_test_split is the default reflex. On malware it is wrong.
Malware arrives in families and campaigns: hundreds of variants of the same code, repacked, re-obfuscated, recompiled. A random split puts variants of the same family on both sides of the boundary. The model sees the family in training and meets it again at test time.
The resulting score answers "can you recognise a family you have already seen?" — not "can you detect something new?". The second question is the one that decides production usefulness.
The gap is not marginal. On a family classifier, moving from a random split to a group-disjoint one — all variants of a family on the same side — can drop a macro-F1 from around 0.7 to something near 0.2. The model did not change. Only the question changed.
That is not a reason to throw the model away: a recogniser of known families has real operational value. It is a reason not to call it a detector of unknown ones.
Bias 3: time
The third is the best documented and the most often ignored. Training on a chronological mixture and testing on the same mixture lets the model learn the future.
Evasion techniques evolve, packers change, campaigns succeed one another. An honest evaluation imposes a temporal boundary: train on everything before a date, test on what follows. The score falls. It becomes comparable to what deployment will produce.
The question to put to any metric
The three biases share a shape. The model learns a variable correlated with the label inside the corpus but which is not the phenomenon — and the evaluation, drawn from the same corpus, validates the correlation instead of challenging it.
Hence a single question, to be asked in front of every good score:
What property of my dataset would make this result false in production?
If no answer comes, it is not that none exists. It is that nobody looked.
In practice that becomes three requirements before believing a number:
- a group-disjoint split (family, campaign, publisher);
- a temporal boundary between training and test;
- a stratified evaluation along the suspicious axes — signing, packer, size,
format — rather than one average.
A single metric on a random split is not a measurement. It is an impression.
See also, for the same idea applied to tests rather than models: the bug that emptied every Android manifest — a test that passed because it was comparing itself to itself.