Google researchers traced deep nets' easy fooling to high-dimensional linearity

By 2014, state-of-the-art neural networks had a known but unexplained failure: change an image by a tiny, deliberately worst-case perturbation — often too small for a human eye to see — and the network classifies it wrong with high confidence. The same tweaked image tended to fool many different architectures trained on different data, which suggested the flaw sat in the training recipe itself rather than in any single network.

Early explanations blamed the supposed extreme nonlinearity of deep networks, plus overfitting, yet standard fixes such as dropout, pretraining and model averaging did not remove the vulnerability. Goodfellow, Shlens and Szegedy tested the opposite assumption: the architectures that train well (ReLU, maxout, LSTM) are deliberately almost linear, and in a high-dimensional input space linearity alone suffices. With hundreds of thousands of pixels, each nudged by less than one bit of image precision, the small terms in a dot product add up to one large change in the network's activation.

That view yielded a fast attack: linearize the cost around the input and perturb it by the sign of the gradient, which costs a single backpropagation pass. On MNIST the method drove a shallow softmax model to 99.9% error and a maxout network to 89.4% misclassification, both at high confidence; on ImageNet a perturbation of just 0.007 per pixel turned a panda into a gibbon. Cheap generation also changed the defense: adversarial training on such examples gave regularization beyond dropout alone and reduced test-set error.

The work demystified the failure for working engineers as well. In late December 2015, Recurse Center's Codewords published Julia Evans' illustrated walkthrough reproducing the attack on GoogLeNet from a Dockerized notebook — a black image became a paper towel and a panda became a vulture — and the Hacker News thread drew 268 points and 66 comments, with readers noting the paper still applied to later waves of neural networks.

Assume linearity: perturb along the gradient's sign, then train on those attacks

Earlier attempts blamed nonlinearity and overfitting; assuming the opposite made adversarial examples predictable instead of mysterious

The mechanism scales with dimension: perturbations below a sensor's precision still sum across hundreds of thousands of pixels into a decisive output change

One backpropagation pass per attack made adversarial training practical, converting a failure mode into a regularizer that beat dropout alone

It explained transfer: any model that behaves linearly in high dimensions shares the blind spot, so one example fools many architectures and training sets

When a model fails in a way that looks mysterious, test the cheapest explanation first — if you can generate the failure cheaply, you can also train against it cheaply.

The first version appeared on arXiv on 20 December 2014 and was revised in March 2015; its panda-to-gibbon image at a perturbation of 0.007 became the canonical picture of an adversarial example. A year later, Julia Evans' Codewords walkthrough let readers reproduce the trick on a GPU-less laptop — a panda became a vulture, a black screen a paper towel — and drew 268 points and 66 comments on Hacker News, where one reader called the paper still largely applicable to later waves of neural networks.