Facebook judged accounts by their friends' behavior so spammers could not game the rules

As Ars Technica reported in March 2020, Facebook sees billions of active users a month and only a small fraction are abusive: fake or compromised accounts, spammers and scammers. People running such accounts can see account data and learn what triggers bans, creating an arms race. Facebook's researchers moved from account data to metadata about an account's connections, such as how many posts a typical friend makes or how often friend requests go out, and combined these into a "deep entity" profile.

The assumption is that normal accounts connect with normal accounts, while abusers connect with bots and unusual accounts. That makes it hard for abusers to know what the algorithm uses and hard to change it. Classification is compute-intensive because it crawls the network graph for each user.

It is a two-tier system. A neural network trained on a large volume of low-quality labels is stopped one layer short of its binary decision, giving a 32-number vector. Decision trees trained on a smaller set of high-quality human labels, one per abuse type such as spammers or hijacked accounts, then make the final call.

Score an account from its neighbours' aggregate behavior, metadata that scammers cannot see or cheaply change, instead of its own features.

Neighbour behavior aggregated across a network graph is not visible to an account owner and is expensive to fake.

Using both the large noisy data set and the small high-quality human-labelled one gets scale and accuracy.

Separate decision trees per abuse type let each tier tune to a specific kind of abuser.

Blocked accounts feed retraining in the background, and the system compares metrics to decide when to replace the production model.

Against adversaries who study your classifier, base decisions on signals that are costly for them to forge, such as who their connections are.

According to the article, Facebook said the system had been in production for a while and blocked at least half a billion accounts per quarter, peaking above 2 billion in Q1 2019 (Facebook's figures, not independently verified). Ars notes that how to apply the technology, for example to misinformation accounts, is a policy question, and Facebook said it was also expanding human moderation.