The encyclopedia · Engineering & Operations · Technical decision · 2018
Alibaba built a simulator of Taobao so its AI could train without real users.
Taobao built Virtual Taobao, a learned replica of its marketplace, so reinforcement learning could experiment on customers who did not exist.
Alibaba · Taobao
the move
Reinforcement learning learns by taking actions and observing outcomes, but on a huge retail site an action is a customer you hope to satisfy and a bad experiment is a sale you lose. Sampling enough real trials to train a policy is expensive and risky.
The Taobao team built Virtual Taobao, a simulator learned from hundreds of millions of genuine customer records, using generative adversarial training to match real behavior and multi-agent imitation to capture how buyers respond. Policies for product search were then trained inside this replica at no live cost.
Because the simulator captured real market behavior, the trained policy transferred to the live site and outperformed the supervised approach it replaced, turning an expensive real-world learning problem into a cheap one solved offline.
why it works
- Simulation lets RL take actions that would be too costly to take live
- The replica is learned from real records, so behavior carries over
- An action-norm constraint stops the policy overfitting the fake world
- Only the strongest imagined actions are ever tried on real users
what transfers
When a system improves by trial and error, trials are the scarce resource; build a cheap stand-in for the real cost and spend your live budget only on what the replica says will work.
what came after
Virtual Taobao became a reference for using learned simulators to train decision-making in high-cost online environments, influencing later work on RL for search, recommendation and pricing where live experimentation is expensive.
references
spotted an error? The archive wants to know.