Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising

@article{Jin2018RealTimeBW,
  title={Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising},
  author={Junqi Jin and Cheng-Ning Song and Han Li and Kun Gai and Jun Wang and Weinan Zhang},
  journal={Proceedings of the 27th ACM International Conference on Information and Knowledge Management},
  year={2018}
}
  • Junqi Jin, Cheng-Ning Song, Weinan Zhang
  • Published 27 February 2018
  • Computer Science
  • Proceedings of the 27th ACM International Conference on Information and Knowledge Management
Real-time advertising allows advertisers to bid for each impression for a visiting user. [] Key Method To deal with a large number of advertisers, we propose a clustering method and assign each cluster with a strategic bidding agent. A practical Distributed Coordinated Multi-Agent Bidding (DCMAB) has been proposed and implemented to balance the tradeoff between the competition and cooperation among advertisers. The empirical study on our industry-scaled real-world data has demonstrated the effectiveness of…

Figures and Tables from this paper

Real-Time Bidding with Soft Actor-Critic Reinforcement Learning in Display Advertising
TLDR
A novel algorithm for calculating the dynamic bid for each impression of the ad in order to maximize the advertiser’s goals is proposed, which takes into account settings of the advertising campaign, budget, the ad lifetime and other parameters.
A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising
TLDR
This paper investigates the competition and cooperation relation among auto-bidding agents, and proposes a temperature-regularized credit assignment to establish a mixed cooperative-competitive paradigm that outperforms several baseline methods in terms of social welfare and revenue.
Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search
TLDR
MACG sets up a carefully designed multi-objective optimization framework where different objectives of advertisers are incorporated and an efficient multi-agent evolutionary strategy for model optimization is designed.
Know Your Enemies and Know Yourself in the Real-Time Bidding Function Optimisation
TLDR
A novel Deep Attentive Survival Analysis model is introduced as the opponent model into the Mean Field Deep Deterministic Policy Gradients (DDPG) algorithm for each agent to learn the optimal bidding strategy and converge to the mean field equilibrium.
Online Causal Inference for Advertising in Real-Time Bidding Auctions*
Real-time bidding (RTB) systems, which leverage auctions to programmatically allocate user impressions to multiple competing advertisers, continue to enjoy widespread success in digital advertising.
Online Inference for Advertising Auctions
TLDR
Simulations show that not only the proposed method successfully accomplishes the advertiser's goals, but also does so at a much lower cost than more conventional experimentation policies aimed at performing causal inference.
Infer Your Enemies and Know Yourself, Learning in Real-Time Bidding with Partially Observable Opponents
TLDR
This paper introduces a novel Deep Attentive Survival Analysis model to infer the censored data in the second price auctions which outperforms start-of-the-art survival analysis and introduces the DASA model as the opponent model into the policy learning process for each agent and develops a mean field equilibrium analysis of the first price auctions.
MoTiAC: Multi-Objective Actor-Critics for Real-Time Bidding
TLDR
This work proposes a multi-objective reinforcement learning algorithm, named MoTiAC, for the problem of bidding optimization with various goals, which outperforms all state-of-the-art baselines and proves that Pareto optimality could be guaranteed.
A Reward Shaping Approach for Reserve Price Optimization using Deep Reinforcement Learning
TLDR
A novel reward shaping approach is developed to provide informative reward signal for the reinforcement learning agent and results show superior performance of the proposed method in terms of revenue compared with the baselines.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 37 REFERENCES
Real-Time Bidding by Reinforcement Learning in Display Advertising
TLDR
A Markov Decision Process framework is built for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment and the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks.
Improving Real-Time Bidding Using a Constrained Markov Decision Process
TLDR
A bidding function is proposed, which outperforms the state-of-the-art bidding functions in terms of the number of clicks when the budget limit is low and is modeled as a Constrained Markov Decision Process based reinforcement learning framework.
Optimal real-time bidding for display advertising
TLDR
The mathematical derivation suggests that optimal bidding strategies should try to bid more impressions rather than focus on a small set of high valued impressions because according to the current RTB market data, compared to the higher evaluated impressions, the lower evaluated ones are more cost effective and the chances of winning them are relatively higher.
An empirical study of reserve price optimisation in real-time bidding
TLDR
This paper empirically examines several commonly adopted algorithms for setting up a reserve price and suggests the proposed game theory based OneShot algorithm performed the best and the superiority is significant in most cases.
Real time bid optimization with smooth budget delivery in online advertising
TLDR
This paper presents an online approach to the smooth budget delivery while optimizing for the conversion performance by distributing the budget optimally across time in an adaptive manner.
LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions
We present LADDER, the first deep reinforcement learning agent that can successfully learn control policies for large-scale real-world problems directly from raw inputs composed of high-level
Bid optimizing and inventory scoring in targeted online advertising
TLDR
This paper presents a bid-optimization approach that is implemented in production at Media6Degrees for bidding on these advertising opportunities at an appropriate price and combines several supervised learning algorithms, as well as second price auction theory, to determine the correct price.
Real-time bidding for online advertising: measurement and analysis
TLDR
An empirical analysis and measurement of a production ad exchange is provided, observing that periodic patterns occur in various statistics including impressions, clicks, bids, and conversion rates, which suggest time-dependent models would be appropriate for capturing the repeated patterns in RTB.
Display Advertising with Real-Time Bidding (RTB) and Behavioural Targeting
TLDR
Topics covered include user response prediction, bid landscape forecasting, bidding algorithms, revenue optimization, statistical arbitrage, dynamic pricing, and ad fraud detection are an invaluable text for researchers and practitioners alike.
Smart Pacing for Effective Online Ad Campaign Optimization
TLDR
A smart pacing approach is proposed in which the delivery pace of each campaign is learned from both offline and online data to achieve smooth delivery and optimal performance goals and can effectively improve campaign performance and achieve delivery goals.
...
1
2
3
4
...