Multi-Armed Bandits for Pricing

A step-by-step guide to applying Multi-Armed Bandits to a surge pricing problem using different exploration strategies.

Libraries Used

This project uses a few key Python libraries. Here is a quick overview of their roles:

• numpy: Used for numerical operations and managing arrays.
• scipy: Used here for generating random variables from a Beta distribution for Thompson Sampling.

1. Define the Problem

• What the code does: It defines a `BanditPricingEnv` to simulate a pricing problem. The `demand_curves` represent the probability of a customer accepting a price.
• Behind the scenes: This is an example of a "multi-armed bandit" problem. Each "arm" is a different price point, and the "reward" is whether a customer buys at that price. The goal is to find the best price to maximize total rewards.
• Abstraction: The `BanditPricingEnv` class abstracts the complex, real-world dynamics of customer demand into a simple, interactive environment.

import numpy as np

class BanditPricingEnv:
    def __init__(self, demand_curves: np.ndarray):
        self.demand_curves = demand_curves
        self.n_actions = len(demand_curves)

    def step(self, action):
        p_buy = self.demand_curves[action]
        if np.random.rand() < p_buy:
            reward = 1.0
        else:
            reward = 0.0
        return reward

2. Implement the Agents

• What the code does: It defines three agents: EpsilonGreedyAgent, UCBAgent, and ThompsonSamplingAgent. Each agent has a `select_action` and an `update` method.
• Behind the scenes: These agents represent different strategies for balancing "exploration" (trying new prices to find better ones) and "exploitation" (sticking with the best-known price). Epsilon-greedy is the simplest, UCB is more sophisticated, and Thompson Sampling is a probabilistic approach.
• Abstraction: The agent classes abstract the complex decision-making logic into a simple, reusable object.

from scipy.stats import beta

class EpsilonGreedyAgent:
    def __init__(self, n_actions, epsilon=0.1):
        self.n_actions = n_actions
        self.epsilon = epsilon
        self.counts = np.zeros(n_actions)
        self.rewards = np.zeros(n_actions)

    def select_action(self):
        if np.random.rand() < self.epsilon:
            return np.random.randint(self.n_actions)
        return np.argmax(self.rewards / (self.counts + 1e-6))

    def update(self, action, reward):
        self.counts[action] += 1
        self.rewards[action] += reward

class UCBAgent:
    def __init__(self, n_actions):
        self.n_actions = n_actions
        self.counts = np.zeros(n_actions)
        self.rewards = np.zeros(n_actions)
        self.t = 0

    def select_action(self):
        self.t += 1
        if 0 in self.counts:
            return np.argmin(self.counts)
        ucb_values = self.rewards / self.counts + np.sqrt(2 * np.log(self.t) / self.counts)
        return np.argmax(ucb_values)

    def update(self, action, reward):
        self.counts[action] += 1
        self.rewards[action] += reward

class ThompsonSamplingAgent:
    def __init__(self, n_actions):
        self.n_actions = n_actions
        self.alphas = np.ones(n_actions)
        self.betas = np.ones(n_actions)

    def select_action(self):
        samples = [beta.rvs(self.alphas[i], self.betas[i]) for i in range(self.n_actions)]
        return np.argmax(samples)

    def update(self, action, reward):
        self.alphas[action] += reward
        self.betas[action] += (1 - reward)

3. Run the Experiment

• What the code does: It creates the environment and an agent, and then simulates the agent's interaction over a number of steps.
• Behind the scenes: For each time step, the agent chooses an action (price), receives a reward (buy or no buy), and then updates its internal state. The `total_rewards` variable tracks the cumulative performance of the agent.
• Abstraction: The simulation loop abstracts the process of an agent repeatedly interacting with an environment over time.

demand_curves = np.array([0.1, 0.3, 0.8, 0.4, 0.2])
env = BanditPricingEnv(demand_curves)
agent = EpsilonGreedyAgent(env.n_actions, epsilon=0.1)

n_steps = 1000
total_rewards = 0

for _ in range(n_steps):
    action = agent.select_action()
    reward = env.step(action)
    agent.update(action, reward)
    total_rewards += reward

print(f"Total rewards after {n_steps} steps: {total_rewards}")