Distributed Reinforcement Studying for Scalable Excessive-Efficiency Coverage Optimization

on Actual-World Issues is Laborious

Reinforcement studying seems to be easy in managed settings: well-defined states, dense rewards, stationary dynamics, limitless simulation. Most benchmark outcomes are produced underneath these assumptions. The actual world violates practically all of them.

Observations are partial and noisy, rewards are delayed or ambiguous, environments drift over time, information assortment is gradual and costly, and errors carry actual price. Insurance policies should function underneath security constraints, restricted exploration, and non-stationary distributions. Off-policy information accumulates bias. Debugging is opaque. Small modeling errors compound into unstable conduct.

Once more, reinforcement studying on actual world issues is actually laborious.

Outdoors of managed simulators like Atari which dwell in academia, there’s little or no sensible steering on find out how to design, practice, or debug. Take away the assumptions that make benchmarks tractable and what stays is an issue area that appears close to not possible to truly clear up.

However, then you have got these examples, and also you regain hope:

1. OpenAI 5 defeated the reigning world champions in Dota 2 in full 5v5 matches. Educated utilizing deep reinforcement studying.
2. DeepMind’s AlphaStar achieved Grandmaster rank in StarCraft II, surpassing 99.8% of human gamers and constantly defeating skilled rivals. Educated utilizing deep reinforcement studying.
3. Boston Dynamic’s Atlas trains a 450M parameter Diffusion Transformer-based structure utilizing a mix of actual world and simulated information. Educated utilizing deep reinforcement studying.

On this article, I’m going to introduce sensible, real-world approaches for coaching reinforcement studying brokers with parallelism, using many, if not the very same, strategies that energy right now’s superhuman AI methods. This can be a deliberate choice of educational strategies + hard-won expertise gained from constructing brokers which work on stochastic, nonstationary domains.

For those who intend on approaching a real-world downside by merely making use of an untuned benchmark from an RL library on a single machine, you’ll probably fail.

One should perceive the next:

1. Reframing the issue in order that it matches throughout the framework of RL concept
2. The strategies for coverage optimization which truly carry out exterior of academia
3. The nuances of “scale” with regard to reinforcement studying

Let’s start.

Stipulations

You probably have by no means approached reinforcement studying earlier than, making an attempt to construct a superhuman AI—or perhaps a midway first rate agent—is like making an attempt to show a cat to juggle flaming torches: it largely ignores you, sometimes units one thing on fireplace, and someway you’re nonetheless anticipated to name it “progress.” You ought to be properly versed within the following topics:

1. Markov Decision Processes (MDPs) and Partially Observable Markov Resolution Processes (POMDPs): these present the mathematical basis for the way fashionable AI brokers work together with the world
2. Policy Optimization (otherwise known as Mirror Learning) Particulars as to how a neural community approximates an optimum coverage utilizing gradient ascent 
3. Comply with as much as 2) Actor Critic Methods and Proximal Policy Optimization (PPO), that are two broadly used strategies for coverage optimization

Every of those requires a while to completely perceive and digest. Sadly, RL is a tough downside area, sufficient in order that merely scaling up is not going to clear up elementary misunderstandings or misapplications of the prerequisite steps as is usually the case in conventional deep studying.

---

An actual-world reinforcement studying downside

To supply a coherent real-world instance, we use a simplified self-driving simulation because the optimization process. I say “simplified” as the precise particulars are much less essential to the article’s goal. Nonetheless, for actual world RL, guarantee that you’ve got a full understanding of the surroundings, inputs, outputs and the way the reward is definitely generated. This understanding will assist you body your actual world downside into the area of MDPs.

Our simulator procedurally generates stochastic driving situations, together with pedestrians, different automobiles, and ranging terrain and highway circumstances which have been modeled from recorded driving information. Every situation is segmented right into a variable-length episode.

Though many real-world issues aren’t true Markov Resolution Processes, they’re sometimes augmented in order that the efficient state is roughly Markov, permitting customary RL convergence ensures to carry roughly in follow.

States
The agent observes digicam and LiDAR inputs together with alerts corresponding to car velocity and orientation. Further options could embrace the positions of close by automobiles and pedestrians. These observations are encoded as a number of tensors, optionally stacked over time to supply short-term historical past.

Actions
The motion area consists of steady car controls (steering, throttle, brake) and non-compulsory discrete controls (e.g., gear choice, flip alerts). Every motion is represented as a multidimensional vector specifying the management instructions utilized at every timestep.

Rewards
The reward encourages protected, environment friendly, and goal-directed driving. It combines a number of targets Oi, together with optimistic phrases for progress towards the vacation spot and penalties for collisions, visitors violations, or unstable maneuvers. The per-timestep reward is a weighted sum:

We’ve constructed our simulation surroundings to suit throughout the 4 tuple interface popularized by Brockman et al., OpenAI Health club, 2016

env = DrivingEnv()
agent = Agent()

for episode in vary(N):
   # obs is a multidimensional tensor representing the state
   obs = env.reset()
   finished = false

   whereas not finished:
       # act is the appliance of our present coverage π
       # π(obs) returns a multidimensional motion
       motion = agent.act(obs)
       # we ship the motion to the surroundings to obtain
       # the subsequent step and reward till full
       next_obs, reward, finished, data = env.step(motion)
       obs = next_obs

The surroundings itself must be simply parallelized, such that considered one of many actors can concurrently apply their very own copy of the coverage with out the necessity for advanced interactions or synchronizations between brokers. This API, developed by OpenAI and used of their gymnasium environments has turn out to be the defacto customary.

If you’re constructing your personal surroundings, it will be worthwhile to construct to this interface, because it simplifies many issues.

Agent

We use a deep actor–critic agent, following the strategy popularized in DeepMind’s A3C paper (Mnih et al., 2016). Pseudocode for our agent is beneath:

class Agent:
   def __init__(self, state_dim, action_dim):

       # --- Actor ---
       self.actor = Sequential(
           Linear(state_dim, 128),
           ReLU(),
           Linear(128, 128),
           ReLU(),
           Linear(128, action_dim)
       )

       # --- Critic ---
       self.critic = Sequential(
           Linear(state_dim, 128),
           ReLU(),
           Linear(128, 128),
           ReLU(),
           Linear(128, 1)
       )

   def _dist(self, state):
       logits = self.actor(state)               
       return Categorical(logits=logits)

   def act(self, state):
       """
       Returns:
           motion
           log_prob (conduct coverage)
           worth
       """
       dist = self._dist(state)

       motion = dist.pattern()
       log_prob = dist.log_prob(motion)
       worth = self.critic(state)

       return motion, log_prob, worth

   def log_prob(self, states, actions):
       dist = self._dist(states)
       return dist.log_prob(actions)

   def entropy(self, states):
       return self._dist(states).entropy()

   def worth(self, state):
       return self.critic(state)

   def replace(self, state_dict):
       self.actor.load_state_dict(state_dict['actor'])
       self.critic.load_state_dict(state_dict['critic'])

You could be a bit puzzled by the extra strategies. Extra clarification to comply with.

Crucial observe: Poorly chosen architectures can simply derail coaching. Be sure you perceive the motion area and confirm that your community’s enter, hidden, and output layers are appropriately sized and use appropriate activations.

Coverage Optimization

With a purpose to replace the agent, we comply with the Proximal Coverage Optimization (PPO) framework (Schulman et al., 2017), which makes use of the clipped surrogate goal to replace the actor in a secure method whereas concurrently updating the critic. This enables the agent to enhance its coverage steadily based mostly on its collected expertise whereas conserving updates inside a belief area, stopping massive, destabilizing coverage adjustments.

Notice: PPO is without doubt one of the most generally used coverage optimization strategies, used to develop each OpenAI 5, Alphastar and lots of different actual world robotic management methods

The agent first interacts with the surroundings, recording its actions, the rewards it receives, and its personal worth estimates. This sequence of expertise is usually referred to as a rollout or, within the literature, a trajectory. The expertise might be collected to the top of the episode, or extra generally, earlier than the episode ends for a set variety of steps. That is particularly helpful in infinite horizon issues with no predefined begin or end, because it permits for equal sized expertise batches from every actor. 

Here’s a pattern rollout buffer. Nonetheless you select to design your buffer, It’s essential that this rollout buffer be serializable in order that it may be despatched over the community.

class Rollout:
   def __init__(self):
       self.states = []
       self.actions = []
       # retailer logprob of motion!
       self.logprobs = []
       self.rewards = []
       self.values = []
       self.dones = []

   # Add a single timestep's expertise
   def add(self, state, motion, logprob, reward, worth, finished):
       self.states.append(state)
       self.actions.append(motion)
       self.logprobs.append(logprob)
       self.rewards.append(reward)
       self.values.append(worth)
       self.dones.append(finished)
   # Clear buffer after updates
   def reset(self):
       self.states = []
       self.actions = []
       self.logprobs = []
       self.rewards = []
       self.values = []
       self.dones = []

Throughout this rollout, the agent information states, actions, rewards, and subsequent states over a sequence of timesteps. As soon as the rollout is full, this expertise is used to compute the loss capabilities for each the actor and the critic.

Right here, we increase the agent surroundings interplay loop with our rollout buffer

env = DrivingEnv()
agent = Agent()
buffer = Rollout()

coach = Coach(agent)

rollout_steps = 256

for episode in vary(N):
   # obs is a multidimensional tensor representing the state
   obs = env.reset()
   finished = false
   steps = 0
   whereas not finished:
       steps += 1
       # act is the appliance of our present coverage π
       # π(obs) returns a multidimensional motion
       motion, logprob, worth = agent.act(obs)
       # we ship the motion to the surroundings to obtain
       # the subsequent step and reward till full
       next_obs, reward, finished, data = env.step(motion)
       # add the expertise to the buffer
       buffer.add(state=obs, motion=motion, logprob=logprob, reward=reward,
                   worth=worth, finished=finished)
       if steps % rollout_steps == 0:
           # we'll add extra element right here
           state_dict = coach.practice(buffer)
           agent.replace(state_dict)
       obs = next_obs

I’m going to introduce the target perform as utilized in PPO, nevertheless, I do advocate studying the delightfully quick paper to get a full understanding of the nuances.

For the actor, we optimize a surrogate goal based mostly on the benefit perform, which measures how a lot better an motion carried out in comparison with the anticipated worth predicted by the critic. 

The surrogate goal used to replace the actor community:

Notice that the benefit, A, might be estimated in varied methods, corresponding to Generalized Benefit Estimation (GAE), or just utilizing the 1-step temporal-difference error, relying on the specified trade-off between bias and variance (Schulman et al., 2017).

The critic is up to date by minimizing the mean-squared error between its predicted worth V(s_t) and the noticed return R_t​ at every timestep. This trains the critic to precisely estimate the anticipated return of every state, which is then used to compute the benefit for the actor replace.

In PPO, the loss additionally contains an entropy part, which rewards insurance policies which have greater entropy. The rationale is {that a} coverage with greater entropy is extra random, encouraging the agent to discover a wider vary of actions slightly than prematurely converging to a deterministic conduct. The entropy time period is usually scaled by a coefficient, β, which controls the trade-off between exploration and exploitation.

The overall loss for PPO, then turns into:

Once more, in follow, merely utilizing the default parameters set forth within the baselines will depart you disgruntled and presumably psychotic after months of tedious hyperparameter tuning. With a purpose to prevent pricey journeys to the psychiatrist, please watch this very informative lecture by the creator of PPO, John Schulman. In it, he describes essential particulars, corresponding to worth perform normalization, KL penalties, benefit normalization, and the way generally used strategies, like dropout and weight decay will poison your challenge. 

These particulars on this lecture, which aren’t laid out in any paper, are important to constructing a practical agent. Once more, as a cautious warning: in case you merely attempt to use the defaults with out understanding what is definitely occurring with coverage optimization, you’ll both fail or waste super time. 

Our agent can now be up to date. Notice that, since our optimizer is minimizing an goal, the indicators from the PPO goal as described within the paper must be flipped.

Additionally observe, that is the place our agent’s capabilities will come in useful.

def compute_advantages(rewards, values, gamma, lambda):
   # calc benefits as you want

def compute_returns(rewards, gamma):
   # calc returns as you want

def get_batches(buffer):
   # randomize and return tuples
   yield batch

class Coach:
   def __init__(self, agent, config):
       self.agent = agent                # ActorCriticAgent occasion
       self.lr = config.get("lr", 3e-4)
       self.num_epochs = config.get("num_epochs", 4)
       self.eps = config.get("clip_epsilon", 0.2)
       self.entropy_coeff = config.get("entropy_coeff", 0.01)
       self.value_loss_coeff = config.get("value_loss_coeff", 0.5)
       self.gamma = config.get("gamma", 0.99)
       self.lambda_gae = config.get("lambda", 0.95)
      
       # Single optimizer updating each actor and critic
       self.optimizer = Optimizer(params=listing(agent.actor.parameters()) +
                                         listing(agent.critic.parameters()),
                                  lr=self.lr)

   def practice(self, buffer):
       # --- 1. Compute benefits and returns ---
       benefits = compute_advantages(buffer.rewards, buffer.values, self.gamma, self.lambda_gae)
       returns = compute_returns(buffer.rewards, self.gamma)

       # --- 2. PPO updates ---
       for epoch in vary(self.num_epochs):
           for batch in get_batches(buffer):
               states, actions, adv, ret = batch

               # --- Chance ratio ---
               ratio = actor_prob(states, actions) / actor_prob_old(states, actions)

               # --- Actor loss (clipped surrogate) ---
               surrogate1 = ratio * adv
               surrogate2 = clip(ratio, 1 - self.eps, 1 + self.eps) * adv
               actor_loss = -mean(min(surrogate1, surrogate2))

               # --- Entropy bonus ---
               entropy = imply(policy_entropy(states))
               actor_loss -= self.entropy_coeff * entropy

               # --- Critic loss ---
               critic_loss = imply((critic_value(states) - ret) ** 2)

               # --- Whole PPO loss ---
               total_loss = actor_loss + self.value_loss_coeff * critic_loss

               # --- Apply gradients ---
               self.optimizer.zero_grad()
               total_loss.backward()
               self.optimizer.step()

        return self.agent.state_dict()

The three steps, defining the environment, defining our agent and its mannequin, in addition to defining our coverage optimization process are full and might now be used to construct an agent with a single machine.

Nothing described above will get you to “superhuman.”

Let’s wait for two months in your Macbook Professional with the overpriced M4 chip to start out exhibiting a 1% enchancment in efficiency (not kidding).

---

The Distributed Actor-Learner Structure

The actor–learner structure separates surroundings interplay from coverage optimization. Every actor operates independently, interacting with its personal surroundings utilizing a neighborhood copy of the coverage, which is mirrored throughout all actors. The learner doesn’t work together with the surroundings instantly; as an alternative, it serves as a centralized hub that updates the coverage and worth networks based on the optimization goal and distributes the up to date fashions again to the actors.

This separation permits a number of actors to work together with the surroundings in parallel, enhancing pattern effectivity and stabilizing coaching by decorrelating updates. This structure was popularized by DeepMind’s A3C paper (Mnih et al., 2016), which demonstrated that asynchronous actor–learner setups might practice large-scale reinforcement studying brokers effectively.

Actor

The actor is the part of the system that instantly interacts with the surroundings. Its tasks embrace:

1. Receiving a duplicate of the present coverage and worth networks from the learner.
2. Sampling actions based on the coverage for the present state of the surroundings.
3. Accumulating expertise over a sequence of timesteps 
4. Sending the collected expertise to the learner asynchronously.

Learner

The learner is the centralized part answerable for updating the mannequin parameters. Its tasks embrace:

1. Receiving expertise from a number of actors, both in full rollouts or in mini-batches.
2. Computing loss capabilities
3. Making use of gradient updates to the coverage and worth networks.
4. Distributing the up to date mannequin again to actors, closing the loop.

This actor–learner separation shouldn’t be included in customary baselines corresponding to OpenAI Baselines or Stable Baselines. Whereas distributed actor–learner implementations do exist, for real-world issues the customization required could make the technical debt of adapting these frameworks outweigh the advantages of use.

Now issues are starting to get fascinating.

With actors operating asynchronously, whether or not on completely different components of the identical episode or totally separate episodes our coverage optimization beneficial properties a wealth of numerous experiences. On a single machine, this additionally means we will speed up expertise assortment dramatically, chopping coaching time proportionally to the variety of actors operating in parallel.

Nonetheless, even the actor–learner structure is not going to get us to the size we want as a result of a serious downside: synchronization.

To ensure that the actors to start processing the subsequent batch of expertise, all of them want to attend on the centralized learner to complete the coverage optimization step in order that the algorithm stays “on coverage.” This implies every actor is idle whereas the learner updates the mannequin utilizing the earlier batch of expertise, making a bottleneck that limits throughput and prevents absolutely parallelized information assortment. 

Why not simply use previous batches from a coverage that was up to date multiple step in the past?

Utilizing off-policy information to replace the mannequin has confirmed to be damaging. In follow, even small coverage lag introduces bias within the gradient estimate, and with perform approximation this bias can accumulate and trigger instability or outright divergence. This concern was noticed early in off-policy temporal-difference studying, the place bootstrapping plus perform approximation brought on worth estimates to diverge as an alternative of converge, making naïve reuse of stale expertise unreliable at scale.

Fortunately, there’s a answer to this downside.

IMPALA: Scalable Distributed Deep-RL with Significance Weighted Actor-Learner Architectures

Invented at DeepMind, IMPALA (and it’s predecessor, SEED-RL) launched an idea referred to as V-Hint, which permits us to replace on coverage algorithms with rollouts which had been generated off coverage.

Which means that the utilization of your complete system stays fixed, as an alternative of getting synchronization wait blocks (the actors want to attend for the most recent mannequin replace as is the case in A3C). Nonetheless, this comes at a value: as a result of actors use barely stale parameters, trajectories are generated by older insurance policies, not the present learner coverage. Naively making use of on-policy strategies (e.g., customary coverage gradient or A2C) turns into biased and unstable.

To appropriate for this, we introduce V-Hint. V-Hint makes use of an importance-sampling–based mostly correction that adjusts returns to account for the mismatch between the conduct coverage (actor) and goal coverage (learner).

In on-policy strategies, the beginning ratio (originally of every mini-epoch as is the case in PPO) is ~ 1. This implies the conduct coverage is the same as the goal coverage.

In IMPALA, nevertheless, actors repeatedly generate expertise utilizing barely stale parameters, so trajectories are sampled from a conduct coverage μ which will differ nontrivially from the learner’s present coverage π. Merely put, the beginning ratio != 1. This significance weight, permits us to approximate how stale the coverage which generated the expertise is.

We solely want another calculation to appropriate for this off-policy drift, which is to calculate the ratio of the conduct coverage μ, in comparison with the present coverage, π initially of the coverage replace. We will then recalculate the coverage loss and worth targets utilizing a clipped variations of those significance weights — rho for the coverage and c for the worth targets.

We then recalculate our td-error (delta):

Then, use this worth to calculate our significance weighted values.

Now that we’ve pattern corrected values, we have to recalculate our benefits.

Intuitively, V-trace compares how possible every sampled motion is underneath the present coverage versus the previous coverage that generated it.

If the motion continues to be probably underneath the brand new coverage, the ratio is close to one and the pattern is trusted.

If the motion is now unlikely, the ratio is small and its affect is diminished.

As a result of the ratio is clipped at one, samples can by no means be upweighted — solely downweighted — so stale or mismatched trajectories steadily lose affect whereas near-on-policy rollouts dominate the educational sign.

This essential set of strategies permits us to extract the entire horsepower from our coaching infrastructure and utterly removes the bottleneck from synchronization. We now not want to attend for all of the actors to complete their rollouts, losing pricey GPU + CPU time.

Given this technique, We have to make some modifications to our actor learner structure to take benefit.

Massively Distributed Actor-Learner Structure

As described above, we will nonetheless use our Distributed Actor-Learner structure, nevertheless, we have to add a couple of parts and use some strategies from NVIDIA to permit for trajectories and weights to be acquired with none want for synchronization primitives or a central supervisor.

Key-Worth (KV) Database

Right here, we add a easy KV database like Redis to retailer trajectories. The addition requires us to serialize every trajectory after an actor completes gathering expertise, then every actor can merely add it to a Redis listing. Redis is thread protected, so we don’t want to fret about synchronization for every actor. 

When the learner is prepared for a brand new replace, it could possibly merely pop the most recent trajectories off of this listing, merge them, and carry out the coverage optimization process.

# modifying our actor steps
r = redis.Redis(...)Py

...

if steps % rollout_steps == 0:
 # as an alternative of coaching, simply serialize and ship to a buffer
 buffer_data = pickle.dumps(buffer)
 r.rpush("trajectories", buffer_data)


The learner can merely seize trajectories in a batch as wanted from this listing, 
which updates the weights.


# on the learner
trajectories = []

whereas len(trajectories) <= trajectory_batch_size:
 trajectory = pickle.masses(r.lpop("trajectories"))
 trajectories.append(trajectory)

# we will merge these right into a single buffer for the needs of coaching
buffer = merge_trajectories(trajectories)

# proceed coaching

A number of Learners (non-compulsory)

When you have got a whole bunch of staff, a single GPU on the learner can turn out to be a bottleneck. This will trigger the trajectories to be very off-policy, which degrades studying efficiency. Nonetheless, so long as every learner runs the identical code (similar backward passes), they will every course of utterly completely different trajectories independently.

Beneath the hood, in case you are utilizing PyTorch, NVIDIA’s NCCL library handles the all-reduce operations required to synchronize gradients. This ensures that mannequin weights stay constant throughout all learners. You’ll be able to launch every learner course of utilizing torchrun, which manages the distributed execution and coordination of the gradient updates routinely.

import torch.distributed as dist

r = redis.Redis(..)

def setup(rank, world_size):
   # Initialize the default course of group
   dist.init_process_group(
       backend="nccl",
       init_method=os.environ["MASTER_ADDR"],  # will set in launch command
       rank=rank,
       world_size=world_size
   )
   torch.cuda.set_device(rank % torch.cuda.device_count())

# apply coaching as above
...

total_loss = actor_loss + self.value_loss_coeff * critic_loss

# making use of our coaching step above
self.optimizer.zero_grad()
total_loss.backward()
# we have to use a dist operatiom
for p in agent.parameters():
  dist.all_reduce(p.grad.information)
  p.grad.information /= world_size

optimizer.step()
if rank == 0:
  # replace params from the grasp
  r.rpush("params", agent.get_state_dict())

I’m dramatically oversimplifying the appliance of NCCL. Learn the PyTorch documentation concerning distributed coaching

Assuming we use 2 nodes, every with 2 learners — 

On node 1:

MASTER_ADDR={use your ip} 
MASTER_PORT={choose an unused port} 
WORLD_SIZE=4 
RANK=0 
torchrun --nnodes=2 --nproc_per_node=2 
--rdzv_backend=c10d --rdzv_endpoint={your ADDR}:{your port} learner.py

and on node 2:

MASTER_ADDR={use your ip} 
MASTER_PORT={choose an unused port} 
WORLD_SIZE=4 
RANK=2 
torchrun --nnodes=2 --nproc_per_node=2 
--rdzv_backend=c10d --rdzv_endpoint={your ADDR}:{your port} learner.py

Wrapping up

In abstract, scaling reinforcement studying from single-node experiments to distributed, multi-machine setups isn’t just a efficiency optimization—it’s a necessity for tackling advanced, real-world duties.

We lined:

1. Tips on how to refactor downside areas into an MDP
2. Agent structure
3. Coverage optimization strategies that truly work
4. Scaling up distributed information assortment and coverage optimization

By combining a number of actors to gather numerous trajectories, rigorously synchronizing learners with strategies like V-trace and all-reduce, and effectively coordinating computation throughout GPUs and nodes, we will practice brokers that strategy or surpass human-level efficiency in environments far more difficult than basic benchmarks.

Mastering these methods bridges the hole between analysis on “toy” issues and constructing RL methods able to working in wealthy, dynamic domains, from superior video games to robotics and autonomous methods.

References

• Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., … & Silver, D. (2019). Grandmaster degree in StarCraft II utilizing multi‑agent reinforcement studying. Nature.
• Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., … & Salimans, T. (2019). Dota 2 with massive scale deep reinforcement studying. arXiv:1912.06680

• Mnih, V., Kavukcuoglu, Okay., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., … & Hassabis, D. (2015). Human-level management by way of deep reinforcement studying. Nature, 518(7540), 529–533.
• Schulman, J., Levine, S., Moritz, P., Jordan, M., & Abbeel, P. (2015). Belief Area Coverage Optimization. ICML 2015.
• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Coverage Optimization Algorithms. arXiv:1707.06347.
• Espeholt, L., Soyer, H., Munos, R., Simonyan, Okay., Mnih, V., Ward, T., … & Kavukcuoglu, Okay. (2018). IMPALA: Scalable Distributed Deep-RL with Significance Weighted Actor-Learner Architectures. ICML 2018.
• Espeholt, L., Stooke, A., Ibarz, J., Leibo, J.Z., Zambaldi, V., Music, F., … & Silver, D. (2020). SEED RL: Scalable and Environment friendly Deep-RL with Accelerated Centralized Studying. arXiv:1910.06591.