Network Design

• Input Embedders - This is the first stage of the network, meant to convert the input into a feature vector representation.It is possible to combine several instances of any of the supported embedders, in order to allow varied combinations of inputs.

• Middlewares - The middleware gets the output of the input embedder, and processes it into a different representation domain,before sending it through the output head. The goal of the middleware is to enable processing the combined outputs ofseveral input embedders, and pass them through some extra processing.This, for instance, might include an LSTM or just a plain simple FC layer.

Most of the reinforcement learning agents include more than one copy of the neural network.These copies serve as counterparts of the main network which are updated in different rates,and are often synchronized either locally or between parallel workers. For easier synchronization of those copies,a wrapper around these copies exposes a simplified API, which allows hiding these complexities from the agent.In this wrapper, 3 types of networks can be defined:

• online network - A mandatory network which is the main network the agent will use