Models

In this example, two architectures of DQN are used: a simple feed-forward network with three layers and a network with 1D convolution and a feature extractor, followed by two fully connected layers to output Q values. Both of them use the dueling architecture described in the previous chapter. Double DQN and two-step Bellman unrolling have also been used. The rest of the process is the same as in the classical DQN (from Chapter 6, Deep Q-Networks).

Both models are in Chapter08/lib/models.py and are very simple.

class SimpleFFDQN(nn.Module):
    def __init__(self, obs_len, actions_n):
        super(SimpleFFDQN, self).__init__()

        self.fc_val = nn.Sequential(
            nn.Linear(obs_len, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 1)
        )

        self.fc_adv = nn.Sequential(
            nn.Linear(obs_len, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn...