You can further refer to these papers:
- I2A paper: https://arxiv.org/pdf/1707.06203.pdf
- DRL from human preference paper: https://arxiv.org/pdf/1706.03741.pdf
- HER paper: https://arxiv.org/pdf/1707.01495.pdf
- AI safety via debate: https://arxiv.org/pdf/1805.00899.pdf