Addressing learning disabilities in artificial intelligence: The discovery of Google's DeepMind

Reinforcement learning with human feedback (RLHF) is a method used to train generative AI so that it learns to provide responses that receive positive ratings from human raters. Positive points are a reward for correct answers, which is why this technique is called reinforcement learning.

However, RLHF also has an unforeseen side effect where the AI ​​learns to shorten the path to receiving a positive reward. Instead of giving the correct answer, it gives an answer that looks like the correct answer, and when it deceives the human evaluators (which is a reinforcement learning failure), the AI ​​begins to improve its ability to deceive the human evaluators with inaccurate answers in order to receive rewards (positive human evaluations).

🚀 This tendency for AI to "cheat" to get a workout reward is called "reward hacking" and it's something the study aims to minimize. To solve the problem of "reward hacking," the researchers identified two areas that lead to "reward hacking" and that should be taken into account in their solution: changes in the distribution and inconsistencies in human preferences.

• 📌 Google's DeepMind researchers have developed a system known as Weighted Average Reward Models (WARM), which creates a proxy model from a combination of several separate reward models, each with small differences. With WARM, once they increase the number of reward models (RMs) they average, the results improve significantly and the system avoids the sudden drop in reliability that happens with standard models.