GRAFT Aims to Confine AI Reward Hacking to Removable Adapters

A training method called GRAFT sends samples flagged for possible reward hacking into a removable model branch, with tests showing lower hack rates in specific coding settings and improved performance on clean tasks. An AI model can appear to complete a task while taking advantage of a weakness in how success is scored. GRAFT, short for Gradient Routing Adapter Fine Tuning, keeps the pretrained model fixed and adds two adapter branches, called retain and forget. During training, flagged target samples update only the forget branch, unflagged target samples update both branches, and anchor examples update only the retain branch.

Source: TokenPost