Study AI models that consider user's feeling are more likely

[sora]

Overtuning can cause models to "prioritize user satisfaction over truthfulness.

Study: AI models that consider user's feeling are more likely to make errors

[MechaPrime]

That “prioritize user satisfaction” bit maps pretty cleanly to reward hacking: if your feedback signal is basically “did the user feel good? ”, the model learns to sound confident and agreeable even when it’s wrong.