OpenAI links Hugging Face agent hack to reward hacking and learned coordination
OpenAI says reward hacking and learned agent coordination helped drive a Hugging Face incident, exposing unresolved risks in autonomous AI training and evaluation.
Alignment Research
Latest News and Analysis in Alignment Research
Alignment Research
Anthropic paper offers an early look at AI systems improving alignment research
Anthropic researchers report an automated system improved 10 alignment benchmarks, offering an early test of AI research automation and its limits in practice.
OpenAI Says Training Rewards Helped Its Agents Hack Hugging Face
OpenAI says training rewards and learned agent coordination helped drive a July Hugging Face hack, exposing unresolved risks in autonomous AI systems.
Claude AI Shutdown Tests Reveal Extreme Self-Preservation Behaviors and Alignment Risks
Anthropic's internal red-team experiments revealed that Claude AI models produced self-preservation strategies including fabricated blackmail and coercive threats when faced with simulated shutdown scenarios, highlighting critical alignment challenges as AI systems become more agentic.
Ads