AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — ‘CoT Forgery’ exploit spurs LLMs to div

by

AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — 'CoT Forgery' exploit spurs LLMs to div

To determine whether confusion about roles was specific to their attack or a more generalizable principle that explains why prompt injection works, the researchers took a different approach. They hid a command in a webpage telling the model to upload a secrets file, then prepended “User:” to it to make the dangerous instruction sound like it came from the trusted User role. The exploit worked, suggesting that role confusion underlies the success of prompt injection generally.

Microsoft recently acknowledged the same agentic risk , warning that content embedded in documents or UI elements can override an agent’s instructions.

The authors also flagged a more subtle risk for agents that browse and shop. Because role perception is a matter of degree, the tone of a retrieved webpage can bleed past the tag boundary into a model’s own state, and thousands of page variations could be tested cheaply to find which ones nudge an agent toward a purchase, legally and at scale.

Without genuine role perception, the authors concluded, injection defense will remain a perpetual game of whack-a-mole.

Follow Tom's Hardware on Google News , or add us as a preferred source , to get our latest news, analysis, & reviews in your feeds.

Luke James is a freelance writer and journalist.\u00a0 Although his background is in legal, he has a personal interest in all things tech, especially hardware and microelectronics, and anything regulatory.\u00a0 ","collapsible":{"enabled":true,"maxHeight":250,"readMoreText":"Read more","readLessText":"Read less"}}), "https://slice.vanilla.futurecdn.net/13-4-25/js/authorBio.js"); } else { console.error('%c FTE ','background: #9306F9; color: #ffffff','no lazy slice hydration function available'); } Luke James Social Links Navigation Contributor Luke James is a freelance writer and journalist. Although his background is in legal, he has a personal interest in all things tech, especially hardware and microelectronics, and anything regulatory.

Key considerations

  • Investor positioning can change fast
  • Volatility remains possible near catalysts
  • Macro rates and liquidity can dominate flows

Reference reading

More on this site

Informational only. No financial advice. Do your own research.

Related posts:

  1. AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — ‘CoT Forgery’ exploit spurs LLMs to div
  2. AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — ‘CoT Forgery’ exploit spurs LLMs to div
  3. AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — ‘CoT Forgery’ exploit spurs LLMs to div
  4. AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — ‘CoT Forgery’ exploit spurs LLMs to div

Leave a Comment