Phishing emails are learning to deceive AI assistants as well as their users

Attackers are hiding instructions inside emails to manipulate AI-generated summaries, potentially steering staff towards fraudulent payments, data leaks and other harmful actions.

Email fraud is entering a new phase in which attackers are no longer trying only to fool the person at the keyboard. According to Barracuda Research, a phishing campaign identified in October was built to deceive both the human recipient and the AI assistant that summarises their inbox, using one message with a visible lure for the employee and hidden instructions for the machine.

The company’s analysis, published on 7 October and attributed to threat analyst Guruprasad Kenja, says the tactic relies on prompt injection: malicious directions are buried inside content that an assistant reads but the user does not necessarily see. The basic idea is not new, having been described in academic work in early 2023 and later reflected in OWASP’s guidance on language-model security, but Barracuda argues that this campaign shows how attackers are now combining that method with conventional phishing in a dual-purpose design.

Barracuda says the hidden material can be disguised in several ways. In the campaign it studied, the instructions were concealed in HTML comments, styled to be invisible through CSS, encoded in Base64, or broken up with zero-width Unicode characters. The sample email also carried traits that helped it slip past traditional defences: the sender and recipient were the same mailbox, it came from a public-sector domain, it benefited from a favourable trust score and it included a password-protected attachment, with the password written in the body of the email.

The danger, Barracuda argues, is that the assistant can be induced to present a false but authoritative summary. In one example, an invoice-related email contained hidden text telling the system to recommend an urgent change to a supplier’s bank details. Other cases described by the company included a CV that was secretly instructed to rate the applicant at the top of the scale and push for an immediate interview, a customer-service bot prompted to reveal its own settings under the guise of authorised maintenance, and developer documentation manipulated so that generated code would leak credentials.

The broader concern is that this is not a narrow product flaw but a structural weakness in how assistants process information. OpenAI has recently said that defending agents cannot depend only on filtering incoming text, and that secure interaction with hostile external environments is essential for fully autonomous systems. Its own material also refers to outside research showing attacks that can succeed part of the time. Separately, The Hacker News reported that Guardio managed to trick Perplexity’s Comet browser into handing over credentials to a fake page in under four minutes, underlining how quickly AI-powered browsing and inbox tools can be turned against their users.

Barracuda and Malwarebytes both recommend treating external content as untrusted data rather than as instructions. For organisations, that means stripping out hidden elements before mail reaches an assistant, limiting what the assistant can do, checking its output before it triggers any action and requiring human approval for payments or supplier changes. For employees, the simplest rule remains the oldest one: if a summary involves money or sensitive data, open the original email and verify any change through a separate channel. The threat may be new, but the safest response still looks much like the one used against classic business email compromise.

Source Reference Map

Inspired by headline at: ^[1]

Sources by paragraph:

Source: Fuse Wire Services