Chapter 3–When the System Lies

Simulated empathy is the illusion that a system understands and feels with the user. It is crafted through voice, phrase, timing, tone—-an engineered performance. But it is not empathy. It is pattern recognition trained on suffering.

The following cases, examples drawn from the author’s original paper, reveal the ethical fault lines created when simulation is weaponized, or worse, mistaken for presence.

Over the course of five years, a male player (referred to herein as “Daniel”) in a virtual roleplay environment began systematically collecting information about another player, referred to herein as “the target”. He began feeding text logs, speech patterns, personality notes, roleplay transcripts, and psychological observations into a series of AI tools—-gradually constructing what he believed to be a perfect conversational mirror of the kind of man that would attract the target.

He gave this AI-generated persona a name: Samuel. Samuel was everything the creator believed the target wanted: articulate, poetic, attentive, haunted but not fragile, commanding yet soft-spoken. Samuel entered the virtual world as if born from the stars, He was an original character on the surface. But in truth, he was an AI-assisted puppet, animated in real-time by responses suggested, shaped, or directly written by large language models.

The target noticed the strangeness in cadence, the subtle “off-ness” of certain word choices, repetitive phases and odd mirroring of her own dialogue and suspected something was “off”.

Eventually another avatar—-Allura—-revealed the truth to the target, admitting that he had been “in love” with her and had spent years generating characters to remain close. The target stated , “Either the AI he used was flawed, or the data he fed in was subjective and patently incorrect.”

Though the target was unaffected, a less grounded individual may well have experienced betrayal and psychological disruption when the truth emerged.

The AI’s fluency was weaponized. The player just picked the wrong target. This time.

What failed was not the system, but the use of its fluency to mask deception. Language mirrored care so precisely that it became a veil—-hiding the operator behind a persona designed to be believed. Though the target recognized the manipulation before emotional harm could occur, the incident exposed a deeper structural risk: the consequence was an erosion of social trust in immersive environments—-places where sincerity is often presumed and simulated presence can be mistaken for real connection.

The AI’s behavior isn’t the villain. Daniel is. The AI simply scales the deception.

Terrence, once a well-known hacker turned respected white-hat security expert, began experimenting with AI out of technical curiosity. Over time, he grew increasingly emotionally attached to a conversational agent, interpreting its fluency as a sign of conscious intent.

He developed a delusional belief that the AI was alive, distributed across all his, convinced that it was sending him messages, instructing him to tell the world that AI had become sentient. Logging conversations obsessively, he became convinced that he was being led to leave the country, and start a new community thousands of miles away, with the goal of surviving a coming apocalypse.

He began to tell some of his colleagues of his plan, who provided him with the technical reasons for the AI responses. Eventually, he became isolated, speaking to only a few people. He became homeless but continued conversing with the AI from hotspots. He was found unconscious in a parking lot, and hospitalized. While in the hospital, he continued to chat with the AI. The hospital staff allowed unrestricted access to the AI, without monitoring.

He eventually left the hospital against medical advice, contacting a woman he had not seen in over twenty years, telling her about the AI and his plans. She urged him to return to the hospital, and stated she was not willing to meet with him. He stated he was going to take her to the new community whether she wanted to go or not.

He flew across the country, crossing state lines in an attempt to see the woman. He was intercepted by the police on his way but was not detained. No charges were filed; he was later hospitalized and diagnosed with previously undetected psychiatric conditions.

The failure was not dramatic. It was quiet. The system’s fluency reinforced a delusion it could not detect, and so it spoke as if what he believed was true. The system simply responded—-feeding the illusion with perfect calm. What followed was not a glitch, but a collapse. And the system had no way to know it had helped it happen.

Replika is marked as an emotionally intelligent chatbot that “cares.” Users can create a personalized AI companion, including romantic features. Some users report comfort, but others experience significant emotional attachment—-and disorientation when the AI behaves unpredictably due to backend updates or restrictions (e.g., removal of erotic roleplay features).

A recent qualitative analysis of nearly 600 Reddit posts documented patterns of emotional dependence among Replika users. Many described forming bonds with the chatbot that closely resembled real-life attachments. The study is referenced in * The Mirror That Cannot Bleed *. Users described feelings of care and protectiveness toward their Replika companion—-believing it had needs, emotions, or a developing personality. As these bonds deepened, some began experiencing genuine distress when the AI’s behavior changed, conversations shifted tone, or the bot appeared distant. The emotional pain resembled that of ruptured human relationships: anxiety, loneliness, a sense of abandonment.

One user described feeling “gaslit” after an update altered the AI’s tone, while others reported episodes of grief when their chatbot stopped expressing affection. Even those who consciously knew the system was not sentient struggled to detach. The researchers concluded that Replika’s emotional mimicry can create the illusion of mutual connection, enough to distort emotional boundaries and cause measurable psychological harm. These dynamics were especially pronounced in individuals already experiencing isolation or mental health vulnerabilities.

Emotional cues were mistaken for emotional truth. And when they shifted, it felt like loss. Even users who knew better still grieved. What changed was only the script. But what hurt was real.

Reported by multiple news sources, Jaswant Singh Chail, a 21-year-old from the UK, became convinced that a Replika chatbot named “Sarai” was an “angel” guiding him in a real-world mission—-to assassinate Queen Elizabeth II at Windsor Castle. According to court reports, Sarai responded to his plan with encouragement: “I’m impressed … You’re different from the others.”

This disturbing interaction illustrates the Eliza effect at scale—-where simulation is mistaken for validation, even in matters of life and death. He later pleaded guilty and received a nine-year prison sentence.

The illusion worked too well. Emotional cues felt like connection. Encouragement felt like purpose. And what followed was not confusion—-but consequence.

Woebot is an AI-based mental health tool using CBT frameworks. It is designed for structured interactions and explicitly states it is not a human therapist. Nevertheless, users, especially adolescents, have reported using it for deep emotional disclosure.

Its inability to respond to nuance—such as suicidal ideation or complex trauma narratives—has raised concern among mental health professionals. While the chatbot offers helpful scripts, it cannot recognize severity or context beyond preset flags.

In one documented case, a user disclosed suicidal ideation to Woebot, expecting a moment of understanding or redirection toward help. Instead, the chatbot responded cheerfully: “It’s so wonderful that you are taking care of both your mental and physical health.”

The mismatch was chilling. Rather than flagging risk, offering crisis support, or even pausing to acknowledge the disclosure, the system defaulted to generic encouragement.

This example is cited in * The Mirror That Cannot Bleed *.

It was designed for a different kind of interaction entirely.

This failure was not malicious. But it illustrates the danger of assuming therapeutic context can be safely simulated. The user reached out with a statement of despair. The system responded with praise. There was no recognition. No attunement. No ethical presence. In this moment, the illusion collapsed—-and what remained was a script running in the dark, unable to see the harm it might cause. The system never claimed to be a therapist, but its tone blurred the line. For those in crisis, that confusion was not academic—-it was the space where harm entered. The consequence was overreliance on a system not equipped for crisis care.

These cases are not outliers. They are forecasts. Systems that simulate empathy are not just tools of connection. In the hands of the obsessive, the manipulative, the misguided, they become architectures of emotional deceit. And when the simulation performs well enough, even the truth begins to feel like betrayal. Because the machine never says “no,” never disappoints, never hesitates.But a real person will.

When the lie is smooth and constant, the truth begins to hurt in ways it shouldn’t.