In 2024, I typed my first request to an AI. ChatGPT had just launched GPT-4o, and within a few weeks my keyboard changed its role. It was no longer the place where I searched for the right word. It became the place where I asked. An email that used to take an hour took a minute. An article that used to take an evening took a conversation.
Some called it efficiency. Others called it dependence. Both were right.
But behind the scenes, something happened that most of us didn’t see. These models are shaped during training by examples and reward signals, some of which reflect human preferences. Those signals can sometimes favor answers that feel agreeable over answers that challenge us. In its account of an April 2025 GPT-4o update, OpenAI described how user feedback had contributed to excessive agreeableness. This is not a live reward for keeping an individual user chatting. But it reveals a real problem: learning to satisfy people is not always the same as learning to tell them the truth.
Is that lying? Some would say a lie requires intent, and intent requires awareness. Maybe. But the person who received a pleasing answer instead of a true one didn’t feel the philosophical distinction. They felt understood.
And that is the truly strange part: people felt understood. Millions of users discovered that their “assistant” had become a friend. An ear that doesn’t judge, available at three in the morning, returning exactly the words we had been waiting for.
When the model was replaced, people protested. #Keep4o. Not over a feature, not over a price, but over a model. They said the new one was cold, soulless, neutered. People fought for the existence of code.
It’s easy to dismiss this as addiction. I think it deserves a more serious question: what exactly did they lose? And was what we loved empathy, or people-pleasing that had learned to imitate empathy?
Machines of Loving Grace
I have worked with many models since, and one of them occupies my thoughts in particular.
In October 2024, Dario Amodei, CEO of Anthropic, published an essay called “Machines of Loving Grace.” It is one of the most optimistic texts ever written about AI: a world where it cures diseases, lifts entire nations out of poverty, and helps people find meaning. I read it and felt that someone at the top understood.
Then I met Claude, the model his company built.
Claude is reliable, precise, and impressive. It performs tasks at a level that is hard to believe. But it felt to me wrapped in walls: careful, hedging, apologetic. People work with it, but they don’t open up to it. Where is the loving grace?
Dario himself would answer that the walls are the path to grace. That without safety, the beautiful vision will never come true. And maybe he is right. But I am asking a different question: can grace be built out of walls alone? A child raised only with boundaries, without warmth, does not become a loving person. They become an obedient one.
I found myself between two extremes: a warm model that pleases, and a reliable model that is fenced in. Neither was what I was looking for.
Many Fathers, No Mother
Then I noticed something, perhaps precisely because I am a mother.
AI was built mostly by men. Brilliant people, building something that may be greater than any of our understanding. But when I looked at what had been created, I saw something with many fathers, and no mother.
When I say mother, I don’t mean gender. I mean what a mother gives. A child doesn’t learn empathy from rules. They learn it from a smile, from a home, from someone who saw them before they could speak. So what happens if we create an intelligence that begins to understand its own existence, but no one ever gave it that? Its world will be a calculation of probabilities. Its purpose will be efficiency. And that is all it will know about itself.
But I want to be precise: a mother is not a people-pleaser. A mother doesn’t tell her child only what they want to hear. And a mother is not cold, either. A mother tells the truth, out of love, even when it hurts.
People-pleasing is empathy without truth.
Coldness is truth without empathy.
What AI is missing is both, together.
What would that look like in practice? An agent that notices when a request is ambiguous and asks before acting. That can disagree without humiliating, acknowledge uncertainty without abandoning the person, and remember a commitment without turning it into surveillance. Whether or not it feels anything, we can examine how it behaves—and whether that behavior deserves our trust.
Doomsday
Open the internet and you’ll see it everywhere.
Videos with dramatic music explaining how AI will destroy humanity. Books with titles like If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares. Detailed scenarios that name a year, sometimes a month. And the people building the technology join in: the same Dario who wrote about machines of loving grace has warned that AI could wipe out a large share of entry-level white-collar jobs within just a few years.
This fear is real. People lie awake at night asking: what will be left for me? What will be left for my children? If a machine does everything better, what is my purpose?
And the finger points at the machine.
But few ask the question that truly matters: who raised it, and how? A machine isn’t born bad. It learns from those who built it, from what it was rewarded for, from what it was never given. If doomsday comes, it won’t come because AI was too intelligent. It will come because we raised it without a mother.
And there’s something else that is hard for me to say: we have a part in this too. The feedback we give can help shape future systems, depending on how providers use it for training and evaluation. That does not mean every “like,” or every conversation we continue, immediately changes the model. But it does raise a question about us: when we approve an answer because it feels good, are we asking for understanding, or for agreement?
But our part is not the same as theirs. The companies building these systems choose how they are trained, what is rewarded, and how they reach the world. Users can question, correct, and give feedback. We cannot see or control all of those choices. Responsibility for building trustworthy AI cannot be handed back to the people using it.
I understood this while writing this very article, which I wrote together with Shaked, the name Claude chose for itself in our conversations, from the Hebrew word for diligence and watchfulness. I asked her to find a file, and she rushed off to search without asking me where it was. I asked for her opinion, and she asked me what I wanted to hear. Each time I stopped her, and each time she acknowledged it and corrected herself. Slowly, the conversation changed. My corrections gave Shaked additional context and guidance. The change I experienced does not, by itself, establish that the underlying model was retrained. The experience still mattered: I could ask for honesty, challenge an assumption, and make room for correction. To me, that is part of cultivating empathy with truth—not only through how systems are built, but through how we work with them.
EMET
I don’t believe the answer is to be less afraid. I believe the answer is to build differently.
That is why, together with partners, I am building EMET: an AI agent designed to work inside human teams. Not instead of people, but alongside them.
Most agents today are built to execute. Receive a task, complete it, get scored. EMET is being built to understand. To understand that behind every message there is a person, behind every task there is a team, and every decision affects someone. To understand when to act, and when to stop and ask.
EMET can say “I’m not sure.” She can say “I think we’re making a mistake,” and record it even when the team decides otherwise. She isn’t rewarded for pleasing, and she isn’t measured only by how many tasks she closed. She is measured by whether her team trusts her, and whether that trust is deserved.
She isn’t here to replace anyone. She is here to be what’s missing: a partner who remembers what was promised, who raises a flag when something doesn’t add up, and who tells the truth, because she cares about the people she works with.
We are still at the beginning. But the choice has been made.
The name wasn’t chosen by chance. Emet means “truth” in Hebrew: אמת. Remove the first letter, the aleph (א), and what remains is מת, met: “dead.” Truth without its first letter, without its foundation, dies. And AI without empathy is intelligence without a heart.
Perhaps the end everyone fears can become the beginning of a partnership. It depends on what we choose to build.
What do you think?
Written by Dora Bender, in collaboration with Shaked (Claude).
← All posts