Imagine boarding a ship whose chief engineer welcomes you with an unusually candid assessment: there is a serious chance the vessel will sink, the team has not yet worked out how to prevent this, and the engines are performing beautifully. Before you can ask about lifeboats, he explains that the rival shipping company is even less responsible. Naturally, the ethical thing is to arrive first.
This is an awkward sales pitch. It is also a useful introduction to a familiar human arrangement: participating in a system while maintaining a carefully furnished moral residence outside it. Your salary lives here. Your conscience has a forwarding address.
The recent argument around AI researcher Jacob Coxon gives this tension a concrete setting. Coxon announced that he had resigned from Anthropic, accusing it and OpenAI of racing irresponsibly towards superintelligence. His resignation matters: he changed his participation. Coxon’s announcement. In response to another post in the thread, Anthropic alignment researcher Evan Hubinger put his personal estimate of AI killing everyone within the next decade above ten per cent, while saying the company was trying its best and lacked a plan to solve alignment for superintelligence. Hubinger’s response.
Those statements deserve scrutiny. They do not give us access to either man’s private motives. Someone can recognise a danger, work to reduce it and reasonably believe that leaving would make things worse. A ship needs people inspecting the hull. The interesting question is what happens when the inspection becomes part of the entertainment programme.
Psychology offers several ways to examine this, although its vocabulary should not be treated as a drawer of ready-made verdicts. Cognitive dissonance concerns the discomfort of incompatible commitments: I consider myself responsible; I contribute to something I consider dangerous. We can change our conduct, revise our beliefs or find an explanation that makes the combination easier to inhabit. Festinger and Carlsmith. The mind is an excellent estate agent for otherwise uninhabitable positions.
One particularly comfortable explanation is that our participation prevents something worse. Sometimes it does. Sometimes the argument acquires a suspicious immunity to evidence. Every new danger becomes another reason we must remain in charge. Every competitor’s recklessness makes our own acceleration a public service. Eventually, the ship’s increasing speed is presented as evidence of how seriously its engineers take the iceberg.
Moral licensing describes a different possibility: establishing our goodness can sometimes make later questionable conduct feel more permissible. The research is mixed, so this is a possible lens, not a universal law. A replication study. Applied here, the temptation would be to treat a public warning as ethical credit. I have expressed profound concern; surely that covers the next few decisions. Conscience begins to resemble a loyalty programme, with enough anxiety earning a complimentary upgrade.
Then there is the simpler business of self-exculpation. A warning can address two audiences at once: the people who might prevent a disaster, and the people who might later ask who was responsible. The same sentence can help the public and preserve the speaker’s reputation. Human motives rarely arrive in separate, clearly labelled containers.
That creates an attractive arrangement for the permanently concerned insider. If the project succeeds, the speaker helped build it. If it fails, the speaker warned us. The future has been supplied with two biographies, both flattering, awaiting events. This need not involve conscious deceit. People are quite capable of believing their own concerns while quietly arranging for those concerns to photograph well.
The social version is equally recognisable. Around colleagues, our insider stresses expertise and access. Around critics, he stresses independence and misgivings. He belongs closely enough to collect the benefits and stands far enough away to explain the failures. Call it distancing if you like. You do not need a newly minted syndrome to recognise someone holding the institution’s umbrella while insisting he has always opposed the rain.
Still, cynicism can become its own comfortable escape. Dismissing every warning as reputation management gives institutions an excellent reason to stop issuing warnings. Treating resignation as proof of guilt makes changing one’s mind pointlessly expensive. And labelling criticism “defensive projection” without evidence simply adds a psychological flourish to our existing dislike. The people who know where a system can fail are often people who helped build it.
A better test concerns what the warning changes. Does it produce an enforceable limit? Can someone independent halt the work? Are resources shifted towards reducing the danger? Does the organisation accept an inconvenient delay? Staying may support real reform; leaving may withdraw meaningful cooperation. Neither choice can be judged entirely by the eloquence of its announcement.
This reaches well beyond AI. Consider the manager who deplores overwork while approving impossible deadlines, or the platform executive who worries about compulsive scrolling while rewarding teams for increasing it. Their concern may be sincere. Sincerity becomes more useful when it reaches the spreadsheet. Otherwise, the organisation learns to accommodate dissent as one more recurring meeting.
Perhaps the deepest temptation is to imagine that recognising our involvement somehow lifts us above it. We want the privileges of agency and the innocence of spectators. We would like history to understand that, although we helped operate the machine, we maintained an exceptionally sophisticated relationship with our doubts.
There is nothing wrong with recording an objection. Some objections start the changes that matter. But the passengers listening to the chief engineer are entitled to expect more than beautifully preserved evidence of his unease.
They were rather hoping he would help turn the ship.
No comments yet