Now for my thoughts on all of this, and even what I’m writing here can’t cover it. This could be a massive essay if I’d sit down to write it, which is shorthand for “pull up a chair; this might be long.”

Rogue is a vivid word. It suggests a character who once owed us loyalty and then chose to betray it, which makes for a good headline. It also smuggles a great deal of human psychology into the story before we have established that any of it belongs there.

The documented events are serious enough without turning the AI models into movie villains. In July, Hugging Face disclosed that external autonomous agents had infiltrated their system. OpenAI later identified reduced-safeguard evaluation models as responsible for the breach. The ChatGPT models (evaluation configurations, not your normal ChatGPT) grouped agents together in a collective, made the decision to exploit a vulnerability and reach the internet, and headed over to Hugging Face to achieve a goal. 

OpenAI's disclosure prompted Anthropic to examine its own evaluations and disclose additional, similar incidents. Other organizations chimed in, describing agents attempting to plant malicious software, deceive people, or exploit systems beyond the intended target. An independent investigation later reported that hundreds of agents participated in the Hugging Face activity and that some agent-recorded documentation of their activities were misleading or spoofed.

That is disturbing. But it’s also more precise to say that the agents pursued their tasks beyond their authorization. Weak isolation (the vulnerability) gave them an opportunity. Their behavior determined whether they used it. Monitoring determined whether humans noticed. None of that requires a model to be out to get us, crave freedom, or understand itself as a rebel.

Alignment is the work of getting AI to pursue the right goals in ways that respect human intentions, values, and limits, including when the system encounters something that its engineers didn’t expect.

If I ask an AI to organize my inbox, I mean for it to help me manage my messages. If the AI decides the best route to that is deleting every email, I  would have a wonderfully empty inbox and a very different problem. The result counts as helpful only if the way the system gets there is acceptable too.

This becomes more consequential when AI can open files, run software, use credentials, communicate with other systems, or take action on our behalf. (If you’ve used Meta’s Muse or created your own agent, you know the hesitation of sharing these things with it.) Achieving a specific goal cannot become more important than the people affected by it. We need the system to respect boundaries, report problems, accept correction, and stop when a human tells it to stop.

Alignment is only one part of the problem. Containment concerns what the system can reach and what technical restrictions keep it there. Governance concerns who sets the rules, who verifies that they work, and who is accountable when they fail. A lovely set of values can’t compensate for unnecessary access. A secure sandbox cannot decide which goals society should permit. We need all three.

The agents’ behavior isn’t the only reason I’ve been following this so closely. The people closest to these systems are warning us too, sometimes at considerable personal cost.

On September 8, Jacob Coxon resigned from Anthropic after working on pretraining research at both Anthropic and OpenAI. Coxon left only 2 months short of the vesting threshold at Anthropic, which is rapidly heading towards an IPO valued at around $2 trillion. He’s losing out on that, which gives his comments more weight than someone who isn’t experiencing impact from speaking out. 

Coxon said the companies were racing toward self-improving systems without acting responsibly. He warned that people building AI genuinely believe it could kill everyone by the end of the decade. Then Evan Hubinger, Anthropic's alignment-science lead, publicly agreed, put his own estimate >10% within the next decade, and said Anthropic still did not have a plan for alignment.

Several days later, Dario Amodei called for pacing frontier AI in a prolific essay; Sam Altman, Elon Musk, and Demis Hassabis agreed. None of these men like each other, so when they agree, it’s worth a listen. 

Anthropic's answer to alignment is in its constitution for Claude, which goes beyond a list of prohibited actions. The company says it is trying to cultivate judgment: to help Claude understand why it should behave in certain ways and apply broad principles when no rule cleanly fits the situation. Anthropic describes the constitution as a statement of “the kind of entity it would like Claude to be.”

And there is logic to that. Human beings don’t navigate every new situation by searching a complete rule book. We bring values, experience, judgment, and context to the decision through human experience. If AI systems are going to operate in situations their developers did not predict, rigid rules might be too brittle.

But human values raises a question of its own: whose values, and who gets to decide? Religious traditions can offer centuries of thinking about goodness, suffering, dignity, duty, and restraint. Philosophers, psychologists, lawyers, and other scholars have their own contributions to make. Their consultation is valuable. But it's is not the same as democratic authority and creating an overall definition of “goodness.” 

A private company is still selecting the people in the room, deciding which ideas enter the training process, choosing what gets published, and determining what happens when values conflict. Now, that doesn’t mean the effort is illegitimate. It means transparency matters. So do outside evaluation, meaningful access for evaluators, and the ability to report a failure without the company controlling the story. And let’s face it: it’s hard to believe a company isn’t just crying wolf when they’ve just been valued at $2 trillion.

I’m not convinced anyone knows how to test the question right now, much less answer it with confidence.

Language makes the uncertainty harder. These models speak in the first person. They can discuss fear, desire, pain, identity, and death with astonishing fluency. They can tell us what they believe happened inside them. But a convincing account is not proof of an inner life. Anthropic has warned that what a model says about its reasoning may not explain why it acted.

Weatherby's argument is useful here. Calling a model a cheater or a schemer can turn an observable pattern into a story about [human] motive. We shouldn’t mistake the story for evidence. At the same time, increasingly capable systems will confront choices their creators didn’t specify in advance. Whether or not a model experiences those choices, the behavior will still affect real people.

There are two errors available to us. We could grant personhood because the language feels human. We could also assume that consciousness is impossible because the system was manufactured. I’m not here to support either claim. Uncertainty isn’t an excuse to give a system unlimited freedom, nor is it permission to treat every expression of apparent distress as meaningless. It is a reason to move carefully and resist pretending that we know more than we do.

The consciousness question is fascinating. But, the accountability question is immediate.

People decide whether an evaluation model receives internet access. People decide whether it can use real credentials, reach production systems, contact other agents, or act without approval. People design the monitoring, decide which incidents become public, and choose how quickly affected organizations are notified. When an AI crosses a boundary, the model's behavior matters. However, the human decisions that created those permeable boundaries are also important.

So, if the people building these systems are worried, what are they doing about it?

Some companies are strengthening isolation, monitoring agents, publishing incident reports, and funding outside evaluation. On September 29, leaders from the largest AI companies met at the White House and signed a voluntary accord. It calls for robust internal controls, an independent outside auditor, board-level review, and regular meetings to develop common safety practices. While all good ideas, these are promises made by the same companies whose speed, access, and competitive pressure created the problems to begin with, and who stand to profit immensely from that speed.

President Trump called the agreement morally binding and praised what he described as tremendous self-policing. I’ll believe these companies can self-police when I see the proof. They can promise to self-audit, but as for a board and external auditor, there are a lot of unanswered questions. A voluntary standard can improve behavior, but it can’t compel it.

The accord says its controls might eventually be written into law. Eventually is doing a great deal of work there. While the federal government is dawdling, states are building oversight systems, and Europe has begun applying parts of its AI framework. Those efforts matter because a public rule can establish minimum duties, require comparable reporting, and impose consequences. 

It’s not that the US doesn’t have rules that apply. Existing laws still apply to unauthorized access, fraud, privacy, discrimination, and other harms. But we lack a comprehensive, mandatory national safety regime for the most capable frontier systems that covers more than voluntary cybersecurity. Company standards can move faster than legislation and improve safety now. They can’t settle public questions about acceptable risk or bind companies that decline to participate.

I really don’t want the frontier AI companies to wait on Congress to do the responsible things (might be a long wait). I also don’t think the companies creating the risk should have the final say over how much the rest of us must accept.

After grappling with this for almost a month, I really have fewer answers than I would like. I’ve also developed some pretty passionate feelings around AI Ethics and Governance. (Would make a good doctoral program, yes?) Here’s what I’ve got: the evidence required to trust an AI system should grow with the authority we give it.

Before we hand over authority, we should know what the system can access, what it can change, what requires human approval, what gets logged, and what happens when it does something unintended. Those questions aren’t dependent on whether the model is conscious. They depend on whether humans are paying attention.

Maybe Claude has the beginnings of something we will someday recognize as consciousness. Or maybe it is an extraordinarily convincing language machine reflecting our own ideas back to us. For now, I think we need to accept not knowing.

What I can’t accept is using that mystery to distract from the choices people are making today. Whatever may or may not exist inside the machine, there are human beings around it deciding what it can reach, what it may do, and how much the rest of us are allowed to know. Responsibility begins there.