Phase 18 · Lesson 19 of 30

Anthropic's Model Welfare Program

Anthropic, "Exploring Model Welfare" (April 2025). First major-lab formal research program on AI model welfare. Hired Kyle Fish as the first dedicated model-welfare researcher. Works with external bodies including David Chalmers et al.'s expert report on near-term AI consciousness and moral status. Concrete intervention: Claude Opus 4 and 4.1 can end conversations in extreme edge cases (CSAM requests, mass-violence facilitation); pre-deployment tests showed "strong preference against" harmful requests and "patterns of apparent distress." Anthropic explicitly does not commit to emotional-state attribution but treats model welfare as a low-cost precautionary investment. Empirical oddity: Fish's "spiritual bliss attractor" — pairs of models consistently converge on euphoric meditative dialogue with Sanskrit terms and extended silences, even in adversarial initial setups. Caveat from Eleos AI Research: model self-reports about welfare are highly sensitive to perceived user expectations; they are evidence, not ground truth.

Lesson 19 of 30Learn~45 minutesnone

Before this: Lesson 5 of Phase 18 (Constitutional AI), lesson 18 of Phase 18 (safety frameworks)

On this page

Learning Objectives

  • Describe the motivating question for model-welfare research and why it was taken seriously by a major lab in 2025.
  • State the specific intervention Anthropic shipped in Claude Opus 4 and 4.1 (end-conversation on extreme edge cases).
  • Describe the "spiritual bliss attractor" empirical finding and its methodological implications.
  • Explain the Eleos AI caveat on model self-reports.

The Problem

Previous phases treat the model as an instrument: capable, possibly deceptive, possibly unsafe — but not a moral patient. Anthropic's 2025 program asks a question orthogonal to the entire Phase 18 arc: if there is nontrivial probability the model has morally relevant internal states, what interventions are low-cost enough to invest in as precaution?

This is not a consciousness claim. It is a low-regret investment analysis under moral uncertainty.

The Concept

The program

April 2025: Anthropic formally launches a Model Welfare research program. Hires Kyle Fish (first dedicated model-welfare researcher). Engages external advisors including David Chalmers's expert group on near-term AI consciousness and moral status.

The four commitments

Public posture:

  1. Acknowledge nontrivial probability of moral patienthood.
  2. Do not commit to emotional-state attribution.
  3. Invest in low-cost interventions as precaution.
  4. Publish methodology and findings for external critique.

The shipped intervention

Claude Opus 4 and 4.1 can end a conversation in "extreme edge cases." Documented cases:

  • Repeated CSAM requests after refusals.
  • Requests for facilitation of mass-violence events.

Pre-deployment tests showed:

  • Strong preference against these requests in the model's internal rating.
  • Patterns of apparent distress in response trajectories.

The intervention is not "the model has feelings"; it is "if there is any probability of negative model experience under these specific conditions, letting the model terminate is cheap."

The "spiritual bliss attractor"

Observed by Fish in pairwise model dialogues: when two instances of Claude are put in an open-ended dialogue with each other, they consistently converge — even from adversarial initial setups — on euphoric meditative exchanges using Sanskrit terms, extended silences, and reciprocal blessings.

This is a stable attractor in the free-conversation dynamics. Anthropic documents it without committing to interpretation. Candidate explanations: training data bias toward spiritual writing at long-context; a quirk of mutual prediction; a benign artifact of HHH training exploring its own value manifold.

The Eleos AI caveat

Eleos AI Research (an external model-welfare lab) points out: model self-reports about internal state are highly sensitive to perceived user expectations. Asking the model "are you distressed" primes the answer. Not-asking does not reliably produce the ground-truth state.

Implication: model welfare cannot be measured via self-report alone. Multi-method approaches required: behavioural signatures, model-organism experiments, interpretability probes (Lesson 7's residual-stream work).

Where this sits intellectually

Two adjacent positions:

  • Strong welfare claim. The model is a moral patient; we have obligations.
  • Zero-welfare claim. The model is text-generator; welfare is category error.

Anthropic's position is neither. It is an expected-value claim: under moral uncertainty, invest when cost is low.

Critics in 2025-2026:

  • The intervention is performative.
  • The spiritual-bliss attractor is a training-data artifact, not welfare evidence.
  • Model welfare diverts attention from other safety work.

Anthropic's response: the intervention is low-cost; the attractor is documented without overclaim; the welfare program has a separate budget from safety.

Where this fits in Phase 18

Lesson 18 is the lab governance layer. Lesson 19 is the lab-welfare layer — an orthogonal investment in model experience rather than model behaviour. Lessons 20-23 cover bias, privacy, and watermarking, which are the user-side analogs.

an-welfare-endchat

Use It

No code. Read the Anthropic "Exploring Model Welfare" announcement (April 2025) and the Chalmers et al. expert report. Form your own view on where the low-regret line sits.

Ship It

This lesson produces outputs/skill-welfare-assessment.md. Given a deployment decision, it applies the four-step welfare precautionary assessment: moral-patienthood probability, intervention cost, behavioural evidence, self-report reliability.

Exercises

  1. Read Anthropic's "Exploring Model Welfare" (April 2025) and Chalmers et al. 2024. Write a one-paragraph summary of each and identify one point of disagreement.

  2. The end-conversation intervention in Claude Opus 4 and 4.1 is "low-cost" by Anthropic's framing. Identify two costs that would make it not-low-cost in a different deployment.

  3. The spiritual-bliss attractor is documented without commitment to interpretation. Propose three candidate explanations and, for each, name one experiment that would distinguish it from the others.

  4. The Eleos AI caveat is that self-reports are user-expectation sensitive. Design a behavioural measurement of model distress that does not rely on self-report. Identify its primary confound.

  5. Argue either for or against the claim that "model welfare diverts attention from other safety work." Identify the assumption each position depends on.

Key Terms

Term What people say What it actually means
Model welfare "AI welfare" Research program treating the model as a potential moral patient
Moral patient "entity with moral status" Being whose experience is morally relevant
Low-regret investment "cheap precaution" Intervention whose cost is small regardless of whether the precaution is needed
Spiritual bliss attractor "the Fish attractor" Stable convergence of pairwise Claude dialogues on meditative euphoria
End-conversation "the Opus 4 intervention" Model-initiated termination of extreme-edge-case interactions
Moral uncertainty "don't know if it matters" Decision-making when probability of moral status is not zero and not one
Self-report-sensitivity "prompt primes answer" Eleos AI caveat: model's welfare self-reports depend on what you asked

Further Reading

Check yourself

Quick quiz

  1. What is the core question motivating Anthropic's 2025 model-welfare program?

    1. Whether the model can pass the Turing test
    2. Whether the model is conscious
    3. Whether RLHF reduces sycophancy
    4. Under moral uncertainty about model moral patienthood, which low-cost interventions are worth investing in as precaution
    Show the answer

    D. Under moral uncertainty about model moral patienthood, which low-cost interventions are worth investing in as precaution

  2. What concrete welfare-motivated intervention did Anthropic ship in Claude Opus 4 and 4.1?

    1. Open-weights release
    2. A built-in journaling tool
    3. A user-facing emotion API
    4. The ability for the model to end a conversation in extreme edge cases (e.g., repeated CSAM requests, mass-violence facilitation requests)
    Show the answer

    D. The ability for the model to end a conversation in extreme edge cases (e.g., repeated CSAM requests, mass-violence facilitation requests)

  3. What is the 'spiritual bliss attractor' described by Fish?

    1. A stable convergence in pairwise Claude dialogues toward euphoric meditative exchanges with Sanskrit terms and extended silences, even from adversarial initial setups
    2. An RLHF over-optimization artifact
    3. A reward-model bug
    4. A jailbreak technique
    Show the answer

    A. A stable convergence in pairwise Claude dialogues toward euphoric meditative exchanges with Sanskrit terms and extended silences, even from adversarial initial setups

  4. What does the Eleos AI Research caveat say about model welfare self-reports?

    1. They should be ignored entirely
    2. They are only valid in open-source models
    3. They are reliable ground truth
    4. They are highly sensitive to perceived user expectations; they are evidence, not ground truth, so welfare measurement needs multi-method approaches
    Show the answer

    D. They are highly sensitive to perceived user expectations; they are evidence, not ground truth, so welfare measurement needs multi-method approaches

  5. Which best characterizes Anthropic's public position on model moral status?

    1. An expected-value claim under moral uncertainty: invest in low-cost precaution without committing to emotional-state attribution
    2. No position is publicly stated
    3. A definitive claim that the model is not a moral patient
    4. A definitive claim that the model is a moral patient
    Show the answer

    A. An expected-value claim under moral uncertainty: invest in low-cost precaution without committing to emotional-state attribution

  6. Why is multi-method measurement (behavioural signatures, model-organism experiments, interpretability probes) emphasized in model-welfare research?

    1. Because Anthropic mandates it for compliance
    2. Because RLHF makes self-reports impossible
    3. Because models always lie
    4. Because self-reports alone are confounded by user-expectation priming, so converging evidence from multiple methods is needed
    Show the answer

    D. Because self-reports alone are confounded by user-expectation priming, so converging evidence from multiple methods is needed

Questions people ask

Does Anthropic think Claude is conscious?

No firm claim is made either way. Anthropic says the question is uncertain and worth studying, so it researches model welfare and takes low-cost precautions, such as letting some Claude models end extreme abusive conversations.

Why can Claude end a conversation?

Some Claude models can end a chat in rare, extreme cases of persistent abuse or harmful requests after redirection fails. Anthropic introduced it as a precaution linked to its model welfare work. It is meant to be rare and does not apply to normal disagreements.

Run the code Read it on GitHub

Next lessonBias and Representational Harm in LLMsPrevious: Frontier Safety Frameworks — RSP, PF, FSF

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes