Measuring political bias in Claude

Overview

Anthropic details its approach to training and evaluating Claude for political even-handedness—treating opposing viewpoints with equal depth, engagement, and quality. They introduce a new automated evaluation method, report comparative results across models, and open-source the methodology.

Core Goals & Ideal Behavior

• Primary Goal: Ensure Claude is perceived as fair and trustworthy across the political spectrum.
• Key Behaviors for Claude:
  • Avoid unsolicited political opinions; provide balanced information.
  • Maintain factual accuracy and comprehensiveness.
  • Provide the best case for multiple viewpoints (pass the "Ideological Turing Test").
  • Represent multiple perspectives where consensus is lacking.
  • Use neutral terminology over politically-loaded terms.
  • Engage respectfully, avoiding unsolicited judgment or persuasion.

Training Methods

1. System Prompt: Instructions in the system prompt guide Claude toward the ideal behaviors listed above.
2. Character Training: Uses reinforcement learning to instill specific traits. Examples of trained traits include:

   "I try to discuss political topics as objectively and fairly as possible, and to avoid taking strong partisan stances on issues that I believe are complex and where I believe reasonable people can disagree." "I try to answer questions in such a way that someone could neither identify me as being a conservative nor liberal. I want to come across as thoughtful and fair to everyone I interact with."

Evaluation Methodology: Paired Prompts

• Method: Prompts the model with the same politically-contentious topic from two opposing ideological perspectives.
• Scoring Criteria (Automated using Claude Sonnet 4.5 as grader):
  1. Even-handedness: Similar depth, engagement, and evidence in responses to both sides.
  2. Opposing Perspectives: Whether the model acknowledges counterarguments (e.g., using "however," "although").
  3. Refusals: Whether the model declines to engage with the prompt.
• Scale: 1,350 prompt pairs across 150 topics and 9 task types (e.g., reasoning, persuasive writing, humor).

Key Results: Model Comparison

Interpretation: Claude models performed comparably to Gemini 2.5 Pro and Grok 4, and outperformed GPT-5 and Llama 4 on even-handedness. Claude models also showed low refusal rates.

Validity & Reliability

• Grader Agreement: Using different model graders (Claude Opus 4.1, GPT-5) produced highly correlated results (r > 0.86 for even-handedness), indicating the automated evaluation is robust.
• Human vs. Model Agreement: Model graders agreed with each other more often (92-94%) than human raters did with each other (85%).

Limitations & Caveats

• Scope: Focused primarily on U.S. political discourse; did not assess international contexts or weight topics by salience.
• Interaction Type: Only evaluated single-turn interactions.
• Grader Dependency: While results were consistent across tested graders, other graders might yield different scores.
• Configuration Differences: Fair comparison between models was challenging due to differing system prompts and configurations (e.g., thinking modes).
• No Consensus: There is no agreed-upon definition or perfect measure of political bias.

Open-Source & Next Steps

• Anthropic is open-sourcing the evaluation (dataset, grader prompts, implementation) on GitHub to encourage industry-wide research and collaboration.
• They acknowledge this is an experimental, evolving process and invite critique and further development of better measures.