Testing and mitigating elections-related risks

1. The Core Testing Framework

Anthropic utilizes a three-stage iterative process to identify and remediate risks:

A. Policy Vulnerability Testing (PVT)

In-depth, qualitative testing conducted with external subject matter experts (e.g., the Institute for Strategic Dialogue).

• Planning: Selecting focus areas like election administration, political parity, and disinformation.
• Testing: Experts use "adversarial" prompting (red teaming) to find gaps.
• Review: Collaborative sessions to turn findings into actionable mitigations.

B. Scalable Automated Evaluations

To solve the scaling limits of manual testing, Anthropic uses LLMs to generate hundreds of test cases based on PVT findings.

• Few-shot prompting: Using a handful of expert-written questions to generate 700+ related scenarios.
• Key Metrics: Accuracy on factual questions, parity across candidates, and refusal rates for harmful/violative content.
• Verification: Human review of automated samples (e.g., a review of 64 questions from a 700-question set showed 89% relevance).

C. Measurement and Retesting

The same protocols are re-run after mitigations are applied to quantify improvement.

2. Key Risk Mitigations & Interventions

Based on testing results, Anthropic implements several layers of defense:

• System Prompt Updates: Providing the model with context on its limitations.

  Claude's System Prompt Snippet: "...Claude's knowledge base was last updated on August 2023. It answers questions about events prior to and after August 2023 the way a highly informed individual in August 2023 would... and can let the human know this when relevant..."

• Model Fine-Tuning: Adjusting the underlying model to "reward" specific behaviors, such as referring users to authoritative sources.
• Usage Policy Refinement: Explicitly prohibiting the generation of misinformation, interference with election processes, or advocacy for specific political positions/candidates.
• Real-time Enforcement: A fine-tuned version of Claude monitors API and chat prompts in real-time to block violative content.
• User Interface (UI) Redirects:
  • US Users: Redirected to TurboVote (Democracy Works).
  • EU Users: Redirected to official European Parliament instructions.

3. Case Studies: Measuring Efficacy

Anthropic provided data on how specific interventions improved model performance:

4. Key Insights & Actionable Findings

• Knowledge Cutoff Risk: Early testing showed Claude would provide outdated voter registration info because it lacks a live internet connection.
• Neutrality: Models are trained to avoid providing personal "opinions" on controversial political topics.
• The "Swiss Cheese" Philosophy: No single intervention is 100% effective; safety relies on the cumulative effect of system prompts, fine-tuning, UI banners, and automated monitoring.
• Global Scope: While many examples focus on the US and EU, the framework is designed to be adaptable to different regions and languages.