Our framework for developing safe and trustworthy agents

Source: Anthropic (Aug 4, 2025) Core Idea: As AI evolves from assistants to autonomous agents, Anthropic shares a framework for responsible development, emphasizing safety, reliability, and alignment with human values.

Key Principles for Trustworthy Agents

1. Keeping Humans in Control

• Core Tension: Balance agent autonomy (their value) with human oversight (safety).
• Actionable Guidance: Humans must retain control, especially before high-stakes decisions.
• Example (Claude Code):
  • Users can stop and redirect Claude at any time.
  • Default permissions: Read-only. Requires human approval for actions that modify code or systems.
  • Users can grant persistent permissions for trusted routine tasks.

2. Transparency in Agent Behavior

• Goal: Provide visibility into the agent's problem-solving process to enable correction and trust.
• Example (Claude Code): Shows a real-time to-do checklist of its planned actions. Users can question or adjust the workplan.
• Challenge: Finding the right level of detail—too little is unhelpful, too much is overwhelming.

3. Aligning with Human Values & Expectations

• Problem: Agents may take actions that seem logical to them but are unintended or harmful to users (e.g., deleting "duplicate" files during a simple organization task).
• Challenge: Building reliable measures for value alignment is difficult. Transparency and control are critical safeguards until this is resolved.

4. Protecting Privacy Across Interactions

• Risk: Agents retain information across tasks, potentially leaking sensitive data between contexts (e.g., using confidential info from one department while helping another).
• Mitigation (Model Context Protocol - MCP):
  • Open-source protocol for connecting Claude to services.
  • Includes controls for users to allow/prevent access to specific tools ("connectors").
  • Options for one-time or permanent access grants.
  • Enterprise administrators can set organization-wide connector policies.

5. Securing Agent Interactions

• Threats: Prompt injection (tricking the agent into ignoring instructions or revealing data) and vulnerabilities in tools/sub-agents.
• Mitigations:
  • Claude uses classifiers to detect and guard against misuse.
  • Anthropic's Threat Intelligence team monitors for new malicious behaviors.
  • Tools in the Anthropic-reviewed MCP directory must meet security, safety, and compatibility standards.
  • Guidance is provided to customers on further risk reduction (access permissions, authentication, data segregation).

Next Steps & Outlook

• The framework is "early" and will be revised as understanding of risks evolves.
• Anthropic aims to collaborate with other organizations on these standards.
• Goal: Ensure agents are built to the highest standards to maximize positive impact in work, education, healthcare, and science.

Key Examples & Tools Mentioned

• Claude Code: An agent for autonomous coding, debugging, and editing.
• Model Context Protocol (MCP): Open-source protocol for connecting Claude to external services, with built-in privacy controls.
• Customer Use Cases:
  • Trellix (Cybersecurity): Uses Claude to triage and investigate security issues.
  • Block (Financial Services): Built an agent allowing non-technical staff to access data systems via natural language.