Claude 3.7 Sonnet & Claude Code Summary

Key Announcement

Claude 3.7 Sonnet is Anthropic's most intelligent model to date and the first hybrid reasoning model on the market. It can produce near-instant responses or extended, step-by-step thinking visible to the user. API users have fine-grained control over thinking duration.

Claude Code is a new command-line tool for agentic coding, available as a limited research preview, enabling developers to delegate substantial engineering tasks directly from their terminal.

Model Details & Philosophy

• Unified Reasoning: Unlike separate reasoning models, Claude 3.7 Sonnet integrates quick response and deep reflection capabilities into a single model, mirroring human cognition.
• Dual Modes:
  • Standard Mode: An upgraded version of Claude 3.5 Sonnet.
  • Extended Thinking Mode: Self-reflects before answering, boosting performance on math, physics, instruction-following, coding, and other tasks.
• API Control: Users can set a token budget (up to 128K tokens) for thinking, trading off speed/cost for answer quality.
• Real-World Focus: Optimized less for competition problems and more for real-world business tasks.

Performance & Benchmarks

• State-of-the-Art on SWE-bench Verified (real-world software issues) and TAU-bench (complex real-world agent tasks).
• Excels in instruction-following, general reasoning, multimodal capabilities, and agentic coding.
• Extended thinking provides a notable boost in math and science.
• Outperformed all previous models in internal Pokémon gameplay tests.

Industry Validation

• Cursor: "Best-in-class for real-world coding tasks," with improvements in complex codebases and tool use.
• Cognition: "Far better than any other model at planning code changes and handling full-stack updates."
• Vercel: "Exceptional precision for complex agent workflows."
• Replit: Successfully deployed to build sophisticated web apps and dashboards from scratch.
• Canva: Consistently produced "production-ready code with superior design taste and drastically reduced errors."

Claude Code (Agentic Coding Tool)

• Capabilities: An active collaborator that can search/read code, edit files, write/run tests, commit/push to GitHub, and use command-line tools.
• Current Use: Already indispensable at Anthropic for test-driven development, debugging, and large-scale refactoring. Completed tasks in a single pass that normally take 45+ minutes of manual work.
• Status: Limited research preview. Planned improvements include tool call reliability, long-running command support, and better in-app rendering.
• Goal: To understand developer usage to inform future model improvements.

Availability & Pricing

• Availability: Now available on all Claude plans (Free, Pro, Team, Enterprise), the Claude Developer Platform, Amazon Bedrock, and Google Cloud Vertex AI.
• Extended Thinking: Available on all surfaces except the free Claude tier.
• Pricing: Same as predecessors: $3 per million input tokens and $15 per million output tokens (includes thinking tokens).

Additional Features

• GitHub Integration: Now available on all Claude plans, allowing direct connection of code repositories for bug fixing, feature development, and documentation.

Safety & Responsibility

• Reduced Refusals: Makes more nuanced distinctions between harmful and benign requests, reducing unnecessary refusals by 45% compared to its predecessor.
• System Card: Details safety evaluations, addresses emerging risks like prompt injection attacks, and examines the safety benefits of reasoning models (understanding model decisions, trustworthiness).

Appendix: Benchmark Methodology

• TAU-bench: Used a prompt addendum to encourage a "planning" tool and increased the maximum step limit from 30 to 100.
• SWE-bench Verified: Used a simple scaffold with a bash tool and a file editing tool. The "high compute" score (70.3%) used parallel attempts, rejection sampling against regression tests, and a scoring model. The base pass@1 score was 63.7%. 11 of 500 test cases were excluded due to infrastructure incompatibility.