Prompt Engineering for Claude's Long Context Window

文章 6 min read · 未标注

Source: Anthropic (Sep 23, 2023) Core Focus: Techniques to maximize Claude's recall over its 100,000 token context window.

Key Techniques Tested

Two primary methods were evaluated to improve information recall from long documents:

Extracting Reference Quotes: Instructing Claude to pull relevant quotes into a "scratchpad" before answering.
Supplementing with Examples: Providing examples of correctly answered questions about other sections of the same document.

Data Source: A daily government document (July 13, 2023, post-training cutoff) containing meeting transcripts.
Q&A Generation: Used a "randomized collage" approach. Claude generated 5 multiple-choice questions per document section. These sections were then randomly stitched into long documents (~75K and ~90K tokens) for testing.
Primary Model: Claude Instant 1.2 (easier to observe improvements; faster for reproducibility). Claude 2 was also tested for comparison.

Careful prompting was needed to avoid pitfalls:

Questions answerable without the document (e.g., "What does the Department of Transportation do?").
Questions Claude inherently struggles with (e.g., counting words/tokens).
Unintentional clues (e.g., making the correct answer overly detailed).
Ambiguous references (e.g., "this document" or "this passage" without specification).
Questions that contain the answer within them.

Solution: A prompt template with few-shot examples and explicit guidelines to ensure questions specify the relevant passage.

Four prompting strategies were tested, each with and without a quote-extraction scratchpad:

Base: Simple question answering.
Nongov Examples: Two fixed, unrelated general knowledge examples.
Two Examples: Two dynamically chosen examples from other sections of the same long document.
Five Examples: Five dynamically chosen examples from other sections.

Variables: Answer position (beginning, middle, end of context) and document length (70K vs. 95K tokens).

Best Performance: Combining quote extraction (scratchpad) with contextual examples yielded the highest accuracy.
Example Type Matters: Generic, unrelated examples did not help. Contextual examples from the same document were effective.
More Examples are Better: Five examples outperformed two.
Position Effect: Performance on information at the beginning and middle of the document improved significantly with techniques. Performance on the end could degrade, possibly because added examples increase the distance between the end of the document and the answer.
Claude 2 Improvement: Prompting techniques reduced Claude 2's error rate by 36% (from 0.939 to 0.961 accuracy).
Latency Trade-off: Quote extraction adds minor latency but improves accuracy. For Claude Instant, this is negligible.

Use a scratchpad to have Claude extract relevant quotes before answering.
Provide multiple, contextual examples of correctly answered questions from the same document.
Place instructions at the end of the prompt to maximize their recall.
Avoid generic examples; ensure examples are relevant to the task and document.
Be explicit in questions, avoiding ambiguous references like "this document."

Anthropic Cookbook: Contains fully reproducible code for this experiment, plus other recipes (Search & Retrieval demo, mock-PDF uploading).
All prompts used in the experiment are available in the linked notebook.

暂无来源