Superforecaster scratchpad prompt (Halawi et al.)

From “Approaching Human-Level Forecasting with Language Models” (Halawi, Zhang, Chen and Steinhardt; NeurIPS 2024). Their system retrieves news, summarises it, then asks the model for a probability several times using scratchpad prompts like this one and aggregates the answers. The paper reports it gets close to, and in some settings beats, the crowd aggregate on competitive forecasting platforms.

The prompt

Structure of BINARY_SCRATCH_PAD_PROMPT_1, lightly condensed — the exact text is in the source file:

Question: {question}
Question Background: {background}
Resolution Criteria: {resolution_criteria}
Today's date: {date_begin}
Question close date: {date_end}
We have retrieved the following information for this question: {retrieved_info}

Instructions:
1. Write down any additional relevant information that is not included above.
   This should be specific facts that you already know the answer to, rather than
   information that needs to be looked up.
2. Provide at least 3 reasons why the answer might be no.
3. Provide at least 3 reasons why the answer might be yes.
4. Rate the strength of each of the reasons given in the last two responses.
   Think like a superforecaster (e.g. Nate Silver).
5. Aggregate your considerations.
6. Output your answer (a number between 0 and 1) with an asterisk at the
   beginning and end of the decimal.

How to use it well

• Write sharp resolution criteria and a close date. Vague questions give vague probabilities.
• Fill {retrieved_info} with dated, sourced material from before “today”. The paper’s gains came from retrieval plus this prompt, not from the prompt alone.
• Run it several times, or across models, and take the median (or a trimmed mean). A single run is noisy.
• Keep a log of questions, probabilities and outcomes so you can check calibration with Brier scores.

Limits

LLMs lean toward hedged answers near 50% and can be confidently wrong about recent events. Treat the output as one input to a decision, not the decision. Not investment advice.