CoT, Thought Chain prompt
Thesis: https://arxiv.org/abs/2205.10625
Print out the thought process or provide one-shot/few-shot, the accuracy of the output will be higher; For example, you can append "Let's think step by step" to the prompt, and the effect will be better.
Tips for chain thinking
Chained thinking prompts excel at a variety of natural language reasoning tasks. But it tends to perform worse on tasks that require solving more difficult problems than the examples in the prompts.
New strategy: tips from easy to hard
In order to overcome this challenge of generalization from simple to complex, a new prompt strategy is proposed, namely "prompt from simple to difficult". The key idea of this strategy is to break down a complex problem into a series of simpler subproblems and then solve them sequentially. Solving each subproblem benefits from the answer to the sub-problem that has been solved before. Experimental results on tasks related to symbolic manipulation, combinatorial generalization, and mathematical reasoning show that simple to difficult prompts can generalize to more difficult problems than the problems seen in the prompts.
Experimental results with GPT-3
It is worth noting that when using the GPT-3 code-davinci-002 model with the simple to difficult prompt, it can solve the combinatorial generalization benchmark SCAN with at least 99% accuracy in any segmentation (including length slicing), while only 14 examples are used, compared to the chained thinking prompt with only 16% accuracy. This is particularly striking because the neuro-symbolic models specifically designed to solve SCAN in the literature are trained on an entire training set containing more than 15,000 examples. We have listed tips for all tasks in the appendix.
For decision-making reasoning problems, CoT is relatively effective, in fact, this emergent effect was discovered rather than designed, and its effectiveness was later demonstrated in this paper.
Thesis: https://arxiv.org/abs/2205.10625
Self-consistency
Paper: https://arxiv.org/abs/2203.11171
Find the one with the highest vote in multiple outputs, for example you can say to ChatGPT:
"For each question, you think of 5 different ideas and then combine them to come up with the best wording, the most comprehensive and accurate answer."
Chained thinking prompts and large language models
Combined with pre-trained large language models, chained thinking prompts have yielded encouraging results on complex reasoning tasks.
New decoding strategy: self-consistency
In this paper, we propose to replace the simple greedy decoding used in chained thinking prompts with a new decoding strategy, self-consistency. It first samples out a diverse set of reasoning paths, rather than just selecting the greedy one, and then selects the most consistent answer by marginalizing the sampled reasoning paths. Self-consistency uses an intuition: a complex reasoning problem is often thought in many different ways, but all result in the same correct answer.
Experimental evaluation results
A large number of empirical evaluations have shown that self-consistency significantly improves the performance of chain thinking prompts on a range of popular arithmetic and common sense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%), and ARC-challenge (+3.9%).
Least to Most Prompting
Thesis: https://arxiv.org/abs/2205.10625
The essence is planning + gradual execution, for complex tasks, the effect of simply using CoT + SC is still not too good, you can let ChatGPT first disassemble the problem by itself and then answer:
"For each question, you first need to answer whether to break the problem into sub-questions, and if not, give the answer directly; If so, break down the questions into sub-questions and combine them to produce the best wording, the most comprehensive and accurate answer. "
"To solve {problem}, we need to...", let the big model open this mode of thinking, constantly refine the problem, the result will be better.
The effect of chained thinking prompts
Chained thinking prompts show amazing performance on natural language reasoning tasks. But performance tends to be poor on tasks that need to be solved more difficult than prompt examples.
New strategy: tips from easy to hard
In order to overcome this generalization problem from simple to difficult, the paper proposes a new prompt strategy, that is, "prompt from simple to difficult". The core idea of this strategy is to break down a complex problem into a series of simple sub-problems and then solve them sequentially. When each subproblem is resolved, the answer to the previously resolved subproblem is helpful.
Experimental results with GPT-3 model
When the GPT-3 code-davinci-002 model is combined with the "simple to difficult prompt" strategy, it can achieve at least 99% accuracy on any segmentation combinatorial generalization benchmark SCAN, using only 14 examples, while the accuracy of chained thinking prompts is only 16%. This finding is compelling because neuro-symbolic models specifically designed to solve SCAN need to be trained on a full training set of more than 15,000 examples.
Multi-Persona Self-Collaboration
Paper: https://arxiv.org/abs/2307.05300
Project code, dataset: https://github.com/MikeWangWZHL/Solo-Performance-Prompting
This is a bit similar to the AutoGen mentioned earlier, allowing multiple agents to talk to each other to solve the problem, except that AutoGen really achieves multi-agent interaction from the engineering level, and what is mentioned here is to let ChatGPT play multiple personalities/roles, such as:
"You can play any role, for the problem I gave, please provide the three most relevant roles, have two rounds of discussion about the problem, and then you synthesize the results of the discussion to summarize the best solution. Please print the discussion process of the three roles and the final proposal. ”
Human intelligence and cognition work synergistically
Human intelligence thrives on cognitive synergy, and cooperation and information integration between different cognitive processes produce better results than a single cognitive process. Although large language models (LLMs) show promise as general-purpose task-solving agents, they still present challenges on tasks that require in-depth domain knowledge and complex reasoning.
Solo Performance Prompt (SPP) strategy
In this study, solo performance cues (SPPs) are proposed to transform a single LLM into a cognitive collaborator through multiple rounds of self-cooperation with multiple characters. A cognitive collaborator is an intelligent agent that works with multiple agents and combines their individual strengths and knowledge to enhance problem-solving and overall performance in complex tasks. By dynamically identifying and simulating different roles based on task inputs, SPP unlocks the potential for cognitive synergy in LLM.
Evaluation and effectiveness of SPP
Studies have found that assigning multiple, fine-grained roles to LLM solves problems better than using a single or fixed number of roles. We evaluate SPP on three challenging tasks: Knowledge Creative Writing, Codenames Collaboration, and Logic Grid Puzzle, which all cover both knowledge-intensive and reasoning-intensive. Unlike previous works that only enhanced LLM's reasoning ability, such as Chain-of-Thought, SPP effectively stimulated internal knowledge acquisition, reduced hallucinations, and maintained strong reasoning ability.