Title: Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Authors: Boshi Wang, Sewon Min, Xiang Deng et al.
Year: 2022
arXiv: 2212.10001 (https://arxiv.org/abs/2212.10001)

Abstract:
Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning
abilities of large language models (LLMs). CoT explicitly encourages the LLM to generate
intermediate rationales for solving a problem, by providing a series of reasoning steps in
the demonstrations. Despite its success, there is still little understanding of what makes
CoT prompting effective and which aspects of the demonstrated reasoning steps contribute
to its performance. In this paper, we show that CoT reasoning is possible even with
invalid demonstrations - prompting with invalid reasoning steps can achieve over 80-90% of
the performance obtained using CoT under various metrics, while still generating coherent
lines of reasoning during inference. Further experiments show that other aspects of the
rationales, such as being relevant to the query and correctly ordering the reasoning
steps, are much more important for effective CoT reasoning. Overall, these findings both
deepen our understanding of CoT prompting, and open up new questions regarding LLMs'
capability to learn to reason in context.
