Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive NLP
claude.iconThis paper proposes a new framework, DEMONSTRATE-SEARCH-PREDICT (DSP), which combines a language model (LM) and a retrieval model (RM). The main points are as follows.
1. DSP can build a pipeline to solve complex knowledge-intensive tasks by exchanging natural language text between LM and RM.
2. the DSP program consists of three stages: automatic generation of demonstrations (DEMONSTRATE), retrieval of relevant information (SEARCH), and grounded forecasting (PREDICT).
3. DEMONSTRATE automatically annotates intermediate pipeline representations from a small number of training examples; SEARCH rewrites questions and performs multi-hop searches that take into account the context of the dialogue; PREDICT merges multiple search results to produce a final output; and DEMONSTRATE automatically annotates the pipeline with a small number of training examples.
4. applied the DSP program to three tasks: open-domain question-and-answer, multi-hop question-and-answer, and dialogue question-and-answer, and achieved 37-120% accuracy improvement over existing methods.
5. showed the potential of DSP to combine pre-trained LM and RM as modules to develop AI systems that can quickly adapt to new domains.
The DSP is a framework that provides the foundation for a more sophisticated in-context learning system by successfully linking LM and RM.
---