Generated Knowledge Prompting
Last updated
Last updated
Image Source: Liu et al. 2022 (opens in a new tab)
LLMs continue to be improved and one popular technique includes the ability to incorporate knowledge or information to help the model make more accurate predictions.
Using a similar idea, can the model also be used to generate knowledge before making a prediction? That's what is attempted in the paper by Liu et al. 2022 (opens in a new tab) -- generate knowledge to be used as part of the prompt. In particular, how helpful is this for tasks such as commonsense reasoning?
Let's try a simple prompt:
Prompt:
Output:
This type of mistake reveals the limitations of LLMs to perform tasks that require more knowledge about the world. How do we improve this with knowledge generation?
First, we generate a few "knowledges":
Prompt:
Knowledge 1:
Knowledge 2:
We are using the prompt provided in the paper by Liu et al. 2022 (opens in a new tab).
The next step is to integrate the knowledge and get a prediction. I reformatted the question into QA format to guide the answer format.
Prompt:
Answer 1 (confidence very high):
Answer 2 (confidence is a lot lower):
Some really interesting things happened with this example. In the first answer, the model was very confident but in the second not so much. I simplify the process for demonstration purposes but there are a few more details to consider when arriving at the final answer. Check out the paper for more.
