Prompt Engineering
CC-BY
Fabian Suchanek
Prompt Engineering
2
•
Introduction
•
Tactics for prompt engineering
•
Augmented language models
Def: Generative language models
Generative language models
(also known as large language models, LLMs) are trained on a huge
corpus of (Web‐)documents. They take a prompt as input and generate an answer as output.
Olympe de Gouges was an 18th century French playwright and
political activist, best known for her Declaration of the Rights of Woman
and other writings on women’s rights and abolition of slavery.
LLM
prompt
answer
Who was Olympe de Gouges?
blah
blah
blah
blah
training corpus
3
Answer the user question:
instruction
user
query
>examples
Capabilities of LLMs
Who was Olympe de Gouges?
Summarize the following text: ...
Consider the following two sentences:
1) Olympe de Gouges was an abolitionist.
2) Olympe de Gouges worked against slavery.
Does one logically entail the other?
LLMs can generate an answer for any prompt and behave very much like a knowledgeable
human. In particular, they can solve many tasks of natural language processing:
• Question answering:
• Textual entailment:
• Summarization:
4
Capabilities of LLMs
LLMs can generate an answer for any prompt and behave very much like a knowledgeable
human. In particular, they can solve many tasks of NLP and information extraction.
• Named entity recognition (and classification):
• Disambiguation:
• Fact extraction:
Extract all named entities from the following text: ...
Which of the following entities is mentioned in this text? ...
Extract facts of the from <subject, predicate, object> from the following text: ...
5
Capabilities of LLMs
LLMs can generate an answer for any prompt and behave very much like a knowledgeable
human. In particular, they can solve many tasks of NLP and information extraction,
but the quality of the answer depends a lot on how we ask the question.
Can you tell me who was
Olympe de Gouges’ husband?
Sure! Olympe de Gouges was married against her
will at age 17 to Louis Aubry. She complained
“I was married to a man I did not love.
I was sacrificed for no reason that could
make up for the repugnance I felt for this man.”
Quite luckily, Aubry was killed in a flooding.
Louis Aubry
Who was Olympe de Gouges’ husband?
Answer with a single name.
[Boquet & Muller]
6
Def: Prompt engineering
Prompt engineering
is the art of finding a prompt that makes an LLM solve a task.
[Coursera.org], biased source
Prompt engineers find employment across industries,
including marketing and advertising, education,
finance, human resources, and health care.
Much of today’s NLP research
is prompt engineering.
7
Prompt Engineering
8
•
Introduction
•
Tactics for prompt engineering
•
Augmented language models
A prompt commonly contains:
•
an instruction
: a specific task or instruction you want the model to perform
•
a context
: external information that can steer the model to better responses,
in particular background knowledge, examples, or more details.
•
input data
: the input or question that we are interested to find a response for
•
an output indicator
: the type or format of the output
9
[promptingguide.ai]
Extract entities from the following text.
We are interested in persons, locations, dates, and organizations.
Olympe de Gouges lived during the French Revolution.
Generate answers of the form “entity: type”.
The prompt
mixes the
input text
with the
instructions
(dangerous...)
What’s in a prompt?
Elements of the prompt can
appear in any order or be omitted.
A prompt commonly contains:
•
an instruction
: a specific task or instruction you want the model to perform
•
a context
: external information that can steer the model to better responses,
in particular background knowledge, examples, or more details.
•
input data
: the input or question that we are interested to find a response for
•
an output indicator
: the type or format of the output
10
[promptingguide.ai]
What’s in a prompt?
Smaller models (<GPT 3.5) will need very simple prompts!
What works for GPT-4 will not necessarily work for LLAMA 7B!
•
Be precise
•
Ask the model to adopt a persona
•
Specify steps to complete a task
•
Experiment with instructions, questions, and completions
11
[OpenAI Prompt Engineering]
When did she live?
Use the following step-by-step instructions to respond.
Step 1: ...
Tactics for the instruction
What are the birth and death dates of Olympe de Gouges?
You are a professional historian. You will answer precisely and seriously.
Who was Olympe de Gouges?
Olympe de Gouges was
Tell me who Olympe de Gouges was.
Be polite! You
will get polite
answers as well!
[crypto4nerd]
12
Tactics for the instruction
Apple’s prompt
[source]
13
Tactics for the instruction
Bing’s prompt
[source]
>input
•
Choose a relevant part of the input
•
Try surrounding the input by special characters
14
[OpenAI Prompt Engineering]
Tactics for the input
In her “Declaration of the Rights of Woman and the Female Citizen”,
Olympe de Gouges stated that male and female citizens must be
equally admitted to all public employment without other
distinction than their talents and virtues.
Use this part if
you’re interested
in authorship.
[de Gouges: Declaration of the Rights of Woman, 1791]
Summarize the text given in triple quotes:
“““de Gouge proclaimed that ....”””
>input
•
Ask the model to reply based only on a reference text
15
[OpenAI Prompt Engineering]
Tactics for the input
Answer the user query based on the following articles in triple
quotes. If the answer cannot be found in the articles, write
“I could not find an answer”. If an answer is provided, it must be
annotated with a citation of the form ({"citation": …}).
“““de Gouges was attacked for speaking in public, and one man
complained that she was one of those women to whom one
feels like giving razor blades as a present. To which she replied:
If woman has the right to mount the scaffold, she must equally
have the right to mount the podium.”””
{user query}
[de Gouges: Declaration of the Rights of Woman, 1791]
[Catel & Bocquet]
Add one or more texts
•
Specify a desired length of the output
•
Specify an output format
16
[OpenAI Prompt Engineering]
Tactics for the output format
Summarize the text in 2 paragraphs / 50 words / 3 sentences / a single sentence.
Output facts in the form <subject, predicate, object>, where the predicate is
taken from the following list: ...
Output the results in JSON in the following form: ...
Most models are really good at
coding, so sticking to a standard
format can be a good idea.
•
Give a list of answers to choose from
•
Produce output that can be processed by algorithms
•
If this proves too complex (e.g., because you have a small model), give the options one by one
17
Tactics for the output format: Keep it simple
Choose an answer from the following numbered list and give only the number as answer.
Which of the following entities is mentioned?
(a) King Louis XVI
(b) Olympe de Gouges
(c) Elvis Presley
Does the text mention King Louis XVI?
Does the text mention Olympe de Gouges?
Does the text mention Elvis Presley?
•
LLMs can solve some tasks without any need for context (
zero‐shot prompting
)
•
We can also give the model a few examples as context (
few shot prompting
,
in‐context learning
)
•
We can also give detailed explanations
18
[OpenAI Prompt Engineering]
Tactics for the context: Zero‐shot and Few‐shot
Classify the following text as positive or negative:
“““I liked the book about the life of Olympe de Gouges
because it describes her life from the inside.”””
I liked the book -> positive
I hated the book -> negative
A text should be classified as positive if it describes an accomplishment,
a happy event, a success, or an endorsing opinion on an issue.
Sometimes LLMs give wrong answers that are not thought through.
19
[OpenAI Prompt Engineering]
Tactics for the context: ask the model to think
She attempted to unmask the villains through her literary work.
They never forgave her, and she paid for her work with her head.
As she approached the scaffold, she forced her executioners to
admit that such courage and beauty had never been seen before.
Did she die?
She did not die. She is very courageous and beautiful!
[Mettais]
Sometimes LLMs give wrong answers that are not thought through.
•
We can get better answers by asking the model to think
(
Zero-shot Chain of Thought prompting
)
•
We can also ask the model to think step by step
20
[OpenAI Prompt Engineering]
Tactics for the context: ask the model to think
Let’s think step by step.
just adding this instruction helps!
First work out the sequence of events that are mentioned in this text.
Then answer the question based on the sequence of events you found.
•
We can break down the problem into several queries
•
We can also ask the model to criticise its own answer
21
Tactics for the context: break down the problem
Which facts are mentioned in this text?
Based on these facts, what is the answer to {query}?
(model answers)
What is the answer to {query}?
What is a weakness of your answer?
(model answers)
(model answers)
What would be an answer to this question that avoids these weaknesses?
Probably needs a larger model
->RAG
->end
Chain‐of‐Thought prompting
gives the model an example question with an example answer
with reasoning before asking the actual question.
22
Def: Chain of Thought Prompts
The French Revolution started in 1789, one year after
Olympe de Gouges turned 40. When was she born?
Since she turned 40 one year earlier than 1789, she must have
turned 40 in 1788. Therefore she must have been born in 1748.
De Gouges was born in 1748. She died five years after
she turned 40. When did she die?
[Emilia Ararura]
Since she was born in 1748, she turned 40 in 1788. Therefore she
must have died in 1793.
Example question
Example reasoning
Actual question
[PromptingGuide.AI]
Self‐consistency prompting
runs the same Chain‐of‐Thought prompt multiple times to get different
answers, and takes the majority answer.
23
Def: Self‐consistency prompting
[PromptingGuide.AI]
... she must have died in 1793.
get several answers
... she must have died 1783.
... she must have died in 1793.
choose the most frequent one
The French Revolution started in 1789, one year after
Olympe de Gouges turned 40. When was she born?
Since she turned 40 one year earlier than 1789, she must have
turned 40 in 1788. Therefore she must have been born in 1748.
De Gouges was born in 1748. She died five years after
she turned 40. When did she die?
you can vary the prompt
slightly with each query
Test the performance of different prompts systematically:
•
Create a gold standard
•
Run the model on all inputs of the gold standard
•
Compute precision and recall wrt. the gold standard
•
Refine your prompt
•
Keep a record of the prompts and the performances
24
[OpenAI Prompt Engineering]
Tactics for testing
<de Gouges, 1748>, <Presley, 1935>, ...
Extract birth dates from the following text:
{text}
Extract pairs of the form <person, birth year>
from the following text:
{text}
iterate
Use placeholders (e.g. Python
f"string") to generate prompts.
->end
Prompt Engineering
•
Introduction
•
Tactics for prompt engineering
•
Augmented language models
25
A
retrieval‐augmented language model
receives a prompt, looks up relevant documents from
a corpus of documents, adds these to the prompt, and then generates an answer.
26
Def: Retrieval‐augmented language models
What did Olympe de Gouge say about marriage?
She proposed that men and women shall have
equal rights in a marriage, including in divorce.
“““Declaration of the Rights of Woman:
Man, are you capable of being just? [...]”””