The NoRDF Project
Fabian Suchanek
Professor at Télécom Paris
Works on: Knowledge Bases,
Reasoning, natural language processing
Past:
• Max Planck Institute for Informatics/Germany
• Microsoft Research/US
• INRIA Saclay/France
2
Fabian Suchanek
A
Language Model
(such as GPT-3, ChatGPT, or GPT-4) is a probability distribution
over sequences of words. It can be used in particular to predict a likely next word
in a sentence, and, by extension, to converse with humans.
Me: My colleagues at work keep avoiding me. They never say a single word to me.
Could I sue them for harassment?
GPT3: That's a possibility, but it might not be the best solution. You could try talking to them first.
What if talking doesn't help?
Then you could consult with a lawyer and see if you have a case.
What would a lawyer say?
They would likely say that you have a case if the avoidance is severe and impacting your work.
...
[example by Goran Frehse on
OpenAI's GPT3
]
3
Language Models
The NoRDF project is concerned with the weaknesses of language models,
and how they can be remedied.
How do we measure how good the output of a language model is?
• We systemized the criteria for the quality of a story from the humanities
• had stories annotated manually by these criteria
4
What are language models actually saying?
Relevance (RE): how well the story matches its prompt
Coherence (CH): how much the story makes sense
Empathy (EM): how well the reader understood the character’s emotions
Surprise (SU): how surprising the end of the story was
...
5
Stories generated by language models, evaluated by humans on criteria from humanities, scale 1-5.
Human performance
Best language model performance
[
Chhun, Colombo, Suchanek, Clavel: A Benchmark of the Evaluation of Story Generation, COLING 2022
]
Models cannot talk as well as humans (yet)
6
Can language models reason?
The lawyer visited the doctor. Did the doctor visit the lawyer?
GTP3: The lawyer did not visit the doctor.
Mary is happy. Mary is rich. If someone is happy and rich, he is not blond. Is Bob blond?
GTP3: Bob is not blond.
The light is on. I eat a pizza. I press the light switch. I press the light switch. I eat a pizza.
Is the light on?
Finetuned BERT: erm... (50% error rate). GPT-3 similar.
We surveyed how language models fare on reasoning tasks. Examples:
[
Helwe, Clavel, Suchanek: “Deep Learning, but Shallow Reasoning”, AKBC 2021
]
Transformer models have systematic limitations
when it comes to logical reasoning!
7
Can language models remember?
Does adding two odd numbers give an odd number?
ChatGPT: Yes. This is because any odd number is of the form 2n+1, and the addition of two odd numbers is
(2n+1)+(2m+1) = 2(n+m) + 2, which is also in the form 2p+1.
Did Elvis Presley die?
GPT3: Yes, Elvis Presley died on August 16, 1977.
Is Elvis Presley alive?
GTP3: There is no definitive answer, but many people believe that Elvis Presley is alive and well.
There have been many sightings of Elvis over the years, but none have been definitively proven.
Language models will forget and invent
information at their own discretion!
They know how to talk, but not what to say!
[Suchanek, Varoquaux: “Language Models and Symbolic Representations”, The Conversation, 2022]
8
We need symbolic knowledge
Elvis Presley died on
August 16, 1977.
Applications in:
- fake news detection
- question answering
- text summarization
- automated reasoning on text
Information Extraction
1977-08-16
diedOnDate
9
Contribution 1: Information Extraction Survey
Elvis Presley died on
August 16, 1977.
1977-08-16
diedOnDate
[Weikum, Dong, Razniewski, Suchanek: Machine Knowledge, Found. & Trends in Databases, 2021]
[Zacchary Sadeddine, Fabian Suchanek: Semantic Parsing — a survey, submitted to TACL 2023]
Information Extraction
Language
Model
We wrote a book and a survey on the state of the art in information extraction.