Publications

Authors:

Bombieri, Marco; Fiorini, Paolo; Ponzetto, Simone Paolo; Rospocher, Marco

Title:

Do LLMs Dream of Ontologies?

Year:

2025

Type of item:

Articolo in Rivista

Tipologia ANVUR:

Articolo su rivista

Language:

Inglese

Format:

Elettronico

Referee:

No

Name of journal:

ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY

ISSN of journal:

2157-6904

Page numbers:

1-22

Keyword:

Generative Artificial Intelligence, Large Language Models, Ontology, Memorization

Short description of contents:

Large Language Models (LLMs) have demonstrated remarkable performance across diverse natural language processing tasks, yet their ability to memorize structured knowledge remains underexplored. In this paper, we investigate the extent to which general-purpose pre-trained LLMs retain and correctly reproduce concept identifier (ID)–label associations from publicly available ontologies. We conduct a systematic evaluation across multiple ontological resources, including the Gene Ontology, Uberon, Wikidata, and ICD-10, using LLMs such as Pythia-12B, Gemini-1.5-Flash, GPT-3.5, and GPT-4. Our findings reveal that only a small fraction of ontological concepts is accurately memorized, with GPT-4 demonstrating the highest performance. To understand why certain concepts are memorized more effectively than others, we analyze the relationship between memorization accuracy and concept popularity on the Web. Our results indicate a strong correlation between the frequency of a concept’s occurrence online and the likelihood of accurately retrieving its ID from the label. This suggests that LLMs primarily acquire such knowledge through indirect textual exposure rather than directly from structured ontological resources. Furthermore, we introduce new metrics to quantify prediction invariance, demonstrating that the stability of model responses across variations in prompt language and temperature settings can serve as a proxy for estimating memorization robustness.

Web page:

https://dl.acm.org/doi/10.1145/3725852

Product ID:

144910

Handle IRIS:

11562/1158628

Last Modified:

March 30, 2025

Bibliographic citation:

Bombieri, Marco; Fiorini, Paolo; Ponzetto, Simone Paolo; Rospocher, Marco, Do LLMs Dream of Ontologies? «ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY» , 2025 , pp. 1-22

Consulta la scheda completa presente nel repository istituzionale della Ricerca di Ateneo