AI-based systems are becoming increasingly widespread, intertwining more deeply with our daily lives. However, a major drawback of current AI solutions is that they rely solely on training data and lack a coherent world model. This limitation restricts their applicability, particularly in critical fields such as healthcare and finance. To address this challenge, researchers at the University of Szeged, in collaboration with two other universities, are conducting research within the framework of the RAItHMA (Reliable AI through Human-Machine Alignment) project, aiming to improve AI systems. We spoke with Dr. Márk Jelasity, head of the Artificial Intelligence Competence Center at the university’s the Centre of Excellence for Interdisciplinary Research, Development, and Innovation (Cluster of Science and Mathematics), about this initiative, which is also supported by a HU-rizont grant.
Q: Could you share some details about your new collaborative project?
A: The collaboration is set to launch on April 1 and will involve the University of Szeged, Rutgers University in the United States, and Ludwig-Maximilians-Universität in Munich. Together with our German and American colleagues, we aim to understand how chatbots like ChatGPT – which are now widely used – represent knowledge. You see, most people instinctively assume that these systems possess human-like intelligence, meaning they have an internal world model that guides their responses to our questions.
Q: Is that not the case?
A: That is only partially true. AI does not have a language ‘in its head’ the way humans do, and this fundamental difference often leads to confusion in human-machine interactions. While chatbots can be fine-tuned to solve highly complex mathematical problems, they still struggle with surprisingly simple tasks. At times, they get stuck on basic multiplication tasks, yet they can also tackle Olympiad-level problem sets with ease – far surpassing the capabilities of the average person. In this context, our goal is to map out what the model actually knows and explore ways to minimize this communication gap between humans and machines. However, to achieve this, we must first confront a fundamental question: What exactly does the model understand?
Photo by Anna Bobkó
Q: How can you explore that?
A: There are several methodologies to explore it. One approach is designing prompts – guided questions – that elicit specific behaviors of interest. By analyzing the model’s responses, we can infer how the ‘brain’ of artificial intelligence operates. We basically challenge it with questions that disrupt its internal balance, revealing inconsistencies. Notably, previous research in this field suggests that these models lack a stable and coherent world model, reinforcing the need for further investigation.
Q: So how are they capable of solving complex problems?
A: That is exactly what we want to investigate. We know that AI systems recognize certain concepts, but we don’t know what those concepts refer to, how they are related to each other, or how they are represented.
For instance, there is a model that was trained on actual taxi routes in Manhattan and was then asked to plan the optimal route from one location to another. The model was able to do so, but the question arises: Did it simply memorize all the routes and respond accordingly, or did it develop a real mental map? You see, when humans plan a route, they rely on a mental map. However, when researchers extracted the map stored in the AI’s ‘mind’ at the end of the experiment, it barely resembled Manhattan’s actual street layout – despite the fact that the AI correctly generated the routes.
At the University of Szeged, we will be focusing precisely on this aspect of the project. We aim to investigate what kind of world these large language models (LLMs) construct internally and what advantages and disadvantages this entails. For example, we can examine whether these models exhibit contradictions or generate conflicting statements. Naturally, this also depends on how frequently a given topic appears on the internet. For instance, it would be extremely difficult to get the model to state that the capital of England is not London but Paris, as it can rely on a vast number of sources in this context.
Another approach is to analyze how the model organizes and interprets concepts, as this could provide valuable insights even for us humans. Since it is a language model, we can also ask it to explain what it understands by a given concept, which may help us better grasp how it processes information.
Q: How is the Ludwig-Maximilians-Universität involved?
A: They are experimenting with an alternative architecture that incorporates memory. Interestingly, most chatbots, including large language models, do not possess short- or long-term memory in the conventional sense. The only context they retain is within a given conversation session. So, when I ask a new question, the model refers back to the entire conversation history within that session, which can now extend up to 60,000 characters. However, despite this expanded capacity, once the session ends, the model retains no information.
Since these bots lack inherent memory and background knowledge, they rely on vast amounts of text data sourced from the internet. However, the reliability of these sources varies widely, as they include everything from scholarly research to conspiracy theories on platforms like Reddit. As a result, the model may occasionally generate false information or fabricate details when data is insufficient – a phenomenon known as hallucination.
Additionally, AI knowledge is fundamentally statistical: it excels at recognizing common patterns but struggles with inconsistencies. For example, it performs frequent calculations like basic addition well but is prone to errors with larger numbers or less commonly occurring scenarios.
Q: Can hallucinations be avoided?
A: One possible solution is retrieval-augmented generation (RAG), which helps refine the model’s background knowledge. In this approach, we provide the model with specific documents as reference material, ensuring that its responses are based on reliable sources. This method significantly reduces hallucinations and improves the accuracy of the model’s answers. However, in the long run, the goal is to develop AI systems that do not require such direct modifications to function reliably.
Another potential approach is integrating memory and additional computational components – features commonly used in modern computers. This is precisely what the researchers in Munich are exploring.
Q: In which fields of science are there applications for these chatbots?
A: One major field of application is medicine. These models, trained on medical data, can assist in interactive diagnostics. For example, one of our partners is exploring stroke prediction by developing a chatbot that, through conversation, can assess a patient’s risk of stroke. However, much of our current work focuses on identifying errors in these systems. We aim to understand how they represent knowledge, what limitations they have, and whether those limitations can be overcome. Accordingly, we are partnering with leading experts across various fields. Our team includes Prof. Dr. Péter Klivényi, head of the Department of Neurology, as well as Dr. Richárd Farkas and Dr. Gábor Berend, associate professors from the Institute of Informatics. Additionally, Dr. György Turán from the HUN-REN Artificial Intelligence Research Group, who is also a professor at the University of Illinois Chicago, plays a key role in our research.
Original Hungarian text by Anna Szabó
Feature photo: pexels.com
Photo by Anna Bobkó
