idw – Informationsdienst Wissenschaft
Nachrichten, Termine, Experten
Nachrichten, Termine, Experten
idw - Informationsdienst
Wissenschaft
- Researchers combine 3D image recognition with language models.
- Artificial intelligence can be integrated into any robot.
- Robots use continuous probability calculations to guide searches.
A robot that can locate lost items on command – this is the latest development at the Technical University of Munich (TUM). It combines knowledge from the internet with a spatial map of its surroundings to efficiently find the objects being sought.
The new robot from Prof. Angela Schoellig’s TUM Learning Systems and Robotics Lab looks like a broomstick on wheels with a camera mounted at the top. It is one of the first robots that not only integrates image understanding but also applies it to a clearly defined task.
To find a pair of glasses misplaced in the kitchen, for example, the robot has to look around and build a three-dimensional image of the room. The camera initially provides two-dimensional images, but these pixels also contain depth information. This creates a spatial map of the environment that is accurate to the centimeter and is constantly updated. A laptop also provides the robot with information about which objects are visible in the image and what significance they have for humans.
“We have taught the robot to understand its surroundings,” says Prof. Angela Schoellig. The head of the Robotics Lab at the TUM Chair of Safety, Performance and Reliability for Learning Systems aims to develop robots that can navigate any environment independently. Humanoid robots working in factories or robots in care settings in private homes require this newly developed basic understanding, which, as Schoellig explains, “is important for all robots that move in spaces that are constantly changing”.
Internet knowledge translated into the robot’s language
The robot therefore understands that a table or window sill can be used to briefly set down a pair of glasses, whereas a stovetop or a sink are not suitable for this purpose. “The language model captures the relationships between the objects and we convert this information into the robot’s language,” explains Prof. Schoellig. Two-digit numbers appear on the three-dimensional map of the environment, constantly recalculating the likelihood that the object being searched for is located there. According to the research results, the robot then searches the probable locations almost 30 per cent more efficiently than if it searched randomly throughout the room. Artificial intelligence is used in two ways: on the one hand in image recognition and on the other hand through the use of a language model.
Another special capability of the robot: it remembers previous images and is able to compare them with new images of its surroundings. If a new object suddenly appears in the kitchen, it recognizes the change with a high degree of certainty (95 per cent) and marks these areas as “highly probable” search locations.
Next step: searching behind cupboard doors
In the next step, the TUM scientist and board member at the Munich Institute of Robotics and Machine Intelligence (TUM MIRMI) also wants to search for objects that are in a drawer or behind a door. To do this, however, the robot will not merely have to draw on knowledge from the internet but will also have to interact with its surroundings. Robotic arms and hands must open a cupboard and determine whether it opens upwards or sideways and how best to grasp the handle. This will enable the robot to search even in closed spaces such as cupboards or drawers.
Further information:
- Prof. Angela Schoellig is a member of the board of the Munich Institute of Robotics and Machine Intelligence (TUM MIRMI). The institute is an integrative research institute at the Technical University of Munich (TUM) that focuses on robotics and AI. The institute brings together expertise in key areas of robotics, including perception and data science. Nearly 80 TUM chairs are networked within MIRMI to develop innovative robotic and AI-supported solutions for the environment, health, mobility, work, as well as security and defense. TUM MIRMI is headed by Prof. Lorenzo Masia. Further information can be found at https://www.mirmi.tum.de/.
- A demonstration of the search robot can be seen until Friday, 13 March at the German Robotics Congress in Cologne: https://robotics-institute-germany.de/event/grc/
Additional material for media outlets:
- Photos for download: https://mediatum.ub.tum.de/1846659
- Scientific video: https://utiasdsl.github.io/semi-static-semantic-exploration/
Prof. Angela Schoellig
Chair of Safety, Performance and Reliability for Learning Systems
Technical University of Munich (TUM)
angela.schoellig@tum.de
Where did I leave my glasses? Open-Vocabulary Semantic Exploration in Real-World Semi-Static Environments; Benjamin Bogenberger, Oliver Harrison, Orrin Dahanaggamaarachchi, Lukas Brunke, Jingxing Qian, Siqi Zhou, Angela P. Schoellig; IEEE Robotics and Automation Letters, 3 March 2026; https://ieeexplore.ieee.org/document/11359697
Criteria of this press release:
Journalists, Scientists and scholars
Electrical engineering, Information technology, Mathematics, Mechanical engineering
transregional, national
Research results
English
You can combine search terms with and, or and/or not, e.g. Philo not logy.
You can use brackets to separate combinations from each other, e.g. (Philo not logy) or (Psycho and logy).
Coherent groups of words will be located as complete phrases if you put them into quotation marks, e.g. “Federal Republic of Germany”.
You can also use the advanced search without entering search terms. It will then follow the criteria you have selected (e.g. country or subject area).
If you have not selected any criteria in a given category, the entire category will be searched (e.g. all subject areas or all countries).
