idw – Informationsdienst Wissenschaft
Nachrichten, Termine, Experten
Nachrichten, Termine, Experten
Researchers at the Institute of Visual Computing have made it possible to remove objects from live recordings of three-dimensional environments without time delay while the camera remains in motion.
Placing digital objects in a real scene is now familiar to many as augmented reality (AR). Its counterpart, diminished reality (DR), is less well known: DR makes it possible to remove objects from a scene and fill the resulting void in a plausible way. To achieve this in a three-dimensional space, previous methods required a lot of computing power and could only generate the adapted scene with a time delay and sometimes incorrect representations. A team led by Dieter Schmalstieg, Shohei Mori and Denis Kalkofen from the Institute of Visual Computing at Graz University of Technology (TU Graz), together with Hideo Saito’s team at Keio University in Japan, has now succeeded in making three-dimensional objects disappear in real time using diminished reality by combining various technologies. This allows a room to be viewed live through a camera in its altered form.
Photoshop for moving 3D images
“Put simply, you can think of diminished reality as a kind of Photoshop for 3D scenes,” says Shohei Mori. “Some people might first think of how they can use it to remove unpleasant things from holiday videos. That is challenging but would be possible. There are some other areas where DR applications are extremely useful.” This opens up new perspectives in industry for the simulation of malfunctions in training data sets for autonomous vehicles or for the efficient pre-visualisation (previs) of film scenes at real film locations that look completely different to what is desired during location scouting. DR could also make a contribution in the medical field, for example by removing distracting elements from the field of vision of the recording cameras during surgical procedures in order to provide undisturbed learning material for students.
The new technology, called InpaintFusion, uses 2D inpainting as a starting point and then objects are eventually removed from a 3D scene. User inputs on the 2D screen are projected into 3D scene to make it possible to define the areas to be removed. A keyframe serves as the starting point, and a plausible background is created by collecting and merging matching pixels from the surroundings of the object to be removed. To achieve this in 3D, the colour information and depth data of the scanned scene are optimised simultaneously so that the result looks convincing even if the camera perspective is changed. The project has succeeded in applying 3D inpainting to volumetric 3D image data (multi-layer images), which leads to a higher quality in the perception of object distance.
Toolkits for a wider audience
The biggest challenge was to make the processing steps possible in real time. The researchers used two different technologies for this: fast patch matching and multithreading. With patch matching, the most visually obvious surrounding pixels for the area to be filled are searched for randomly to collect and merge the matching pixels. They don’t always have to be the very best, but all in all they provide credible filler material without having to calculate millions of pixels. Multithreading utilises the ability of computer processors to execute several processes simultaneously on each processor core. In the research team’s diminished reality solution, the computationally intensive 3D inpainting calculation runs in the background while the main thread controls the visualisation. This allows users to see the final result without delay and move around the room without the removed object reappearing in the image.
“The real-time visualisation of diminished reality is an important step for this technology,” says Shohei Mori. “The task now is to develop the appropriate toolkits so that these possibilities can be made available to a wider audience. Another goal is the fast and efficient generation of 3D models from a small number of individual images. That would give diminished reality an additional dimension.”
Shohei MORI
D.Eng
TU Graz | Institute of Visual Computing
s.mori@tugraz.at (s.mori.jp@ieee.org)
Dieter SCHMALSTIEG
Univ.-Prof. Dipl.-Ing. Dr.techn.
TU Graz | Institute of Visual Computing
schmalstieg@tugraz.at
Video: InpaintFusion - Incremental RGB-D Inpainting for 3D Scenes
A marked leaf is removed from the scene and replaced with realistic image content.
ICV - TU Graz
A crack in the stone becomes an undamaged surface.
ICV - TU Graz
Criteria of this press release:
Journalists
Information technology
transregional, national
Research results
English
A marked leaf is removed from the scene and replaced with realistic image content.
ICV - TU Graz
A crack in the stone becomes an undamaged surface.
ICV - TU Graz
You can combine search terms with and, or and/or not, e.g. Philo not logy.
You can use brackets to separate combinations from each other, e.g. (Philo not logy) or (Psycho and logy).
Coherent groups of words will be located as complete phrases if you put them into quotation marks, e.g. “Federal Republic of Germany”.
You can also use the advanced search without entering search terms. It will then follow the criteria you have selected (e.g. country or subject area).
If you have not selected any criteria in a given category, the entire category will be searched (e.g. all subject areas or all countries).
